devops on Saurabh Kumar

Three Things That Bite You When You Ship gRPC to Production

Fri, 24 Jul 2026 00:00:00 +0000

I recently spent some time going deeper into gRPC, past the quickstart and into HTTP/2 framing, Protobuf encoding, and the parts of the system that sit around the service itself. I came away liking it. Contract-first APIs are useful, generated clients remove an entire class of mistakes, and streaming is part of the protocol rather than something attached later.

I also came away with a list of things I wish the quickstart had made harder to miss. They are not obscure bugs, and they are not arguments against gRPC. They are direct consequences of the choices that make it good.

Connections Are Not Requests

For a long time I carried a simple model of load balancing: a request arrives, the load balancer picks a backend, and the backend handles it. That model is often close enough for HTTP/1.1 systems, even though the machinery underneath is balancing TCP connections rather than individual requests.

gRPC makes the difference visible. It usually keeps an HTTP/2 connection open and carries many concurrent RPCs over that connection as streams. Reusing the connection avoids handshakes and lets several calls make progress at once. This is one of the reasons gRPC performs well.

An L4 load balancer cannot see those streams. It sees one TCP connection, chooses one backend when that connection opens, and keeps sending the connection there for its lifetime. If a client resolves a single virtual IP and holds one long-lived connection to it, hundreds or thousands of calls can remain pinned to one backend while other instances have very little to do.

The metrics look absurd: one pod is hot, several pods are quiet, and adding more pods changes almost nothing. The cluster has capacity, but the client has no path to it.

It is like assigning a delivery truck to a warehouse when it leaves the depot, then insisting that every package on that truck go to the same warehouse. Adding warehouses does not help. The decision was already made at the wrong level.

The fix is to balance one level higher. A gRPC client can resolve individual backends, keep subchannels to them, and choose one for each RPC with a policy such as round_robin. In Kubernetes that often means a headless service rather than a normal ClusterIP service. An HTTP/2-aware proxy such as Envoy, or a service mesh built around one, can also distribute streams across backend connections.

Both approaches work, and both add something I now have to operate: endpoint discovery and health checking in the client, or another proxy in the request path. That may be a good trade. It is still a trade.

A load balancer that understands connections does not necessarily understand gRPC.

Browser HTTP/2 Is Not gRPC

I made what seemed like an obvious inference: gRPC uses HTTP/2, browsers support HTTP/2, therefore browser JavaScript can call a gRPC service.

It cannot, at least not as native gRPC through the browser APIs available to application code.

The browser may speak HTTP/2 on the network, but JavaScript does not control that connection as an HTTP/2 peer. gRPC places its final status and error details in trailing headers after the response body, and fetch and XMLHttpRequest do not give application code the trailer behavior native gRPC expects. Client streaming and bidirectional streaming also need the request and response to remain active together, with the client writing while it reads. Browser request APIs have historically not offered that form of full-duplex transport in a portable way.

The browser owns the connection. My code gets a window into it, not the keys.

grpc-web exists to bridge that boundary. It adapts the protocol so trailers can travel inside the response body as a marked frame that browser code can read. A proxy such as Envoy commonly translates between grpc-web on the browser side and native gRPC on the service side, although some servers support grpc-web directly. Unary calls work, and server streaming can work, but client and bidirectional streaming remain constrained by the browser transport.

Connect takes a different approach. A Connect server can expose browser-friendly HTTP alongside gRPC and grpc-web, with JSON or Protobuf payloads, and the browser can call it with ordinary fetch. That removes a dedicated translation proxy in many deployments and makes a service much easier to inspect with tools such as curl. It does not grant JavaScript arbitrary control over the browser’s HTTP/2 connection; available streaming modes still depend on the browser transport.

I do not consider this a flaw in gRPC. It was designed primarily for environments where both ends are programs with control over their network stack, and server-to-server RPC is where its tradeoffs make the most sense. A browser is someone else’s runtime, with someone else’s security model and someone else’s API surface.

The mistake is not that gRPC has a boundary. The mistake is drawing an architecture diagram that pretends the boundary is not there.

Field Numbers Never Forget

Protobuf field names are for people. Field numbers are for the wire.

message User {
 string name = 1;
 string email = 2;
}

The encoded message does not contain the words name or email. It contains field numbers, wire types, and values. Renaming name to full_name while keeping field number 1 does not change the wire contract. Changing the number does.

This is easy to understand while looking at one version of a .proto file. It becomes harder after field 2 was deleted two years ago, the person who deleted it has moved to another team, and the current file makes number 2 look pleasantly unused.

It is not unused. It is retired.

Suppose a later version reuses field 2 for another length-delimited value:

message User {
 string name = 1;
 bytes avatar_hash = 2; // reused field number
}

An old client can still send an email address in field 2. The new server sees the same field number and a compatible wire type, and accepts those bytes as avatar_hash. Protobuf cannot object because the old meaning is not present on the wire. Unless validation catches it, perfectly valid bytes can acquire the wrong meaning and travel further into the system.

That is worse than a failed decode.

The fix is a tombstone:

message User {
 reserved 2;
 reserved "email";

 string name = 1;
 bytes avatar_hash = 3;
}

reserved tells the compiler that the number and name may not be used again. It turns historical knowledge into a rule the tooling can enforce. Without it, the only thing protecting the schema is memory, and organizational memory is not a compatibility mechanism.

The rest of the discipline follows from the same fact. I do not renumber fields. I reserve every removed number and name. I prefer additive changes while old and new binaries must coexist, and I introduce a new message or RPC when the meaning genuinely needs to break. Proto3’s removal of required fields came from the same hard-earned lesson: one service should not make every older peer invalid merely by evolving its schema.

A field number is small, but it is permanent.

The Abstraction Ends Somewhere

These three problems looked unrelated to me at first. One was about Kubernetes, one about browsers, and one about schema files. The common thread is that gRPC’s guarantees stop at the layer that understands them.

A TCP load balancer does not understand HTTP/2 streams. Browser JavaScript does not control the browser’s HTTP/2 implementation. Protobuf understands field numbers and wire types, but it does not understand that some bytes used to mean an email address and now mean an avatar hash.

Put those boundaries together and gRPC becomes less forgiving than its clean interface suggests. The generated method call may look like a local function, but it still crosses a network, a load balancer, a runtime, and a schema history. Each layer has constraints the service design needs to account for.

I still like gRPC, especially for internal communication across services written in different languages. The contract is real, the tooling is mature, and native streaming is valuable. I do not want to give those things up merely because the surrounding system needs care.

I need to account for those boundaries when choosing clients, load balancers, and schema changes.

Lobste.rs moved to SQLite

Sat, 18 Jul 2026 12:00:00 +0000

Lobste.rs moved to SQLite, and the HN thread had sharper lessons than the post itself:

WAL checkpoints only run during “reader gaps.” Keep reads/writes continuously overlapping and the WAL file grows unbounded, a DoS vector kgeist reproduced with short-lived reads/writes, not just long-lived readers as the docs imply.
Writes are fully serialized, one fsync each, yet app-layer batching still gets 100k+ writes/sec.
Plain rsync on a live .db file can silently corrupt it since it doesn’t understand SQLite transactions. Use sqlite3_rsync instead.
The real weak point is schema migrations: treat SQLite like a KV store, or reach for CQRS, to keep the churn low.

SecretSpec

Wed, 15 Jul 2026 12:00:00 +0000

SecretSpec declares secrets once in secretspec.toml (names + descriptions, never values) and resolves them from any of 11 backends: keyring, 1Password, Vault, AWS, GCP, .env. Same command everywhere: secretspec run --profile production -- npm start. Per-secret fallback chains (providers = ["vault", "keyring", "env"]) and a ref = { item, field } form points at a secret you already have elsewhere, no renaming. Sharpest bit: agents get held to a stricter bar by default. require_reason = "agents" forces secretspec run --reason "..." before an AI agent can read a secret, appended to a local audit log; humans running interactively are unaffected. Rust core, SDKs for Python/Go/Node/Ruby/PHP/Haskell. Worth piloting on a project drowning in .env sprawl.

Dynamic workflows

Fri, 29 May 2026 12:00:00 +0000

Dynamic workflows in Claude Code. Claude dynamically writes orchestration scripts that spawn tens to hundreds of parallel subagents. It plans, decomposes into subtasks, fans out agents, and checks results before returning a coordinated answer. Adversarial agents try to refute findings, iterating until convergence. Enable via ultracode setting or ask Claude to “create a workflow”. Notable: Jarred Sumner used it to port Bun from Zig to Rust (~750K lines, 11 days). Available on Max, Team, Enterprise plans. Consumes significantly more tokens than typical sessions.

mise

Sat, 04 Apr 2026 12:00:00 +0000

mise let’s you manage dependencies at OS level and project level. Thinking to manage my dev machine. The hard part is documenting what I have installed.

Hono

Tue, 10 Feb 2026 12:00:00 +0000

Hono based on web standards is a great option for js runtimes esp. if you plan to deploy your code to cloudflare workers, bun or deno. You get faster startup time and cross-platform compatibility.

Free-threaded Python in uv and GitHub Actions

Mon, 12 Jan 2026 12:00:00 +0000

The free-threaded version of python can be added to uv and github actions with t suffix. e.g. 3.14t and 3.13t source

New tools

Sat, 20 Dec 2025 12:00:00 +0000

New tools:

gh-actions-lockfile to generate and verify lockfiles for gh-actions
Proxyman is more user-friendly http debugging proxy than Charles Proxy on MacOS

actions/checkout

Tue, 10 Dec 2024 12:00:00 +0000

actions/checkout saves the workflow token (GITHUB_TOKEN or a custom one) in the checked-out repo’s .git/config by default. Workflows that upload the repo as an artifact leak it, and Unit 42 found many projects doing this. GitHub calls it intended behaviour. Turn it off, and scan workflows with zizmor, which would have caught the Ultralytics injection.

- uses: actions/checkout@v4
 with:
 persist-credentials: false

Efficiently Managing Mass URL Redirects with Nginx

Mon, 25 Sep 2017 00:00:00 +0000

I use Nginx’s map directive to keep URL redirects in a separate configuration file. It creates hash tables for fast lookups.

Creating Your Redirect Configuration

First, create a file called redirects-map.conf where each line represents a single redirect rule in this format:

<old_location> <new_url>;

You can use two types of location specifications:

Simple path redirects:

/contact-success/ /contact;

Regex-based redirects (prefixed with ~):

~^/showcase/(.*)?$ /projects/;

Here’s an example of a redirects-map.conf file:

# /etc/nginx/redirects-map.conf

# Simple redirects
/contact-success/ /contact;
/old-page/ /new-page;
/outdated-content/ https://newdomain.com/content;

# Pattern-based redirects
~^/showcase/(.*)?$ /projects/;
~^/articles/(\d+)/(.*)$ /blog/$2;

Configuring Nginx

Step 1: Upload the configuration

Upload your redirects-map.conf to your server at /etc/nginx/

Step 2: Set up the map directive

Add the following at the top of your nginx virtualhost file (outside any server blocks):

map $uri $redirected_url {
 default "none";
 include /etc/nginx/redirects-map.conf;
}

Step 3: Add the rewrite rule

Inside your server block, add this redirection rule:

# Handle redirects defined in the map
if ($redirected_url != "none") {
 rewrite ^ $redirected_url permanent;
}

Your complete virtualhost configuration should look something like this:

map $uri $redirected_url {
 default "none";
 include /etc/nginx/redirects-map.conf;
}

server {
 listen 443 ssl http2;
 server_name example.org;

 # Handle redirects defined in the map
 if ($redirected_url != "none") {
 rewrite ^ $redirected_url permanent;
 }

 # Your other server configuration directives
 # ...
}

Step 4: Testing

After saving your changes, test your Nginx configuration:

nginx -t

If successful, reload Nginx:

nginx -s reload

Handling Large Redirect Sets

Important: Nginx has memory limitations for map files, controlled by the map_hash_bucket_size and map_hash_max_size directives. If you encounter an error like [emerg]: could not build the map_hash, you’ll need to increase these values.

Add these directives to the http block in your nginx.conf:

http {
 # Increase these values for large redirect sets
 map_hash_bucket_size 128; # Default is 32/64 depending on architecture
 map_hash_max_size 4096; # Default is 2048

 # Rest of your http configuration
 # ...
}

For very large redirect sets (e.g., 30KB file), you might need values like:

map_hash_bucket_size 256;
map_hash_max_size 30720;

Performance Considerations

The map module is highly efficient because:

It uses a hash table for O(1) lookups regardless of how many redirects you have
It loads the entire redirect configuration into memory once during startup
It avoids the performance penalties of using multiple if or location blocks

This approach is ideal for sites migrating to new URL structures or consolidating content from multiple domains.