The GIL Bottleneck Hiding in Self-Hosted LLM Serving

Batch a hundred requests into one GPU forward pass and the GPU does not care that there are a hundred of them. It is still one matrix multiply, sized up. That is the whole promise of continuous batching in modern LLM serving engines: throughput scales with batch size almost for free.

Constrained decoding breaks that promise in a way that is easy to miss until it shows up in your p99. If your serving stack enforces JSON output, a grammar, or any other per-token constraint through a custom logits processor, and that processor runs once per request in Python, you have quietly reintroduced a sequential bottleneck underneath a system that was supposed to be embarrassingly parallel.

Netflix's AI infrastructure team hit exactly this running vLLM and NVIDIA Triton in production, and their writeup of the fix is a useful case study in a failure mode that has nothing to do with Netflix specifically. Anyone self-hosting constrained decoding at scale will hit the same wall.

What a logits processor actually does

Before a language model picks its next token, the serving engine computes a probability distribution (the "logits") over the entire vocabulary. A logits processor is a hook that runs on that distribution before sampling: mask out tokens that would produce invalid JSON, zero out anything that breaks a grammar, push down tokens that were already used, and so on. Structured output modes - "return valid JSON," "match this regex," "pick from these five labels" - are almost always implemented this way.

The GPU has already done the expensive part by the time the logits processor runs: one batched forward pass produced logits for every request in the batch at once. The processor's job is comparatively cheap per token. The trap is in how "per token" turns into "per request."

Where the GIL bites

In vLLM's original (V0) design, a custom logits processor ran once per request, in a per-request Python object, sequentially. That is the natural way to write one: you subclass a base class, you get called with one request's state, you mask its logits, you return. It reads cleanly and it is wrong for the same reason a lot of clean-looking Python is wrong at scale - the GIL means only one of those calls executes at a time, no matter how many CPU cores are sitting idle.

The GPU forward pass batches all N requests together in roughly constant wall time. The logits-processing step that follows it does not: it is N sequential Python calls, each holding the GIL for its duration. CPU time for that step therefore grows linearly with batch size, and because it sits between the GPU pass and the token being returned to the client, it lands directly in tail latency. You can have a perfectly batched GPU and a CPU-bound request path at the same time, and the profiler will point at Python, not CUDA.

flowchart TB
  A["Batch of N requests"] --> B["One matmul, batched
(cost ~ constant per request)"]
  B --> C{"Custom logits processor"}
  C --> D1["Request 1: mask (Python, GIL held)"]
  D1 --> D2["Request 2: mask (Python, GIL held)"]
  D2 --> D3["... sequential, one at a time"]
  D3 --> DN["Request N: mask (Python, GIL held)"]
  DN --> E["Sample + return tokens"]

The batch-level fix

vLLM's V1 architecture restructures the hook itself: instead of a per-request object called N times, the processor operates on batch-level data structures and computes masks for the whole batch in one call, tracking which requests are currently in the batch through an explicit update_state(batch_update) API rather than being handed one request's state at a time.

That restructuring is what makes reimplementing the hot path in C++ worthwhile. A single call over a batch-shaped array is something you can push into a multi-threaded C++ extension and step around the GIL entirely, because the GIL only serializes Python bytecode execution - it says nothing about native code running underneath it. Netflix's team did exactly that: batch-level masking, computed across many requests together, with the actual mask computation moved into multi-threaded C++.

flowchart LR
  subgraph GPU forward pass
    A[Batch of N requests] --> B["One matmul, batched"]
  end
  B --> C["Batch-level logits processor
(one call, C++ multithreaded)"]
  C --> E[Sample + return tokens]

The general lesson holds well outside LLM serving: a per-item Python hook inside a hot loop that is otherwise batched is a bottleneck waiting for load, and the fix is rarely "make Python faster" - it is restructuring the interface so the hot path can leave Python's execution model entirely.

Two complications the batch rewrite exposes

Moving to batch-level state tracking is not free, and Netflix's writeup names two operational issues that only appear once you commit to it.

Partial prefills. vLLM V1 supports chunked prefill, where a long prompt is processed across several scheduler steps rather than one. The batch-update interface was not granular enough to express "this request's prefill is only partially done," which broke the state tracking's assumption that a request either has not started or is fully in the decode phase. Netflix added internal tracking to cover the gap.

Preemption. Under memory pressure, the scheduler can evict a request's KV cache mid-generation and later resume it from an earlier point, effectively shrinking its token history. A stateful logits processor (say, one tracking which tokens have already been emitted to enforce a grammar) has to detect that its expected token history no longer matches what it is seeing, reset its internal state machine, and reinitialize from the new, shorter prompt. Skipping this produces state that silently drifts from the actual generation.

Both are instances of the same rule: any per-request state you keep outside the serving engine has to track the engine's actual execution model, including the parts of it (chunked prefill, preemption) that exist purely for GPU memory management and have nothing to do with your feature.

A quieter bug: the field the gateway dropped

Netflix put an OpenAI-compatible HTTP frontend in front of their Triton/vLLM stack, using NVIDIA's own Triton OpenAI-compatible frontend rather than writing one from scratch. response_format - the field that requests JSON-mode output - is a documented part of that schema and was accepted by it, but silently dropped before the request reached vLLM. Nothing errored. Callers requesting JSON output simply stopped getting the guided decoding they asked for.

The fix was to patch the frontend to translate response_format into vLLM's own guided-decoding parameters. The broader point is about API gateways generally: "the field is in the schema and the request validates" is not the same claim as "the field reaches the component that implements it." Anything that sits between a client and an engine and re-encodes the request is a place where a field can be accepted, validated, and dropped, and the failure mode looks exactly like the feature working normally at the API layer while doing nothing underneath.

Cold starts: pre-place the weights, don't fetch them

Large model weights loading from S3 or straight from Hugging Face at deploy time was slow enough to be a real cold-start cost - the kind of thing that turns "roll out a new model version" into a multi-minute wait before the first request can be served. Netflix's fix was to materialize model weights onto Amazon FSx ahead of time, at the point a model is announced rather than at the point it is deployed, so a warm start reads from a high-performance shared filesystem instead of pulling from object storage.

This is a prefetch pattern, not an LLM-specific one: if you can identify the moment a large artifact becomes known (a model is announced, a build finishes, a dataset is published) separately from the moment it is needed (a deploy, a traffic event), moving the transfer to the earlier moment turns a request-time cost into a background one.

One serving stack, two sets of metrics

Running vLLM behind Triton means running two engines that each expose their own Prometheus metrics, and the built-in bridge between them was lossy: it surfaced 9 of vLLM's 40-plus metrics, missing exactly the ones that matter for tuning an LLM serving stack - token throughput, KV cache utilization, prefix cache hit rate.

The fix Netflix describes is a small, general observability pattern: a lightweight HTTP proxy that fetches Triton's own metrics over HTTP, reads vLLM's metrics from disk using Prometheus's multiprocess collector (vLLM runs multiple worker processes, and Prometheus's client library handles that by writing each worker's metrics to shared files rather than serving them from a single in-process registry), and merges both into a single /metrics endpoint.

flowchart TB
  T[Triton /metrics HTTP] --> P[Merge proxy]
  V["vLLM worker metric files
(Prometheus multiprocess mode)"] --> P
  P --> M["Single /metrics endpoint
40+ vLLM metrics + Triton metrics"]

Whenever a serving stack is composed of two components that each ship their own metrics story, the built-in bridge between them is a good place to check what it is actually forwarding before you build dashboards on top of it.

The shape of the lesson

None of these five fixes are about vLLM or Triton specifically. They are about what happens the moment a system that batches well acquires a per-item Python hook, a request-rewriting gateway, a large artifact that has to exist somewhere before it is needed, or more than one component emitting its own telemetry. Self-hosting inference at any real scale surfaces all four, and the fixes are the same ones that apply anywhere else in a distributed system: push the hot path out of the interpreter, verify the gateway forwards what it validates, prefetch what you can, and merge your metrics before you build on top of them.