Continuous Batching: Keeping the GPU Busy

PagedAttention solved where many KV caches live so they fit in one GPU. This post is the other half: when each sequence actually runs, so the GPU never sits idle while requests of different lengths come and go. The technique is continuous batching, and together with paging it’s why a modern server gets the throughput it does.

First: why batch at all?

Recall from the request’s journey that decode is memory-bandwidth-bound — every single token step streams the entire model out of VRAM. If you serve one request at a time, you pay that whole weight-read to produce one token. If you run many sequences together, you pay it once and get one token from each:

Same memory traffic, 32× the useful output. That’s why batching is the throughput lever. The whole problem is how to batch when requests have different lengths and arrive at different times.

Static batching, and why it stalls

The naïve approach: gather a batch, run it together until every sequence in it finishes, then start the next batch. Here are 6 requests on a GPU with room for 3 at once. Watch what static batching does — note that R2 is a long generation, and R4–R6 are stuck waiting:

Two failures are visible in that chart:

• Ragged completion (head-of-line blocking). R1 and R3 finish early, but their slots can’t be reused — the batch runs until the slowest (R2) is done. Look at the slots strip: columns 4–8 are running at 1/3. Two-thirds of the GPU is idle, held hostage by one long sequence.
• No mid-flight admission. R4–R6 sit in the queue for the entire first batch, even though slots are free from iteration 3 onward. Their time-to-first-token is terrible for no reason.

Result: 12 iterations, only 61% utilisation. And real workloads have far more length variance than this toy, so it’s usually worse.

Continuous batching: decide every iteration

The fix is to make the batching decision every token step instead of once per batch — what the Orca paper called iteration-level scheduling. After each decode step:

• Any sequence that hit its stop token leaves immediately and its result is returned.
• Waiting requests join the running batch the same iteration a slot opens up.

Same 6 requests, same 3 slots — step through it and watch the freed slots get refilled at once:

8 iterations at 92% utilisation, versus static’s 12 and 61%. Same work, done in two-thirds the time, with the GPU near-full the whole way — and R4–R6 started decoding as soon as a slot freed instead of waiting out the first batch, so their latency drops too. The batch is a living thing now: its membership changes on every single token.

How it fits with everything else

Continuous batching doesn’t stand alone — it’s the scheduler sitting on top of the memory system from the last post:

• PagedAttention makes it possible. The batch’s membership changes every iteration and each sequence’s KV cache grows independently. That only fits in memory because paging allocates KV blocks on demand and packs them tightly. Paging = where the caches live; continuous batching = when the sequences run.
• It trades against latency. A bigger batch means more throughput, but each step now does more compute, so time-per-output-token (TPOT) rises. Admitting a fresh prefill adds a blip to in-flight decodes. There’s a knee: past some batch size you’re trading real latency for diminishing throughput.

Check yourself

What’s next

We’ve now maxed the throughput axis: paging packs memory, continuous batching keeps the GPU busy. The other axis is latency — and it’s in tension with everything we just did. Next we’ll pin down the two numbers every serving SLA is written in, TTFT and TPOT, where each comes from, and how batch size and prefill scheduling let you trade one against the other.

• Batching amortises the per-step weight-read from VRAM — it’s the throughput lever for bandwidth-bound decode.
• Static batching stalls on ragged completion and blocks new arrivals; utilisation collapses with length variance.
• Continuous batching schedules every iteration: finished sequences leave, queued ones join — ~90%+ utilisation.
• It rides on PagedAttention and trades throughput against per-token latency.