TTFT, TPOT, and the Latency Trade-off

The last two posts were about throughput: paging packs memory, continuous batching keeps the GPU busy. But a user never feels throughput — they feel how long the answer takes to start, and how fast it streams. Those are the two latency numbers every SLA is written in, and they’re in direct tension with the throughput we just maximised.

The two numbers every SLA is written in

• TTFT — time to first token. From arrival to the first token appearing. It’s queue wait + prefill: how long before a slot opens, plus the one pass over the whole prompt. It grows with prompt length (longer prompt = heavier prefill) and with load (more queueing). This is what makes a chat feel responsive or sluggish to start.
• TPOT — time per output token (also called ITL, inter-token latency). The steady gap between streamed tokens during decode — essentially one decode-step time. It grows with batch size (more sequences share each step) and with context length (a bigger KV cache to read). This is the streaming speed the user watches.

end-to-end latency  =  TTFT  +  (N_output − 1) × TPOT

# a 500-token reply with TTFT 300 ms and TPOT 40 ms:
#   300 ms  +  499 × 40 ms  ≈  20.3 s   ← TPOT dominates long replies

The trade-off: throughput vs latency

Here’s the tension. From the batching post, a decode step streams the whole model from VRAM regardless of batch size, so a bigger batch turns one weight-read into more tokens — higher throughput. But each extra sequence adds a little compute to the step, so the step gets longer — higher TPOT. Model it simply:

TPOT(B)       = base + k·B              (base = weight streaming, fixed; k = per-sequence cost)
throughput(B) = B × 1000 / TPOT(B)      → approaches a ceiling of 1000/k as B grows

Drag the batch size and watch both move at once:

Two things the curve makes obvious. Throughput saturates — doubling the batch from 64 to 128 adds only ~13% more throughput (762 → 865 tok/s). And TPOT never stops rising — that same doubling nearly doubles per-token latency (84 → 148 ms), making every user’s tokens noticeably slower. Cross the SLA line and you’re technically serving more tokens per second while every individual user has a worse experience.

Prefill vs decode: where the two metrics collide

The reason TTFT and TPOT interfere is that prefill and decode are completely different workloads sharing one GPU:

When a big prefill lands in an iteration, it monopolises the GPU and every in-flight decode stalls for that step — a TPOT spike felt by everyone already streaming. Two ways out:

• Chunked prefill (from the last post): split the prompt into chunks and interleave them with ongoing decode steps. Smoother TPOT for everyone, at a slightly higher TTFT for the arriving request. It’s a dial between the two metrics.
• Prefill/decode disaggregation: run prefill and decode on separate instances (tuned differently — prefill boxes for compute, decode boxes for bandwidth) and ship the KV cache between them. They stop fighting entirely, so TTFT and TPOT can be optimised independently. This is what large-scale systems (DistServe, Splitwise, and modern production stacks) do.

Measure the tail, and pick the operating point

• Report percentiles, not the mean. Bursts and long prompts create a long tail; p99 TTFT is what an unlucky user actually experiences. A great average with a terrible p99 is a bad service.
• The workload picks where you sit on the curve. Interactive chat → tight TTFT/TPOT, smaller batch, live near the knee. Offline/batch jobs (bulk summarisation, evals) → latency doesn’t matter, so crank the batch to the throughput ceiling. Same server, two different operating points.

Check yourself

What’s next

We can now reason about both axes: throughput (paging + batching) and latency (TTFT/TPOT and their trade-off). The obvious next question is whether we can cheat the decode bottleneck itself — get more than one token per expensive weight-read, and cut TPOT without shrinking the batch. That’s speculative decoding, and it’s the next post.

• TTFT = queue + prefill (grows with prompt length); TPOT = decode-step time (grows with batch size).
• End-to-end ≈ TTFT + (N−1)·TPOT — TPOT dominates any long reply.
• Bigger batch → more throughput but higher TPOT; throughput saturates, so the SLA caps the batch. Optimise goodput.
• Prefill (bursty, compute-bound) fights decode (steady, bandwidth-bound); chunked prefill and disaggregation resolve it.