The Journey of a Request: How LLM Serving Works

This series starts from a real question: when you’re handed a GPU box from a cloud provider and told to “deploy the model,” what do you actually need to know? The honest answer is a lot — GPUs, drivers, serving engines, the KV cache, quantization, parallelism, autoscaling, cost — and it happens to be exactly the material that shows up in infrastructure interviews. So rather than a checklist, we’re going to build the intuition, one layer at a time, following a single request all the way down to the silicon and back.

We’re not training anything here, and we’re not tied to one cloud vendor — the ideas transfer across Lambda, RunPod, CoreWeave, AWS, or a bare-metal box in a rack. This first post is the foundation the rest stands on: the mental model of what happens when a model serves a request. Get this right and every later topic — batching, PagedAttention, quantization, tensor parallelism — is just a detail hanging off it.

The one sentence to internalize

Generating text is not one big calculation. It’s a loop that produces one token at a time, and each new token depends on every token before it. Picture writing a sentence where you may only ever add the next word, and to choose it you re-read everything written so far. That “re-read everything” is the expensive part — and essentially the entire craft of LLM serving is about making that re-read cheap.

This property is called autoregressive generation, and it splits every request into two phases that could not be more different.

Follow one request

Here is the whole journey, from an HTTP call to text streaming back. Read the top band first, then the bottom.

The five steps: your text is tokenized into integer IDs (cheap, on the CPU); then prefill runs one big pass over the entire prompt; then the decode loop emits tokens one by one until it hits a stop condition; each token is detokenized back into text and streamed to you as it’s ready. The two that matter are prefill and decode, and the single most useful table in this whole series is the one that contrasts them:

Why the KV cache exists

Naively, every decode step would re-run the whole sequence so far — the prompt plus everything generated — which makes the total work grow with the square of the length. Instead, the model caches the Key and Value tensors that each token produces inside the attention layers, so the next step only has to do one token’s worth of new work and attend against the stored cache. That’s the KV cache. It is the reason decode is fast enough to be practical — and, as we’ll see in a moment, the reason your GPU runs out of memory.

The hardware you’re renting

Now the layer underneath. A GPU has two resources that matter for serving, and — this is the key insight — they run out independently:

• VRAM (also called HBM) — the GPU’s on-board memory. It holds the weights, the KV cache, and the transient activations. This decides whether the model fits at all. An H100 has 80 GB; an A100 has 40 or 80 GB.
• Memory bandwidth — how fast bytes move between VRAM and the compute cores (an H100 does about 3.35 TB/s). This decides how fast decode runs.
• FLOPs / Tensor cores — raw matmul throughput (H100 ≈ 990 TFLOPS in BF16). This decides how fast prefill runs.

That gives a back-of-the-envelope ceiling for a single request’s decode speed:

max tokens/sec  ≈  memory_bandwidth  /  model_size_in_bytes

# 13B model in FP16 (26 GB) on an H100 (3.35 TB/s):
#   3350 GB/s  /  26 GB  ≈  ~129 tokens/sec   (single-stream upper bound)

Real numbers land lower, but the shape is what matters, and two enormous levers fall straight out of it:

• Quantize the model → move fewer bytes → decode gets faster. Halve the model’s size and you roughly double the decode ceiling. That’s why quantization (a later post) is a latency win, not just a memory one.
• Batch many requests together → amortize the weight read. You’re already paying to stream all the weights out of VRAM; you may as well push dozens of users’ tokens through them on the same pass. That’s why batching is the throughput lever, and it gets its own post.

One more hardware term to bank for later: NVLink vs PCIe. When a model is too big for one GPU and gets split across several, those GPUs must constantly swap partial results. NVLink is the fast, direct GPU-to-GPU highway; PCIe is the slow road through the host. Tensor parallelism only works well over NVLink — a detail that quietly decides which boxes are worth renting.

Will it fit? The VRAM math

The calculation every interviewer eventually asks for. Total VRAM is three buckets plus overhead:

Total VRAM  ≈  Weights  +  KV cache  +  Activations  +  Overhead

Bucket 1 — the weights

weights_bytes  =  num_params  ×  bytes_per_param

The reflex to build: params × 2 = GB in FP16. 7B → 14, 70B → 140. Instant, no calculator.

Bucket 2 — the KV cache (the one people forget)

kv_bytes_per_token  =  2 (K and V)  ×  num_layers  ×  num_kv_heads  ×  head_dim  ×  bytes
total_kv            =  kv_bytes_per_token  ×  seq_len  ×  batch_size

Worked for Llama-2-13B in FP16 (40 layers, 40 KV heads, head_dim 128):

per token          = 2 × 40 × 40 × 128 × 2  =  819,200 bytes  ≈  0.8 MB / token
4096-token context =  0.8 MB × 4096          ≈  3.2 GB   (one sequence)
same, batch of 32  =  3.2 GB × 32            ≈  105 GB   ← larger than the 26 GB of weights!

There’s the headline: at real serving batch sizes, the KV cache can dwarf the model itself. This is the single fact that motivates half of the engine internals we’ll cover — PagedAttention exists to manage exactly this memory, and it’s why long context is expensive.

Buckets 3 & 4 — activations and overhead

Activations (the transient tensors of a single forward pass) are modest at inference compared to training. Overhead — the CUDA context, the framework, memory fragmentation, the allocator’s reserved pool — is worth budgeting a couple of gigabytes for. A serviceable quick estimate:

Total  ≈  (Weights + peak KV cache)  ×  ~1.2

Check yourself

What’s next

You now have the map: a request is a cheap parallel prefill followed by an expensive sequential decode; the GPU limits you by VRAM (fit) and bandwidth (speed); and the KV cache is the quiet memory hog. Every later post zooms into one band of that diagram.

Next, we get hands-on: the bring-up stack on a fresh box — driver, CUDA, the NVIDIA Container Toolkit — and serving a first model, with a free local stand-in so you can follow along without renting a GPU yet. That’s where this stops being a diagram and starts being something running on a port.

• Serving = prefill (parallel, compute-bound, TTFT) then decode (sequential, bandwidth-bound, TPOT).
• The KV cache makes decode affordable — and can outgrow the weights at real batch sizes.
• A GPU runs out of VRAM and bandwidth separately; know which one you’re hitting.
• params × 2 = GB in FP16, plus a KV-cache estimate, is the whole “will it fit?” answer.