LLM Infra · Series

Hosting LLMs, From Scratch

How to host and serve large language models — following one request from the API down to the GPU and back. The mental model, the serving engine internals (batching, the KV cache, PagedAttention), quantization, parallelism, autoscaling, and cost. Built for real intuition and for the questions that come up on the job and in interviews.

0 of 2 posts complete0%

The Journey of a Request: How LLM Serving Works

~14 min read

Phase 0Concept

The foundation for the series: a request is a parallel prefill then a sequential decode loop, prefill is compute-bound (TTFT) and decode is memory-bandwidth-bound (TPOT), the KV cache makes decode affordable but can outgrow the weights, and the VRAM math that answers “will it fit?”.

What Actually Lives in the KV Cache

~15 min read

Phase 2Deep Dive

A deep dive on the thing everyone gets slightly wrong: from a token ID you look up an embedding — but that is not what gets cached. On a tiny 4-dim model where every number is real, watch prefill fill the cache token by token and decode reuse it (Q·K → softmax → weighted V). The cache stores K and V, one pair per token per layer, and never the embedding or Q.