LLM Infra · Series · 02 of 6

What “13B Parameters” Actually Means

Phase 0TypeConceptTime~9 min readPrereqThe Journey of a Request

// TL;DR

"13B parameters" means 13 billion numbers in total — the B is billion. It is the whole count, not squared, not 13B × 13B.
Those 13 billion are every weight in the model, not just the Q/K/V matrices: embeddings, all four attention projections, the feed-forward (MLP) blocks, and the output head. The MLP blocks hold the most — more than attention.
A parameter count does not tell you the model’s dimension. d_model for a 13B model is 5120, not √13B ≈ 114,000 — because the weights are spread over hundreds of matrices across dozens of layers, not one giant square.
Memory = parameter count × bytes per parameter, and the format sets the bytes: FP32 = 4, FP16/BF16 = 2, INT8 = 1, INT4 = 0.5. So 13B in FP16 ≈ 26 GB.
Play with the calculator: change hidden dim and layers, watch the total, the embedding/attention/MLP split, and the memory at each precision.

The first post leaned on a reflex — “params × 2 = GB in FP16” — but skipped a fair question: what is a “parameter,” and what does “13B” actually count? Two wrong mental models are common, and both are worth killing before the VRAM math means anything.

13B = 13 billion numbers, total

A trained model is one enormous pile of numbers — the weights it learned. A “parameter” is one of those numbers. “13B parameters” is simply the count of them: 13,000,000,000. That’s it. The B means billion. It isn’t squared, and it isn’t 13B × 13B — the 13B is the final tally.

What’s actually in the pile

Four kinds of weights make up the total. Their relative sizes surprise people:

Embedding + output head — the token-id→vector lookup table and the final vector→vocabulary layer. A small slice.
Attention — per layer, four matrices: W_Q, W_K, W_V, and W_O (each ~d × d).
MLP / feed-forward — per layer, two or three wide matrices. This is the biggest bucket — bigger than attention.
LayerNorm scales — a handful of numbers per layer. Negligible.

Change the shape below and watch the split — notice attention is only about a third, and the MLP dominates:

preset:

d_model (hidden dim)5120layers40

total parameters ≈13B12,910,592,000

embedding + head · 328M (3%) attention (Q,K,V,O) · 4.2B (32%) MLP / feed-forward · 8.4B (65%)

format	bytes/param	weights in memory
FP32	4	52 GB
FP16 / BF16	2	26 GB
FP8 / INT8	1	13 GB
INT4	0.5	6.5 GB

And note: d_model = 5120, but √(total) = 113,625. The dimension is nowhere near the square root of the parameter count — more on why below.

// question

A model is described as “13B parameters.” What does that number count?

Dimension is NOT the square root of the count

The second trap: assuming d_model = √(params). For a 13B model, √13B ≈ 114,000 — but the real hidden dimension is 5120. Off by more than 20×. Here’s why the square-root intuition fails:

text

√(params) = dimension   ONLY IF the model were a single d × d matrix.

It isn't. A 13B model is ~hundreds of matrices across ~40 layers:

   total  ≈  (number of matrices) × d²          (roughly)
   d      ≈  √( total / number_of_matrices )     ← not √(total)

check:  √(13e9 / ~480 matrices)  =  √(27e6)  ≈  5200  ≈  5120  ✓

d_model is an architecture choice (5120 for a 13B, 4096 for a 7B, 8192 for a 70B), picked by the model designers — not something you derive from the parameter count. The parameter count is the consequence of choosing a dimension and a number of layers, not the other way around.

// question

A 13B model has hidden dimension d_model = 5120. Why isn’t d_model equal to √13B ≈ 114,000?

From count to memory: the format sets the bytes

Once you have the count, memory is one multiply:

text

memory = parameter_count × bytes_per_parameter

Each parameter is a real number like 0.0327 or −1.84. To store a real number you pick a format, and the format decides how many bits (and therefore bytes — remember 8 bits = 1 byte) each one takes, trading memory against precision:

Format	Bits	Bytes	What it is
FP32	32	4	“full precision” float — most precise, original training format
FP16 / BF16	16	2	“half precision” float — half the memory, slightly less precise
FP8 / INT8	8	1	8-bit float / integer (quantized)
INT4	4	0.5	4-bit integer — smallest, most lossy

FP = floating point (a number with a decimal point). INT = integer, via quantization (squeeze the values into whole-number buckets). Fewer bits = less memory but coarser rounding, so the model gets a little less accurate — the whole trade-off behind quantization (its own post later).

Check yourself

What a parameter count means

0/3 answered

// question

A 7B model is loaded in FP16. Roughly how much GPU memory do the weights take?

// question

In a typical decoder-only LLM, which part holds the most parameters?

// question

How is a model’s hidden dimension (d_model) determined?

What’s next

Now “13B” is concrete: 13 billion weights, mostly in the MLP, at a dimension the architects chose — and multiplying by bytes-per-parameter gives the memory. That’s the weights half of the VRAM math; the other half, the KV cache, is its own post. The precision trade-off — how far you can drop the bytes before the model degrades — becomes the quantization deep dive later in the series.

“13B” = 13 billion total weights — the whole model, not just Q/K/V, not squared.
Most of them live in the MLP/feed-forward blocks, not attention.
d_model is a chosen dimension (5120 for a 13B), not √(param count).
Memory = count × bytes/param; the format (FP16 = 2 bytes) sets the bytes.

series progress0%