Linear Algebra, Intuitively

Linear algebra sounds scary. It isn't. Three ideas — vectors, the dot product, and matrix multiplication — are enough to understand what every layer of a neural network actually does. This is the intuition, not the proofs.

Learning objectives

• Picture a vector as a point / direction in space — and see why an embedding is just a vector
• Read the dot product as a similarity score
• See matrix multiplication as a batch of dot products (and as a transformation)
• Tell when vectors are linearly independent, and why rank — how many directions are real — powers tricks like LoRA
• Connect it all to what a Transformer computes when it "pays attention"

The problem

Open any model and it's linear algebra all the way down: tokens become vectors (embeddings), a layer multiplies them by a weight matrix, and attention scores every vector against every other with dot products. If those three operations feel natural, deep learning stops being a black box. You don't need to prove theorems — you need a feel for the shapes and what they mean.

The concept

Everything scales up from the vector. A scalar is one number. A vector is a list of numbers — a point in space, or an arrow from the origin. A matrix is a grid: either a stack of vectors, or a machine that transforms vectors. Multiplication is how vectors get compared and combined.

Build it

Step 1 — Vectors are meaning

A vector is just a list of numbers, but you can treat each number as a coordinate. Two words with similar meaning end up as vectors pointing in similar directions — that's the whole idea behind embeddings.

And in AI, everything gets this treatment — the vector is the universal container for meaning:

import numpy as np

# Toy 4-D "embeddings" — real ones have hundreds/thousands of dims
king  = np.array([0.9, 0.8, 0.1, 0.7])
queen = np.array([0.9, 0.2, 0.1, 0.8])
apple = np.array([0.1, 0.1, 0.9, 0.2])

print(king.shape)          # (4,)  -> a point in 4-D space
print(np.linalg.norm(king))  # its length (magnitude)

Step 2 — The dot product is similarity

Multiply the vectors element-wise and add it all up. One number comes out — and its sign already tells a story:

This one operation is quietly running half the modern stack: vector search embeds your query and dots it against every stored document, recommender systems dot user vectors against item vectors, and RAG retrieval is "return the chunks with the highest scores." Different products — same multiply-and-add.

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

print(round(cosine_similarity(king, queen), 3))  # high  -> similar
print(round(cosine_similarity(king, apple), 3))  # low   -> different

Step 3 — Linear independence: how many directions are real?

Give me a set of vectors. The question that matters isn't how many there are — it's how many point in genuinely new directions. A vector is redundant if you can build it by scaling and adding the others; it takes you nowhere the rest couldn't already reach.

v1 = np.array([1, 0, 0])
v2 = np.array([0, 1, 0])
v3 = np.array([2, 1, 0])       # = 2*v1 + v2  ->  adds no new direction

V = np.stack([v1, v2, v3])
print(np.linalg.matrix_rank(V))   # 2, not 3

v1 and v2 are independent — two real directions. But v3 = 2·v1 + v2, so it's old news: three vectors, yet only two real directions. They all lie flat in the x-y plane and can never reach [0, 0, 1]. That count of genuinely independent directions is the rank — here it's 2.

Here's the same idea as a diagnosis table — what a matrix's rank tells you about your model:

Step 4 — Matrix × vector is a transformation

A matrix times a vector produces a new vector — rotated, scaled, or projected into a different space. A linear layer in a network is exactly this: y = W x (plus a bias). Training is the search for the weight matrix W that maps inputs to useful outputs.

Sit with that for a second, because it flips how you see models: the matrices are the model. The weights are matrices that transform vectors, attention is a matrix of scores deciding what to look at, and the embedding table is a matrix mapping token IDs to meaning. When you download "the weights," you're downloading a stack of transformations.

W = np.array([[1., 0., 0., 0.],
              [0., 1., 0., 0.]])   # (2, 4): projects 4-D down to 2-D

y = W @ king                       # matrix @ vector
print(y, y.shape)                  # -> a new 2-D vector

Step 5 — Matrix × matrix is many dot products at once

Matrix multiplication is nothing more than every row of the first, dotted with every column of the second. That's why shapes have to line up: an (m × k) times a (k × n) gives an (m × n).

A = np.random.rand(2, 3)   # (2, 3)
B = np.random.rand(3, 4)   # (3, 4)
C = A @ B                  # (2, 4)  — inner 3's cancel
print(C.shape)

Step 6 — This is attention

Here's the payoff. In self-attention, each token is a vector. To decide how much token i should attend to token j, the model takes the dot product of their query and key vectors — a similarity score. Doing that for every pair at once is a single matrix multiply: scores = Q @ K.T.

Q = np.stack([king, queen, apple])   # (3, 4): one query vector per token
K = Q.copy()                          # (3, 4): the keys

scores = Q @ K.T                      # (3, 3): every query vs every key
print(scores.shape)
print(np.round(scores, 2))            # row i = how much token i matches each token

Where each idea shows up

Nothing in this lesson is math for math's sake — each idea is doing a specific job in systems you'll build:

Use it

Run the demo to see similarities and an attention-score matrix print out:

python foundations/linear-algebra/vectors.py

Read it inline without leaving the page: </>vectors.py.

Ship it

This lesson produces:

• </>vectors.py — vectors, cosine similarity, a linear layer, and attention scores in ~30 lines of numpy
• </>linear_algebra.ipynb — the same, cell by cell, to poke at

Exercises

1. Make two toy 3-D "word" vectors you think are similar and two that aren't; compute cosine similarity and check your intuition.
2. Write matmul(A, B) with plain Python loops (no numpy) and verify it matches A @ B.
3. Given Q and K of shape (5, 8), what shape is Q @ K.T? Compute it and say what a single entry means.
4. Normalize a batch of vectors to unit length in one numpy expression.

Key terms