Vectors & Matrices: The Operations

Last lesson was the intuition — what vectors, dot products, and rank mean. This one is the toolkit: the actual operations you type, the shapes that have to line up, and the two multiplications everyone confuses. By the end you’ll build a full neural-network layer from scratch — and read tensor-shape errors like plain English, which is half of every ML interview and most of the debugging you’ll ever do.

Learning objectives

• Do the core operations by hand — add, scale, transpose, multiply — and know which rule each one follows
• Tell element-wise (*) apart from matrix multiply (@) instantly, and know when each applies
• Read a shape mismatch and fix it — the single most useful debugging reflex in ML
• Explain determinant (area/volume scaling) and inverse (undo), and why det = 0 breaks both
• See why the identity matrix is a residual connection and how broadcasting adds bias for free
• Build one dense layer — relu(W @ x + b) — with no libraries, then in PyTorch
• Walk into an interview able to reason out loud about shapes, commutativity, transposes, and invertibility

The problem

You open a neural network and the whole forward pass of a layer is one line:

output = activation(W @ x + b)

That @ is matrix multiplication. W is a matrix of weights, x is the input vector, b is a bias, and activation bends the result. If those operations are fuzzy, this line is magic. If they’re not, it’s just transform, shift, bend — three moves. Every image is a grid of numbers, every embedding is a vector, every layer is a matrix transformation. Fluency here is like knowing variables before you write code — non-negotiable.

The operations map

Seven operations carry almost all of deep learning. Here’s the whole territory before we walk it:

Warm up: vector operations

Adding two vectors is tip-to-tail: walk along the first, then along the second; where you land is the sum. Scaling multiplies the length (and flips direction if the scalar is negative). No shape rules to worry about yet — same-length in, same-length out.

import torch

a = torch.tensor([3., 1.])
b = torch.tensor([1., 2.5])

print(a + b)       # tensor([4.0000, 3.5000])  -> tip-to-tail
print(2 * a)       # tensor([6., 2.])          -> scaled (twice as long)
print(a * b)       # tensor([3.0000, 2.5000])  -> ELEMENT-WISE: positions multiplied
print(a @ b)       # tensor(5.5000)            -> DOT PRODUCT: one number

Look at those last two lines. Same two vectors, one * away from each other, completely different objects: a * b is a vector (positions multiplied), a @ b is a single number (multiply and add). That split is the whole ballgame once we get to matrices.

The distinction that trips everyone: * vs @

There are two ways to “multiply” matrices and they are not variations on a theme — they’re different operations with different rules and different shapes out.

Element-wise (Hadamard): matching positions

Line the two grids up and multiply cell by cell. Both matrices must be the same shape, and the result is that same shape.

| 1  2 |     | 5  6 |     | 1·5  2·6 |   |  5  12 |
| 3  4 |  *  | 7  8 |  =  | 3·7  4·8 | = | 21  32 |

Matrix multiply: rows into columns

Each output cell is a dot product: row i of the left with column j of the right. This is the operation from last lesson — a whole grid of dot products at once.

| 1  2 |     | 5  6 |     | 1·5+2·7  1·6+2·8 |   | 19  22 |
| 3  4 |  @  | 7  8 |  =  | 3·5+4·7  3·6+4·8 | = | 43  50 |

Shapes are the grammar

Matrix multiply has one hard rule, and internalizing it will save you more time than any other single thing in this course:

(m × k) @ (k × n) = (m × n)
         ↑     ↑
      these must match — and they cancel

Write the shapes side by side; the inner pair has to be equal and it disappears, and you keep the outer pair. It’s dimensional cancellation, like units in physics.

(128 × 784) @ (784 × 1) = (128 × 1)
   weights       input       output
inner: 784 = 784  ✓ valid   ->  784-dim input becomes a 128-dim output

Matrix multiply is composition: do B, then A

Here’s the why under the mechanics. From last lesson, a matrix is a transformation — vector in, moved vector out. So what is A @ B? It’s a single matrix that does both transforms back to back: (A @ B) x = A (B x) — first B transforms x, then A transforms the result. You read it right to left: the matrix nearest the vector goes first.

That one idea explains two rules you just took on faith:

• Why the inner dims must match. B maps a vector into some space, and A has to accept that space as its input. So B’s output size = A’s input size — the inner numbers meet in the middle because that’s exactly where one transform hands off to the next.
• Why order matters. Doing B then A generally lands somewhere different from A then B, so AB ≠ BA. It isn’t a quirk of the arithmetic — it’s the plain fact that the order you apply operations in changes the outcome.

Transpose: flip it

Transpose swaps rows and columns — cell (i, j) becomes (j, i). An (m × n) matrix becomes (n × m). That’s it mechanically, but it’s everywhere: it’s the Kᵀ in attention’s Q @ Kᵀ, and it’s how gradients travel backward through a layer (the backward pass multiplies by Wᵀ).

A = torch.tensor([[1, 2, 3],
                  [4, 5, 6]])   # (2 × 3)

print(A.T)                      # (3 × 2)
# tensor([[1, 4],
#         [2, 5],
#         [3, 6]])

One property is a genuine interview favorite because you can reason it out instead of memorizing it: transposing a product reverses the order.

(A @ B).T  ==  B.T @ A.T

Determinant: how much space gets stretched

Feed the two columns of a 2×2 matrix in as vectors; they span a parallelogram. The determinant is the (signed) area of that parallelogram — how much the transformation blows up or shrinks space. In 3-D it’s a volume; in general it’s the volume-scaling factor of the transform.

For a 2×2 the formula is short: det([[a, b], [c, d]]) = ad − bc. What the number says matters more than how you compute it:

Inverse: undo the transformation

If a matrix A maps x to A @ x, its inverse A⁻¹ maps it straight back: A⁻¹ @ (A @ x) = x. Equivalently A @ A⁻¹ = I. It exists only when det ≠ 0 — you can’t undo a transform that already threw a dimension away.

A = torch.tensor([[4., 7.],
                  [2., 6.]])

Ainv = torch.linalg.inv(A)
print(torch.round(A @ Ainv))   # the identity -> confirmed inverse
# tensor([[1., 0.],
#         [0., 1.]])

singular = torch.tensor([[1., 2.],
                         [2., 4.]])   # row 2 = 2·row 1  ->  det = 0
# torch.linalg.inv(singular)  ->  LinAlgError: singular matrix

Identity: the do-nothing matrix (and residual connections)

The identity I has 1s down the diagonal and 0s everywhere else. A @ I = A, I @ x = x — it’s multiplication by 1. Sounds boring until you notice what I + something does.

I = torch.eye(3)
print(I)
# tensor([[1., 0., 0.],
#         [0., 1., 0.],
#         [0., 0., 1.]])

x = torch.tensor([5., 7., 9.])
print(I @ x)     # tensor([5., 7., 9.])  -> unchanged

Broadcasting: free shape-stretching

When you add a bias vector to a whole matrix of outputs, the shapes don’t match — yet it just works. Broadcasting stretches the smaller array along the missing dimension so the two line up, without ever copying the data in memory.

outputs = torch.tensor([[1, 2, 3],
                        [4, 5, 6]])   # (2 × 3)  — two examples
bias    = torch.tensor([10, 20, 30])  # (3,)     — one per feature

print(outputs + bias)
# bias is stretched across both rows:
# tensor([[11, 22, 33],
#         [14, 25, 36]])

The rule: line the shapes up from the right; each dimension must be equal, or one of them must be 1 (a size-1 or missing dim gets stretched). That’s exactly how W @ x + b adds one bias per output unit across an entire batch, for free.

The payoff: build a dense layer from scratch

Every piece is now on the table. A dense (fully-connected) layer is exactly relu(W @ x + b): transform the input with a weight matrix, shift it with a bias (broadcast), bend it with an activation. Here’s the whole thing with a hand-built Matrix — no libraries — so nothing is hidden:

class Matrix:
    def __init__(self, data):
        self.data = [list(r) for r in data]
        self.rows, self.cols = len(self.data), len(self.data[0])
        self.shape = (self.rows, self.cols)

    def __add__(self, other):                       # broadcasts a column bias
        return Matrix([[self.data[i][j] + other.data[i][0]
                        for j in range(self.cols)] for i in range(self.rows)])

    def matmul(self, other):
        return Matrix([[sum(self.data[i][k] * other.data[k][j]
                            for k in range(self.cols))
                        for j in range(other.cols)] for i in range(self.rows)])


def relu(m):
    return Matrix([[max(0.0, v) for v in row] for row in m.data])


x = Matrix([[0.5], [0.8], [0.2]])                   # (3 × 1) input
W = Matrix([[0.2, -0.4, 0.1], [-0.3, 0.9, 0.5]])    # (2 × 3) weights
b = Matrix([[0.1], [0.1]])                          # (2 × 1) bias

out = relu(W.matmul(x) + b)                         # (2 × 3)@(3 × 1) + (2 × 1)
print(out.shape, out.data)                          # (2, 1) -> the layer's output

Then the same thing in PyTorch — identical math, orders of magnitude faster:

x = torch.tensor([[0.5], [0.8], [0.2]])   # (3, 1)
W = torch.randn(2, 3)                      # (2, 3)
b = torch.tensor([[0.1], [0.1]])           # (2, 1)

out = torch.relu(W @ x + b)                # relu(W @ x + b)
print(out.shape)                           # torch.Size([2, 1])

Think like an interviewer

This section is here on purpose: interviews don’t test whether you memorized ad − bc — they test whether you can reason about shapes and operations out loud. Here’s the thought process that makes you sound fluent, plus the traps that catch people.

The shape-debugging reflex

Ninety percent of real ML bugs are shape bugs, and interviewers know it. When you hit mat1 and mat2 shapes cannot be multiplied (32×784 and 128×784), don’t guess — run the checklist out loud:

Saying that sequence while you debug signals more competence than any clever one-liner. The fix above is X @ W.T: (32 × 784) @ (784 × 128) = (32 × 128).

Questions you can reason out (not memorize)

Where each idea shows up

Use it

Run the from-scratch Matrix and the PyTorch mirror, then watch the diagrams regenerate:

python foundations/matrix-operations/matrices.py     # from-scratch Matrix + a dense layer, then PyTorch
python foundations/matrix-operations/visualize.py    # element-wise vs matmul, determinant-as-area, broadcasting

Read them inline without leaving the page: </>matrices.py · </>matrix_operations.ipynb.

Ship it

This lesson produces:

• </>matrices.py — a from-scratch Matrix (add, scalar, element-wise, matmul, transpose, determinant, inverse, identity), a dense layer, and the PyTorch equivalents side by side
• </>matrix_operations.ipynb — the walkthrough, cell by cell
• </>exercises.py / </>solutions.py — verify the inverse, extend to 3×3, and build a two-layer network with no libraries
• </>visualize.py — regenerates the operation diagrams as PNGs

Exercises

1. Verify the inverse. Multiply A @ inv(A) for three different 2×2 matrices and confirm you get the identity. Then try one with det = 0 — what happens, and why?
2. Prove the transpose rule. Pick any non-square A and B that can multiply, and check numerically that (A @ B).T == B.T @ A.T. Then explain, from shapes alone, why A.T @ B.T usually isn’t even legal.
3. Element-wise vs matmul. Take two 2×2 matrices and compute both A * B and A @ B. Write one sentence on why a forget gate wants the first and a layer wants the second.
4. Build a two-layer network. Using only your Matrix class (no libraries), wire input (3) → hidden (4) → output (2) with random weights, run a forward pass, and assert every intermediate shape is what you predicted.
5. Break it on purpose. Add a shape (3,) tensor to a shape (3, 1) tensor in PyTorch. Predict the output shape before you run it, then explain the broadcasting that produced it.

Key terms