What Is Machine Learning?

This is the phase where the math starts paying off. You’ve built vectors, matrices, and distance; now you’ll use them to make a computer learn from examples instead of following rules you wrote by hand. We’ll map the whole field in one sitting — the three kinds of learning, the workflow, the traps — and then build the simplest thing that actually learns, from scratch.

Learning objectives

• Explain the shift from writing rules to learning from data, and what “the model” actually is
• Tell supervised, unsupervised, and reinforcement learning apart — and place self-supervised (how LLMs train)
• Distinguish classification from regression and pick the right loss
• Split data into train / validation / test and say why the test set is sacred
• Diagnose overfitting vs underfitting through the bias–variance lens
• Judge when a problem shouldn’t use ML at all
• Build a nearest-centroid classifier from scratch and beat a random baseline

The problem

You want to build a spam filter. The old way: sit down and write rules. “If it says FREE MONEY, mark spam. If it has 3+ exclamation marks, mark spam.” You spend weeks on rules. Spammers change one word. Your rules break. You write more rules. The cycle never ends.

Machine learning flips it. You hand the computer thousands of emails already labeled spam or not spam, and it figures out the patterns itself — patterns you’d never have hand-coded. When spammers adapt, you retrain on new data instead of rewriting logic. That shift — from programming rules to learning from data — is the whole idea, and it powers every recommender, voice assistant, and language model you’ve used.

Learning from data, not rules

Traditional programming and machine learning run in opposite directions. In classic programming, you supply the rules; the program applies them to data to make output. In ML, you supply the data and the desired output, and the algorithm produces the rules.

The model that falls out of training is the rules — encoded as numbers (weights / parameters). It generalizes from the examples it saw to make predictions on data it has never seen. When you “download the weights” of a model, you’re downloading learned rules.

The three families of learning

Most of what you’ll build is supervised. Unsupervised learning shows up in preprocessing and exploration. Reinforcement learning powers game AI, robotics, and the RLHF that aligns language models.

Classification vs regression

The two main supervised tasks — and the one distinction that decides your loss function:

The ML workflow

Every project — no matter the algorithm — walks the same loop:

A humbling reality: cleaning and feature engineering usually eat 60–80% of the time, not the modeling. Good features beat fancy algorithms, and this is exactly why data management came first. Deployment isn’t the end either — data drifts, models decay, and monitoring tells you when to retrain.

Train, validation, test — and the sacred test set

This is the concept beginners most often get wrong. You must judge a model on data it never saw while training, or you’re measuring memorization.

Overfitting, underfitting, and the bias–variance dial

The tell-tale sign of overfitting: a big gap between training and test scores. Bias is error from wrong assumptions (a straight line for a curved truth → underfit); variance is sensitivity to the particular training sample (a wiggly curve through every point → overfit). Total error ≈ bias² + variance + irreducible noise, and you’re hunting the sweet spot between them.

When NOT to use machine learning

ML is powerful, not universal. Before reaching for a model, ask if you actually need one.

Build it: the simplest thing that learns

Meet the nearest-centroid classifier — arguably the simplest ML algorithm there is. Learn: compute the center (mean) of each class. Predict: assign a new point to the nearest center. That “nearest” is the Euclidean distance √Σ(a−b)² — the vector idea straight out of the linear-algebra lesson. No gradient descent, no iteration, no hyperparameters.

import torch

class NearestCentroid:
    def fit(self, X, y):                       # "learn" = one mean per class
        self.classes = torch.unique(y)
        self.centroids = torch.stack([X[y == c].mean(dim=0) for c in self.classes])
        return self

    def predict(self, X):                      # "predict" = nearest center
        dists = torch.cdist(X, self.centroids)  # (n_points, n_classes) distances
        return self.classes[dists.argmin(dim=1)]

# two Gaussian blobs, one per class
torch.manual_seed(42)
X = torch.cat([torch.randn(100, 2) + 1.0,      # class 0 around (+1, +1)
               torch.randn(100, 2) - 1.0])      # class 1 around (-1, -1)
y = torch.tensor([0] * 100 + [1] * 100)

clf = NearestCentroid().fit(X, y)
acc = (clf.predict(X) == y).float().mean()
print(f"accuracy: {acc:.2f}")                  # ~0.94

That’s the entire algorithm: fit computes two means, predict measures distances. And it already captures the pattern every model shares — learn a representation, predict with it, evaluate.

What it can’t do — and why that’s the syllabus

The nearest-centroid classifier assumes each class is a single round blob and draws a straight decision boundary. It breaks when:

• a class has several clusters (the digit “1” is written many ways) — motivates k-nearest neighbors
• the boundary is curved (one class wraps around another) — motivates decision trees / neural nets
• features are on wildly different scales (distance drowns in the biggest one) — motivates feature scaling

Each limitation is the reason a later algorithm exists. That’s the whole path from here to transformers.

Use it

Run the from-scratch classifier and watch it beat the baselines as the classes separate:

python ml-fundamentals/what-is-machine-learning/ml_intro.py

Read it inline without leaving the page: </>ml_intro.py · </>what_is_machine_learning.ipynb.

Ship it

This lesson produces:

• </>ml_intro.py — a from-scratchNearestCentroid, synthetic data, train/test split, and random + majority baselines
• </>what_is_machine_learning.ipynb — the walkthrough, cell by cell
• </>exercises.py / </>solutions.py — task-type framing, baselines, and a train/test-split diagnosis
• </>prompt-ml-problem-framer.md — a prompt that turns a vague business ask into a concrete ML task (learning type, target, features, metric, baseline, pitfalls)

Exercises

1. Take any dataset (Iris, Titanic) and split it 70/15/15 into train/val/test. In one sentence, explain why tuning on the test set invalidates your final number.
2. List three real-world problems. For each, label it classification / regression / clustering and supervised / unsupervised. Defend the trickiest one.
3. A model gets 99% on training data and 60% on test. Diagnose it in one word, then list three fixes you’d try in order.
4. Change the blob separation in ml_intro.py from far to nearly overlapping. Plot accuracy vs. separation and explain the trend through the bias–variance lens.

Key terms