The intimidating version of this topic is a four-year syllabus. The useful version is much smaller: four areas, and within each, a handful of ideas that show up over and over in model code. You do not need to prove theorems. You need enough fluency to read a paper's equations, debug a training run that will not converge, and know why a metric is lying to you.
Linear algebra — the language of the data
Every batch of data is a matrix; every layer is a matrix multiply followed by something nonlinear. What matters is shape intuition — being able to say what dimensions go in and come out of any operation — because roughly a third of practical ML debugging is shape errors. Beyond that: dot products as similarity (the basis of every embedding search you will ever build), matrix multiplication as composed linear maps, transposes and broadcasting, norms as measures of size (L1 and L2 regularization are literally these), and eigenvectors/SVD as the machinery behind PCA and low-rank approximation. LoRA fine-tuning is, at bottom, the observation that an update matrix can be approximated by the product of two skinny ones.
import numpy as np
# cosine similarity — the whole of semantic search in three lines
def cosine(a: np.ndarray, b: np.ndarray) -> np.ndarray:
a = a / np.linalg.norm(a, axis=-1, keepdims=True)
b = b / np.linalg.norm(b, axis=-1, keepdims=True)
return a @ b.T
# attention, stripped to its skeleton
def attention(Q, K, V):
scores = (Q @ K.T) / np.sqrt(K.shape[-1]) # scale keeps softmax sane
weights = np.exp(scores - scores.max(-1, keepdims=True))
weights /= weights.sum(-1, keepdims=True)
return weights @ VCalculus — only enough for gradients
Training is: measure how wrong you are, compute the direction that reduces the wrongness, take a small step, repeat. The derivative is the slope, the gradient is the vector of partial derivatives, and the chain rule is what lets error at the output propagate back through every layer. Autograd computes all of this for you; what you need is the intuition to diagnose it. Vanishing gradients (signal dies in deep stacks — hence residual connections and ReLU), exploding gradients (hence clipping), and the reason learning rate is the hyperparameter that matters most: it is the step size along that direction.
Probability and statistics — where the judgment lives
This is the area engineers most often skip and most often regret. Models output distributions, not answers, and every claim about performance is a statistical claim.
- Random variables, expectation, variance — and the bias/variance decomposition that explains under- and overfitting in one picture.
- Conditional probability and Bayes' rule — the reason a 99%-accurate test for a rare disease is mostly false positives, and the reason your fraud model's precision collapses in production.
- Common distributions and their uses: Bernoulli/Binomial for clicks, Normal for noise and initialization, Poisson for counts and arrivals, Exponential for waiting times.
- Maximum likelihood — cross-entropy loss is not arbitrary; it is negative log-likelihood, and mean squared error is the same thing under Gaussian noise.
- Sampling, confidence intervals, and hypothesis testing — everything you need to say whether the new model is really better or you got lucky.
- KL divergence and entropy — how far one distribution is from another, which shows up in variational methods, distillation, and drift monitoring.
Optimization — how the fitting happens
Convexity tells you when a single minimum is guaranteed (linear and logistic regression, SVMs) and neural networks tell you what to do when it is not: stochastic gradient descent on mini-batches, momentum to smooth the path, Adam to adapt per-parameter step sizes, learning rate schedules with warmup and decay. Regularization — L1 for sparsity, L2 for shrinkage, dropout, early stopping — is the practical answer to a model that fits the training set too well. You will tune these far more often than you will derive them.
What you can skip, and how to study
You can safely skip measure theory, most real analysis, manual proofs of convergence, and hand-computing determinants. Learn instead by implementing: write linear regression with gradient descent in NumPy, then logistic regression, then a two-layer network with hand-written backpropagation. It takes a weekend and permanently changes how you read papers. Pair that with 3Blue1Brown for geometric intuition, Mathematics for Machine Learning (Deisenroth et al., free online) as the reference, and Boyd's Convex Optimization when you need the depth. Then get back to the data — that is where the outcomes actually come from.