Descent

A visual curriculum for machine learning

Descent

From Shannon’s guessing game to transformers, reinforcement learning from human feedback, and the questions no one has answered yet. Built for people starting further back than a PhD — and aiming further forward.

Begin at Base Camp See the route
Parts
24
Lessons
126
Reading time
101 h
Pace
12 months
Descent in twenty seconds
01

Motivation before mechanism

Every idea arrives as the answer to a problem someone actually had. You’ll know why neural networks, attention and the KV cache exist before you learn how they work.

02

Every symbol decoded

Equations come with a colour-coded decoder: hover a symbol to see what it means and how to say it out loud. Notation stops being a wall and becomes a compression format.

03

See it, then run it

Interactive plates you can drag and scrub, Manim animations, and Python you can run right here in the browser. Nothing stays purely conceptual.

The route

24 parts, one continuous descent

Each part assumes only the ones above it. The months are a suggested pace for a year of serious part-time study.

  1. 0
    Reading the languageWeek 1

    Base Camp

    How to use this curriculum, and a field guide to the symbols that make papers feel impenetrable. Learn to read an equation the way you read a sentence.

    1. How to use Descent
    2. Reading the symbols
    3. Shapes, indices and sums
    4. The year plan
  2. I
    Information and predictionMonth 1

    Shannon's Game

    In 1948 Claude Shannon measured information in bits and, a few years later, asked people to guess the next letter of English text. Every modern language model is still playing his game.

    1. Surprise, bits and entropy
    2. Predicting the next letter
    3. Cross-entropy, KL and perplexity
    4. From n-grams to neural language models
  3. II
    Linear algebra, calculus, probabilityMonths 1–2

    The Toolkit

    The three languages every paper is written in. Vectors and matrices as geometry, derivatives as sensitivity, probability as bookkeeping for uncertainty.

    1. Vectors and the dot product
    2. Matrices as transformations
    3. Derivatives, gradients and the chain rule
    4. Matrix calculus you actually need
    5. Probability as bookkeeping
    6. Maximum likelihood: where losses come from
    7. Eigenvectors, SVD and PCA
  4. III
    Linear models, closed forms, descentMonth 2

    Fitting Lines

    Linear regression is the hydrogen atom of machine learning. Solve it exactly with the normal equations, then solve it again by walking downhill — and see why the second way is the one that scales.

    1. Linear regression
    2. The closed form: normal equations
    3. Gradient descent
    4. Logistic regression and softmax
    5. Generalization, bias–variance and regularization
  5. IV
    Neighbours, trees, ensembles, kernelsMonth 3

    Beyond Lines

    Before deep learning, and still the first thing to try on a spreadsheet: methods that bend the straight line into neighbourhoods, trees, votes and kernels, and the reasons each one works.

    1. Nearest neighbours and naive Bayes
    2. Decision trees
    3. Bagging and random forests
    4. Gradient boosting
    5. Support vector machines and kernels
    6. Generalized linear models
  6. V
    Metrics, validation, statisticsMonth 3

    Measuring Honestly

    A model is only as good as the way you measured it. Precision and recall, calibration, validation that doesn’t leak, what to do when classes are rare or the world shifts, and how sure you can be of a result.

    1. Precision, recall and ROC
    2. Calibration and proper scoring rules
    3. Validation design and leakage
    4. Imbalance and distribution shift
    5. How sure are you? Bootstrap and tests
  7. VI
    Clustering, embeddings, anomaliesMonth 4

    Finding Structure

    No labels, only data. Grouping it, flattening it into pictures you can see, estimating where it is dense, and spotting the points that don’t belong.

    1. k-means, properly
    2. Gaussian mixtures and EM
    3. Hierarchical and density-based clustering
    4. Spectral clustering and graph Laplacians
    5. t-SNE, UMAP and manifold learning
    6. Density estimation
    7. Anomaly detection
  8. VII
    Bayesian inference, graphical models, samplingMonths 4–5

    Thinking in Distributions

    Treat every unknown as a distribution and learning becomes inference. Priors and posteriors, graphs of dependence, hidden states, and the two great tools for intractable integrals: variational inference and Markov chain Monte Carlo.

    1. Bayesian inference
    2. Graphical models
    3. Latent variables, HMMs and Kalman filters
    4. Variational inference
    5. Markov chain Monte Carlo
    6. Gaussian processes
  9. VIII
    Convexity, duality, convergenceMonth 5

    The Shape of Loss

    Every model is fitted by an optimizer. When a bowl has one bottom, what Lagrange multipliers mean, why Newton’s method is fast and expensive, and what can be proved about stochastic gradient descent.

    1. Convexity
    2. Lagrange multipliers, duality and KKT
    3. Newton, quasi-Newton and natural gradients
    4. Why SGD converges
    5. Constraints, projections and proximal methods
  10. IX
    PAC, VC, bounds, banditsMonth 5

    Why Learning Works

    When can a finite sample tell you anything about the future? Probably-approximately-correct learning, capacity measured by VC dimension and Rademacher complexity, the no-free-lunch theorem, and learning online against an adversary.

    1. PAC learning
    2. VC dimension
    3. Rademacher complexity and generalization bounds
    4. No free lunch and inductive bias
    5. Online learning and bandits
  11. X
    From perceptrons to backpropMonth 6

    Why Neural Networks

    Linear models cannot learn their own features. A stack of simple nonlinear units can. The history of that idea — its winters, its revival — and the chain-rule machinery that makes it trainable.

    1. A short history of the neuron
    2. The limits of linear
    3. The multilayer perceptron
    4. Backpropagation
    5. Training in practice
  12. XI
    CNNs, RNNs, LSTMsMonth 6

    Seeing and Remembering

    Architectures are assumptions about the world baked into wiring. Convolutions assume locality; recurrence assumes sequence. Why each was invented, and where each broke.

    1. Convolution
    2. The CNN lineage: LeNet to ResNet
    3. Recurrent networks
    4. LSTM and GRU
    5. Seq2seq and the bottleneck
  13. XII
    The transformer and next-token predictionMonth 7

    Attention

    Attention began as a patch for translation and became the whole architecture. Queries, keys and values; heads and residual streams; and how GPT turned Shannon’s game into a trillion-parameter industry.

    1. Attention: learning where to look
    2. Scaled dot-product and multi-head attention
    3. The transformer block
    4. Positional information
    5. GPT and next-token prediction
  14. XIII
    KV cache, sparse attention, efficiencyMonth 7

    Making It Fast

    Attention is quadratic and generation is sequential. The engineering that makes large models usable: caching, sharing heads, skipping most of the attention matrix, and respecting the memory hierarchy.

    1. The cost of attention
    2. The KV cache
    3. Sparse attention
    4. FlashAttention and the memory hierarchy
    5. Mixture of experts
  15. XIV
    VAEs, GANs, flows, diffusionMonth 8

    Making New Things

    Language models predict the next token; other generative models learn to turn noise into images, sound and molecules. Autoencoders, adversarial games, invertible flows, and diffusion — the idea behind modern image generators.

    1. What generative modelling is
    2. Autoencoders and VAEs
    3. Generative adversarial networks
    4. Normalizing flows
    5. Diffusion models
    6. Score matching and flow matching
  16. XV
    Vision, graphs, audio, time, recommendationMonth 8

    Other Worlds

    The same ideas, bent to different data. Images beyond classification, molecules and networks as graphs, sound, sequences in time, what to show whom, and models that join text to pictures.

    1. Vision transformers, detection and segmentation
    2. Graph neural networks
    3. Speech and audio
    4. Time series and forecasting
    5. Recommender systems
    6. Multimodal models
  17. XVI
    Supervised, unsupervised, RL, RLHF, RLVRMonth 9

    Learning Signals

    Where does the learning signal come from? Labels, structure in the data itself, or reward. From the Bellman equation to the pipelines that turn a text predictor into an assistant and a reasoner.

    1. Three kinds of learning signal
    2. Reinforcement learning foundations
    3. Policy gradients and PPO
    4. RLHF: learning from preferences
    5. RLVR and reasoning models
  18. XVII
    Deep RL, planning, exploration, imitationMonth 9

    Deeper Reinforcement

    Reinforcement learning past the basics: deep value and actor–critic methods, planning with learned models, the exploration problem, learning from demonstrations and logged data, and many agents at once.

    1. Deep Q-learning and actor–critic methods
    2. Model-based RL and planning
    3. Exploration
    4. Imitation and offline RL
    5. Multi-agent RL
  19. XVIII
    Fine-tuning, retrieval, agents, scaleMonth 10

    Building with LLMs

    What it takes to adapt, shrink, ground, evaluate and train large language models in practice: parameter-efficient fine-tuning, quantization, retrieval-augmented generation, tool use, benchmarks, and parallelism across thousands of chips.

    1. Fine-tuning and LoRA
    2. Quantization and distillation
    3. Embeddings, retrieval and RAG
    4. Agents and tool use
    5. Evaluating language models
    6. Training at scale
  20. XIX
    NumPy, SciPy, PyTorch, JAX, KerasMonth 10

    Under the Hood

    Frameworks are autodiff engines wrapped around fast array libraries. Build a tiny one yourself, then open up the real ones: strides, dispatchers, tapes, tracers and compilers.

    1. Arrays: NumPy and SciPy
    2. Autodiff from scratch
    3. PyTorch internals
    4. JAX: tracing and transformations
    5. Keras, and choosing a framework
  21. XX
    Width, depth, double descent, grokking, scalingMonth 11

    Strange Phenomena

    Deep learning keeps doing things classical theory said it shouldn’t. What happens as networks get wider or deeper, why test error can fall twice, why generalization can arrive long after memorization — and where it all might end.

    1. Width versus depth
    2. Double descent
    3. Grokking
    4. Scaling laws and their limits
    5. The open questions
  22. XXI
    Graphs, interventions, counterfactualsMonth 11

    Cause and Effect

    Prediction asks what will happen; causality asks what would happen if we acted. Confounders, causal graphs, the do-operator, potential outcomes, and why a model that predicts perfectly can still mislead.

    1. Why correlation isn’t enough
    2. Causal graphs and d-separation
    3. Interventions and the do-operator
    4. Potential outcomes and treatment effects
    5. Causal discovery and causal ML
  23. XXII
    Interpretability, robustness, fairness, privacy, safetyMonth 12

    Trust

    Deployed models are used by people and attacked by some of them. Looking inside networks, adversarial examples, what fairness can and can’t mean, privacy guarantees, calibrated uncertainty, and the problem of aligning capable systems.

    1. Interpretability
    2. Adversarial examples and robustness
    3. Fairness
    4. Privacy and differential privacy
    5. Uncertainty in deep learning
    6. Alignment and safety
  24. XXIII
    Reading researchMonth 12

    The Summit

    Everything converges here: take a real paper apart line by line, and leave with a method for reading the next hundred.

    1. How to read an ML paper
    2. Worked reading: Attention Is All You Need
    3. Where to go next

262 years in one line

How we got here

  1. 1763Bayes' method of inference appears, two years after his deathThomas Bayes
  2. 1805Legendre publishes the method of least squaresAdrien-Marie Legendre
  3. 1847Cauchy's method of steepest descentAugustin-Louis Cauchy
  4. 1901Pearson's “On Lines and Planes of Closest Fit to Systems of Points in Space”Karl Pearson
  5. 1948Shannon publishes A Mathematical Theory of CommunicationClaude Shannon
  6. 1951Shannon's guessing game: Prediction and Entropy of Printed EnglishClaude Shannon
  7. 1951Kuhn and Tucker publish "Nonlinear programming"Harold Kuhn, Albert Tucker
  8. 1953Metropolis et al. publish the Metropolis algorithmN. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, E. Teller
  9. 1958Rosenblatt's perceptron learns its weights from examplesFrank Rosenblatt
  10. 1960Kalman's recursive filterR. E. Kalman
  11. 1971Vapnik and Chervonenkis: uniform convergenceVladimir Vapnik & Alexey Chervonenkis
  12. 1974Potential outcomes for observational studiesDonald B. Rubin
  13. 1977The EM algorithm is namedA. P. Dempster, N. M. Laird, D. B. Rubin
  14. 1979Efron introduces the bootstrapBradley Efron
  15. 1983Nesterov's accelerated gradient methodYurii Nesterov
  16. 1984Valiant: A Theory of the LearnableLeslie G. Valiant
  17. 1986Rumelhart, Hinton and Williams popularise backpropagationDavid Rumelhart, Geoffrey Hinton, Ronald Williams
  18. 1995Causal diagrams and the do-calculusJudea Pearl
  19. 1995Support-vector networksCorinna Cortes & Vladimir Vapnik
  20. 1997Long short-term memorySepp Hochreiter, Jürgen Schmidhuber
  21. 2001Random forestsLeo Breiman
  22. 2005NumPy unifies Numeric and NumarrayNumPy developers
  23. 2006Differential privacy definedCynthia Dwork, Frank McSherry, Kobbi Nissim & Adam Smith
  24. 2007Keshav's “How to Read a Paper”S. Keshav
  25. 2008t-SNELaurens van der Maaten, Geoffrey Hinton
  26. 2012AlexNet wins ImageNetAlex Krizhevsky, Ilya Sutskever, Geoffrey Hinton
  27. 2013Adversarial examplesChristian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow & Rob Fergus
  28. 2014Generative adversarial networksIan Goodfellow and co-authors
  29. 2014Bahdanau, Cho and Bengio let the decoder look back: soft attentionDzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio
  30. 2017PyTorch beta releasedPyTorch team
  31. 2017Graph convolutional networksThomas Kipf, Max Welling
  32. 2017Modern neural networks are overconfidentChuan Guo, Geoff Pleiss, Yu Sun & Kilian Q. Weinberger
  33. 2017“Attention Is All You Need”: the TransformerVaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin
  34. 2018AlphaZero masters chess, shogi and Go from self-playDavid Silver et al. (DeepMind)
  35. 2019Sparse Transformers: factorised sparse attentionRewon Child et al. (OpenAI)
  36. 2019AlphaStar reaches Grandmaster in StarCraft IIOriol Vinyals et al. (DeepMind)
  37. 2019Double descentMikhail Belkin, Daniel Hsu, Siyuan Ma & Soumik Mandal
  38. 2020Scaling laws for neural language modelsJared Kaplan, Sam McCandlish et al. (OpenAI)
  39. 2020Retrieval-augmented generation (RAG)Patrick Lewis et al. (Facebook AI Research, UCL, NYU)
  40. 2020Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter Abbeel
  41. 2021CLIPAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, et al.
  42. 2021LoRA: low-rank adaptationEdward Hu, Yelong Shen, Phillip Wallis et al. (Microsoft)
  43. 2022InstructGPT: RLHF on GPT-3Ouyang et al. (OpenAI)
  44. 2022FlashAttention: exact attention, aware of the memory hierarchyTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré
  45. 2025DeepSeek-R1 and R1-ZeroDeepSeek-AI
Full timeline →