Descent

The route

How the ideas connect

Every lesson, and every “builds on” link between them. Hover (or tap) a lesson to light up everything it depends on and everything that depends on it. Filled dots are lessons you’ve marked complete.

0Base CampReading the language
  1. 0.1How to use Descent
  2. 0.2Reading the symbols
  3. 0.3Shapes, indices and sums
  4. 0.4The year plan
IShannon's GameInformation and prediction
  1. I.1Surprise, bits and entropy
  2. I.2Predicting the next letter
  3. I.3Cross-entropy, KL and perplexity
  4. I.4From n-grams to neural language models
IIThe ToolkitLinear algebra, calculus, probability
  1. II.1Vectors and the dot product
  2. II.2Matrices as transformations
  3. II.3Derivatives, gradients and the chain rule
  4. II.4Matrix calculus you actually need
  5. II.5Probability as bookkeeping
  6. II.6Maximum likelihood: where losses come from
  7. II.7Eigenvectors, SVD and PCA
IIIFitting LinesLinear models, closed forms, descent
  1. III.1Linear regression
  2. III.2The closed form: normal equations
  3. III.3Gradient descent
  4. III.4Logistic regression and softmax
  5. III.5Generalization, bias–variance and regularization
IVBeyond LinesNeighbours, trees, ensembles, kernels
  1. IV.1Nearest neighbours and naive Bayes
  2. IV.2Decision trees
  3. IV.3Bagging and random forests
  4. IV.4Gradient boosting
  5. IV.5Support vector machines and kernels
  6. IV.6Generalized linear models
VMeasuring HonestlyMetrics, validation, statistics
  1. V.1Precision, recall and ROC
  2. V.2Calibration and proper scoring rules
  3. V.3Validation design and leakage
  4. V.4Imbalance and distribution shift
  5. V.5How sure are you? Bootstrap and tests
VIFinding StructureClustering, embeddings, anomalies
  1. VI.1k-means, properly
  2. VI.2Gaussian mixtures and EM
  3. VI.3Hierarchical and density-based clustering
  4. VI.4Spectral clustering and graph Laplacians
  5. VI.5t-SNE, UMAP and manifold learning
  6. VI.6Density estimation
  7. VI.7Anomaly detection
VIIThinking in DistributionsBayesian inference, graphical models, sampling
  1. VII.1Bayesian inference
  2. VII.2Graphical models
  3. VII.3Latent variables, HMMs and Kalman filters
  4. VII.4Variational inference
  5. VII.5Markov chain Monte Carlo
  6. VII.6Gaussian processes
VIIIThe Shape of LossConvexity, duality, convergence
  1. VIII.1Convexity
  2. VIII.2Lagrange multipliers, duality and KKT
  3. VIII.3Newton, quasi-Newton and natural gradients
  4. VIII.4Why SGD converges
  5. VIII.5Constraints, projections and proximal methods
IXWhy Learning WorksPAC, VC, bounds, bandits
  1. IX.1PAC learning
  2. IX.2VC dimension
  3. IX.3Rademacher complexity and generalization bounds
  4. IX.4No free lunch and inductive bias
  5. IX.5Online learning and bandits
XWhy Neural NetworksFrom perceptrons to backprop
  1. X.1A short history of the neuron
  2. X.2The limits of linear
  3. X.3The multilayer perceptron
  4. X.4Backpropagation
  5. X.5Training in practice
XISeeing and RememberingCNNs, RNNs, LSTMs
  1. XI.1Convolution
  2. XI.2The CNN lineage: LeNet to ResNet
  3. XI.3Recurrent networks
  4. XI.4LSTM and GRU
  5. XI.5Seq2seq and the bottleneck
XIIAttentionThe transformer and next-token prediction
  1. XII.1Attention: learning where to look
  2. XII.2Scaled dot-product and multi-head attention
  3. XII.3The transformer block
  4. XII.4Positional information
  5. XII.5GPT and next-token prediction
XIIIMaking It FastKV cache, sparse attention, efficiency
  1. XIII.1The cost of attention
  2. XIII.2The KV cache
  3. XIII.3Sparse attention
  4. XIII.4FlashAttention and the memory hierarchy
  5. XIII.5Mixture of experts
XIVMaking New ThingsVAEs, GANs, flows, diffusion
  1. XIV.1What generative modelling is
  2. XIV.2Autoencoders and VAEs
  3. XIV.3Generative adversarial networks
  4. XIV.4Normalizing flows
  5. XIV.5Diffusion models
  6. XIV.6Score matching and flow matching
XVOther WorldsVision, graphs, audio, time, recommendation
  1. XV.1Vision transformers, detection and segmentation
  2. XV.2Graph neural networks
  3. XV.3Speech and audio
  4. XV.4Time series and forecasting
  5. XV.5Recommender systems
  6. XV.6Multimodal models
XVILearning SignalsSupervised, unsupervised, RL, RLHF, RLVR
  1. XVI.1Three kinds of learning signal
  2. XVI.2Reinforcement learning foundations
  3. XVI.3Policy gradients and PPO
  4. XVI.4RLHF: learning from preferences
  5. XVI.5RLVR and reasoning models
XVIIDeeper ReinforcementDeep RL, planning, exploration, imitation
  1. XVII.1Deep Q-learning and actor–critic methods
  2. XVII.2Model-based RL and planning
  3. XVII.3Exploration
  4. XVII.4Imitation and offline RL
  5. XVII.5Multi-agent RL
XVIIIBuilding with LLMsFine-tuning, retrieval, agents, scale
  1. XVIII.1Fine-tuning and LoRA
  2. XVIII.2Quantization and distillation
  3. XVIII.3Embeddings, retrieval and RAG
  4. XVIII.4Agents and tool use
  5. XVIII.5Evaluating language models
  6. XVIII.6Training at scale
XIXUnder the HoodNumPy, SciPy, PyTorch, JAX, Keras
  1. XIX.1Arrays: NumPy and SciPy
  2. XIX.2Autodiff from scratch
  3. XIX.3PyTorch internals
  4. XIX.4JAX: tracing and transformations
  5. XIX.5Keras, and choosing a framework
XXStrange PhenomenaWidth, depth, double descent, grokking, scaling
  1. XX.1Width versus depth
  2. XX.2Double descent
  3. XX.3Grokking
  4. XX.4Scaling laws and their limits
  5. XX.5The open questions
XXICause and EffectGraphs, interventions, counterfactuals
  1. XXI.1Why correlation isn’t enough
  2. XXI.2Causal graphs and d-separation
  3. XXI.3Interventions and the do-operator
  4. XXI.4Potential outcomes and treatment effects
  5. XXI.5Causal discovery and causal ML
XXIITrustInterpretability, robustness, fairness, privacy, safety
  1. XXII.1Interpretability
  2. XXII.2Adversarial examples and robustness
  3. XXII.3Fairness
  4. XXII.4Privacy and differential privacy
  5. XXII.5Uncertainty in deep learning
  6. XXII.6Alignment and safety
XXIIIThe SummitReading research
  1. XXIII.1How to read an ML paper
  2. XXIII.2Worked reading: Attention Is All You Need
  3. XXIII.3Where to go next