Library

Go to the source

The papers, books, notes and documentation behind each lesson — 474 of them. When a lesson makes something click, read the original next: you now have the vocabulary for it.

0Base Camp

Test-enhanced learning: Taking memory tests improves long-term retentionRoediger & Karpicke · 2006The experiment behind the “Check yourself” boxes.From: How to use Descent
Manim Community documentation2026The community-maintained fork of the animation engine Grant Sanderson (3Blue1Brown) created; it renders the animations here.From: How to use Descent
Deep Learning — NotationGoodfellow, Bengio & Courville · 2016The one-page reference most of this lesson's conventions are checked against. The LaTeX source is public at github.com/goodfeli/dlbook_notation.From: Reading the symbols
Mathematics for Machine Learning — Table of SymbolsDeisenroth, Faisal & Ong · 2020Two pages of symbols with their typical meanings, at the front of a free textbook written to close exactly this gap.From: Reading the symbols
Standard notations for Deep Learning (course notes)deeplearning.ai course notes, transcribedWhere the $W^{[l]}$ and $x^{(i)}$ conventions you will meet in many tutorials come from.From: Reading the symbols
numpy.einsum — documentationNumPy developersThe definitive statement of the subscript rules used in this lesson.From: Shapes, indices and sums
Broadcasting — NumPy user guideNumPy developersThe two-line rule, with pictures.From: Shapes, indices and sums
nanoGPT model.pyAndrej KarpathyReal transformer code that annotates tensor shapes in comments, (B, T, C) style — worth imitating.From: Shapes, indices and sums
Improving Students’ Learning With Effective Learning TechniquesDunlosky, Rawson, Marsh, Nathan & Willingham · 2013A review of ten study techniques; practice testing and distributed practice came out on top.From: The year plan
Neural Networks: Zero to HeroAndrej KarpathyLectures that build micrograd, makemore and a GPT from scratch, in code.From: The year plan
Mathematics for Machine LearningDeisenroth, Faisal & Ong · 2020From: The year plan
Dive into Deep LearningZhang, Lipton, Li & Smola · 2023From: The year plan
Understanding Deep LearningSimon J. D. Prince · 2023From: The year plan

IShannon's Game

A Mathematical Theory of CommunicationC. E. Shannon · 1948The founding paper. Sections 6 (entropy) and 9 (the noiseless coding theorem) are very readable.From: Surprise, bits and entropy
A Method for the Construction of Minimum-Redundancy CodesD. A. Huffman · 1952Four pages. The optimal question-asking strategy, found as a term paper.From: Surprise, bits and entropy
Stanford EE274 Data Compression notes: entropy and Huffman codesShubham Chandak, Pulkit Tandon, Kedar Tatwawadi & Tsachy Weissman (Stanford EE274) · 2023Clear proofs that entropy lower-bounds code length, with Python.From: Surprise, bits and entropy
Information Theory course scriptChristian Schaffner (University of Amsterdam) · 2016Lecture notes with the twenty-questions view of Huffman coding.From: Surprise, bits and entropy
Prediction and Entropy of Printed EnglishC. E. Shannon · 1951Fifteen pages. The guessing game, the N-gram entropy table and the entropy bounds.From: Predicting the next letter
An Example of Statistical Investigation of the Text Eugene Onegin Concerning the Connection of Samples in ChainsA. A. Markov (English translation, 2006) · 1913Markov's 1913 lecture, translated in Science in Context 19(4).From: Predicting the next letter
Speech and Language Processing, ch. 3: N-gram Language ModelsDan Jurafsky & James H. Martin · 2026The standard textbook treatment, free online.From: Predicting the next letter
First Links in the Markov ChainBrian Hayes (American Scientist) · 2013The story of Markov's poem-counting, for a general reader.From: Predicting the next letter
On Information and SufficiencyS. Kullback & R. A. Leibler · 1951Annals of Mathematical Statistics 22(1), 79–86. Where the divergence comes from.From: Cross-entropy, KL and perplexity
An Estimate of an Upper Bound for the Entropy of EnglishP. F. Brown, V. J. Della Pietra, R. L. Mercer, S. A. Della Pietra & J. C. Lai · 1992Ten pages that use cross-entropy exactly as this lesson does, as an upper bound on the entropy of English.From: Cross-entropy, KL and perplexity
Stanford EE274 notes: context-based coding and LLM compressionShubham Chandak, Pulkit Tandon, Kedar Tatwawadi & Tsachy Weissman (Stanford EE274) · 2023Why a cross-entropy loss is a compression rate.From: Cross-entropy, KL and perplexity
A Neural Probabilistic Language ModelY. Bengio, R. Ducharme, P. Vincent & C. Jauvin · 2003Journal of Machine Learning Research 3, 1137–1155. Sections 1 and 2 are the whole idea.From: From n-grams to neural language models
Speech and Language Processing, ch. 3 (smoothing) and ch. 5 (embeddings)Dan Jurafsky & James H. Martin · 2026From: From n-grams to neural language models
Efficient Estimation of Word Representations in Vector SpaceT. Mikolov, K. Chen, G. Corrado & J. Dean · 2013word2vec, and the famous king − man + woman ≈ queen.From: From n-grams to neural language models

IIThe Toolkit

Mathematics for Machine Learning, chapters 2–3Deisenroth, Faisal & Ong · 2020Free textbook. Chapter 3 defines norms, inner products and angles carefully.From: Vectors and the dot product
Deep Learning, chapter 2 (Linear Algebra)Goodfellow, Bengio & Courville · 2016A compact summary of the linear algebra deep learning uses.From: Vectors and the dot product
Dive into Deep Learning: Geometry and linear algebraic operationsZhang, Lipton, Li & Smola · 2023The dot product, angles and cosine similarity, with code.From: Vectors and the dot product
Attention Is All You NeedVaswani et al. · 2017Where dot products become attention scores.From: Vectors and the dot product
PyTorch nn.Linear (source and docstring)PyTorch contributors · 2026How a real linear layer stores its matrix, and which way round it multiplies.From: Matrices as transformations
The Matrix Calculus You Need For Deep LearningTerence Parr & Jeremy Howard · 2018Starts from single-variable rules and builds to the vector chain rule. Good for this lesson and the next.From: Derivatives, gradients and the chain rule
Probabilistic Machine Learning: An Introduction, §7.8 (Matrix calculus)Kevin P. Murphy · 2022Compact reference for gradients, Jacobians, JVPs/VJPs and common identities. Same convention as this lesson.From: Matrix calculus you actually need
The Matrix CookbookKaare Brandt Petersen & Michael Syskind Pedersen · 2012A lookup table of identities. Check its conventions before mixing its results with another source's.From: Matrix calculus you actually need
scikit-learn: PCA (source and docstring)scikit-learn developers · 2026How a production PCA is computed (centre, then SVD).From: Eigenvectors, SVD and PCA

IIIFitting Lines

Mathematics for Machine Learning, ch. 9 (Linear Regression)Deisenroth, Faisal & Ong · 2020The maximum-likelihood derivation used here, in full detail; free PDF.From: Linear regression
Probabilistic Machine Learning: An Introduction, ch. 11Kevin P. Murphy · 2022Least squares, its geometry and its numerics; free PDF.From: Linear regression
Dive into Deep Learning, §3.1 Linear RegressionZhang, Lipton, Li & SmolaThe same model from the deep-learning side, with code.From: Linear regression
numpy.linalg.lstsqNumPy developersWhat the standard solver returns, including the minimum-norm solution for rank-deficient problems.From: The closed form: normal equations
Why Momentum Really WorksGabriel Goh · 2017Gradient descent on a quadratic, solved exactly in the eigenbasis; interactive and beautiful.From: Gradient descent
Closed-form solutions vs. gradient descent and SGDSebastian RaschkaA short, practical comparison.From: Gradient descent
Dive into Deep Learning, §4.1 Softmax RegressionZhang, Lipton, Li & SmolaSoftmax, cross-entropy and the gradient with respect to logits, with code.From: Logistic regression and softmax
The Origins of Logistic RegressionJ. S. Cramer · 2002A short history, from Verhulst's population curve to modern statistics.From: Logistic regression and softmax
Regression Shrinkage and Selection via the LassoRobert Tibshirani · 1996The lasso paper. Readable, and full of good pictures.From: Generalization, bias–variance and regularization
Ridge Regression: Biased Estimation for Nonorthogonal ProblemsA. E. Hoerl & R. W. Kennard · 1970Adding a little to the diagonal of $X^{⊤} X$ , and the ridge trace.From: Generalization, bias–variance and regularization
Dive into Deep Learning, §3.6 GeneralizationZhang, Lipton, Li & SmolaTraining vs test error, model selection and cross-validation, from the deep-learning side.From: Generalization, bias–variance and regularization

IVBeyond Lines

Nearest neighbor pattern classificationT. M. Cover & P. E. Hart · 1967The factor-of-two bound on the nearest-neighbour rule's error.From: Nearest neighbours and naive Bayes
Probabilistic Machine Learning: An Introduction, §9.3 and §16.1Kevin P. Murphy · 2022Naive Bayes classifiers, and k-NN with the curse of dimensionality. Free online.From: Nearest neighbours and naive Bayes
On the Optimality of the Simple Bayesian Classifier under Zero-One LossP. Domingos & M. Pazzani · 1997Why naive Bayes can classify optimally even when its independence assumption is badly wrong.From: Nearest neighbours and naive Bayes
When Is "Nearest Neighbor" Meaningful?Beyer, Goldstein, Ramakrishnan & Shaft · 1999Distances concentrating in high dimensions, stated as a theorem.From: Nearest neighbours and naive Bayes
scikit-learn user guide: Nearest Neighbors and Naive BayesPractical details, including the smoothing formula used below.From: Nearest neighbours and naive Bayes
The Elements of Statistical Learning, §9.2 Tree-Based MethodsHastie, Tibshirani & Friedman · 2009The clearest short account of CART, impurity measures and cost-complexity pruning. Free PDF.From: Decision trees
Induction of decision treesJ. R. Quinlan · 1986ID3 and information gain.From: Decision trees
scikit-learn user guide: Decision TreesPractical advice, the ID3/C4.5/CART comparison, and minimal cost-complexity pruning.From: Decision trees
Bagging predictorsLeo Breiman · 1996The original paper. Short and readable.From: Bagging and random forests
Random ForestsLeo Breiman · 2001Strength and correlation, out-of-bag estimates, variable importance.From: Bagging and random forests
scikit-learn user guide: Permutation feature importanceWhy impurity-based importances are biased, and what permutation importance does instead.From: Bagging and random forests
Greedy function approximation: A gradient boosting machineJerome H. Friedman · 2001The gradient boosting paper (Annals of Statistics 29, 1189–1232).From: Gradient boosting
XGBoost: A Scalable Tree Boosting SystemTianqi Chen & Carlos Guestrin · 2016Regularised objective, second-order split finding, and the systems work that made it fast.From: Gradient boosting
Why do tree-based models still outperform deep learning on typical tabular data?Grinsztajn, Oyallon & Varoquaux · 2022A benchmark of 45 datasets, and experiments on why.From: Gradient boosting
Support-vector networksCorinna Cortes & Vladimir Vapnik · 1995The soft-margin support vector machine.From: Support vector machines and kernels
A training algorithm for optimal margin classifiersBoser, Guyon & Vapnik · 1992Maximum margin plus kernels — the SVM's first paper.From: Support vector machines and kernels
A Generalized Representer TheoremSchölkopf, Herbrich & Smola · 2001Why the solution lives in the span of the training points.From: Support vector machines and kernels
scikit-learn user guide: Support Vector MachinesPractical advice on scaling, C and gamma.From: Support vector machines and kernels
Generalized Linear ModelsJ. A. Nelder & R. W. M. Wedderburn · 1972The paper that unified the family (Journal of the Royal Statistical Society, Series A 135, 370–384).From: Generalized linear models
scikit-learn user guide: Generalized Linear ModelsUnit deviances, which family for which target, and exposure as sample weights.From: Generalized linear models

VMeasuring Honestly

Probabilistic Machine Learning: An Introduction, §5.1.2–5.1.4Kevin P. Murphy · 2022Confusion matrices, cost-sensitive thresholds, ROC and PR curves, F-scores and class imbalance, in eight tight pages.From: Precision, recall and ROC
An introduction to ROC analysisTom Fawcett · 2006Pattern Recognition Letters 27, 861–874. The standard tutorial on ROC graphs for machine learning.From: Precision, recall and ROC
The meaning and use of the area under a receiver operating characteristic (ROC) curveJames A. Hanley & Barbara J. McNeil · 1982Radiology 143(1), 29–36. AUC as a ranking probability, and its link to the Wilcoxon statistic.From: Precision, recall and ROC
The relationship between Precision-Recall and ROC curvesJesse Davis & Mark Goadrich · 2006ICML 2006. Why dominance transfers between the two spaces, and why you must not linearly interpolate a PR curve.From: Precision, recall and ROC
scikit-learn user guide: Metrics and scoring2026Exact definitions of every metric used here, including the averaging options.From: Precision, recall and ROC
On Calibration of Modern Neural NetworksChuan Guo, Geoff Pleiss, Yu Sun & Kilian Q. Weinberger · 2017ICML 2017. Reliability diagrams, ECE, and temperature scaling for deep networks.From: Calibration and proper scoring rules
Strictly Proper Scoring Rules, Prediction, and EstimationTilmann Gneiting & Adrian E. Raftery · 2007JASA 102, 359–378. The theory of scores that reward honesty.From: Calibration and proper scoring rules
Predicting Good Probabilities With Supervised LearningAlexandru Niculescu-Mizil & Rich Caruana · 2005ICML 2005. Which classical learners are miscalibrated and how; Platt scaling vs isotonic regression.From: Calibration and proper scoring rules
Revisiting the Calibration of Modern Neural NetworksMatthias Minderer et al. · 2021NeurIPS 2021. Newer architectures are among the best calibrated; the 2017 story is not the whole story.From: Calibration and proper scoring rules
scikit-learn user guide: Probability calibration2026Calibration curves, the sigmoid, isotonic and temperature calibrators, and their trade-offs.From: Calibration and proper scoring rules
Leakage in Data Mining: Formulation, Detection, and AvoidanceShachar Kaufman, Saharon Rosset, Claudia Perlich & Ori Stitelman · 2012ACM TKDD 6(4). The classic taxonomy, with real competition disasters.From: Validation design and leakage
Do ImageNet Classifiers Generalize to ImageNet?Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt & Vaishaal Shankar · 2019ICML 2019. New test sets for CIFAR-10 and ImageNet, a decade of test-set reuse later.From: Validation design and leakage
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance EvaluationGavin C. Cawley & Nicola L. C. Talbot · 2010JMLR 11. Why selecting models by cross-validation can overfit, and why nested evaluation is needed.From: Validation design and leakage
Leakage and the reproducibility crisis in machine-learning-based scienceSayash Kapoor & Arvind Narayanan · 2023Patterns. Leakage across 17 scientific fields, and a taxonomy of eight kinds.From: Validation design and leakage
scikit-learn user guide: Cross-validation, and Common pitfalls2026Data leakage during preprocessing, grouped and time-series splitters, nested cross-validation.From: Validation design and leakage
Dive into Deep Learning, §4.6 Generalization in ClassificationZhang, Lipton, Li & SmolaHow big a test set must be, and what goes wrong when you reuse it.From: Validation design and leakage
Dive into Deep Learning, §4.7 Environment and Distribution ShiftZhang, Lipton, Li & SmolaCovariate, label and concept shift, with the corrections derived in this lesson.From: Imbalance and distribution shift
Dataset Shift in Machine LearningJ. Quiñonero-Candela, M. Sugiyama, A. Schwaighofer & N. D. Lawrence (eds.) · 2009MIT Press. The edited volume that consolidated the field.From: Imbalance and distribution shift
WILDS: A Benchmark of in-the-Wild Distribution ShiftsPang Wei Koh et al. · 2021ICML 2021. Ten real datasets where test data come from new hospitals, cameras, places or times.From: Imbalance and distribution shift
The harm of class imbalance corrections for risk prediction modelsR. van den Goorbergh, M. van Smeden, D. Timmerman & B. Van Calster · 2022JAMIA 29(9). Why resampling ruins calibrated risks, and why moving the threshold is usually enough.From: Imbalance and distribution shift
Failing Loudly: An Empirical Study of Methods for Detecting Dataset ShiftStephan Rabanser, Stephan Günnemann & Zachary C. Lipton · 2019NeurIPS 2019. Which two-sample tests actually catch shift in high dimensions.From: Imbalance and distribution shift
Bootstrap Methods: Another Look at the JackknifeBradley Efron · 1979Annals of Statistics 7(1), 1–26. The paper that introduced the bootstrap.From: How sure are you? Bootstrap and tests
Approximate Statistical Tests for Comparing Supervised Classification Learning AlgorithmsThomas G. Dietterich · 1998Neural Computation 10(7). Which tests for "is algorithm B better?" can be trusted, and which can't.From: How sure are you? Bootstrap and tests
Accounting for Variance in Machine Learning BenchmarksXavier Bouthillier et al. · 2021MLSys 2021. How much data sampling, initialisation and hyperparameter choice move benchmark results.From: How sure are you? Bootstrap and tests
Adding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsEvan Miller · 2024Standard errors, clustering and paired differences for LLM benchmark scores.From: How sure are you? Bootstrap and tests

VIFinding Structure

Probabilistic Machine Learning: An Introduction, §21.3Kevin P. Murphy · 2022K-means, vector quantization, k-means++, k-medoids and choosing K, with figures.From: k-means, properly
scikit-learn user guide: K-meansscikit-learn developers · 2026The inertia criterion, its assumptions, and how the production implementation initialises and stops.From: k-means, properly
k-means++: The Advantages of Careful SeedingArthur & Vassilvitskii · 2007The seeding rule and its O(log k) guarantee.From: k-means, properly
Silhouettes: a graphical aid to the interpretation and validation of cluster analysisPeter J. Rousseeuw · 1987From: k-means, properly
Maximum Likelihood from Incomplete Data via the EM AlgorithmDempster, Laird & Rubin · 1977The paper that named the algorithm and gave it its general form.From: Gaussian mixtures and EM
scikit-learn user guide: Gaussian mixture modelsscikit-learn developers · 2026Covariance types, initialisation, singularities and BIC in a production implementation.From: Gaussian mixtures and EM
scikit-learn user guide: Hierarchical clustering, DBSCAN, HDBSCANscikit-learn developers · 2026Practical behaviour of each linkage, DBSCAN's parameters, and the HDBSCAN construction.From: Hierarchical and density-based clustering
A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with NoiseEster, Kriegel, Sander & Xu · 1996The DBSCAN paper (KDD-96).From: Hierarchical and density-based clustering
Density-Based Clustering Based on Hierarchical Density EstimatesCampello, Moulavi & Sander · 2013HDBSCAN.From: Hierarchical and density-based clustering
A Tutorial on Spectral ClusteringUlrike von Luxburg · 2007The standard introduction (Statistics and Computing 17(4)). Graph types, the three Laplacians, derivations and practical advice.From: Spectral clustering and graph Laplacians
Normalized Cuts and Image SegmentationShi & Malik · 2000From: Spectral clustering and graph Laplacians
On Spectral Clustering: Analysis and an algorithmNg, Jordan & Weiss · 2001NIPS 14. The row-normalised algorithm implemented here.From: Spectral clustering and graph Laplacians
How to Use t-SNE EffectivelyWattenberg, Viégas & Johnson · 2016Interactive, and the single best guide to misreading t-SNE plots. Much of the last section follows it.From: t-SNE, UMAP and manifold learning
Visualizing Data using t-SNEvan der Maaten & Hinton · 2008JMLR 9. The original paper.From: t-SNE, UMAP and manifold learning
UMAP: Uniform Manifold Approximation and Projection for Dimension ReductionMcInnes, Healy et al. · 2018From: t-SNE, UMAP and manifold learning
Initialization is critical for preserving global data structure in both t-SNE and UMAPKobak & Linderman · 2021From: t-SNE, UMAP and manifold learning
scikit-learn user guide: Density estimationscikit-learn developers · 2026From histograms to kernel density estimation, with kernels and bandwidth.From: Density estimation
A Tutorial on Kernel Density Estimation and Recent AdvancesYen-Chi Chen · 2017Bias, variance, convergence rates and bandwidth selection, carefully.From: Density estimation
Isolation ForestLiu, Ting & Zhou · 2008ICDM 2008. Short, readable, and the source of the path-length score.From: Anomaly detection
LOF: Identifying Density-Based Local OutliersBreunig, Kriegel, Ng & Sander · 2000SIGMOD 2000.From: Anomaly detection
Anomaly Detection: A SurveyChandola, Banerjee & Kumar · 2009ACM Computing Surveys 41(3). The taxonomy of anomaly types.From: Anomaly detection
scikit-learn user guide: Novelty and outlier detectionscikit-learn developers · 2026From: Anomaly detection

VIIThinking in Distributions

Probabilistic Machine Learning: An Introduction, §4.6 (Bayesian statistics) and §11.7 (Bayesian linear regression)Kevin P. Murphy · 2022Conjugate priors, posterior predictives, credible intervals, and the Bayesian linear regression figures this lesson's plate imitates.From: Bayesian inference
Probabilistic Machine Learning: Advanced Topics, ch. 3 and ch. 7Kevin P. Murphy · 2023Bayesian statistics at research level, and a map of the approximate-inference algorithms the rest of this part teaches.From: Bayesian inference
Bayesian InterpolationDavid J. C. MacKay · 1992The evidence framework for choosing regularisers and models, with Occam's razor falling out of the maths.From: Bayesian inference
Reverend Bayes on Inference Engines: A Distributed Hierarchical ApproachJudea Pearl · 1982Pearl's early paper on propagating evidence by local messages in tree-structured networks, the seed of belief propagation.From: Graphical models
A Tutorial on Hidden Markov Models and Selected Applications in Speech RecognitionLawrence R. Rabiner · 1989Proceedings of the IEEE 77(2). The classic tutorial, organised around the three problems of evaluation, decoding and learning.From: Latent variables, HMMs and Kalman filters
A New Approach to Linear Filtering and Prediction ProblemsR. E. Kalman · 1960Journal of Basic Engineering 82(1), 35–45. The original paper.From: Latent variables, HMMs and Kalman filters
Variational Inference: A Review for StatisticiansBlei, Kucukelbir & McAuliffe · 2017Journal of the American Statistical Association 112(518). The standard review, from the ELBO and CAVI to stochastic VI.From: Variational inference
Auto-Encoding Variational BayesKingma & Welling · 2013The reparameterised lower bound, and the variational autoencoder.From: Variational inference
Rank-normalization, folding, and localization: An improved R-hat for assessing convergence of MCMCVehtari, Gelman, Simpson, Carpenter & Bürkner · 2021Bayesian Analysis 16(2). Why R-hat below 1.1 is not enough, and the diagnostics Stan and ArviZ use today.From: Markov chain Monte Carlo
Inference from Iterative Simulation Using Multiple SequencesGelman & Rubin · 1992Statistical Science 7(4). Run several chains from overdispersed starts and compare them.From: Markov chain Monte Carlo
Gaussian Processes for Machine LearningCarl Edward Rasmussen & Christopher K. I. Williams · 2006The standard book, free online. Chapter 2 (regression) and chapter 5 (model selection) are this lesson in depth.From: Gaussian processes

VIIIThe Shape of Loss

Convex OptimizationStephen Boyd & Lieven Vandenberghe · 2004The standard text, free from the authors. Chapters 2–3 (sets, functions) and §4.2 (local = global) are this lesson in full detail.From: Convexity
Probabilistic Machine Learning: An Introduction, §8.1Kevin Murphy · 2022Convexity, strong convexity, smoothness and subgradients in ten pages, in ML notation.From: Convexity
Handbook of Convergence Theorems for (Stochastic) Gradient Methods, §2Guillaume Garrigos & Robert M. Gower · 2023Every property of convex, strongly convex and smooth functions used in this part, each with a short proof.From: Convexity
Deep Learning, §8.2 (Challenges in neural network optimization)Goodfellow, Bengio & Courville · 2016What changes when the loss is not convex.From: Convexity
Mathematics for Machine Learning, §7.2 and §12.3Deisenroth, Faisal & Ong · 2020Lagrange multipliers and duality in ML notation, then the SVM dual worked through.From: Lagrange multipliers, duality and KKT
Numerical Optimization (2nd ed.)Jorge Nocedal & Stephen J. Wright · 2006The standard reference for line searches, trust regions, quasi-Newton methods (BFGS, L-BFGS) and nonlinear least squares. SciPy's BFGS cites it.From: Newton, quasi-Newton and natural gradients
Natural Gradient Works Efficiently in LearningShun-ichi Amari · 1998The paper that introduced the natural gradient to machine learning.From: Newton, quasi-Newton and natural gradients
Probabilistic Machine Learning: Advanced Topics, §6.4Kevin Murphy · 2023Natural gradient descent, the Fisher matrix, its connection to Gauss–Newton, and K-FAC.From: Newton, quasi-Newton and natural gradients
Optimization Methods for Large-Scale Machine LearningLéon Bottou, Frank E. Curtis & Jorge Nocedal · 2018The standard review of stochastic gradient methods for ML: why SGD, its convergence theory, noise reduction and second-order extensions (SIAM Review 60(2)).From: Why SGD converges
ELE 522 lecture notes: accelerated gradient, stochastic gradient, variance reductionYuxin Chen (Princeton) · 2019Slide-by-slide proofs, including Nesterov's method and the lower bounds.From: Why SGD converges
Dive into Deep Learning, §12.4 (Stochastic Gradient Descent)Zhang, Lipton, Li & SmolaA gentle version of the O(1/√T) proof for convex SGD.From: Why SGD converges
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse ProblemsAmir Beck & Marc Teboulle · 2009FISTA — proximal gradient with Nesterov's momentum (SIAM Journal on Imaging Sciences 2(1)).From: Constraints, projections and proximal methods
Decoupled Weight Decay RegularizationIlya Loshchilov & Frank Hutter · 2019Why weight decay and L2 regularisation differ for adaptive methods, and AdamW.From: Constraints, projections and proximal methods

IXWhy Learning Works

Understanding Machine Learning: From Theory to Algorithms, ch. 2–4Shai Shalev-Shwartz & Shai Ben-David · 2014The clearest modern textbook on this material, free from the authors. This lesson follows its chapters 2–4.From: PAC learning
A Theory of the LearnableLeslie G. Valiant · 1984The paper that introduced the PAC model, in Communications of the ACM.From: PAC learning
Probability Inequalities for Sums of Bounded Random VariablesWassily Hoeffding · 1963The concentration inequality that does the work in the agnostic case.From: PAC learning
On the Uniform Convergence of Relative Frequencies of Events to Their ProbabilitiesV. N. Vapnik & A. Ya. Chervonenkis · 1971The original paper (English translation in Theory of Probability and Its Applications), reprinted with commentary.From: VC dimension
Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training DataGintare Karolina Dziugaite & Daniel M. Roy · 2017Optimises a PAC-Bayes bound directly and gets a non-vacuous guarantee for a large network.From: Rademacher complexity and generalization bounds
The Lack of A Priori Distinctions Between Learning AlgorithmsDavid H. Wolpert · 1996The supervised-learning no-free-lunch paper, in Neural Computation 8(7).From: No free lunch and inductive bias
Bandit AlgorithmsTor Lattimore & Csaba Szepesvári · 2020The modern reference, free online. Chapters 4, 6–8, 11, 15–16, 18 and 36 cover everything in this lesson and much more.From: Online learning and bandits
Finite-time Analysis of the Multiarmed Bandit ProblemP. Auer, N. Cesa-Bianchi & P. Fischer · 2002UCB1 and its finite-time logarithmic regret bound, in Machine Learning 47.From: Online learning and bandits

XWhy Neural Networks

The perceptron: a probabilistic model for information storage and organization in the brainFrank Rosenblatt · 1958Psychological Review 65, 386–408. The paper that started the first wave.From: A short history of the neuron
Learning representations by back-propagating errorsRumelhart, Hinton & Williams · 1986Four pages in Nature that made backpropagation famous.From: A short history of the neuron
Who invented the reverse mode of differentiation?Andreas Griewank · 2012A numerical analyst's history of the algorithm behind backprop.From: A short history of the neuron
The Perceptron ControversyYuxi LiuA long, opinionated essay arguing that Perceptrons did not "kill" neural networks.From: A short history of the neuron
The Bitter LessonRich Sutton · 2019One page. Read it after this lesson.From: A short history of the neuron
Deep Learning, chapter 1 (historical trends)Goodfellow, Bengio & Courville · 2016The source of the "three waves" framing used in the timeline plate.From: A short history of the neuron
Kernel methods in machine learningHofmann, Schölkopf & Smola · 2008A careful survey, including the history from Aizerman et al. (1964) to support vector machines.From: The limits of linear
Support-vector networksCorinna Cortes & Vladimir Vapnik · 1995The support vector machine paper — a linear classifier in a high-dimensional feature space.From: The limits of linear
Neural Networks, Manifolds, and TopologyChris Olah · 2014The inspiration for Plate X.3.4 — layers as deformations of space, with beautiful animations.From: The multilayer perceptron
A visual proof that neural nets can compute any functionMichael NielsenChapter 4 of Neural Networks and Deep Learning. Builds universal approximation out of steps and bumps.From: The multilayer perceptron
Approximation by superpositions of a sigmoidal functionGeorge Cybenko · 1989The universal approximation theorem for one hidden layer of sigmoidal units.From: The multilayer perceptron
TensorFlow PlaygroundSmilkov, Carter et al. · 2016The original in-browser network playground that Plate X.3.5 pays homage to.From: The multilayer perceptron
3Blue1Brown — Neural networks (video series)Grant SandersonWatch chapter 1 alongside this lesson.From: The multilayer perceptron
How the backpropagation algorithm works (Neural Networks and Deep Learning, ch. 2)Michael NielsenThe four fundamental equations of backprop, derived slowly, with code.From: Backpropagation
Calculus on Computational Graphs: BackpropagationChris Olah · 2015Forward-mode versus reverse-mode differentiation, with pictures. The fan-out example in Plate X.4.1 is his.From: Backpropagation
Automatic differentiation in machine learning: a surveyBaydin, Pearlmutter, Radul & Siskind · 2018Where backprop sits within automatic differentiation; the bridge to Autodiff from scratch, later in the route.From: Backpropagation
Understanding the difficulty of training deep feedforward neural networksXavier Glorot & Yoshua Bengio · 2010The paper behind "Xavier" initialisation — and a careful look at activations and gradients layer by layer.From: Training in practice
Adam: A Method for Stochastic OptimizationDiederik P. Kingma & Jimmy Ba · 2014From: Training in practice
Decoupled Weight Decay RegularizationLoshchilov & Hutter · 2019From: Training in practice
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftSergey Ioffe & Christian Szegedy · 2015From: Training in practice
How Does Batch Normalization Help Optimization?Santurkar, Tsipras, Ilyas & Madry · 2018Argues that batch norm's benefit has little to do with the "internal covariate shift" it was named after.From: Training in practice
On the importance of initialization and momentum in deep learningSutskever et al. · 2013The paper PyTorch cites for its Nesterov momentum.From: Training in practice

XISeeing and Remembering

Gradient-Based Learning Applied to Document RecognitionLeCun, Bottou, Bengio & Haffner · 1998Section II makes the case against fully connected nets for images, in the inventors' own words.From: Convolution
Deep Learning, ch. 9: Convolutional NetworksGoodfellow, Bengio & Courville · 2016Sparse interactions, parameter sharing, equivariance and pooling, carefully.From: Convolution
Dive into Deep Learning, ch. 7: Convolutional Neural NetworksZhang, Lipton, Li & Smola · 2023Derives convolution from locality and translation invariance, with code in several frameworks.From: Convolution
Understanding the Effective Receptive Field in Deep Convolutional Neural NetworksLuo, Li, Urtasun & Zemel · 2016Why the part of the receptive field that actually matters is smaller than the formula says.From: Convolution
ImageNet Classification with Deep Convolutional Neural NetworksKrizhevsky, Sutskever & Hinton · 2012AlexNet. Short and readable; section 3 lists the tricks, in order of importance.From: The CNN lineage: LeNet to ResNet
Deep Residual Learning for Image RecognitionHe, Zhang, Ren & Sun · 2015The degradation problem and residual learning. Figure 1 is worth a thousand words.From: The CNN lineage: LeNet to ResNet
Identity Mappings in Deep Residual NetworksHe, Zhang, Ren & Sun · 2016The gradient argument for the shortcut, in two equations.From: The CNN lineage: LeNet to ResNet
Dive into Deep Learning, ch. 8: Modern Convolutional Neural NetworksZhang, Lipton, Li & Smola · 2023AlexNet, VGG, GoogLeNet, batch norm and ResNet, each with runnable code.From: The CNN lineage: LeNet to ResNet
Finding Structure in TimeJeffrey L. Elman · 1990The simple recurrent network, and a beautiful argument for representing time implicitly.From: Recurrent networks
On the difficulty of training recurrent neural networksPascanu, Mikolov & Bengio · 2013Vanishing and exploding gradients made precise, and the case for gradient clipping.From: Recurrent networks
The Unreasonable Effectiveness of Recurrent Neural NetworksAndrej Karpathy · 2015Character-level RNNs writing Shakespeare, LaTeX and C, plus a 100-line NumPy implementation.From: Recurrent networks
Dive into Deep Learning, ch. 9: Recurrent Neural NetworksZhang, Lipton, Li & Smola · 2023Language models, RNNs from scratch, and a careful derivation of BPTT.From: Recurrent networks
Long Short-Term MemoryHochreiter & Schmidhuber · 1997The original. Section 3 derives the constant error carrousel; section 4 adds the gates.From: LSTM and GRU
Understanding LSTM NetworksChristopher Olah · 2015The classic illustrated walk-through of the cell, gate by gate, with variants.From: LSTM and GRU
Learning to Forget: Continual Prediction with LSTMGers, Schmidhuber & Cummins · 2000Where the forget gate comes from.From: LSTM and GRU
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine TranslationCho et al. · 2014Introduces the gated recurrent unit (and the encoder–decoder of the next lesson).From: LSTM and GRU
Sequence to Sequence Learning with Neural NetworksSutskever, Vinyals & Le · 2014Deep LSTMs, reversed source sentences, and the first neural system to beat a phrase-based baseline at scale.From: Seq2seq and the bottleneck
On the Properties of Neural Machine Translation: Encoder–Decoder ApproachesCho, van Merriënboer, Bahdanau & Bengio · 2014A careful look at where encoder–decoders fail — long sentences and unknown words.From: Seq2seq and the bottleneck
Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau, Cho & Bengio · 2014The paper that removed the bottleneck. The Attention part starts here.From: Seq2seq and the bottleneck
Dive into Deep Learning, ch. 10: Modern Recurrent Neural NetworksZhang, Lipton, Li & Smola · 2023Encoder–decoders, teacher forcing, beam search and BLEU, with code.From: Seq2seq and the bottleneck

XIIAttention

Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau, Cho & Bengio · 2014The paper that introduced soft alignment (attention) for translation; published at ICLR 2015.From: Attention: learning where to look
Effective Approaches to Attention-based Neural Machine TranslationLuong, Pham & Manning · 2015Global and local attention, and the dot / general / concat scoring functions.From: Attention: learning where to look
Attention? Attention!Lilian Weng · 2018A careful survey of the attention family with consistent notation.From: Attention: learning where to look
Dive into Deep Learning, ch. 11: Attention Mechanisms and TransformersZhang, Lipton, Li & Smola · 2023Queries, keys and values introduced through a database analogy, with runnable code.From: Attention: learning where to look
Speech and Language Processing, ch. 13 (RNNs and LSTMs), §13.8 AttentionJurafsky & Martin · 2025The textbook account of the bottleneck and of dot-product attention in encoder–decoders.From: Attention: learning where to look
Attention Is All You NeedVaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser & Polosukhin · 2017Section 3.2 is this lesson. The summit's worked reading of this paper goes through it line by line.From: Scaled dot-product and multi-head attention
The Annotated TransformerRush; updated by Huang, Subramanian, Sum, Almubarak & Biderman · 2022The paper's text interleaved with a working PyTorch implementation.From: Scaled dot-product and multi-head attention
Dive into Deep Learning, §11.3 and §11.5: scoring functions and multi-head attentionZhang, Lipton, Li & Smola · 2023From: Scaled dot-product and multi-head attention
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be PrunedVoita, Talbot, Moiseev, Sennrich & Titov · 2019Evidence that heads specialise — and that many are dispensable after training.From: Scaled dot-product and multi-head attention
On Layer Normalization in the Transformer ArchitectureXiong, Yang, He, Zheng, Zheng, Xing, Zhang, Lan, Wang & Liu · 2020Why Post-LN needs learning-rate warm-up and Pre-LN doesn't (ICML 2020).From: The transformer block
A Mathematical Framework for Transformer CircuitsElhage, Nanda, Olsson et al. · 2021The source of the residual-stream way of seeing a transformer.From: The transformer block
Scaling Laws for Neural Language ModelsKaplan et al. · 2020Section 2.1 gives the ≈ 12 n_layer d²_model parameter count used here.From: The transformer block
Self-Attention with Relative Position RepresentationsShaw, Uszkoreit & Vaswani · 2018NAACL 2018 (short paper).From: Positional information
RoFormer: Enhanced Transformer with Rotary Position EmbeddingSu, Lu, Pan, Murtadha, Wen & Liu · 2021From: Positional information
Train Short, Test Long: Attention with Linear Biases Enables Input Length ExtrapolationPress, Smith & Lewis · 2022ICLR 2022; the repository links the paper and explains the method.From: Positional information
The Impact of Positional Encoding on Length Generalization in TransformersKazemnejad, Padhi, Natesan, Das & Reddy · 2023NeurIPS 2023. Finds that no explicit encoding (NoPE) often generalises best in decoder-only models.From: Positional information
Dive into Deep Learning, §11.6 Self-Attention and Positional EncodingZhang, Lipton, Li & Smola · 2023From: Positional information
Neural Machine Translation of Rare Words with Subword UnitsSennrich, Haddow & Birch · 2016Byte-pair encoding for open-vocabulary translation (ACL 2016).From: GPT and next-token prediction
Language Models are Few-Shot LearnersBrown et al. · 2020GPT-3, and the term in-context learning.From: GPT and next-token prediction
BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingDevlin, Chang, Lee & Toutanova · 2018From: GPT and next-token prediction
GPT-2 model card and codeOpenAI · 2019The released model, its byte-level BPE encoder and its sampling code.From: GPT and next-token prediction
How to generate text: using different decoding methodsPatrick von Platen (Hugging Face) · 2020Greedy, beam, temperature, top-k and top-p, with examples from GPT-2.From: GPT and next-token prediction

XIIIMaking It Fast

How To Scale Your Model, ch. 1 (All About Rooflines) and ch. 7 (All About Transformer Inference)Austin, Douglas, Frostig, Levskaya, Chen, Vikram, Lebron, Choy, Ramasesh, Webson & Pope (Google DeepMind) · 2025The clearest long-form treatment of rooflines, prefill and decode. Most numbers in this lesson follow it.From: The cost of attention
Large Transformer Model Inference OptimizationLilian Weng · 2023A survey of every trick in this part, with references.From: The cost of attention
Optimizing your LLM in productionPatrick von Platen (Hugging Face) · 2023Hands-on numbers for attention memory, the KV cache, MQA and GQA (also in the Transformers docs as "Optimizing LLMs for Speed and Memory").From: The cost of attention
Understanding and Coding the KV Cache in LLMs from ScratchSebastian Raschka · 2025Readable from-scratch PyTorch code, with and without a cache.From: The KV cache
Fast Transformer Decoding: One Write-Head is All You NeedNoam Shazeer · 2019Multi-query attention.From: The KV cache
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsAinslie, Lee-Thorp, de Jong, Zemlyanskiy, Lebrón & Sanghai · 2023From: The KV cache
Efficient Memory Management for Large Language Model Serving with PagedAttentionKwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang & Stoica · 2023The vLLM paper (SOSP 2023).From: The KV cache
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelDeepSeek-AI · 2024Introduces multi-head latent attention (MLA).From: The KV cache
Cache strategies (Hugging Face Transformers documentation)2025From: The KV cache
Generating Long Sequences with Sparse TransformersRewon Child et al. (OpenAI) · 2019From: Sparse attention
Longformer: The Long-Document TransformerBeltagy, Peters & Cohan · 2020From: Sparse attention
Big Bird: Transformers for Longer SequencesZaheer et al. · 2020From: Sparse attention
Mistral 7BJiang et al. · 2023From: Sparse attention
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse AttentionYuan et al. (DeepSeek-AI, Peking University, University of Washington) · 2025ACL 2025.From: Sparse attention
The Transformer Family Version 2.0Lilian Weng · 2023A careful survey of sparse and efficient attention patterns, with the Sparse Transformer's formulas.From: Sparse attention
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessDao, Fu, Ermon, Rudra & Ré · 2022NeurIPS 2022.From: FlashAttention and the memory hierarchy
FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri Dao · 2023ICLR 2024.From: FlashAttention and the memory hierarchy
Online normalizer calculation for softmaxMilakov & Gimelshein (NVIDIA) · 2018The running-maximum trick, originally for softmax layers.From: FlashAttention and the memory hierarchy
Self-attention Does Not Need O(n²) MemoryRabe & Staats · 2021From: FlashAttention and the memory hierarchy
How To Scale Your Model, ch. 4, Appendix A: How does Flash Attention work?Austin et al. (Google DeepMind) · 2025From: FlashAttention and the memory hierarchy
Flash Attention (annotated implementation)labml.aiA readable Triton implementation of FlashAttention-2 with the equations alongside.From: FlashAttention and the memory hierarchy
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerShazeer, Mirhoseini, Maziarz, Davis, Le, Hinton & Dean · 2017From: Mixture of experts
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityFedus, Zoph & Shazeer · 2021From: Mixture of experts
Mixtral of ExpertsJiang et al. (Mistral AI) · 2024From: Mixture of experts
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDai et al. (DeepSeek-AI) · 2024From: Mixture of experts
Mixture of Experts ExplainedHugging Face · 2023A clear survey of the whole line of work, with the practical trade-offs.From: Mixture of experts

XIVMaking New Things

Probabilistic Machine Learning: Advanced Topics, ch. 20 (Generative models: an overview)Kevin P. Murphy · 2023The map of model families (Table 20.1) and the section on evaluating generative models by Rosca, Mohamed and Lakshminarayanan. The source for most of this lesson.From: What generative modelling is
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash EquilibriumHeusel, Ramsauer, Unterthiner, Nessler & Hochreiter · 2017NeurIPS 2017. The paper that introduced the Fréchet Inception Distance; the link is the authors' code and FID implementation.From: What generative modelling is
On Aliased Resizing and Surprising Subtleties in GAN EvaluationParmar, Zhang & Zhu · 2022CVPR 2022. How image resizing and quantisation change FID, and a library that standardises it.From: What generative modelling is
Auto-Encoding Variational BayesDiederik P. Kingma & Max Welling · 2013ICLR 2014. The VAE and the reparameterized lower bound.From: Autoencoders and VAEs
From Autoencoder to Beta-VAELilian Weng · 2018A clear walk from plain autoencoders to the VAE family.From: Autoencoders and VAEs
Generative Adversarial NetworksGoodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville & Bengio · 2014NIPS 2014. The original paper; short, and the proofs of the optimal discriminator are in it.From: Generative adversarial networks
Wasserstein GANArjovsky, Chintala & Bottou · 2017ICML 2017. The Wasserstein distance as a GAN objective.From: Generative adversarial networks
From GAN to WGANLilian Weng · 2017The derivations in this lesson, worked through with care, plus the problems of GAN training.From: Generative adversarial networks
Density estimation using Real NVPLaurent Dinh, Jascha Sohl-Dickstein & Samy Bengio · 2016ICLR 2017. Affine coupling layers with checkerboard and channel masks.From: Normalizing flows
Flow-based Deep Generative ModelsLilian Weng · 2018NICE, RealNVP, Glow and autoregressive flows, side by side.From: Normalizing flows
Relaxing Bijectivity Constraints with Continuously Indexed Normalising FlowsCornish, Caterini, Deligiannidis & Doucet · 2020ICML 2020. Why a single bijection struggles when the data's support has a different topology from the base.From: Normalizing flows
Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain & Pieter Abbeel · 2020NeurIPS 2020. The paper that made diffusion competitive; its Algorithms 1 and 2 are this lesson's code.From: Diffusion models
What are Diffusion Models?Lilian Weng · 2021Every derivation in this lesson, in full, with later updates on guidance and latent diffusion.From: Diffusion models
High-Resolution Image Synthesis with Latent Diffusion ModelsRombach, Blattmann, Lorenz, Esser & Ommer · 2021CVPR 2022. Diffusion in an autoencoder's latent space, conditioned by cross-attention; the basis of Stable Diffusion.From: Diffusion models
Score-Based Generative Modeling through Stochastic Differential EquationsSong, Sohl-Dickstein, Kingma, Kumar, Ermon & Poole · 2021ICLR 2021. The SDE framework that unifies score-based models and DDPM; the link is the authors' code and summary.From: Score matching and flow matching
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified FlowXingchao Liu, Chengyue Gong & Qiang Liu · 2022ICLR 2023. Straight-line interpolation, and the reflow procedure that straightens a learned flow.From: Score matching and flow matching
Flow Matching Guide and Codebase (facebookresearch/flow_matching)2024A PyTorch library for continuous and discrete flow matching, with 2-D example notebooks; its README links the accompanying guide (arXiv 2412.06264).From: Score matching and flow matching

XVOther Worlds

An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleDosovitskiy et al. · 2021The Vision Transformer. Section 3.1 is one page and contains the whole model; section 4.3 is the data-size study.From: Vision transformers, detection and segmentation
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksRen, He, Girshick & Sun · 2015Anchors and the region proposal network — the template for two-stage detectors.From: Vision transformers, detection and segmentation
You Only Look Once: Unified, Real-Time Object DetectionRedmon, Divvala, Girshick & Farhadi · 2016Detection as a single regression; section 2.4 lists its limitations honestly.From: Vision transformers, detection and segmentation
U-Net: Convolutional Networks for Biomedical Image SegmentationRonneberger, Fischer & Brox · 2015Eight pages; Figure 1 is the architecture that became everyone's default encoder–decoder.From: Vision transformers, detection and segmentation
Object Recognition for Dummies, parts 3 and 4Lilian Weng · 2017A clear walk through R-CNN, Fast/Faster/Mask R-CNN, and then YOLO and SSD.From: Vision transformers, detection and segmentation
COCO evaluation code (cocoeval.py)COCO consortiumThe definition of COCO AP is this file. Read accumulate() once.From: Vision transformers, detection and segmentation
A Gentle Introduction to Graph Neural NetworksSanchez-Lengeling, Reif, Pearce & Wiltschko · 2021Distill. Graphs as data, the three task levels, and a playground — the best first read.From: Graph neural networks
Understanding Convolutions on GraphsDaigavane, Ravindran & Aggarwal · 2021Distill. Equivariance, the graph Laplacian, and spectral versus spatial convolutions.From: Graph neural networks
Graph Convolutional Networks (blog post)Thomas Kipf · 2016The author's own derivation of the propagation rule, and the random-weights karate-club surprise.From: Graph neural networks
Semi-Supervised Classification with Graph Convolutional NetworksKipf & Welling · 2017ICLR 2017. The GCN paper.From: Graph neural networks
How Powerful are Graph Neural Networks?Xu, Hu, Leskovec & Jegelka · 2019ICLR 2019. The Weisfeiler–Lehman connection and GIN.From: Graph neural networks
PyTorch Geometric: Creating message passing networksPyG teamThe general message-passing equation and how a library implements it.From: Graph neural networks
Sequence Modeling with CTCAwni Hannun · 2017Distill article (source repository). The clearest explanation of CTC's alignments, loss and decoding.From: Speech and audio
Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural NetworksGraves, Fernandez, Gomez & Schmidhuber · 2006ICML 2006. The original CTC paper.From: Speech and audio
Robust Speech Recognition via Large-Scale Weak SupervisionRadford, Kim, Xu, Brockman, McLeavey & Sutskever · 2022The Whisper paper. Read it alongside the model card in the official repository.From: Speech and audio
Whisper (official code and model card)OpenAIaudio.py is a 150-line, readable log-mel front end; model-card.md states the limitations.From: Speech and audio
librosa documentation: stft, mel filterslibrosa developersThe reference Python implementations of STFT and mel scales, with notes on competing mel conventions.From: Speech and audio
Forecasting: Principles and Practice (3rd edition)Rob J. Hyndman & George AthanasopoulosFree online textbook; the practical reference for ARIMA, exponential smoothing and forecast evaluation.From: Time series and forecasting
M4 competition: methods, forecasts and evaluation codeMakridakis, Spiliotis & Assimakopoulos (organisers)The data, every submission's forecasts, the benchmarks' R code and the official scores.From: Time series and forecasting
Are Transformers Effective for Time Series Forecasting?2023AAAI 2023. One-layer linear models against long-horizon forecasting transformers.From: Time series and forecasting
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence ModelingBai, Kolter & Koltun · 2018Temporal convolutional networks versus recurrent networks.From: Time series and forecasting
statsmodels: time series analysis (tsa)statsmodels developersAutoReg, ARIMA, exponential smoothing, ACF/PACF and unit-root tests in Python.From: Time series and forecasting
Dive into Deep Learning, chapter 21: Recommender SystemsZhang, Lipton, Li & SmolaCollaborative filtering, matrix factorisation, AutoRec and ranking losses, with code.From: Recommender systems
TensorFlow Recommenders: retrieval and ranking tutorialsTensorFlow teamThe two-stage architecture and a two-tower model, end to end.From: Recommender systems
Microsoft Recommenders: evaluation metrics and glossaryRecommenders contributorsReference implementations of recall@k, NDCG@k and friends, and a glossary of the field's vocabulary.From: Recommender systems
Learning Transferable Visual Models From Natural Language SupervisionRadford, Kim, Hallacy, Ramesh, Goh et al. · 2021CLIP (ICML 2021). Figure 3's pseudocode is the whole training objective in a dozen lines.From: Multimodal models
Contrastive Representation LearningLilian Weng · 2021InfoNCE, SimCLR, MoCo, CLIP and more in one careful survey.From: Multimodal models
Generalized Visual Language ModelsLilian Weng · 2022ClipCap, Flamingo and the other ways of wiring a vision encoder to a language model.From: Multimodal models
Visual Instruction Tuning (LLaVA)Liu, Li, Wu & Lee · 2023NeurIPS 2023. A vision encoder, a projection, and a language model, trained in two stages.From: Multimodal models
CLIP (official code and model card)OpenAImodel-card.md lists the limitations and bias findings.From: Multimodal models

XVILearning Signals

Reinforcement Learning: An Introduction (2nd ed.), ch. 1Sutton & Barto · 2018The standard textbook, free online. Chapter 1 contrasts reinforcement learning with supervised and unsupervised learning.From: Three kinds of learning signal
scikit-learn user guide: K-means2026A clear, practical account of Lloyd's algorithm, its objective and its failure modes.From: Three kinds of learning signal
A Simple Framework for Contrastive Learning of Visual Representations (SimCLR)Chen, Kornblith, Norouzi & Hinton · 2020From: Three kinds of learning signal
Learning with not Enough Data Part 1: Semi-Supervised LearningLilian Weng · 2021From: Three kinds of learning signal
Reinforcement Learning from Human Feedback (book), ch. 7Nathan Lambert · 2026Where the “cake” analogy is recounted, and how modern post-training stacks the paradigms.From: Three kinds of learning signal
Spinning Up in Deep RL — Part 1: Key Concepts in RLOpenAI (Joshua Achiam) · 2018A compact, careful statement of the definitions and Bellman equations used here.From: Reinforcement learning foundations
Human-level control through deep reinforcement learningMnih et al. · 2015From: Reinforcement learning foundations
A (Long) Peek into Reinforcement LearningLilian Weng · 2018From: Reinforcement learning foundations
Spinning Up in Deep RL — Part 3: Intro to Policy OptimizationOpenAI (Joshua Achiam) · 2018The derivation this lesson follows, with the EGLP lemma, reward-to-go and baselines.From: Policy gradients and PPO
Simple statistical gradient-following algorithms for connectionist reinforcement learningRonald J. Williams · 1992REINFORCE. Machine Learning 8, 229–256.From: Policy gradients and PPO
Proximal Policy Optimization AlgorithmsSchulman, Wolski, Dhariwal, Radford & Klimov · 2017From: Policy gradients and PPO
Trust Region Policy OptimizationSchulman, Levine, Moritz, Jordan & Abbeel · 2015From: Policy gradients and PPO
Training language models to follow instructions with human feedback (InstructGPT)Ouyang et al. · 2022From: RLHF: learning from preferences
Deep reinforcement learning from human preferencesChristiano, Leike, Brown, Martic, Legg & Amodei · 2017From: RLHF: learning from preferences
Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafailov, Sharma, Mitchell, Manning, Ermon & Finn · 2023From: RLHF: learning from preferences
Scaling Laws for Reward Model OveroptimizationGao, Schulman & Hilton · 2022From: RLHF: learning from preferences
Reward Hacking in Reinforcement LearningLilian Weng · 2024From: RLHF: learning from preferences
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-AI (Guo et al.) · 2025Later published in Nature as “DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning”.From: RLVR and reasoning models
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsShao et al. · 2024Introduces GRPO.From: RLVR and reasoning models
Tülu 3: Pushing Frontiers in Open Language Model Post-TrainingLambert et al. · 2024Names RLVR.From: RLVR and reasoning models
Learning to reason with LLMsOpenAI · 2024From: RLVR and reasoning models
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?Yue et al. · 2025From: RLVR and reasoning models

XVIIDeeper Reinforcement

Reinforcement Learning: An Introduction (2nd ed.), ch. 11 and §16.5Sutton & Barto · 2018Off-policy divergence, Baird's counterexample and the deadly triad; the DQN case study.From: Deep Q-learning and actor–critic methods
Human-level control through deep reinforcement learningMnih et al. · 2015From: Deep Q-learning and actor–critic methods
Deep Reinforcement Learning with Double Q-learningvan Hasselt, Guez & Silver · 2016From: Deep Q-learning and actor–critic methods
Deep Reinforcement Learning and the Deadly Triadvan Hasselt, Doron, Strub, Hessel, Sonnerat & Modayil · 2018From: Deep Q-learning and actor–critic methods
Spinning Up in Deep RL — DDPG, TD3 and SACOpenAI (Joshua Achiam) · 2018Clear statements of the three algorithms' losses; this lesson's equations follow them.From: Deep Q-learning and actor–critic methods
A general reinforcement learning algorithm that masters chess, shogi and Go through self-playSilver et al. · 2018Preprint of the Science paper (Science 362, 1140–1144).From: Model-based RL and planning
Mastering Atari, Go, Chess and Shogi by Planning with a Learned ModelSchrittwieser et al. · 2020From: Model-based RL and planning
World ModelsHa & Schmidhuber · 2018The interactive version at worldmodels.github.io is a joy.From: Model-based RL and planning
When to Trust Your Model: Model-Based Policy OptimizationJanner, Fu, Zhang & Levine · 2019From: Model-based RL and planning
Unifying Count-Based Exploration and Intrinsic MotivationBellemare, Srinivasan, Ostrovski, Schaul, Saxton & Munos · 2016From: Exploration
Exploration by Random Network DistillationBurda, Edwards, Storkey & Klimov · 2018From: Exploration
First return, then exploreEcoffet, Huizinga, Lehman, Stanley & Clune · 2021Preprint of the Nature paper (Nature 590, 580–586).From: Exploration
Deep Exploration via Bootstrapped DQNOsband, Blundell, Pritzel & Van Roy · 2016From: Exploration
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online LearningRoss, Gordon & Bagnell · 2011DAgger. AISTATS 2011.From: Imitation and offline RL
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open ProblemsLevine, Kumar, Tucker & Fu · 2020From: Imitation and offline RL
Off-Policy Deep Reinforcement Learning without ExplorationFujimoto, Meger & Precup · 2019From: Imitation and offline RL
Conservative Q-Learning for Offline Reinforcement LearningKumar, Zhou, Tucker & Levine · 2020From: Imitation and offline RL
Probabilistic Machine Learning: Advanced Topics, §35.6.3 (imitation learning)Kevin P. Murphy · 2023From: Imitation and offline RL
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and AlgorithmsZhang, Yang & Başar · 2021From: Multi-agent RL
Grandmaster level in StarCraft II using multi-agent reinforcement learningVinyals et al. · 2019From: Multi-agent RL
Dota 2 with Large Scale Deep Reinforcement LearningBerner et al. (OpenAI) · 2019From: Multi-agent RL
Emergent Tool Use From Multi-Agent AutocurriculaBaker et al. · 2020From: Multi-agent RL
Cycles in Adversarial Regularized LearningMertikopoulos, Papadimitriou & Piliouras · 2018From: Multi-agent RL

XVIIIBuilding with LLMs

LoRA: Low-Rank Adaptation of Large Language ModelsEdward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang & Weizhu Chen · 2021ICLR 2022. Sections 4 and 7 are the heart of it.From: Fine-tuning and LoRA
QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman & Luke Zettlemoyer · 2023From: Fine-tuning and LoRA
Parameter-Efficient Transfer Learning for NLPNeil Houlsby, Andrei Giurgiu, Stanisław Jastrzębski et al. · 2019Adapters (ICML 2019).From: Fine-tuning and LoRA
LoRA Learns Less and Forgets LessDan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz et al. · 2024The most careful LoRA-versus-full-fine-tuning comparison (TMLR).From: Fine-tuning and LoRA
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-TuningArmen Aghajanyan, Luke Zettlemoyer & Sonal Gupta · 2020From: Fine-tuning and LoRA
PEFT (Hugging Face library)2026The reference implementation most people use.From: Fine-tuning and LoRA
LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleTim Dettmers, Mike Lewis, Younes Belkada & Luke Zettlemoyer · 2022From: Quantization and distillation
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth & Song Han · 2022ICML 2023.From: Quantization and distillation
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersElias Frantar, Saleh Ashkboos, Torsten Hoefler & Dan Alistarh · 2022ICLR 2023.From: Quantization and distillation
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and AccelerationJi Lin, Jiaming Tang, Haotian Tang, Shang Yang et al. · 2023MLSys 2024 best paper.From: Quantization and distillation
Distilling the Knowledge in a Neural NetworkGeoffrey Hinton, Oriol Vinyals & Jeff Dean · 2015From: Quantization and distillation
Large Transformer Model Inference OptimizationLilian Weng · 2023A readable survey of quantization, pruning and distillation.From: Quantization and distillation
Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus et al. · 2020NeurIPS 2020. The paper that named RAG.From: Embeddings, retrieval and RAG
Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oğuz, Sewon Min et al. · 2020EMNLP 2020.From: Embeddings, retrieval and RAG
Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphsYu. A. Malkov & D. A. Yashunin · 2016HNSW; published in IEEE TPAMI.From: Embeddings, retrieval and RAG
Lost in the Middle: How Language Models Use Long ContextsNelson F. Liu, Kevin Lin, John Hewitt et al. · 2023From: Embeddings, retrieval and RAG
The Faiss libraryMatthijs Douze, Alexandr Guzhva, Chengqi Deng et al. · 2024A clear tour of IVF, product quantization and graph indexes by the library's authors.From: Embeddings, retrieval and RAG
How to Build an Open-Domain Question Answering System?Lilian Weng · 2020From: Embeddings, retrieval and RAG
ReAct: Synergizing Reasoning and Acting in Language ModelsShunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan & Yuan Cao · 2022ICLR 2023.From: Agents and tool use
Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì et al. · 2023From: Agents and tool use
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?Carlos E. Jimenez, John Yang, Alexander Wettig et al. · 2023ICLR 2024.From: Agents and tool use
Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt InjectionKai Greshake, Sahar Abdelnabi, Shailesh Mishra et al. · 2023From: Agents and tool use
Building effective agentsErik Schluntz & Barry Zhang (Anthropic) · 2024Practical patterns; the page itself notes that tooling has changed since.From: Agents and tool use
LLM Powered Autonomous AgentsLilian Weng · 2023From: Agents and tool use
Harness Engineering for Self-ImprovementLilian Weng · 2026From: Agents and tool use
Adding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsEvan Miller · 2024Short, practical, and the source of most of this lesson's statistics advice.From: Evaluating language models
Lessons from the Trenches on Reproducible Evaluation of Language ModelsStella Biderman, Hailey Schoelkopf, Lintang Sutawika et al. · 2024From the developers of the lm-eval harness.From: Evaluating language models
Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaLianmin Zheng, Wei-Lin Chiang, Ying Sheng et al. · 2023NeurIPS 2023 Datasets and Benchmarks.From: Evaluating language models
Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart et al. · 2020MMLU (ICLR 2021).From: Evaluating language models
Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceWei-Lin Chiang, Lianmin Zheng, Ying Sheng et al. · 2024From: Evaluating language models
The Leaderboard IllusionShivalika Singh, Yiyang Nan, Alex Wang et al. · 2025A critique of how arena rankings can be gamed.From: Evaluating language models
How To Scale Your ModelJacob Austin, Sholto Douglas, Roy Frostig et al. (Google DeepMind) · 2025The best single guide to the arithmetic of parallelism; source of this lesson's hardware figures.From: Training at scale
ZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase & Yuxiong He · 2019From: Training at scale
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper & Bryan Catanzaro · 2019From: Training at scale
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMDeepak Narayanan, Mohammad Shoeybi, Jared Casper et al. · 2021From: Training at scale
GPipe: Easy Scaling with Micro-Batch Pipeline ParallelismYanping Huang, Youlong Cheng, Ankur Bapna et al. · 2018From: Training at scale
The Llama 3 Herd of Models (§3.3, Infrastructure, Scaling, and Efficiency)Llama Team, AI @ Meta · 2024An unusually detailed first-hand account of a 16,000-GPU run.From: Training at scale

XIXUnder the Hood

The N-dimensional array (ndarray) — NumPy referenceNumPy developersThe official description of the memory layout and the strided indexing formula used in this lesson.From: Arrays: NumPy and SciPy
Broadcasting — NumPy user guideNumPy developersThe rule, stated exactly, with pictures.From: Arrays: NumPy and SciPy
Copies and views — NumPy user guideNumPy developersFrom: Arrays: NumPy and SciPy
Array programming with NumPyHarris, Millman, van der Walt, Gommers et al. · 2020The NumPy paper in Nature; history from Numeric and Numarray to NumPy.From: Arrays: NumPy and SciPy
SciPy 1.0: fundamental algorithms for scientific computing in PythonVirtanen et al. · 2020From: Arrays: NumPy and SciPy
Python Data Science Handbook, ch. 2 (NumPy)Jake VanderPlas · 2016Why loops are slow and ufuncs are fast, with timings.From: Arrays: NumPy and SciPy
micrograd — a tiny scalar-valued autograd engineAndrej KarpathyThe inspiration for this lesson's engine. MIT licensed; read engine.py after this lesson and compare.From: Autodiff from scratch
The Autodiff Cookbook (JAX documentation)Alex Wiltschko, Matthew JohnsonJVPs, VJPs, and why reverse mode wins for loss functions — with costs.From: Autodiff from scratch
Autograd tutorial: what is going on under the hoodHIPS Autograd authorsA short, clear account of recording a graph and sweeping it in reverse.From: Autodiff from scratch
Forward-mode automatic differentiation (PyTorch tutorial)PyTorchDual tensors in a production framework.From: Autodiff from scratch
Autograd mechanics (PyTorch notes)PyTorchThe official account of the recorded graph, saved tensors, requires_grad and no-grad mode.From: PyTorch internals
PyTorch internalsEdward Z. Yang · 2019The classic long-form tour of tensors, strides, dispatch and autograd, linked from PyTorch's own Tensor Views documentation.From: PyTorch internals
Extending dispatcher for a new backend / The dispatcher (tutorial)PyTorchFrom: PyTorch internals
PyTorch dispatcher walkthrough (PyTorch wiki)PyTorch developers, adapted from Edward Z. YangFrom: PyTorch internals
PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph CompilationAnsel et al. · 2024The paper behind torch.compile (ASPLOS 2024); PyTorch's preferred citation.From: PyTorch internals
Key concepts (JAX documentation)JAX authorsTransformations, tracers, jaxprs and pytrees in one page.From: JAX: tracing and transformations
Just-in-time compilation (JAX documentation)JAX authorsFrom: JAX: tracing and transformations
🔪 JAX — The Sharp Bits 🔪JAX authorsPurity, in-place updates, randomness and control flow, with examples.From: JAX: tracing and transformations
Autodidax: JAX core from scratchJAX authorsBuilds jvp, vmap, jaxprs, jit and vjp as interpreters in pure Python. The best next step after this lesson.From: JAX: tracing and transformations
Compiling machine learning programs via high-level tracingFrostig, Johnson & Leary · 2018The SysML 2018 paper describing the first version of JAX (autodiff plus compilation to XLA).From: JAX: tracing and transformations
About Keras 3Keras teamThe design principles in the Keras team's own words.From: Keras, and choosing a framework
Introducing Keras 3.0Keras team · 2023The multi-backend rewrite, keras.ops, and the stateless API.From: Keras, and choosing a framework
The Functional API (Keras guide)François CholletFrom: Keras, and choosing a framework
Keras source: the JAX backend trainerKeras teamfit() on JAX, in about a screen of code — compare with the JAX lesson.From: Keras, and choosing a framework

XXStrange Phenomena

Benefits of depth in neural networksMatus Telgarsky · 2016The depth-separation theorem (COLT 2016).From: Width versus depth
The Power of Depth for Feedforward Neural NetworksRonen Eldan & Ohad Shamir · 2016One extra layer can be worth exponentially many units.From: Width versus depth
Neural Tangent Kernel: Convergence and Generalization in Neural NetworksArthur Jacot, Franck Gabriel & Clément Hongler · 2018From: Width versus depth
On Lazy Training in Differentiable ProgrammingLénaïc Chizat, Edouard Oyallon & Francis Bach · 2019From: Width versus depth
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter TransferGreg Yang, Edward Hu et al. · 2021From: Width versus depth
Finite Depth and Width Corrections to the Neural Tangent KernelBoris Hanin & Mihai Nica · 2020ICLR 2020. Why the NTK picture needs depth ≪ width.From: Width versus depth
Infinite Limits of Neural Networks (Kempner Institute overview)Kempner Institute, Harvard · 2024From: Width versus depth
Reconciling modern machine-learning practice and the classical bias–variance trade-offMikhail Belkin, Daniel Hsu, Siyuan Ma & Soumik Mandal · 2019PNAS 116(32). Named the phenomenon.From: Double descent
Deep Double Descent: Where Bigger Models and More Data HurtPreetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak & Ilya Sutskever · 2019From: Double descent
Surprises in High-Dimensional Ridgeless Least Squares InterpolationTrevor Hastie, Andrea Montanari, Saharon Rosset & Ryan J. Tibshirani · 2022Annals of Statistics 50(2). The exact theory for linear and random-features models.From: Double descent
Double Descent DemystifiedRylan Schaeffer et al. · 2023The SVD explanation used in this lesson, with ablations.From: Double descent
A U-turn on Double Descent: Rethinking Parameter Counting in Statistical LearningAlicia Curth, Alan Jeffares & Mihaela van der Schaar · 2023NeurIPS 2023. The main critique.From: Double descent
Grokking: Generalization Beyond Overfitting on Small Algorithmic DatasetsAlethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin & Vedant Misra · 2022From: Grokking
Progress measures for grokking via mechanistic interpretabilityNeel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith & Jacob Steinhardt · 2023ICLR 2023. The reverse-engineered algorithm and the three phases.From: Grokking
Do Machine Learning Models Memorize or Generalize?Adam Pearce, Asma Ghandeharioun, Nada Hussein, Nithum Thain, Martin Wattenberg & Lucas Dixon (Google PAIR) · 2023An interactive explainer with a five-neuron construction.From: Grokking
Explaining grokking through circuit efficiencyVikrant Varma et al. (DeepMind) · 2023From: Grokking
Omnigrok: Grokking Beyond Algorithmic DataZiming Liu, Eric J. Michaud & Max Tegmark · 2023From: Grokking
Scaling Laws for Neural Language ModelsJared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown et al. · 2020From: Scaling laws and their limits
Training Compute-Optimal Large Language ModelsJordan Hoffmann, Sebastian Borgeaud, Arthur Mensch et al. · 2022The Chinchilla paper (NeurIPS 2022).From: Scaling laws and their limits
Scaling Laws, CarefullyLilian Weng · 2026A careful review, including why the fits disagree.From: Scaling laws and their limits
Chinchilla Scaling: A replication attemptTamay Besiroglu, Ege Erdil et al. (Epoch AI) · 2024From: Scaling laws and their limits
Are Emergent Abilities of Large Language Models a Mirage?Rylan Schaeffer, Brando Miranda & Sanmi Koyejo · 2023From: Scaling laws and their limits
Will we run out of data? Limits of LLM scaling based on human-generated dataPablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim & Marius Hobbhahn · 2024From: Scaling laws and their limits
Understanding deep learning requires rethinking generalizationChiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht & Oriol Vinyals · 2017From: The open questions
Toy Models of SuperpositionNelson Elhage et al. (Anthropic) · 2022From: The open questions
Towards Monosemanticity: Decomposing Language Models With Dictionary LearningTrenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn et al. (Anthropic) · 2023From: The open questions
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 SonnetAdly Templeton, Tom Conerly et al. (Anthropic) · 2024From: The open questions
In-context Learning and Induction HeadsCatherine Olsson et al. (Anthropic) · 2022From: The open questions
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNsTimur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov & Andrew Gordon Wilson · 2018From: The open questions
Why Language Models HallucinateAdam Tauman Kalai, Ofir Nachum, Santosh S. Vempala & Edwin Zhang · 2025From: The open questions
Concrete Problems in AI SafetyDario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman & Dan Mané · 2016From: The open questions

XXICause and Effect

Causal Inference: What If, Part I (chapters 1–3)Miguel A. Hernán & James M. Robins · 2020Free. Potential outcomes, randomised experiments and observational studies, told slowly and carefully.From: Why correlation isn’t enough
Elements of Causal Inference: Foundations and Learning AlgorithmsJonas Peters, Dominik Janzing & Bernhard SchölkopfFree (open access). The machine-learning-minded introduction to structural causal models.From: Why correlation isn’t enough
Confounding and Simpson’s paradoxS. A. Julious & M. A. Mullee · 1994The two-page BMJ note that turned the kidney-stone study into the textbook example.From: Why correlation isn’t enough
Causal diagrams for empirical researchJudea Pearl · 1995The Biometrika paper (with discussion) that brought causal diagrams to statisticians.From: Causal graphs and d-separation
Causal Inference for the Brave and True, ch. 4 (Graphical Causal Models)Matheus FacureA free, code-first companion with plenty of worked graphs.From: Causal graphs and d-separation
Pearl’s calculus of intervention is completeYimin Huang & Marco Valtorta · 2006The proof that the three rules of the do-calculus are enough to identify every identifiable effect.From: Interventions and the do-operator
The central role of the propensity score in observational studies for causal effectsPaul R. Rosenbaum & Donald B. Rubin · 1983The paper that introduced the propensity score.From: Potential outcomes and treatment effects
Double/debiased machine learning for treatment and structural parametersChernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey & Robins · 2018The paper behind double ML. The DoubleML package documentation is the gentlest companion.From: Causal discovery and causal ML
Invariant Risk MinimizationMartin Arjovsky, Léon Bottou, Ishaan Gulrajani & David Lopez-Paz · 2019Read with Rosenfeld et al. (2021) and Kamath et al. (2021) for the limits.From: Causal discovery and causal ML

XXIITrust

Axiomatic Attribution for Deep NetworksMukund Sundararajan, Ankur Taly & Qiqi Yan · 2017Integrated gradients. The authors' how-to guide is at github.com/ankurtaly/Integrated-Gradients.From: Interpretability
Interpretable Machine Learning (book)Christoph Molnar · 2025Clear chapters on LIME, Shapley values, SHAP and saliency maps, including their failures.From: Interpretability
Sanity Checks for Saliency MapsJulius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt & Been Kim · 2018From: Interpretability
Probing Classifiers: Promises, Shortcomings, and AdvancesYonatan Belinkov · 2022From: Interpretability
Zoom In: An Introduction to CircuitsChris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov & Shan Carter · 2020From: Interpretability
In-context Learning and Induction HeadsCatherine Olsson et al. (Anthropic) · 2022From: Interpretability
Intriguing properties of neural networksChristian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow & Rob Fergus · 2013From: Adversarial examples and robustness
Explaining and Harnessing Adversarial ExamplesIan Goodfellow, Jonathon Shlens & Christian Szegedy · 2015From: Adversarial examples and robustness
Towards Deep Learning Models Resistant to Adversarial AttacksAleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras & Adrian Vladu · 2018From: Adversarial examples and robustness
Certified Adversarial Robustness via Randomized SmoothingJeremy Cohen, Elan Rosenfeld & Zico Kolter · 2019From: Adversarial examples and robustness
Adversarial Attacks on LLMsLilian Weng · 2023From: Adversarial examples and robustness
Probabilistic Machine Learning: Advanced Topics, §19.8Kevin Murphy · 2023From: Adversarial examples and robustness
Fairness and Machine Learning: Limitations and OpportunitiesSolon Barocas, Moritz Hardt & Arvind Narayanan · 2022The standard textbook; chapter 3 ("Classification") backs most of this lesson.From: Fairness
Equality of Opportunity in Supervised LearningMoritz Hardt, Eric Price & Nathan Srebro · 2016From: Fairness
Inherent Trade-Offs in the Fair Determination of Risk ScoresJon Kleinberg, Sendhil Mullainathan & Manish Raghavan · 2016From: Fairness
Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentsAlexandra Chouldechova · 2016From: Fairness
Fairness Through AwarenessCynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold & Richard Zemel · 2012From: Fairness
How to deploy machine learning with differential privacy?Nicolas Papernot & Abhradeep Thakurta (DifferentialPrivacy.org) · 2021From: Privacy and differential privacy
Deep Learning with Differential PrivacyMartín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar & Li Zhang · 2016From: Privacy and differential privacy
Extracting Training Data from Large Language ModelsNicholas Carlini et al. · 2021From: Privacy and differential privacy
Membership Inference Attacks against Machine Learning ModelsReza Shokri, Marco Stronati, Congzheng Song & Vitaly Shmatikov · 2017From: Privacy and differential privacy
Communication-Efficient Learning of Deep Networks from Decentralized DataH. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson & Blaise Agüera y Arcas · 2017From: Privacy and differential privacy
Simple and Scalable Predictive Uncertainty Estimation using Deep EnsemblesBalaji Lakshminarayanan, Alexander Pritzel & Charles Blundell · 2017From: Uncertainty in deep learning
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?Alex Kendall & Yarin Gal · 2017From: Uncertainty in deep learning
Dropout as a Bayesian ApproximationYarin Gal & Zoubin Ghahramani · 2016From: Uncertainty in deep learning
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset ShiftYaniv Ovadia et al. · 2019From: Uncertainty in deep learning
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty QuantificationAnastasios N. Angelopoulos & Stephen Bates · 2021From: Uncertainty in deep learning
Specification gaming: the flip side of AI ingenuityVictoria Krakovna et al. (DeepMind) · 2020From: Alignment and safety
Concrete Problems in AI SafetyDario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman & Dan Mané · 2016From: Alignment and safety
Reward Hacking in Reinforcement LearningLilian Weng · 2024From: Alignment and safety
Risks from Learned Optimization in Advanced Machine Learning SystemsEvan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse & Scott Garrabrant · 2019From: Alignment and safety
AI safety via debateGeoffrey Irving, Paul Christiano & Dario Amodei · 2018From: Alignment and safety
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak SupervisionCollin Burns et al. · 2024From: Alignment and safety
AI as Normal TechnologyArvind Narayanan & Sayash Kapoor · 2025From: Alignment and safety

XXIIIThe Summit

How to Read a PaperS. Keshav · 2007Two pages, ACM SIGCOMM Computer Communication Review 37(3). The source of the three-pass method.From: How to read an ML paper
Troubling Trends in Machine Learning ScholarshipZachary C. Lipton & Jacob Steinhardt · 2018Four patterns to watch for when you read — and to avoid when you write.From: How to read an ML paper
NeurIPS Paper ChecklistNeurIPSWhat reviewers are asked to look for. An excellent checklist for readers too.From: How to read an ML paper
A Recipe for Training Neural NetworksAndrej Karpathy · 2019Why reproducing results is harder than it looks, from someone who does it a lot.From: How to read an ML paper
Attention Is All You NeedVaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser & Polosukhin · 2017The paper itself. Read it alongside this lesson; the version history is on the same page.From: Worked reading: Attention Is All You Need
The Annotated TransformerRush (original); Huang, Subramanian, Sum, Almubarak & Biderman (2022 update)The paper's text interleaved with a working PyTorch implementation, line by line.From: Worked reading: Attention Is All You Need
tensor2tensorGoogleThe code base the paper's conclusion names as the one used for its experiments.From: Worked reading: Attention Is All You Need
An Opinionated Guide to ML ResearchJohn Schulman · 2020How to choose problems, organise your time, and keep growing — from a researcher who has advised many others.From: Where to go next
Research DebtChris Olah & Shan Carter · 2017Why good explanation is research, not a lesser activity.From: Where to go next
You and Your ResearchRichard HammingA classic talk on doing important work, recommended by Schulman.From: Where to go next
Principles of Effective ResearchMichael NielsenAlso recommended by Schulman; a companion to Hamming.From: Where to go next