From 1763 to now

Murphy dates “Bayes’ original method” to 1763; Bayes (1702–1761) was an English mathematician and Presbyterian minister. Bayes’ rule turns a likelihood into a posterior.

In a book on determining the orbits of comets, Legendre introduces the method of least squares and gives it its name.

In Theoria motus corporum coelestium Gauss presents the method and says he had used it since 1795, starting a famous priority dispute with Legendre.

Verhulst models a population whose growth levels off, introducing the S-shaped curve he later calls logistic. It was largely ignored until Pearl and Reed rediscovered it in 1920.

A three-page note in the Comptes Rendus, “Méthode générale pour la résolution des systèmes d'équations simultanées”, to which the idea of stepping along the negative gradient is usually dated.

Modelled Walter Weldon's lopsided shore-crab measurements as two mixed normal curves, fitted by the method of moments (Phil. Trans. R. Soc. A).

Published in the Philosophical Magazine: the origin of principal component analysis, posed as finding the line or plane with the smallest squared distance to the data.

In a lecture on 23 January 1913 Markov classified 20,000 letters of Pushkin's novel in verse as vowels or consonants and showed that each letter's type depends on the previous one: after a vowel, a vowel followed with probability about 0.128; after a consonant, about 0.663.

Analysing field experiments, Neyman describes each plot by the yields it would give under every variety, only one of which is observed; an English translation appeared in Statistical Science in 1990.

To separate supply from demand for butter, Wright looks for a variable that shifts one curve but not the other: the first instrumental-variable estimate.

A Biometrika paper on deciding which of two unknown probabilities is larger proposes the allocation rule now called Thompson sampling; Lattimore and Szepesvári call it the first paper on bandits.

“Analysis of a Complex of Statistical Variables into Principal Components”, Journal of Educational Psychology 24, 417–441.

Non-negativity, normalisation and countable additivity: the axioms on which modern probability is built.

Randomisation, replication and blocking, illustrated by the lady tasting tea: eight cups in random order, a 1 in 70 chance of guessing all correctly.

“The Approximation of One Matrix by Another of Lower Rank”, Psychometrika 1(3), 211–218. Truncating the SVD gives the closest rank-$k$ matrix.

An unpublished University of Chicago thesis, "Minima of functions of several variables with inequalities as side constraints", derives the optimality conditions later named after Karush, Kuhn and Tucker.

“A Logical Calculus of the Ideas Immanent in Nervous Activity” shows that networks of threshold units can compute logical functions.

“Application of the Logistic Function to Bio-Assay” (JASA) proposes the logistic function for dose–response data and names its inverse the logit, by analogy with Bliss's probit.

"A method for the solution of certain non-linear problems in least squares" (Quarterly of Applied Mathematics) adds damping to Gauss–Newton steps; Marquardt's 1963 paper completes what is now called Levenberg–Marquardt.

Shows that two unrelated diseases can appear associated in hospital data, because admission depends on both: selection on a common effect.

A note in Psychometrika on the sampling error of the difference between correlated proportions gives the test now used to compare two classifiers on the same test set.

Published in two parts (July and October) in the Bell System Technical Journal. It defines entropy, introduces the word 'bit' (credited to J. W. Tukey), proves the limits of compression and of reliable communication, and generates fake English from letter and word statistics.

The Organization of Behavior states a postulate about learning in neural connections — an idea, not yet an algorithm.

In Monthly Weather Review, Brier scores probabilistic weather forecasts by squared error, the Brier score, a proper scoring rule still used to evaluate classifiers.

People guess the next letter of a passage; from their guess counts Shannon bounds the entropy of literary English at roughly 0.6 to 1.3 bits per letter with 100 letters of context, 'of the order of one bit per letter'.

A short paper on interpreting interactions in contingency tables; the reversal it describes was named Simpson's paradox by Colin Blyth in 1972.

A US Air Force School of Aviation Medicine technical report introduces a nonparametric rule for classifying a new observation from its nearest labelled neighbours, and proves it consistent.

'On Information and Sufficiency' (Annals of Mathematical Statistics 22) defines the mean information for discrimination between two distributions, now called the KL divergence.

“A Stochastic Approximation Method” (Annals of Mathematical Statistics) finds roots from noisy measurements by iteration: the ancestor of stochastic gradient descent.

At the Second Berkeley Symposium on Mathematical Statistics and Probability they give optimality conditions for inequality-constrained problems: the KKT conditions behind constrained ML models such as the support vector machine.

"Some aspects of the sequential design of experiments" formally restates the multi-armed bandit problem and introduces the notion of regret.

'A Method for the Construction of Minimum-Redundancy Codes' (Proceedings of the IRE) gives the optimal prefix code by repeatedly merging the two least likely symbols. It began as a term paper in Robert Fano's course.

“Equation of State Calculations by Fast Computing Machines”, Journal of Chemical Physics 21. A random walk that accepts or rejects moves by a density ratio, run on the Los Alamos MANIAC: the first MCMC method.

“Remarks on some nonparametric estimates of a density function” (Annals of Mathematical Statistics 27), one of the two founding papers of kernel density estimation.

Dynamic programming for sequential decisions — the source of the Bellman equations behind value-based reinforcement learning.

A Bell Labs technical report on least-squares quantization for pulse-code modulation, describing the alternating algorithm now called k-means; published in IEEE Transactions on Information Theory in 1982.

A JRSS-B paper on sequences of 0s and 1s whose chance of being 1 depends on explanatory variables — the regression setting of logistic regression.

Published in Psychological Review after a 1957 Cornell Aeronautical Laboratory report; the Navy's press demonstration drew extravagant headlines.

“Receptive fields of single neurones in the cat's striate cortex”, Journal of Physiology 148. Part of the work honoured by the 1981 Nobel Prize.

“A New Approach to Linear Filtering and Prediction Problems”, Journal of Basic Engineering 82(1). Exact Bayesian tracking of a hidden state in linear-Gaussian systems, soon adapted for Apollo navigation at NASA Ames.

“Adaptive switching circuits” trains a linear unit by gradient descent on squared error.

Journal of Physiology 160: cells tuned to an oriented bar at one position (simple) and anywhere in their field (complex), proposed as successive stages of a hierarchy.

If a separating hyperplane exists, the perceptron rule makes finitely many mistakes before finding one.

“On estimation of a probability density function and mode” (Annals of Mathematical Statistics 33); KDE is also called the Parzen window estimator.

In the Journal of the American Statistical Association, Automatic Interaction Detection splits survey data recursively to predict a numeric response.

Exponential bounds on how far an average of bounded independent variables strays from its mean — the concentration tool behind finite-sample learning guarantees.

"An algorithm for least-squares estimation of nonlinear parameters" (Journal of SIAM) — the other half of the Levenberg–Marquardt method.

“Hierarchical grouping to optimize an objective function” (JASA 58): merge the pair of groups that loses least under an objective.

“Some methods of speeding up the convergence of iteration methods” adds momentum to gradient descent.

They prove that with unlimited data the 1-nearest-neighbour error rate is at most twice the Bayes error rate.

Published in IEEE Transactions on Information Theory as a decoder for convolutional codes; the same max-product dynamic program finds the most probable hidden path in an HMM.

“Some methods for classification and analysis of multivariate observations” (Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability) introduced the term k-means.

Theorems on parity and connectedness for single-layer perceptrons; only an “intuitive judgment” about multilayer ones.

Adding small positive quantities to the diagonal of XᵀX gives biased but stable estimates for strongly correlated predictors; the ridge trace shows the effect on every coefficient.

A master's thesis on accumulated rounding error contains the algorithm that backpropagation is a special case of.

Four separately published 1970 papers arrive at the same rank-two update of an approximate inverse Hessian, still the default quasi-Newton method.

“Monte Carlo Sampling Methods Using Markov Chains and Their Applications”, Biometrika 57(1): the correction for asymmetric proposals, giving Metropolis–Hastings.

Published in The Annals of Mathematical Statistics 41; the method now called Baum–Welch, an instance of EM for hidden Markov models.

The full paper (announced in 1968) on when frequencies converge to probabilities uniformly over a class of events; its combinatorial dimension became the VC dimension.

Normal, binomial, Poisson and gamma regressions become one model, fitted by iterative weighted least squares, with a generalisation of the analysis of variance based on log-likelihoods.

Harvard PhD thesis often credited with backpropagation for neural networks; how far it went is disputed, and his 1982 paper applied the method to them explicitly.

Rubin's paper in the Journal of Educational Psychology defines causal effects as comparisons of potential outcomes and argues that carefully controlled non-randomised data can estimate them.

“Maximum likelihood from incomplete data via the EM algorithm” (JRSS B 39) gave the general form, the monotone-likelihood property and examples including finite mixtures.

'Bootstrap Methods: Another Look at the Jackknife' (Annals of Statistics) estimates sampling variability by resampling the data with replacement, making error bars available for almost any statistic.

Alternating S-cell and C-cell layers give shift-tolerant pattern recognition, learned without a teacher (Biological Cybernetics 36).

In Radiology they show the area under an ROC curve is the probability that a random diseased case is rated above a random healthy one, the quantity the Wilcoxon statistic estimates.

“Reverend Bayes on Inference Engines: A Distributed Hierarchical Approach”, AAAI-82: updating beliefs by local, asynchronous message passing, the seed of belief propagation.

Adjusting for the probability of treatment given the covariates is sufficient to remove bias from all observed covariates (Biometrika).

"A method of solving a convex programming problem with convergence rate O(1/k²)" (Soviet Mathematics Doklady): one gradient per step, and a rate that matches the lower bound for first-order methods.

"Problem complexity and method efficiency in optimization" (Wiley): lower bounds for first-order methods, and mirror descent.

Classification and Regression Trees sets out binary recursive partitioning, impurity-based splitting and cost-complexity pruning.

“Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images”, IEEE PAMI 6(6): Gibbs sampling, invented independently in the context of Ising-type models.

Introduces probably approximately correct (PAC) learning in Communications of the ACM; Valiant received the 2010 Turing Award for it.

"Asymptotically efficient adaptive allocation rules": consistent bandit policies must have regret growing at least logarithmically, with a constant set by KL divergences between arms.

A BMJ comparison of open surgery and percutaneous nephrolithotomy whose stone-size split became the textbook example of Simpson's paradox.

Induction of Decision Trees, in the first volume of the journal Machine Learning, chooses each question to maximise the entropy reduction.

“Learning representations by back-propagating errors” (Nature) shows hidden units learning useful features.

“Statistics and Causal Inference” (JASA) names Rubin's model and the impossibility of observing both potential outcomes for one unit.

“Hybrid Monte Carlo”, Physics Letters B 195(2): simulated Hamiltonian dynamics plus a Metropolis test. Introduced to statisticians by Radford Neal in 1993.

“Learning to predict by the methods of temporal differences” (Machine Learning 3): learn from the difference between successive predictions.

Pearl's book on networks of plausible inference, with belief propagation for trees and the warning that messages on loopy graphs may not converge.

Handwritten ZIP-code digits from the U.S. Postal Service, with task knowledge built into the architecture (Neural Computation 1(4)).

One hidden layer of sigmoidal units can approximate any continuous function on a cube — given enough units.

"On the limited memory BFGS method for large scale optimization" (Mathematical Programming), building on Nocedal's 1980 limited-storage updates: quasi-Newton steps from a few stored vector pairs.

Introduced in his Cambridge PhD thesis, Learning from Delayed Rewards; Watkins and Dayan proved convergence in 1992.

“A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition”, Proceedings of the IEEE 77(2), organised around evaluation, decoding and learning.

“Finding Structure in Time” (Cognitive Science 14): hidden units fed back through context units; its hidden states cluster words into nouns and verbs.

The Strength of Weak Learnability proves that an algorithm barely better than guessing can be combined into one with arbitrarily high accuracy.

“Sampling-based approaches to calculating marginal densities”, JASA 85; Murphy credits it with making MCMC well known to the wider statistical community.

“Integrated architectures for learning, planning, and reacting based on approximating dynamic programming” (Proceedings of the 7th International Workshop on Machine Learning, 1990): the same update learns from real and model-simulated experience.

Two DAGs imply the same conditional independences if and only if they share their skeleton and their v-structures (UAI 1990).

Diploma thesis at TU Munich, written in German: error signals flowing back through time shrink or blow up.

Recovers a sparse causal graph's equivalence class from conditional-independence tests (Social Science Computer Review).

A preprint server where many ML papers appear first. Its moderation checks relevance and scholarly standards but is explicitly not peer review.

A word-trigram model trained on 583 million words has a cross-entropy of 1.75 bits per character on the Brown Corpus, presented as an upper bound on the entropy of printed English.

“Neural networks and the bias-variance dilemma” frames the trade-off between fitting the training data and generalising.

“Simple statistical gradient-following algorithms for connectionist reinforcement learning” (Machine Learning 8): the likelihood-ratio policy gradient.

“Bayesian Interpolation”, Neural Computation 4(3): choosing regularisers and models by their evidence, with Occam's razor falling out automatically.

“Inference from Iterative Simulation Using Multiple Sequences”, Statistical Science 7(4): compare several chains from overdispersed starts; the origin of R-hat.

At COLT '92 they combine the maximum-margin hyperplane with kernel functions, expressing the solution through the training points closest to the boundary.

IEEE Transactions on Neural Networks 5(2): gradient-based learning gets harder as the dependencies to be captured get longer.

Multiplicative weights for prediction with expert advice, published in Information and Computation (with Vovk's 1990 work, the start of exponential weighting in online learning).

A neural network trained by temporal-difference learning on games against itself (Neural Computation 6).

“Causal diagrams for empirical research” (Biometrika) introduces the do-calculus for deciding when causal effects can be computed from observational data.

The first practical boosting algorithm reweights the training examples each round so that the next weak learner focuses on the mistakes (journal version 1997).

The soft-margin SVM: a maximum-margin linear classifier in a high-dimensional feature space that tolerates non-separable data through slack variables.

Bootstrap aggregating averages predictors trained on bootstrap resamples; the key, Breiman writes, is the instability of the base method.

Averaged uniformly over all targets, every learning algorithm has the same off-training-set error; for any two algorithms there are as many targets favouring one as the other.

“Regression Shrinkage and Selection via the Lasso” (JRSS-B): bound the sum of absolute coefficients and some become exactly zero.

In Bayesian Learning for Neural Networks, a one-hidden-layer network with random weights becomes a Gaussian process as its width goes to infinity.

Density-based clustering of arbitrary shapes with explicit noise (KDD-96); SIGKDD Test of Time Award in 2014.

Neural Computation 9(8): a gated memory cell with a “constant error carrousel” bridges time lags of more than 1,000 steps.

"Natural gradient works efficiently in learning" (Neural Computation) measures steps by the Fisher information of the model instead of Euclidean distance in parameter space.

Proceedings of the IEEE 86(11): seven layers, about 60,000 trainable parameters, 0.95% test error on MNIST.

Neural Computation paper measuring the type I error of five tests; finds McNemar's test has low type I error, proposes the 5x2 cross-validation test, and warns that two widely used tests should never be used.

The first PAC-Bayes bounds: generalization guarantees for a posterior over hypotheses, paying the KL divergence from a prior fixed before seeing data.

“Learning to Forget” (Neural Computation 12(10)) lets an LSTM cell learn to reset itself on continual input streams.

Weighting the log-likelihood by the ratio of input densities improves prediction when the test inputs follow a different distribution from the training inputs.

“Normalized cuts and image segmentation” (IEEE TPAMI 22): graph partitioning solved through a generalised eigenvalue problem.

“A global geometric framework for nonlinear dimensionality reduction” (Science 290): geodesic distances from a neighbourhood graph, then MDS.

“LOF: identifying density-based local outliers” (SIGMOD): outlierness relative to the neighbourhood's density.

Bagged trees that also choose each split from a random subset of features, with out-of-bag error estimates and permutation importance.

Greedy Function Approximation recasts boosting as gradient descent in function space, fitting each tree to the negative gradient of any differentiable loss.

Per the SciPy 1.0 paper (Virtanen et al., 2020), SciPy's initial release was in 2001; it went on to become a de facto standard for scientific algorithms in Python.

“On spectral clustering: analysis and an algorithm” (NIPS 14): normalised Laplacian eigenvectors, row normalisation, then k-means.

A two-phase greedy search over equivalence classes that, in the large-sample limit, identifies the generating DAG's equivalence class (JMLR).

Simple upper-confidence-bound index policies with logarithmic regret at every finite horizon (Machine Learning 47).

Scores a translation by its n-gram overlap with human reference translations; the metric the transformer paper reports.

Learned word feature vectors plus a neural network predict the next word, fighting the curse of dimensionality of n-gram tables. Test perplexity about 24% lower than the best n-gram on the Brown corpus (JMLR 3).

Online gradient descent has regret growing like the square root of the number of rounds for any sequence of convex losses with bounded gradients on a bounded set (ICML 2003).

The Cambridge University Press textbook, freely available from the authors, that became the standard reference for convexity, duality and Newton's method.

An ICML study finds max-margin methods distort probabilities in a sigmoid shape while the neural nets of the time were well calibrated, and compares Platt scaling with isotonic regression.

Per the NumPy paper in Nature (Harris et al., 2020), NumPy emerged in 2005 as a 'best of both worlds' unification of the Numeric and Numarray array packages.

“Estimation of Non-Normalized Statistical Models by Score Matching” (JMLR 6) fits a model by matching the gradient of its log-density, which never involves the intractable normalising constant.

“Calibrating noise to sensitivity in private data analysis” (Theory of Cryptography Conference): add noise scaled to how much one record can change the answer, and an adversary learns almost the same whether or not you are in the data.

Two independent proofs that every identifiable causal effect can be derived with the three rules of the do-calculus (UAI 2006, AAAI 2006).

With linear mechanisms, no hidden confounders and non-Gaussian noise, the complete causal graph is identifiable from observational data (JMLR).

ICML 2006. Adds a blank symbol and sums over every alignment by dynamic programming, so a network can be trained on transcripts without knowing which frame produced which label.

A one-million-dollar prize for improving the Cinematch recommender by 10%. The grand prize went to BellKor's Pragmatic Chaos; matrix factorisation played a critical role in the winning blend.

Greedy layer-by-layer training of deep networks; the start of the “deep learning” wave.

The MIT Press book on GP regression, classification and kernel learning by marginal likelihood; free online.

Coulom's MCTS paper and Kocsis and Szepesvári's “Bandit based Monte-Carlo planning” (ECML 2006), which applies a bandit UCB rule at every node of the search tree.

“k-means++: the advantages of careful seeding” (SODA): D² sampling gives an expected cost within O(log k) of optimal.

Statistics and Computing 17(4): the standard introduction to graph Laplacians and spectral clustering.

Two pages in ACM SIGCOMM Computer Communication Review introducing the three-pass approach: a five-to-ten-minute survey, up to an hour for content, and a virtual re-implementation.

“Visualizing data using t-SNE” (JMLR 9): heavy-tailed similarities in the map ease the crowding problem.

ICDM 2008: isolate anomalies with random cuts instead of modelling normal data; score from the average path length.

CVPR 2009: 3.2 million images in 5,247 WordNet categories at publication; its challenge (ILSVRC) used about 1.2 million training images in 1,000 classes.

IEEE Computer. Users and items as vectors in one latent-factor space, with biases and regularisation: the model behind the Netflix Prize era.

Adds Nesterov's momentum to iterative soft-thresholding (ISTA) for smooth-plus-non-smooth objectives such as the lasso, improving the rate from O(1/k) to O(1/k²).

Theano combined a computer-algebra system with an optimizing compiler for array expressions, with symbolic differentiation; described at the SciPy 2010 conference.

Tracks activations and gradients layer by layer and proposes the variance-preserving “Xavier” initialisation.

“A connection between score matching and denoising autoencoders” (Neural Computation 23(7)) shows that learning to point noisy points back toward clean ones estimates the score of the noisy data.

Training on the learner's own states, labelled by the expert, turns behavioural cloning's worst-case error bound from quadratic to linear in the horizon, when single mistakes are recoverable (AISTATS 2011).

A variant won ILSVRC-2012 with 15.3% top-5 error against 26.2% for the second-best entry: ReLU, dropout, augmentation and two GPUs.

A taxonomy of leakage, 'information about the target that should not be legitimately available', illustrated with public data-mining competitions such as the KDD Cup 2008 patient-ID leak.

Individual fairness as a Lipschitz condition: people who are similar for the task should get similar distributions over outcomes (ITCS 2012).

'Efficient Estimation of Word Representations in Vector Space' simplifies neural language models to learn word vectors from 1.6 billion words in under a day.

“On the difficulty of training recurrent neural networks” (ICML 2013): spectral conditions for vanishing and exploding gradients, and clipping as the fix for explosion.

A control-variate gradient estimate that lets constant-step stochastic gradient methods converge linearly on finite sums (NIPS 2013).

“Density-based clustering based on hierarchical density estimates” (PAKDD): DBSCAN across all density levels.

“Playing Atari with Deep Reinforcement Learning”: a convolutional Q-network learned seven Atari 2600 games from raw pixels.

arXiv 1312.6114 (ICLR 2014): the reparameterised variational lower bound, optimised with stochastic gradients; the variational autoencoder.

“Intriguing properties of neural networks”: hardly perceptible perturbations, found by maximising the network’s error, cause misclassification — and transfer to networks trained on different data.

arXiv 1406.1078: an encoder–decoder used to score phrase pairs in a translation system, with a new gated unit (reset and update gates).

CVPR 2014. Classifies CNN features of about 2,000 region proposals per image, improving PASCAL VOC 2012 mean average precision by more than 30% relative, to 53.3%.

A generator is trained to fool a discriminator that is trained to catch it. Against a perfect discriminator the generator minimises the Jensen–Shannon divergence to the data (NIPS 2014).

Argue that in high-dimensional non-convex problems the main difficulty is the proliferation of saddle points surrounded by plateaus, and propose the saddle-free Newton method.

Score-function gradients make VI work for any model you can evaluate (AISTATS 2014); NUTS makes HMC tune its own path length (JMLR 15).

16–19 layers of 3×3 convolutions (VGG); 22-layer Inception network (GoogLeNet), which won ILSVRC-2014 with 6.67% top-5 error.

Two deep LSTMs and a reversed source reach 34.8 BLEU on WMT'14 English–French, above a phrase-based baseline (33.3).

Arguing that a fixed-length vector is a bottleneck, they let a translation decoder (soft-)search all encoder states at every step (arXiv 1409.0473; ICLR 2015).

Autograd differentiates native Python and NumPy code, including loops and branches, in reverse and forward mode. Its public repository starts in November 2014.

Adaptive estimates of the first and second moments of the gradient, with bias correction.

“Explaining and Harnessing Adversarial Examples” (ICLR 2015) blames the networks’ linearity and attacks with one signed gradient step.

Standardising each feature over the mini-batch allows much higher learning rates; why it works is still debated.

One algorithm, network and set of hyperparameters reached a level comparable to a professional games tester across 49 games.

Committed on 27 March 2015 as a 'Theano-based Deep Learning library', 'a minimalist, highly modular neural network library'.

Train a small model on a large model's softened output distribution (softmax at a raised temperature $T$), scaling the soft-target gradients by $T^2$. On MNIST a small net trained this way made 74 test errors instead of 146.

MICCAI 2015. A contracting path for context and a symmetric expanding path with skip connections for precise segmentation, trained from very few annotated images.

NeurIPS 2015. A network proposes object boxes from shared convolutional features using anchors, making proposals nearly cost-free; 5 frames per second with VGG-16.

“Deep Unsupervised Learning using Nonequilibrium Thermodynamics” (ICML 2015) introduces generative models that learn to reverse a gradual noising process.

Kronecker-factored approximate curvature (ICML 2015) approximates each layer's block of the Fisher matrix by a Kronecker product of two small matrices, making natural-gradient steps practical for networks.

EMNLP 2015: dot, general and concat scoring functions; local attention gained 5.0 BLEU over non-attentional systems on English–German.

TensorFlow's release notes list 0.5.0 as the 'Initial release of TensorFlow'; the 0.5.0 tag dates from November 2015.

y = F(x) + x fixes the degradation problem; 152 layers, and an ensemble wins ILSVRC 2015 with 3.57% top-5 error.

Policy and value networks trained with supervised learning on expert games and RL from self-play, plus tree search; described in Nature in January 2016.

Coupling layers with a scale and a shift give normalizing flows with exact likelihood, exact inversion and one-pass sampling on natural images (ICLR 2017).

CVPR 2016. Detection framed as a single regression from the image to a grid of boxes and class probabilities; 45 frames per second.

COLT 2016: with random initialization, gradient descent converges to a local minimizer almost surely, proved with the Stable Manifold Theorem.

“Why Should I Trust You?” (KDD 2016) explains any classifier’s prediction by fitting a simple model to its outputs near the input — and shows a husky/wolf classifier that learned to detect snow.

DP-SGD (CCS 2016): clip each example’s gradient, add Gaussian noise, and track the privacy cost with the moments accountant.

Letting the online network choose and the target network evaluate removes most of DQN's upward bias from the max (AAAI 2016).

Counts derived from a density model let an agent explore 15 of the 24 rooms of Montezuma's Revenge's first level (NIPS 2016).

At COLT 2016, two papers prove that some functions computed by small deep networks need exponentially many units in shallower ones.

ACL 2016: Gage's 1994 compression algorithm, repurposed to build open-vocabulary subword tokens by repeatedly merging the most frequent pair.

An interactive Distill article on how t-SNE plots mislead: cluster sizes and gaps can mean nothing.

A regularised, second-order, highly optimised gradient tree boosting system; its paper reports it was used in 17 of 29 Kaggle challenge-winning solutions published in 2015.

A generative model of raw audio, one sample at a time, built from stacks of causal dilated convolutions.

Calibration within groups and balance for both classes can hold together only with perfect prediction or equal base rates (ITCS 2017); Chouldechova derived a related identity for recidivism scores the same autumn.

“Understanding deep learning requires rethinking generalization” shows standard networks easily fit random labels, so capacity cannot explain why they generalise (ICLR 2017).

PyTorch v0.1.6 ('Beta is here') was released on 21 January 2017: NumPy-like tensors with GPU acceleration and a tape-based autograd that re-records the graph on every forward pass.

Replaces the discriminator with a Lipschitz critic that estimates the earth mover's distance, which keeps a useful gradient even when the model's and the data's distributions don't overlap (ICML 2017).

Noisy top-$k$ gating chooses a few of up to thousands of expert networks per example, making conditional computation practical at scale: a 137-billion-parameter LSTM. Communication costs and training instability remained open problems.

“Axiomatic Attribution for Deep Networks” (ICML 2017): attributions from integrating the gradient along a path from a baseline, the unique path method satisfying their axioms.

Argues that poor exposition and bad notation are a cost every newcomer pays, and that distilling ideas is research work.

ICLR 2017. The propagation rule σ(D̃^−1/2 (A+I) D̃^−1/2 H W): message passing with self-loops and symmetric degree normalisation, for semi-supervised node classification.

ICML 2017. Unifies earlier graph networks as message, update and readout functions, applied to predicting molecular properties.

ICCV 2017. Adds a mask branch to Faster R-CNN for instance segmentation, with RoIAlign to remove the misalignment caused by rounding.

'On Calibration of Modern Neural Networks' (ICML) finds deep networks poorly calibrated and shows a single temperature parameter fitted on validation data largely fixes it.

Introduced in a paper on training GANs with a two time-scale update rule (NeurIPS 2017), FID compares Gaussians fitted to Inception features of real and generated images, and became the standard image-generation score.

By optimising a PAC-Bayes bound directly, a non-vacuous generalization bound for stochastic neural networks with millions of parameters trained on tens of thousands of examples (UAI 2017).

A reward model learned from people choosing between pairs of clips; feedback on less than 1% of the agent's interactions.

“Variational Inference: A Review for Statisticians”, Journal of the American Statistical Association 112(518).

Robustness as a min–max problem, with projected gradient descent as the inner attacker (ICLR 2018).

A translation model built entirely from attention and feed-forward layers, with no recurrence: 28.4 BLEU on WMT 2014 English–German (arXiv 1706.03762; NIPS 2017).

A clipped surrogate objective giving TRPO-like stability with first-order updates.

Theano's documentation records that after Theano 1.0.0 (15 November 2017), MILA would stop developing it, providing support for one year.

Computes softmax's running maximum and normalizer together in a single pass, rescaling the partial sum whenever the maximum grows. The rule FlashAttention later used to process attention one block at a time.

Proposes training two agents to debate before a human judge, so that judging a debate substitutes for checking a hard answer directly.

Orthogonal scores and cross-fitting give valid inference on causal parameters when nuisance functions are fitted with machine learning (Econometrics Journal).

Random forests adapted to heterogeneous treatment effects, with honest estimation and asymptotically Gaussian estimates (JASA).

Structure learning as continuous optimisation, with acyclicity written as the smooth constraint tr(exp(W∘W)) − d = 0 (NeurIPS 2018).

"Optimization Methods for Large-Scale Machine Learning" (SIAM Review) consolidates the convergence theory of stochastic gradient methods and their noise-reduced and second-order extensions.

One algorithm, given only the rules, trained by self-play and tree search, convincingly defeated a world-champion program in each of the three games (Science, 2018).

Twin critics against overestimation (TD3) and maximum-entropy RL (SAC) became the standard off-policy methods for continuous control (both ICML 2018).

In the infinite-width limit, training a network follows a kernel that stays constant during training.

NAACL 2018: learned vectors for clipped relative distances, added to keys and values instead of absolute position encodings.

A 12-layer, 768-dimensional decoder-only transformer pre-trained on BooksCorpus by next-token prediction, then fine-tuned per task.

“UMAP: Uniform Manifold Approximation and Projection for dimension reduction” (arXiv 1802.03426).

Names four patterns to read critically for: explanation versus speculation, unidentified sources of gains, mathiness, and misuse of language.

Predicts 15% of tokens hidden at random from both sides; BERT-base 110M and BERT-large 340M parameters (arXiv 1810.04805).

Composable transformations — grad, jit, vmap — of pure Python and NumPy-style functions, compiled with XLA. A first version was described at SysML 2018; the public repository begins in November 2018.

Small bottleneck modules with near-identity initialisation, inserted into a frozen BERT, came within 0.4% of full fine-tuning on GLUE while adding 3.6% parameters per task.

A classifier that votes over Gaussian-noised copies of its input is provably robust within an L2 radius σΦ⁻¹(p) — a certified defence that scales to ImageNet.

48 layers, d_model 1600, 1,024-token context and a 50,257-token byte-level vocabulary, trained on WebText; smaller 124M–774M versions also released.

General methods that leverage computation — search and learning — have beaten hand-built knowledge “by a large margin”.

Strided and fixed attention patterns let each position attend to about $2\sqrt{n}$ others while every position stays within two hops, cutting attention cost to $O(n\sqrt{n})$; with block-sparse kernels and recomputation it trained networks with hundreds of layers on sequences of up to 16,384.

Self-play PPO at very large scale, trained for 10 months, defeated Team OG on 13 April 2019.

A practical process for training and reproducing models, built on two observations: training is a leaky abstraction, and it fails silently.

“The Curious Case of Neural Text Degeneration” (ICLR 2020): sample from the smallest set of tokens whose probability reaches p.

ICLR 2019. Message-passing GNNs are at most as powerful as the Weisfeiler–Lehman test at telling graphs apart; the sum-based GIN reaches that bound.

New test sets built like the originals cost published models 3–15 points on CIFAR-10 and 11–14 on ImageNet, yet gains carried over: the drop reflected harder images rather than adaptive overfitting.

“Risks from Learned Optimization” separates aligning the training objective with intent (outer) from aligning a learned optimiser’s own objective with the training objective (inner).

League training with exploiter agents, from a supervised start; rated above 99.8% of ranked human players for all three races.

A PNAS paper (preprint December 2018) reconciles the classical U-shaped risk curve with interpolating models: test error peaks at the interpolation threshold and falls again beyond it.

Proposes learning representations whose optimal classifier is the same across training environments; later analyses found it rarely beats standard training.

Splitting each MLP's first matrix by columns and second by rows (and attention by heads) needs only two all-reduces per layer in each direction; an 8.3B-parameter model trained on 512 GPUs at 76% scaling efficiency.

Data-parallel training stored 16 bytes per parameter on every GPU. Sharding the copies instead cuts per-GPU model-state memory up to linearly in the number of GPUs, at the same or 1.5× the communication.

All query heads share a single key and value head, dividing the KV cache, and the memory traffic of every decoding step, by the number of heads.

Double descent in modern deep networks, as a function of model size, training epochs and — in a critical regime — dataset size, where more data can hurt.

Language-model loss falls as a power law in parameters, data and compute, with some trends spanning more than seven orders of magnitude; depth versus width matters little within a wide range.

Self-supervised image features matched a supervised ResNet-50 under linear evaluation (76.5% top-1 on ImageNet).

“On Layer Normalization in the Transformer Architecture” (ICML 2020): Post-LN has large gradients near the output at initialisation; Pre-LN does not.

A local windowed attention combined with task-motivated global attention scales linearly with sequence length and is a drop-in replacement for standard self-attention.

Reviews the causes of, and a list of about 60 cases of, agents satisfying the letter of an objective without its intent, from a flipped Lego block to a boat circling for points.

A BART generator reads passages retrieved by a DPR dense index over 21M Wikipedia chunks, marginalising over the top $k$. State of the art on three open-domain QA tasks; swapping the index updates what the model knows.

175 billion parameters, 96 layers, 2,048-token context; tasks specified by a few examples in the prompt, with no gradient updates (arXiv 2005.14165).

Predicting the added noise, with a simplified unweighted loss, makes diffusion models produce high-quality images from plain regression training (NeurIPS 2020).

Replacing the softmax similarity with a product of feature maps lets the sums be reordered, giving $O(n{d}^2)$ cost; the causal form is a recurrent network with a fixed-size state.

Tree search over a learned model that predicts only reward, policy and value matched AlphaZero without being given the rules, and set a new state of the art on 57 Atari games (Nature, 2020).

A regulariser that pushes down the values of actions the dataset does not support, so the learned Q-function lower-bounds the policy's true value (2020).

A sparse pattern with linear cost, shown to be a universal approximator of sequence functions and Turing complete, like full attention; handles sequences up to 8 times longer on similar hardware.

arXiv 2010.11929: image patches as tokens in a plain transformer, with no convolutional layers; “large scale training trumps inductive bias”.

Of 1,800 suspicious generations inspected, 604 reproduced training text verbatim, including one person’s contact details (USENIX Security 2021).

Simplified mixture-of-experts routing to the single best expert, with a load-balancing loss and bfloat16 training; models built on T5-Base and T5-Large pre-trained up to 7x faster with the same computational resources, and pre-training was scaled to trillion-parameter models.

Contrastive pre-training on 400 million image–text pairs; zero-shot ImageNet accuracy matching the original ResNet-50 without using its 1.28 million labelled training images.

Rotates queries and keys by position-dependent angles so attention scores depend only on relative offset (arXiv 2104.09864); later used in LLaMA and PaLM.

Ten datasets whose test data come from new hospitals, cameras, places or times; standard training does substantially worse out of distribution on every one.

Score-based models and DDPM are shown to be discretisations of one stochastic differential equation, reversible given the score; the probability-flow ODE gives exact likelihoods (ICLR 2021).

Freeze the pretrained weights and learn each update as $\Delta W=BA$ with rank $r\ll d$. On GPT-3 175B: 10,000× fewer trainable parameters, 3× less GPU memory, no added inference latency after merging.

Bayesian Analysis 16(2): rank-normalised split-R-hat, a 1.01 threshold instead of 1.1, and at least four chains by default.

Remembering promising states and returning to them before exploring solved the Atari games algorithms had not previously beaten humans on (Nature 590, 2021; preprint 2019).

Diffusion in the latent space of an autoencoder, conditioned through cross-attention (CVPR 2022); the basis of Stable Diffusion, which uses a frozen CLIP ViT-L/14 text encoder.

Small transformers on algorithmic tasks generalise perfectly long after memorising their training data; weight decay greatly improves data efficiency. Presented at an ICLR 2021 workshop, posted to arXiv in January 2022.

SFT, a reward model from human rankings, then PPO. Outputs of the 1.3B model were preferred to those of the 175B GPT-3.

For compute-optimal training, model size and training tokens should be scaled equally; a 70B model trained on 1.4T tokens outperforms the 280B Gopher at the same compute.

In the maximal update parametrization (µP), optimal hyperparameters stay stable as models grow; tuning a 40M model beat published 6.7B GPT-3 numbers at 7% of the pretraining cost.

Tiling, an online softmax and recomputation keep the $n\times n$ attention matrix out of GPU main memory: the same result, memory linear in sequence length, and 2 to 4 times faster than standard attention at 128 to 4K tokens on an A100.

A benchmark of 45 tabular datasets finds tree-based models remain state of the art at around 10,000 samples, and traces it to irregular targets, uninformative features and meaningful feature directions.

Abilities absent in smaller models and present in larger ones, which cannot be predicted by extrapolating smaller models (TMLR 2022).

ICLR 2022: no position embeddings, just a fixed per-head penalty proportional to distance, aimed at extrapolating beyond the training length.

Found activation outliers up to 20× larger than other dimensions, concentrated in a few feature dimensions from about 6.7B parameters; keeping those dimensions in 16-bit made 8-bit inference lossless up to 175B.

Learns an ODE along straight paths between noise and data samples, and a “reflow” step that straightens the paths toward one-step generation (ICLR 2023); one of several independent flow-matching papers.

Small ReLU networks represent more sparse features than they have dimensions, at the cost of interference.

Prompting a frozen model to alternate thoughts, tool actions and observations reduced hallucination on question answering and beat imitation and RL agents on ALFWorld and WebShop by 34 and 10 points.

Quantizing weights column by column while compensating the error in the remaining ones compressed 175B-parameter models to 3–4 bits in about four GPU hours with negligible accuracy loss.

An encoder–decoder transformer trained on 680,000 hours of weakly supervised web audio in 98 languages besides English, transferring zero-shot to speech benchmarks.

Harmlessness training where the only human oversight is a list of principles; AI preferences train the reward model.

A transformer that groks modular addition uses discrete Fourier transforms and trigonometric identities; training splits into memorisation, circuit formation and cleanup (ICLR 2023).

Showed that instructions planted in data an LLM application retrieves (web pages, emails) can hijack it, including real systems such as Bing's GPT-4-powered chat — the security problem of tool-using agents.

Released 15 March 2023: TorchDynamo captures graphs from Python bytecode, AOTAutograd captures the backward pass ahead of time, and TorchInductor generates Triton or C++ code — while eager mode stays the default.

Argues that apparent emergent abilities largely come from nonlinear or discontinuous metrics applied to smoothly improving models.

Groups of query heads share key and value heads, a middle ground between multi-head and multi-query attention; existing checkpoints can be uptrained with about 5% of the original pre-training compute.

Backpropagating through a frozen 4-bit (NF4) model into LoRA adapters, with double quantization and paged optimizers, fine-tuned a 65B model on a single 48 GB GPU.

Solves the KL-regularised RLHF objective with a classification loss on preference pairs — no reward model, no RL loop.

The KV cache is stored in fixed-size blocks mapped through a block table, like virtual-memory pages, cutting waste from fragmentation and over-reservation to under 4%. Published at SOSP 2023.

GPT-4 as a judge agreed with human preferences over 80% of the time, about as often as humans agree with each other, but showed position, verbosity and possible self-enhancement biases.

The same exact algorithm with better parallelism and work partitioning; GPT training with it reached up to 225 TFLOPs/s per A100, 72% model FLOPs utilisation.

A 4,096-token attention window per layer gives a theoretical span of about 131K tokens after 32 layers, and a rolling-buffer cache 8 times smaller at 32K tokens.

2,294 real GitHub issues from 12 Python repositories, checked by running the projects' tests. At release the best model resolved 1.96% of them.

A full rewrite that runs Keras workflows on JAX, TensorFlow or PyTorch (and OpenVINO for inference), with a backend-agnostic keras.ops NumPy API and a stateless API for JAX.

Strong models fine-tuned on a weaker model’s labels outperform their supervisor but fall short of their own capability — an analogy for humans supervising superhuman systems (ICML 2024).

An open-weight mixture of experts routing each token to 2 of 8 feed-forward experts per layer: 47B parameters in total, 13B active per token.

Splits experts into many smaller ones and keeps a few always-on shared experts; the 16B model (2.8B active) is comparable to LLaMA2 7B with about 40% of its computation.

Group Relative Policy Optimization: PPO without a critic, using the group of sampled answers as the baseline.

Keys and values are cached as a 512-dimensional latent plus a 64-dimensional positional key per layer, instead of per-head keys and values; the paper reports a 93.3% smaller KV cache than DeepSeek 67B.

On the Corr2Cause benchmark, seventeen language models performed close to random at inferring causal structure from stated correlations (ICLR 2024).

If trends continue, models will be trained on datasets roughly equal to the stock of public human text between 2026 and 2032.

A reasoning model trained with large-scale RL; performance improved with both train-time RL compute and test-time thinking.

“Reinforcement Learning with Verifiable Rewards”: RL against a checker instead of a learned reward model.

Open reasoning models trained with GRPO and rule-based rewards; R1-Zero's AIME 2024 pass@1 rose from 15.6% to 71.0% during RL.

Learned, hardware-aligned sparsity trained from the start: each query combines compressed-block, selected-block and sliding-window attention through learned gates. Published at ACL 2025.

Base models beat their RLVR-trained versions at large pass@k in their experiments — the start of an ongoing debate.