Field guide

Notation

Every symbol used in Descent, what it means, and how to say it out loud. Papers are written in a compressed dialect; this is the dictionary. Filter by typing a word you half-remember.

Conventions and decorations

$θ$: “theta (bold)”The model's parameters: every adjustable number, stacked into one vector. Plain $θ$ is a single parameter.In $f (x; θ)$ , $θ$ holds all the weights.Taught in: Reading the symbols
$α$: “alpha”A scalar coefficient; in the Adam optimiser, the step size (default $0.001$ ).Taught in: Reading the symbols
$ϵ$: “epsilon”A small positive constant that keeps a computation finite (Adam adds $1 0^{- 8}$ to a denominator) — or, in diffusion models, random noise. The paper decides which.Taught in: Reading the symbols
$λ$: “lambda”The strength of a regulariser, $λ \geq 0$ ; in linear algebra, an eigenvalue or a Lagrange multiplier.Ridge regression adds $λ ∥ θ ∥_{2}^{2}$ to the loss.Taught in: Reading the symbols
$y^$: “y hat”A prediction or estimate: the hat means 'computed from data'. (Occasionally a paper uses a hat for something else — check its definition.) $θ^$ are fitted parameters; $y^$ is a predicted target.Taught in: Reading the symbols
$x ˉ$: “x bar”An average, usually the sample mean. $x ˉ = \frac{1}{N} \sum_{n = 1}^{N} x_{n}$ Taught in: Reading the symbols
$θ^{⋆}$: “theta star”The optimum: the value that minimises (or maximises) the objective.Taught in: Reading the symbols
$x ~$: “x tilde”Another version of $x$ — modified, noisy, approximate or simply a second one. No fixed meaning; the paper defines it.Taught in: Reading the symbols
$f^{'} (x)$: “f prime of x”The derivative of $f$ . On a variable, $x^{'}$ may just mean 'another $x$ ' — and in some statistics papers $X^{'}$ is a transpose.Taught in: Reading the symbols
$:=$: “is defined as”Introduces a new name for the expression on the right. $x ˉ := \frac{1}{N} \sum_{n} x_{n}$ Taught in: Reading the symbols
$\equiv$: “is identical to”Identical: equal for every value of the variables, not just for particular ones. Check the paper's usage.Taught in: Reading the symbols
$\approx$: “is approximately equal to”Close, with the approximation usually explained nearby.Taught in: Reading the symbols
$\propto$: “is proportional to”Equal up to a constant factor that is being ignored. $p (θ ∣ D) \propto p (D ∣ θ) p (θ)$ Taught in: Reading the symbols
$\leftarrow$: “is replaced by”Assignment in pseudo-code: overwrite the left side with the value on the right.Adam: $θ_{t} \leftarrow θ_{t - 1} - α m^_{t} / (v^_{t} + ϵ)$ Taught in: Reading the symbols
$1 [condition]$: “indicator of the condition”1 if the condition is true, 0 otherwise.Accuracy $= \frac{1}{n} \sum_{i} 1 [y^_{i} = y_{i}]$ Taught in: Reading the symbols
$O (n^{2})$: “big-O of n squared”Grows at most proportionally to $n^{2}$ for large $n$ (constant factors ignored).A self-attention layer costs $O (n^{2} \cdot d)$ for $n$ positions of width $d$ .Taught in: Reading the symbols
$m (f) = 2595 lo g_{10} (1 + f /700)$: “mel of f”The HTK mel scale, one of several conventions for warping frequency toward perceived pitch. $m (1000 Hz) \approx 1000$ mel.Taught in: Speech and audio
$ϵ (blank)$: “blank”In CTC, the blank symbol: it outputs nothing and separates repeated characters. $h e l ϵ l o \to$ hello.Taught in: Speech and audio
$y^_{T + h ∣ T}$: “y hat T plus h given T”The forecast for time $T + h$ made with data up to time $T$ ; $h$ is the horizon.Taught in: Time series and forecasting
$f32[2,3]$: “f32 of shape 2 by 3”An abstract value as printed in a jaxpr (or an FX graph): the dtype and shape of an array, without its numbers. Tracing works on these. $b:f32[] = sin a$ records a sine of a float32 scalar.Taught in: JAX: tracing and transformations
$s = \frac{m a x _{i} ∣ x _{i} ∣}{2 ^{b - 1} - 1}$: “the absmax scale”The scale for symmetric absmax quantization to $b$ bits: values are stored as $round (x_{i} / s)$ and read back as $s$ times that integer. The largest value sets the step for everything it covers.For int8, $2^{b - 1} - 1 = 127$ ; a single outlier of 20 makes the step $20/127 \approx 0.16$ .Taught in: Quantization and distillation
$W8A8, W4A16$: “W eight A eight; W four A sixteen”Quantization settings: the bit width of the weights (W) and of the activations (A).SmoothQuant targets W8A8; GPTQ and AWQ are usually used as W4A16 (4-bit weights, 16-bit activations).Taught in: Quantization and distillation
$w.r.t.$: “with respect to”Names the variable a derivative or optimisation is taken over.Adam's Algorithm 1: 'Get gradients w.r.t. stochastic objective at timestep $t$ '.Taught in: How to read an ML paper
$e.g.$: “for example”Latin exempli gratia: introduces examples, not a complete list.Taught in: How to read an ML paper
$i.e.$: “that is”Latin id est: introduces a restatement or the complete meaning.Taught in: How to read an ML paper
$arXiv:1706.03762v7$: “arXiv seventeen-oh-six dot oh-three-seven-six-two, version seven”An arXiv identifier: YYMM of first submission, a sequence number, and the version. Cite the version you read.1706 = June 2017, the first version of the transformer paper.Taught in: How to read an ML paper
$c_{i}$: “c i”The cluster that point $i$ is assigned to, a whole number from $1$ to $K$ . $c_{7} = 3$ : point 7 belongs to cluster 3.Taught in: k-means, properly

Sets and spaces

$a \in A$: “a is in A” $a$ is an element of the set $A$ ; $\in /$ means 'is not in'.Taught in: Reading the symbols
$R$: “R”The real numbers.Taught in: Reading the symbols
$R^{n}$: “R n”Vectors of $n$ real numbers. $x \in R^{3}$ is a list of three reals.Taught in: Reading the symbols
$R^{n \times d}$: “R n by d”Real matrices with $n$ rows and $d$ columns.A data matrix $X \in R^{n \times d}$ : one row per example.Taught in: Reading the symbols
${1, \dots, K}$: “the set one to K”The integers from 1 to $K$ ; curly braces list or build a set.Taught in: Reading the symbols
$[a, b], (a, b]$: “closed interval; half-open interval” $[a, b]$ includes both ends; $(a, b]$ excludes $a$ and includes $b$ .Taught in: Reading the symbols
$A \subset B$: “A is a subset of B”Every element of $A$ is also in $B$ .Taught in: Reading the symbols
$A ∖ B$: “A minus B”The elements of $A$ that are not in $B$ .Taught in: Reading the symbols
$\emptyset$: “the empty set”The set with no elements.Taught in: Reading the symbols
$∣ S ∣$: “the size of S”For a set, its number of elements; around a number, absolute value; around a matrix, sometimes the determinant.A vocabulary $V$ has $∣ V ∣$ tokens.Taught in: Reading the symbols
$\forall$: “for all”Universal quantifier: the statement holds for every value.Taught in: Reading the symbols
$\exists$: “there exists”Existential quantifier: at least one value makes the statement true.Taught in: Reading the symbols
$f : R^{n} \to R$: “f maps R n to R” $f$ takes a vector of $n$ reals and returns one real — the function's type.Taught in: Reading the symbols
$g \circ f$: “g after f”Function composition: $(g \circ f) (x) = g (f (x))$ . A few sources use $\circ$ for the element-wise product instead.Taught in: Reading the symbols
$f (x; θ)$: “f of x, parametrised by theta”A function of the input $x$ whose behaviour is set by the parameters $θ$ (held fixed while it is applied).Taught in: Reading the symbols
$D$: “script D”A dataset. $D = {(x_{i}, y_{i})}_{i = 1}^{n}$ Taught in: Reading the symbols
$N_{k} (x)$: “N k of x”The neighbourhood of $x$ : the indices of the $k$ training points closest to it.With $k = 5$ , k-NN predicts the majority label among the 5 points in $N_{5} (x)$ .Taught in: Nearest neighbours and naive Bayes
$R_{m}$: “R m”The region (axis-aligned box) of input space that ends at leaf $m$ of a tree. $f^(x) = \sum_{m} c_{m} 1 [x \in R_{m}]$ .Taught in: Decision trees
$H_{k}$: “H k, the RKHS”The reproducing kernel Hilbert space of kernel $k$ : a space of functions with $f (x) = ⟨ f, k (\cdot, x)⟩$ .The representer theorem: regularised minimisers over $H_{k}$ have the form $\sum_{i} α_{i} k (x_{i}, \cdot)$ .Taught in: Support vector machines and kernels
$N (i)$: “the neighbourhood of i”The set of nodes joined to node $i$ by an edge.Message passing aggregates over $j \in N (i)$ , often including $i$ itself.Taught in: Graph neural networks
$B \times T \times d$: “B by T by d”A shape written out in a paper: here a batch of $B$ sequences of $T$ tokens with $d$ features each. Broadcasting rules compare such shapes from the right.Adding a bias of shape $(d,)$ to an activation of shape $B \times T \times d$ broadcasts over $B$ and $T$ .Taught in: Arrays: NumPy and SciPy
$∣ V ∣$: “the size of the vocabulary”The number of distinct symbols (letters, words or tokens) a language model can predict. The softmax output has one entry per vocabulary item.Shannon's alphabet has $∣ V ∣ = 27$ : 26 letters and the space.Taught in: From n-grams to neural language models
$H$: “script H”The hypothesis class: the set of prediction rules a learner may output, fixed before seeing data.All thresholds $x \mapsto 1 [x \geq a]$ .Taught in: PAC learning
$VCdim (H)$: “the VC dimension of H”The size of the largest set of points the class can shatter (label in every possible way).Half-planes in $R^{2}$ : 3. Half-spaces in $R^{d}$ : $d + 1$ .Taught in: VC dimension
$H_{C}$: “H restricted to C”The set of label vectors the class can produce on the finite point set $C$ . $H$ shatters $C$ when $∣ H_{C} ∣ = 2^{∣ C ∣}$ .Thresholds on 3 points give 4 label vectors, not 8.Taught in: VC dimension
$x \in R^{d}$: “x is in R d” $x$ is a vector of $d$ real numbers. Bold lowercase means a vector, written as a column.A $28 \times 28$ greyscale image flattened into a vector lives in $R^{784}$ .Taught in: Vectors and the dot product
$d_{single} (G, H) = min_{i \in G, j \in H} d_{ij}$: “single linkage”Distance between clusters as their closest pair (complete: farthest pair; average: mean over pairs).Taught in: Hierarchical and density-based clustering
$ε, minPts$: “epsilon, min points”DBSCAN's radius and the number of points (itself included, in scikit-learn) needed within it for a core point. $ε = 0.2$ , minPts $= 5$ .Taught in: Hierarchical and density-based clustering

Sums, products and indices

$\sum_{i = 1}^{n} a_{i}$: “the sum from i equals 1 to n of a i”Add $a_{1} + a_{2} + \dots + a_{n}$ : a for-loop with an accumulator starting at 0.Taught in: Shapes, indices and sums
$\prod_{i = 1}^{n} a_{i}$: “the product from i equals 1 to n of a i”Multiply $a_{1} a_{2} \dots a_{n}$ : a loop whose accumulator starts at 1. Its log is a sum of logs.Taught in: Shapes, indices and sums
$x^{(i)}$: “x superscript i”The $i$ -th training example; the brackets say it is not a power.Taught in: Reading the symbols
$W^{[l]}$: “W superscript l”The weights of layer $l$ (a common course convention).Taught in: Reading the symbols
$x_{t}$: “x sub t”The value at time step $t$ (also written $x^{⟨ t ⟩}$ ).Taught in: Reading the symbols
$C_{ij} = \sum_{k} A_{ik} B_{k j}$: “C i j equals the sum over k of A i k B k j”Matrix product in index notation: $i, j$ are free (they index the result), $k$ is summed away. Count free indices to know the shape.Taught in: Shapes, indices and sums
$⨁_{j \in N (i)}$: “aggregate over the neighbours of i”A permutation-invariant aggregation (sum, mean or max) of messages from node $i$ 's neighbours.Taught in: Graph neural networks
$X [m, k] = \sum_{n} x [n + m H] w [n] e^{- 2 π ik n / N}$: “X of m, k, the short-time Fourier transform”The DFT of frame $m$ (hop $H$ , window $w$ of length $N$ ): how much of frequency $k f_{s} / N$ is present in that frame.Whisper: $N = 400$ , $H = 160$ at 16 kHz (25 ms windows, 10 ms hop).Taught in: Speech and audio
$o + \sum_{k} s_{k} n_{k}$: “o plus the sum over k of s k times n k”The strided indexing formula: where item $(n_{0}, \dots, n_{N - 1})$ of an array lives, measured from the start of the buffer ( $o$ is the array's starting offset).Item $[2, 1]$ of an int64 $(3, 4)$ array sits at $32 \cdot 2 + 8 \cdot 1 = 72$ bytes.Taught in: Arrays: NumPy and SciPy
$x_{< t}$: “x before t”All the symbols before position $t$ : $x_{1}, \dots, x_{t - 1}$ (the context or history). Also written $x_{1 : t - 1}$ or, in Bengio et al. (2003), $w_{1}^{t - 1}$ .For the text 'cat', at $t = 3$ the context $x_{< 3}$ is 'ca'.Taught in: Predicting the next letter
$τ_{H} (m)$: “tau H of m (growth function)”The largest number of distinct labellings the class can produce on any $m$ points. Written $Π_{H} (m)$ by Mohri et al.Sauer–Shelah: if $VCdim = d$ , $τ (m) \leq \sum_{i \leq d} (i m)$ .Taught in: VC dimension

Vectors and matrices

$ρ (W)$: “rho of W, the spectral radius”The largest absolute value of the eigenvalues of a square matrix $W$ . It governs whether $W^{k}$ shrinks ( $ρ < 1$ ) or can grow ( $ρ > 1$ ).For $W = 0.9 Q$ with $Q$ orthogonal, $ρ (W) = 0.9$ .Taught in: Recurrent networks
$a ⊙ b$: “a Hadamard b (elementwise product)”Elementwise multiplication of two vectors of the same shape. Gates act on signals this way. $(0.5, 1) ⊙ (4, 3) = (2, 3)$ Taught in: LSTM and GRU
$x^{⊤}$: “x transpose”Turns a column vector into a row; $X^{⊤}$ swaps a matrix's rows and columns. Also written $X^{T}$ or $X^{'}$ .Taught in: Reading the symbols
$⟨ x, y ⟩$: “the inner product of x and y” $\sum_{i} x_{i} y_{i}$ , also written $x^{⊤} y$ : two vectors in, one number out.Taught in: Reading the symbols
$∥ x ∥_{2}$: “the two-norm of x”Euclidean length $\sum_{i} x_{i}^{2}$ ; a bare $∥ x ∥$ usually means this.Taught in: Reading the symbols
$∥ x ∥_{1}$: “the one-norm of x” $\sum_{i} ∣ x_{i} ∣$ .Taught in: Reading the symbols
$A ⊙ B$: “A Hadamard B”Element-wise product of same-shaped arrays. NumPy: A * B.Taught in: Reading the symbols
$(n \times k) (k \times m) \to n \times m$: “n by k times k by m gives n by m”The shape rule of matrix multiplication: inner sizes must match and disappear.Taught in: Shapes, indices and sums
$I_{n}$: “the n by n identity”The square matrix with ones on the diagonal and zeros elsewhere.Taught in: Reading the symbols
$d_{M} (x, x^{'})$: “Mahalanobis distance” $(x - x^{'})^{⊤} M (x - x^{'})$ for a positive definite matrix $M$ ; $M = I$ gives Euclidean distance.With $M = diag (1, 16)$ , a difference of 1 in the second feature counts as much as 4 in the first.Taught in: Nearest neighbours and naive Bayes
$k (x, x^{'})$: “k of x and x prime”A kernel: a similarity equal to an inner product $ϕ (x)^{⊤} ϕ (x^{'})$ in some feature space. Valid kernels make every Gram matrix positive semidefinite.RBF: $k (x, x^{'}) = exp (- γ ∥ x - x^{'} ∥^{2})$ .Taught in: Support vector machines and kernels
$K_{ij} = k (x_{i}, x_{j})$: “the Gram matrix”The $n \times n$ matrix of kernel values between all pairs of training points.Kernel ridge regression: $α = (K + λ I)^{- 1} y$ .Taught in: Support vector machines and kernels
$f (P X, P A P^{⊤}) = P f (X, A)$: “f of P X, P A P transpose equals P f of X, A”Permutation equivariance: relabelling a graph's nodes (permutation matrix $P$ ) relabels a node-level output the same way.A graph-level readout instead satisfies $g (P X, P A P^{⊤}) = g (X, A)$ (invariance).Taught in: Graph neural networks
$A ~ = A + I, D ~_{ii} = \sum_{j} A ~_{ij}$: “A tilde, D tilde”The adjacency matrix with self-loops, and its diagonal degree matrix (each node's degree plus one).Taught in: Graph neural networks
$r^_{u i} = μ + b_{u} + b_{i} + p_{u}^{⊤} q_{i}$: “r hat u i”Biased matrix factorisation: global mean plus user and item biases plus the dot product of user and item latent factors.Taught in: Recommender systems
$s_{ij} = v_{i}^{⊤} t_{j}$: “s i j”Cosine similarity between normalised image embedding $v_{i}$ and text embedding $t_{j}$ ; CLIP's logits are $s_{ij} / τ$ .Taught in: Multimodal models
$s_{k}$: “s sub k, the stride of axis k”How far to move in memory when the index along axis $k$ increases by one. NumPy counts strides in bytes, PyTorch in elements. Zero means the axis is broadcast; negative means it runs backwards.An int64 array of shape $(3, 4)$ in C order has strides $(32, 8)$ bytes; its transpose has $(8, 32)$ .Taught in: Arrays: NumPy and SciPy
$itemsize \prod_{j > k} d_{j}$: “item size times the product of the sizes after axis k”The stride of axis $k$ for a freshly created row-major (C-order) array with shape $(d_{0}, \dots, d_{N - 1})$ .float32 (4 bytes), shape $(2, 3, 4)$ : strides $(48, 16, 4)$ .Taught in: Arrays: NumPy and SciPy
$y^= w^{⊤} x + b$: “y hat equals w transpose x plus b”The linear model: a weighted sum of the features plus a bias. With a leading 1 appended to $x$ it becomes a single dot product $θ^{⊤} x ~$ .With $w = 0.5$ , $b = 1$ and $x = 4$ : $y^= 3$ .Taught in: Linear regression
$X θ$: “X theta”All $n$ predictions at once: the data (design) matrix $X$ , one example per row with a leading column of ones, times the parameter vector. $X$ is $n \times d$ and $θ$ is $d \times 1$ , so $X θ$ is $n \times 1$ .Taught in: Linear regression
$X^{⊤} X θ^= X^{⊤} y$: “X transpose X theta hat equals X transpose y”The normal equations: the gradient of the squared error is zero, equivalently the residual is orthogonal (normal) to every column of $X$ .With one feature and an intercept they give $b = y ˉ - w x ˉ$ and $w = \sum (x_{i} - x ˉ) (y_{i} - y ˉ) / \sum (x_{i} - x ˉ)^{2}$ .Taught in: The closed form: normal equations
$θ^= (X^{⊤} X)^{- 1} X^{⊤} y$: “theta hat equals X transpose X inverse, X transpose y”The closed-form (OLS) least-squares solution, valid when the columns of $X$ are linearly independent. Compute it with QR or the SVD (np.linalg.lstsq), not an explicit inverse.One column $x$ , no intercept: $θ^= x^{⊤} y / x^{⊤} x$ .Taught in: The closed form: normal equations
$X^{+}$: “X plus, the pseudoinverse of X”The Moore–Penrose pseudoinverse. $X^{+} y$ is the least-squares solution of smallest norm, defined even when $X^{⊤} X$ is singular. Via the SVD, $X^{+} = V Σ^{+} U^{⊤}$ .If $X$ has independent columns, $X^{+} = (X^{⊤} X)^{- 1} X^{⊤}$ .Taught in: The closed form: normal equations
$H = X (X^{⊤} X)^{- 1} X^{⊤}$: “the hat matrix”The orthogonal projection onto the column space of $X$ : it maps the targets to the fitted values, $y^= H y$ . Symmetric, and $H^{2} = H$ .Projecting twice changes nothing: $H (H y) = H y$ .Taught in: The closed form: normal equations
$κ (A) = σ_{m a x} / σ_{m i n}$: “kappa of A, the condition number”The ratio of the largest to the smallest singular value; large values mean a nearly singular matrix whose solutions are sensitive to small errors. For a Hessian, the ratio of extreme eigenvalues, which controls how fast gradient descent converges. $κ (X^{⊤} X) = κ (X)^{2}$ , so forming $X^{⊤} X$ squares it.Taught in: The closed form: normal equations
$cos (q, d)$: “cosine similarity of q and d”The dot product of two vectors divided by the product of their lengths: the cosine of the angle between them. For unit-length embeddings it is just the dot product.Dense retrieval returns the documents whose embeddings have the highest cosine similarity to the query's.Taught in: Embeddings, retrieval and RAG
$A ⪰ 0$: “A is positive semidefinite”The symmetric matrix $A$ has $v^{⊤} A v \geq 0$ for every $v$ , equivalently all its eigenvalues are $\geq 0$ . $A ⪰ B$ means $A - B ⪰ 0$ ; $≻$ is the strict (positive definite) version.A twice-differentiable $f$ is convex exactly when $\nabla^{2} f (x) ⪰ 0$ everywhere.Taught in: Convexity
$A \otimes B$: “A Kronecker B”The Kronecker product: the block matrix whose $(i, j)$ block is $a_{ij} B$ . Satisfies $(A \otimes B)^{- 1} = A^{- 1} \otimes B^{- 1}$ , which is why K-FAC can invert large curvature blocks cheaply.Taught in: Newton, quasi-Newton and natural gradients
$∥ δ ∥_{\infty} \leq ε$: “the infinity norm of delta is at most epsilon”An adversarial budget: no input coordinate (pixel) may change by more than $ε$ .CIFAR-10 attacks commonly use $ε = 8/255$ .Taught in: Adversarial examples and robustness
$Φ^{+}$: “Phi plus (pseudo-inverse)”The Moore–Penrose pseudo-inverse. $Φ^{+} y$ is the least-squares solution, and the minimum-norm one when there are many. $Φ^{+} = \sum_{r} σ_{r}^{- 1} v_{r} u_{r}^{⊤}$ over non-zero singular values.Taught in: Double descent
$σ_{m i n} (Φ)$: “sigma min of Phi”The smallest non-zero singular value of the feature matrix; it approaches zero near the interpolation threshold, which amplifies noise.Dividing noise by $σ_{m i n} = 1 0^{- 3}$ multiplies it by a thousand.Taught in: Double descent
$x_{i}$: “x sub i”The $i$ -th component (entry) of the vector $x$ , a single number, so written in italic.If $x = [3, 1]$ then $x_{1} = 3$ and $x_{2} = 1$ .Taught in: Vectors and the dot product
$∥ x ∥_{2}$: “the two-norm of x”Euclidean (straight-line) length: $\sum_{i} x_{i}^{2}$ . Plain $∥ x ∥$ usually means this one. $∥ [3, 4] ∥_{2} = 5$ .Taught in: Vectors and the dot product
$∥ x ∥_{1}$: “the one-norm of x”Taxicab length: the sum of absolute values of the components. Used for L1 (lasso) penalties. $∥ [3, - 4] ∥_{1} = 7$ .Taught in: Vectors and the dot product
$∥ x ∥_{\infty}$: “the infinity-norm of x”The largest absolute component. $∥ [3, - 4] ∥_{\infty} = 4$ .Taught in: Vectors and the dot product
$a \cdot b$: “a dot b”The dot product: $\sum_{i} a_{i} b_{i} = ∥ a ∥∥ b ∥ cos θ$ . A single number that is large when the vectors point the same way. $[3, 1] \cdot [1, 2.5] = 5.5$ .Taught in: Vectors and the dot product
$⟨ a, b ⟩$: “the inner product of a and b”Another way to write the dot product (and its generalisations). $⟨ x, x ⟩ = ∥ x ∥^{2}$ .Taught in: Vectors and the dot product
$a^{⊤} b$: “a transpose b”The dot product written as a matrix product: a $1 \times d$ row times a $d \times 1$ column.Attention scores $Q K^{⊤}$ are a table of such products.Taught in: Vectors and the dot product
$cos θ = \frac{a \cdot b}{∥ a ∥∥ b ∥}$: “cosine similarity”Cosine similarity: the dot product with lengths divided out, between $- 1$ and $1$ .A vector and $3$ times itself have cosine similarity $1$ .Taught in: Vectors and the dot product
$proj_{a} b$: “projection of b onto a”The closest point to $b$ on the line through $a$ : $\frac{a \cdot b}{a \cdot a} a$ . $proj_{[1, 0]} [2, 3] = [2, 0]$ .Taught in: Vectors and the dot product
$span {v, w}$: “the span of v and w”Every vector reachable as a linear combination $c_{1} v + c_{2} w$ .Two non-parallel vectors in the plane span the whole plane.Taught in: Vectors and the dot product
$A \in R^{m \times n}$: “A is an m by n real matrix”A matrix with $m$ rows and $n$ columns; as a transformation it maps $R^{n} \to R^{m}$ . Its columns are where the basis vectors land.A layer mapping 784 inputs to 128 outputs has a $128 \times 784$ matrix (in the $W x$ convention).Taught in: Matrices as transformations
$A_{ij}$: “A i j”The entry of $A$ in row $i$ , column $j$ . $(A B)_{ij} = \sum_{k} A_{ik} B_{k j}$ .Taught in: Matrices as transformations
$A B$: “A B”Matrix product = composition: apply $B$ first, then $A$ . Needs inner dimensions to match: $(n \times d) (d \times k) = n \times k$ . In general $A B \neq = B A$ .Shear after rotation differs from rotation after shear.Taught in: Matrices as transformations
$A^{⊤}$: “A transpose”Rows and columns swapped: $(A^{⊤})_{ij} = A_{j i}$ .Gradients through a layer $W x$ flow back through $W^{⊤}$ .Taught in: Matrices as transformations
$det A$: “determinant of A”The factor by which $A$ scales area (volume); negative if it flips orientation; zero if it squashes space flat. Also written $∣ A ∣$ . $det [a c b d] = a d - b c$ .Taught in: Matrices as transformations
$A^{- 1}$: “A inverse”The transformation that undoes $A$ : $A^{- 1} A = I$ . Exists exactly when $det A \neq = 0$ .Solve $A x = b$ as $x = A^{- 1} b$ (in code: np.linalg.solve).Taught in: Matrices as transformations
$I$: “the identity matrix”The matrix that leaves every vector unchanged: ones on the diagonal, zeros elsewhere. $A I = I A = A$ .Taught in: Matrices as transformations
$rank (A)$: “rank of A”The dimension of the column space: how many independent directions survive the transformation.A matrix that squashes the plane onto a line has rank 1.Taught in: Matrices as transformations
$ker (A)$: “the kernel (null space) of A”All inputs that $A$ sends to zero. Rank–nullity: $rank (A) + dim ker (A) = n$ . $[1 0.5 21]$ sends every multiple of $[- 2, 1]$ to zero.Taught in: Matrices as transformations
$⊙$: “element-wise (Hadamard) product”Multiply two same-shaped arrays entry by entry.Backprop through an activation: $\nabla_{z} L = σ^{'} (z) ⊙ \nabla_{a} L$ .Taught in: Matrix calculus you actually need
$A v = λ v$: “A v equals lambda v”The eigenvalue equation: $v$ (nonzero) is an eigenvector, a direction $A$ only stretches, by the eigenvalue $λ$ . $[2112]$ has eigenvalues $3$ and $1$ .Taught in: Eigenvectors, SVD and PCA
$A = Q Λ Q^{⊤}$: “A equals Q Lambda Q transpose”Spectral decomposition of a symmetric matrix: orthonormal eigenvectors in $Q$ , real eigenvalues on the diagonal of $Λ$ .Every covariance matrix has one.Taught in: Eigenvectors, SVD and PCA
$A = U Σ V^{⊤}$: “A equals U Sigma V transpose”Singular value decomposition: rotate ( $V^{⊤}$ ), stretch by singular values ( $Σ$ ), rotate ( $U$ ). Exists for every real matrix.PCA directions are the columns of $V$ for centred data.Taught in: Eigenvectors, SVD and PCA
$σ_{i}$: “sigma sub i”The $i$ -th singular value, sorted $σ_{1} \geq σ_{2} \geq \dots \geq 0$ . (In statistics, $σ$ alone is a standard deviation.) $∥ A ∥_{2} = σ_{1}$ .Taught in: Eigenvectors, SVD and PCA
$∥ A ∥_{F}$: “the Frobenius norm of A” $\sum_{ij} A_{ij}^{2} = \sum_{i} σ_{i}^{2}$ .Truncated SVD error: $∥ A - A_{k} ∥_{F} = \sum_{i > k} σ_{i}^{2}$ .Taught in: Eigenvectors, SVD and PCA
$∥ A ∥_{2}$: “the spectral norm of A”The most $A$ can lengthen a unit vector: its largest singular value. Also $∥ A ∥_{op}$ .Eckart–Young: $∥ A - A_{k} ∥_{2} = σ_{k + 1}$ .Taught in: Eigenvectors, SVD and PCA
$tr (A)$: “trace of A”Sum of the diagonal entries; equals the sum of the eigenvalues.Total variance of a dataset = $tr$ of its covariance.Taught in: Eigenvectors, SVD and PCA
$W, w_{ij}$: “W, w i j”The weight (affinity) matrix of a similarity graph: $w_{ij} \geq 0$ is the strength of the edge between points $i$ and $j$ .A Gaussian kernel $w_{ij} = e^{- ∥ x_{i} - x_{j} ∥^{2} /2 σ^{2}}$ .Taught in: Spectral clustering and graph Laplacians
$D = diag (d_{1}, \dots, d_{n})$: “D”The degree matrix: $d_{i} = \sum_{j} w_{ij}$ on the diagonal.Taught in: Spectral clustering and graph Laplacians
$L = D - W$: “L equals D minus W”The graph Laplacian. $f^{⊤} L f = \frac{1}{2} \sum_{ij} w_{ij} (f_{i} - f_{j})^{2}$ ; its zero eigenvalues count connected components.Taught in: Spectral clustering and graph Laplacians
$L_{sym} = I - D^{- 1/2} W D^{- 1/2}$: “L sym”The symmetric normalised Laplacian used by the Ng–Jordan–Weiss algorithm ( $L_{rw} = I - D^{- 1} W$ is the random-walk version).Taught in: Spectral clustering and graph Laplacians
$1_{S}$: “indicator of S”The indicator vector of a set of nodes: 1 on $S$ , 0 elsewhere.Taught in: Spectral clustering and graph Laplacians

Calculus

$\prod_{t = k + 1}^{T} \frac{\partial h _{t}}{\partial h _{t - 1}}$: “the product of the step-to-step Jacobians”How a late state depends on an early one. A long product of Jacobians tends to vanish or explode: the core difficulty of training RNNs.For a tanh RNN each factor is $diag (1 - h_{t}^{2}) W_{h}$ .Taught in: Recurrent networks
$\frac{df}{d x}$: “d f d x”The derivative of a one-input function $f$ with respect to $x$ .Taught in: Reading the symbols
$\frac{\partial f}{\partial x _{j}}$: “partial f partial x j”How $f$ changes when only $x_{j}$ moves, all other inputs held fixed.Taught in: Reading the symbols
$a + b ε, ε^{2} = 0$: “a plus b epsilon, with epsilon squared equal to zero”A dual number: a primal value $a$ carrying a tangent $b$ . Arithmetic on dual numbers applies the chain rule automatically — the basis of forward-mode autodiff. $f (a + b ε) = f (a) + f^{'} (a) b ε$ ; seed $b = 1$ to read off $f^{'} (a)$ .Taught in: Autodiff from scratch
$v ˉ$: “v bar, the adjoint of v”The adjoint of an intermediate value $v$ : $\partial L / \partial v$ , the sensitivity of the final output to $v$ . Reverse mode computes adjoints from the output backwards, summing over every place $v$ is used.For $L = e \cdot e$ : $e ˉ = L ˉ e + L ˉ e = 2 e$ .Taught in: Autodiff from scratch
$x ˙$: “x dot, a tangent”A tangent vector: a direction in input space along which forward mode differentiates.A one-hot $x ˙$ makes $J x ˙$ one column of the Jacobian.Taught in: Autodiff from scratch
$J x ˙$: “J times x dot, a Jacobian-vector product”A JVP: what forward-mode autodiff computes in one pass, about as costly as evaluating the function a few times. JAX writes it $(x, v) \mapsto \partial f (x) v$ .For $f : R^{3} \to R^{1000}$ , three JVPs give the whole Jacobian.Taught in: Autodiff from scratch
$y ˉ^{⊤} J$: “y bar transpose times J, a vector-Jacobian product”A VJP: what reverse-mode autodiff (backpropagation) computes in one backward pass. For a scalar loss with $y ˉ = 1$ it is the whole gradient. JAX writes it $(x, v) \mapsto v \partial f (x)$ , with $v$ a cotangent.One VJP of a training loss gives $\nabla_{θ} L$ for all parameters at once.Taught in: Autodiff from scratch
$\partial f (x)$: “partial f of x, the Jacobian of f at x”JAX documentation's notation for the Jacobian of $f : R^{n} \to R^{m}$ at $x$ , an $m \times n$ matrix (or the linear map it represents).A loss $f : R^{n} \to R$ has $\partial f (x) \in R^{1 \times n}$ , a very wide Jacobian — reverse mode's best case.Taught in: JAX: tracing and transformations
$∣ det J_{f} (u)∣$: “absolute determinant of the Jacobian of f at u”How much an invertible map $f$ scales volume near $u$ . Normalizing flows divide the base density by it: $lo g p_{x} (x) = lo g p_{u} (u) - lo g ∣ det J_{f} (u)∣$ .For an affine coupling layer it is $exp (\sum_{i} s_{i})$ , the product of the scales.Taught in: Normalizing flows
$d x = f (x, t) d t + g (t) d w$: “d x equals f dt plus g d w”A stochastic differential equation: a drift $f$ plus Brownian noise of size $g$ . Diffusion models' forward processes are SDEs of this form, reversible given the score.DDPM's limit: $f = - \frac{1}{2} β (t) x$ , $g = β (t)$ .Taught in: Score matching and flow matching
$\frac{L ( θ + h ) - L ( θ - h )}{2 h}$: “centred difference”A finite-difference estimate of a derivative, with error of order $h^{2}$ . Used to check backpropagation code.With 64-bit floats, $h \approx 1 0^{- 5}$ works well.Taught in: Backpropagation
$\partial h (x)$: “the subdifferential of h at x”The set of subgradients of a convex function: slopes $g$ with $h (z) \geq h (x) + g^{⊤} (z - x)$ for all $z$ . A single gradient where $h$ is differentiable, an interval at a kink. $\partial ∣ x ∣$ at $x = 0$ is $[- 1, 1]$ .Taught in: Constraints, projections and proximal methods
$f^{'} (x) = \frac{df}{d x}$: “f prime of x; d f d x”The derivative: how much $f$ changes per unit nudge of $x$ . $f (x + h) \approx f (x) + f^{'} (x) h$ .For $f (x) = x^{2}$ , $f^{'} (x) = 2 x$ .Taught in: Derivatives, gradients and the chain rule
$\frac{\partial f}{\partial x _{i}}$: “partial f partial x i”Partial derivative: nudge only $x_{i}$ , hold the other inputs fixed.For $f = x^{2} y + 3 y$ : $\partial f / \partial x = 2 x y$ .Taught in: Derivatives, gradients and the chain rule
$\nabla f$: “grad f, or nabla f”The gradient: the vector of partial derivatives, same shape as the input. Points in the direction of steepest ascent, perpendicular to contour lines. $f (x, y) = x^{2} + y^{2}$ has $\nabla f = [2 x, 2 y]$ .Taught in: Derivatives, gradients and the chain rule
$\nabla_{θ} L$: “grad of L with respect to theta”Gradient of the loss with respect to the parameters; the subscript says which variables are nudged.Gradient descent: $θ \leftarrow θ - η \nabla_{θ} L$ .Taught in: Derivatives, gradients and the chain rule
$D_{u} f$: “directional derivative of f along u”Rate of change of $f$ moving in unit direction $u$ : $\nabla f \cdot u$ .Largest when $u$ points along $\nabla f$ .Taught in: Derivatives, gradients and the chain rule
$\frac{\partial f}{\partial x}$: “the Jacobian of f with respect to x”The Jacobian: an $m \times n$ matrix with entries $\partial f_{i} / \partial x_{j}$ (numerator layout). The matrix of the best local linear approximation.The Jacobian of $A x$ is $A$ .Taught in: Matrix calculus you actually need
$J_{f} (x)$: “J f of x”Another name for the Jacobian of $f$ at $x$ . Chain rule: $J_{g \circ f} = J_{g} J_{f}$ . $∣ det J ∣$ is the local area scaling of a map.Taught in: Matrix calculus you actually need
$\frac{\partial L}{\partial W}$: “d L by d W”Gradient of a scalar with respect to a matrix: same shape as $W$ , entry $(i, j)$ is $\partial L / \partial W_{ij}$ .For $y = W x$ : $\partial L / \partial W = (\nabla_{y} L) x^{⊤}$ .Taught in: Matrix calculus you actually need

Probability and statistics

$μ$: “mu”A mean (bold $μ$ for a mean vector). $N (μ, σ^{2})$ is a Gaussian with mean $μ$ .Taught in: Reading the symbols
$σ$: “sigma”A standard deviation — or, written as a function $σ (z)$ , the logistic sigmoid.Taught in: Reading the symbols
$p (y ∣ x)$: “p of y given x”The probability of $y$ given $x$ . Different from $p (x ∣ y)$ .Taught in: Reading the symbols
$p (x; θ)$: “p of x, parametrised by theta”A distribution whose shape is set by $θ$ ; also written $p_{θ} (x)$ .Taught in: Reading the symbols
$x \sim p$: “x is distributed as p” $x$ is a random draw from the distribution $p$ .Taught in: Reading the symbols
$P (y ∣ d o (x))$: “P of y given do x”The interventional distribution: the distribution $Y$ would have if $X$ were set to $x$ by an outside action, for every unit. Not the same as $P (y ∣ x)$ .If a hidden cause drives both $X$ and $Y$ , $P (y ∣ x)$ reflects it but $P (y ∣ d o (x))$ does not.Taught in: Interventions and the do-operator
$X_{j} := f_{j} (PA_{j}, U_{j})$: “X j is assigned f j of its parents and its noise”One assignment of a structural causal model: a variable is computed from its direct causes and an independent noise term. The $:=$ marks an assignment, not an equation. $Y := 2 X + U_{Y}$ says $Y$ is computed from $X$ , not the reverse.Taught in: Causal graphs and d-separation
$PA_{j}, pa (j)$: “the parents of j”The direct causes of variable $j$ in a causal graph: the nodes with an arrow into $j$ .In $Z \to X \to Y \leftarrow Z$ , $PA_{Y} = {X, Z}$ .Taught in: Causal graphs and d-separation
$X ⊥ ⊥_{G} Y ∣ S$: “X and Y are d-separated by S in G”d-separation: every path between $X$ and $Y$ in the graph $G$ is blocked by $S$ . It implies conditional independence in any distribution generated by the graph.In $X \to M \leftarrow Y$ , $X$ and $Y$ are d-separated by the empty set but not by ${M}$ .Taught in: Causal graphs and d-separation
$M_{x}, G_{X}$: “M sub x; G with X over-barred”The model after the intervention $d o (X = x)$ , and its graph: all arrows into $X$ deleted (graph surgery). $G_{X}$ deletes the arrows out of $X$ .Randomising a treatment turns the graph into $G_{T}$ .Taught in: Interventions and the do-operator
$Y_{i} (1), Y_{i} (0)$: “Y i of one, Y i of zero”Potential outcomes: what unit $i$ 's outcome would be under treatment and under control. Only one is ever observed.A patient's blood pressure with the drug, $Y (1)$ , and without it, $Y (0)$ .Taught in: Potential outcomes and treatment effects
$τ_{i} = Y_{i} (1) - Y_{i} (0)$: “tau i”The individual treatment effect. Never observable, by the fundamental problem of causal inference.If $Y_{i} (1) = 7$ and $Y_{i} (0) = 5$ , then $τ_{i} = 2$ .Taught in: Potential outcomes and treatment effects
$ATE, ATT$: “A T E, A T T”Average treatment effect $E [Y (1) - Y (0)]$ over everyone, and the average among the treated, $E [Y (1) - Y (0) ∣ T = 1]$ .In a trial, the difference in mean outcomes between arms estimates the ATE.Taught in: Potential outcomes and treatment effects
$τ (x)$: “tau of x”The conditional average treatment effect (CATE): the average effect among units with covariates $x$ . Used for personalisation and uplift modelling.A campaign might have $τ (x) > 0$ for persuadables and $< 0$ for sleeping dogs.Taught in: Potential outcomes and treatment effects
$e (x) = P (T = 1 ∣ X = x)$: “e of x, the propensity score”The probability of receiving treatment given the covariates. Units with the same propensity score have the same covariate distribution in both arms.Inverse propensity weighting gives each treated unit weight $1/ e (x)$ .Taught in: Potential outcomes and treatment effects
$(Y (0), Y (1)) ⊥ ⊥ T ∣ X$: “the potential outcomes are independent of treatment given X”Ignorability (unconfoundedness): among units with the same covariates, treatment is as good as random. Untestable from data.Holds by design in a trial randomised within strata of $X$ .Taught in: Potential outcomes and treatment effects
$θ^_{DML}$: “theta hat D M L”The double/debiased machine learning estimate: regress the outcome residual $Y - ℓ^(X)$ on the treatment residual $D - m^(X)$ , with both nuisance models cross-fitted.With lasso nuisance models it corrects the shrinkage bias of a single lasso.Taught in: Causal discovery and causal ML
$h (W) = tr (e^{W \circ W}) - d$: “h of W”The NOTEARS acyclicity function: zero exactly when the weighted adjacency matrix $W$ has no directed cycles.For a single edge $W_{12} = 1$ , $W \circ W$ is nilpotent, so $e^{W \circ W} = I + W \circ W$ and $h = 0$ .Taught in: Causal discovery and causal ML
$p (x ∣ y) = \prod_{j} p (x_{j} ∣ y)$: “p of x given y equals the product over j of p of x j given y”The naive Bayes assumption: features are conditionally independent given the class.For two binary words with $P (free ∣ spam) = 0.3$ and $P (cash ∣ spam) = 0.2$ , the model gives $P (both ∣ spam) = 0.06$ .Taught in: Nearest neighbours and naive Bayes
$G = 1 - \sum_{k} p_{k}^{2}$: “Gini impurity”Impurity of a tree node with class proportions $p_{k}$ : the chance of mislabelling a random example if labels were drawn from the node's proportions.A node with 5 blue and 5 gold examples has $G = 0.5$ ; a pure node has $G = 0$ .Taught in: Decision trees
$ρ σ^{2} + \frac{1 - ρ}{B} σ^{2}$: “rho sigma squared plus one minus rho over B sigma squared”Variance of an average of $B$ estimators with variance $σ^{2}$ and pairwise correlation $ρ$ .With $σ^{2} = 1$ , $ρ = 0.3$ , $B = 10$ : variance $0.37$ ; it can never fall below $0.3$ .Taught in: Bagging and random forests
$OOB$: “out-of-bag”Training examples left out of a given bootstrap sample (about 37% of them); predicting each example with only the trees that did not see it gives an error estimate. $(1 - 1/ n)^{n} \to e^{- 1} \approx 0.368$ .Taught in: Bagging and random forests
$g (μ) = η$: “g of mu equals eta”The link function of a GLM: connects the mean response $μ$ to the linear predictor $η = w^{⊤} x + b$ .Poisson regression uses the log link, $lo g μ = η$ .Taught in: Generalized linear models
$A (θ)$: “A of theta”The log-partition function of an exponential family; its derivatives give the mean, $A^{'} (θ)$ , and variance, $ϕ A^{''} (θ)$ .Poisson: $A (θ) = e^{θ}$ , so mean = variance = $e^{θ} = μ$ .Taught in: Generalized linear models
$D$: “deviance”Twice the log-likelihood gap between the saturated model and the fitted model; for a Gaussian it is the residual sum of squares.Poisson unit deviance: $2 (y lo g \frac{y}{μ ^} - y + μ^)$ .Taught in: Generalized linear models
$IoU (B_{p}, B_{g})$: “intersection over union”Area of overlap of two boxes divided by the area of their union; 1 for identical boxes, 0 for disjoint ones. Detections count as correct above a threshold.COCO's AP averages over IoU thresholds $0.50, 0.55, \dots, 0.95$ .Taught in: Vision transformers, detection and segmentation
$AP_{50}, AP_{75}, AP_{S}$: “A P fifty, A P seventy-five, A P small”Average precision at IoU threshold 0.50 or 0.75, or restricted to small objects (area below $3 2^{2}$ pixels). Plain “AP” in COCO averages over IoU thresholds and categories.Taught in: Vision transformers, detection and segmentation
$p (Y ∣ X) = \sum_{A \in B^{- 1} (Y)} \prod_{t} p_{t} (a_{t} ∣ X)$: “the CTC probability”Probability of transcript $Y$ : the sum over all frame-level alignments $A$ that collapse to $Y$ (merge repeats, drop blanks) of the product of per-frame probabilities.Taught in: Speech and audio
$WER = (S + D + I) / N$: “word error rate”Substitutions, deletions and insertions in the best word alignment, divided by the number of reference words. Can exceed 1.Taught in: Speech and audio
$r_{k}$: “r k, the autocorrelation at lag k”Sample correlation between a series and itself shifted by $k$ steps.For an AR(1) with coefficient $ϕ$ , the autocorrelation at lag $k$ is $ϕ^{k}$ .Taught in: Time series and forecasting
$ARIMA (p, d, q)$: “ARIMA p d q”An autoregressive model of order $p$ with $q$ moving-average (past error) terms, fitted to the series differenced $d$ times.Taught in: Time series and forecasting
$MASE$: “mean absolute scaled error”Out-of-sample mean absolute error divided by the in-sample mean absolute error of the seasonal-naive forecast (M4 definition).Taught in: Time series and forecasting
$sMAPE = \frac{200}{h} \sum_{j} \frac{∣ y _{j} - y ^ _{j} ∣}{∣ y _{j} ∣ + ∣ y ^ _{j} ∣}$: “symmetric mean absolute percentage error”A percentage error between 0 and 200, as defined in the M4 competition code.Taught in: Time series and forecasting
$NDCG@ k$: “N D C G at k”Discounted cumulative gain of the top $k$ (relevance divided by $lo g_{2} (position + 1)$ ), divided by its ideal value; between 0 and 1.Taught in: Recommender systems
$Recall@ k$: “recall at k”Fraction of a user's relevant items that appear in the top $k$ recommendations.Taught in: Recommender systems
$TP, FP, FN, TN$: “true positives, false positives, false negatives, true negatives”The four cells of a binary confusion matrix: counts of each combination of true label and predicted label.A false negative is a positive example predicted as negative.Taught in: Precision, recall and ROC
$TPR = \frac{TP}{TP + FN}$: “true-positive rate”Recall or sensitivity: the fraction of actual positives the classifier flags. It does not depend on how common positives are.Catching 90 of 100 sick patients gives TPR = 0.9.Taught in: Precision, recall and ROC
$FPR = \frac{FP}{FP + TN}$: “false-positive rate”The fraction of actual negatives wrongly flagged; one minus the specificity. The x-axis of a ROC curve.Flagging 9 of 100 healthy people gives FPR = 0.09.Taught in: Precision, recall and ROC
$precision = \frac{TP}{TP + FP}$: “precision”Of the examples flagged positive, the fraction that really are. It depends on the prevalence.90 true positives among 981 flagged gives precision 0.092.Taught in: Precision, recall and ROC
$F_{β}$: “F beta”A weighted harmonic mean of precision and recall; $β$ says how many times more recall matters than precision. $F_{1}$ weighs them equally. $F_{1} = 2 P R / (P + R)$ .Taught in: Precision, recall and ROC
$AUC = P (s (x^{+}) > s (x^{-}))$: “area under the ROC curve”The probability that a randomly chosen positive gets a higher score than a randomly chosen negative (ties count half).0.5 for random scores, 1 for a perfect ranking.Taught in: Precision, recall and ROC
$R^{2}$: “R squared”One minus the ratio of a regression model's squared error to that of predicting the mean. Can be negative.Predicting the mean everywhere gives $R^{2} = 0$ .Taught in: Precision, recall and ROC
$P (Y = 1 ∣ p^(X) = p) = p$: “calibration”A model is calibrated if, among inputs where it predicts probability $p$ , a fraction $p$ are positive.Of days forecast at 70% rain, it rains on about 70%.Taught in: Calibration and proper scoring rules
$ECE$: “expected calibration error”The bin-size-weighted average gap between confidence and observed frequency (or accuracy) in a reliability diagram. Depends on binning and sample size.ECE = 0.02 means predictions are off by about 2 points on average.Taught in: Calibration and proper scoring rules
$BS = \frac{1}{n} \sum_{i} (q_{i} - y_{i})^{2}$: “Brier score”Mean squared error between predicted probabilities and 0/1 outcomes: a proper scoring rule.Its expectation is $(q - p)^{2} + p (1 - p)$ , smallest at $q = p$ .Taught in: Calibration and proper scoring rules
$w (x) = \frac{p _{T} ( x )}{p _{S} ( x )}$: “importance weight”The density ratio between target and source input distributions; weighting training losses by it corrects for covariate shift.Estimated from a classifier that tells source from target inputs.Taught in: Imbalance and distribution shift
$p_{S}, p_{T}$: “source and target distributions”The distribution a model was trained (and usually tested) on, and the one it is deployed on.Covariate shift: $p_{T} (x) \neq = p_{S} (x)$ but $p_{T} (y ∣ x) = p_{S} (y ∣ x)$ .Taught in: Imbalance and distribution shift
$se (p^) = p^(1 - p^) / n$: “standard error of p hat”The standard deviation of a measured accuracy across test sets of size $n$ . $p^= 0.9$ , $n = 10, 000$ : se = 0.003.Taught in: How sure are you? Bootstrap and tests
$n_{01}, n_{10}$: “discordant counts”In McNemar's test, the numbers of test examples where only model B, or only model A, is correct. Only these carry information about which is better.Under no difference, $n_{01} \sim Binomial (n_{01} + n_{10}, \frac{1}{2})$ .Taught in: How sure are you? Bootstrap and tests
$p_{data} (x), p_{θ} (x)$: “p data of x; p theta of x”The (unknown) distribution the data came from, and the model's distribution with parameters $θ$ . A generative model tries to make $p_{θ}$ close to $p_{data}$ . Diffusion papers write the data side as $q$ .Maximum likelihood minimises $D_{KL} (p_{data} ∥ p_{θ})$ .Taught in: What generative modelling is
$FID$: “Fréchet Inception Distance” $∥ μ_{r} - μ_{g} ∥^{2} + tr (Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2})$ for Gaussians fitted to Inception-network features of real and generated images. Lower is better; depends on sample size and implementation.Two samples from the same distribution give FID near 0 only when the samples are large.Taught in: What generative modelling is
$p_{θ} (x ∣ z)$: “p theta of x given z”A VAE's decoder (generative network): a distribution over data given a latent code.A Bernoulli probability for each pixel, output by a network from $z$ .Taught in: Autoencoders and VAEs
$L (θ, ϕ; x)$: “the ELBO, L of theta and phi at x”The evidence lower bound of a VAE: $E_{q_{ϕ}} [lo g p_{θ} (x ∣ z)] - D_{KL} (q_{ϕ} (z ∣ x) ∥ p (z))$ . It is at most $lo g p_{θ} (x)$ , and the gap is the KL from the encoder to the true posterior.The toy VAE of the lesson reaches a negative ELBO of about 23.9 nats per digit image.Taught in: Autoencoders and VAEs
$W_{1} (p, q)$: “Wasserstein-1 distance, earth mover's distance”The least total mass-times-distance needed to move $p$ onto $q$ ; equivalently the largest $E_{p} [f] - E_{q} [f]$ over 1-Lipschitz $f$ . Used by Wasserstein GANs.A distribution and a copy shifted by $μ$ are $∥ μ ∥$ apart.Taught in: Generative adversarial networks
$β_{t}, α_{t}, α ˉ_{t}$: “beta t, alpha t, alpha bar t”A diffusion model's noise schedule: $β_{t}$ is the noise added at step $t$ , $α_{t} = 1 - β_{t}$ , and $α ˉ_{t} = \prod_{s \leq t} α_{s}$ is the fraction of signal variance left after $t$ steps.DDPM: $β$ rises linearly from $1 0^{- 4}$ to $0.02$ over $T = 1000$ steps, giving $α ˉ_{1000} \approx 4 \times 1 0^{- 5}$ .Taught in: Diffusion models
$\nabla_{x} lo g p (x)$: “the score; grad x of log p of x”The score function: the gradient of the log-density with respect to the data. It does not depend on the normalising constant. (Not the Fisher score $\nabla_{θ} lo g p_{θ}$ .)For $N (μ, σ^{2} I)$ the score is $- (x - μ) / σ^{2}$ .Taught in: Score matching and flow matching
$E_{θ} (x), Z_{θ}$: “the energy E theta of x; the partition function Z theta”An energy-based model defines $p_{θ} (x) = e^{- E_{θ} (x)} / Z_{θ}$ with any network $E_{θ}$ ; $Z_{θ} = \int e^{- E_{θ}}$ is usually intractable.Adding two energies multiplies the densities (a product of experts).Taught in: Score matching and flow matching
$p (x_{t} ∣ x_{< t})$: “p of x t given x before t”The probability of the next symbol $x_{t}$ given everything before it. By the chain rule, a language model that supplies these can assign a probability to any text: $p (x_{1}, \dots, x_{T}) = \prod_{t} p (x_{t} ∣ x_{< t})$ . $p (m ∣ she stepped into the roo)$ is close to 1.Taught in: Predicting the next letter
$c (w_{t - 1}, w_{t})$: “the count of w t minus one followed by w t”How many times a sequence occurs in the training text. An n-gram model estimates $p (w_{t} ∣ w_{t - 1}) = c (w_{t - 1}, w_{t}) / c (w_{t - 1})$ ; add-one smoothing uses $(c + 1) / (c (w_{t - 1}) + V)$ .If 'the' occurs 100 times and 'the cat' 5 times, the bigram estimate of $p (cat ∣ the)$ is 0.05.Taught in: From n-grams to neural language models
$R (h)$: “R of h”True risk (generalization error) of a hypothesis: the probability it misclassifies a fresh example from the data distribution. Written $L_{D} (h)$ in Shalev-Shwartz & Ben-David.A threshold that disagrees with the truth on 5% of inputs has $R (h) = 0.05$ .Taught in: PAC learning
$R^_{S} (h)$: “R hat S of h”Empirical risk (training error): the fraction of the sample $S$ that $h$ gets wrong. Also $L_{S} (h)$ .3 mistakes on 100 examples: $R^_{S} (h) = 0.03$ .Taught in: PAC learning
$(ε, δ)$: “epsilon, delta”PAC accuracy and confidence: with probability at least $1 - δ$ over the sample, the learned hypothesis has error at most $ε$ (or within $ε$ of the best in the class). $ε = 0.05$ , $δ = 0.05$ .Taught in: PAC learning
$n_{H} (ε, δ)$: “n sub H of epsilon, delta”Sample complexity: the number of examples that suffices to learn $H$ to accuracy $ε$ with confidence $1 - δ$ . Written $m_{H}$ in most texts.Finite class, realizable: at most $⌈ ln (∣ H ∣/ δ) / ε ⌉$ .Taught in: PAC learning
$R^_{S} (H)$: “empirical Rademacher complexity of H on S”The expected best correlation between the class's outputs on the sample and independent random signs: how well the class fits noise on your inputs.Any class that fits every labelling of the sample has $R^_{S} = 1$ .Taught in: Rademacher complexity and generalization bounds
$σ_{i} \in {- 1, + 1}$: “sigma i (Rademacher variable)”A fair random sign, $+ 1$ or $- 1$ with probability $\frac{1}{2}$ each.Coin flips used as fake labels to measure capacity.Taught in: Rademacher complexity and generalization bounds
$ε_{app}, ε_{est}$: “approximation error, estimation error”The risk of ERM splits into the best risk achievable in the class (approximation) and the excess due to learning from a sample (estimation).More data shrinks $ε_{est}$ ; only a better class shrinks $ε_{app}$ .Taught in: No free lunch and inductive bias
$r_{i} = y_{i} - y^_{i}$: “r i equals y i minus y hat i”The residual of example $i$ : measured value minus prediction. Positive when the point lies above the fitted line.If $y_{i} = 5$ and $y^_{i} = 4.2$ , then $r_{i} = 0.8$ .Taught in: Linear regression
$lo g \frac{p}{1 - p}$: “the log-odds, or logit, of p”The inverse of the sigmoid. Logistic regression models the log-odds of class 1 as a linear function, $w^{⊤} x + b$ . $p = 0.5$ gives log-odds 0; $p = 0.9$ gives $lo g 9 \approx 2.2$ .Taught in: Logistic regression and softmax
$σ^{2} + bias^{2} + variance$: “noise plus bias squared plus variance”The decomposition of expected squared test error at an input: irreducible noise, the squared gap between the average prediction and the truth, and the spread of predictions across training sets.A degree-11 polynomial through 12 noisy points: tiny bias, enormous variance.Taught in: Generalization, bias–variance and regularization
$f ˉ (x) = E_{D} [f^_{D} (x)]$: “f bar of x, the average prediction”The model's prediction at $x$ averaged over all training sets $D$ it might have been trained on. Bias compares it with the truth; variance measures spread around it.Taught in: Generalization, bias–variance and regularization
$K -fold CV$: “K-fold cross-validation”Split the training data into $K$ folds; train on $K - 1$ , validate on the remaining one, rotate through all $K$ , and average. Used to choose hyperparameters such as $λ$ without touching the test set. $K = 5$ with 30 examples: five rounds, each validating on 6 examples.Taught in: Generalization, bias–variance and regularization
$q_{i} = \frac{e ^{z_{i} / T}}{\sum _{j} e ^{z_{j} / T}}$: “softmax with temperature T”Softmax of logits $z$ at temperature $T$ . In distillation the teacher's outputs at a high $T$ are the student's soft targets; higher $T$ gives a flatter distribution.Hinton et al. multiply the soft-target gradients by $T^{2}$ because they shrink like $1/ T^{2}$ .Taught in: Quantization and distillation
$p_{η} (z ∣ x)$: “p eta of z given x”In RAG, the retriever: a distribution over passages $z$ given the input $x$ , proportional to $exp (d (z)^{⊤} q (x))$ . Paired with a generator $p_{θ} (y ∣ x, z)$ .RAG-Sequence: $p (y ∣ x) \approx \sum_{z \in top- k} p_{η} (z ∣ x) p_{θ} (y ∣ x, z)$ .Taught in: Embeddings, retrieval and RAG
$idf (w)$: “inverse document frequency of w”A weight that grows as a word gets rarer in the collection; in Lucene's BM25, $lo g (1 + \frac{N - n _{w} + 0.5}{n _{w} + 0.5})$ for $n_{w}$ of $N$ documents containing $w$ .A word in every document gets an idf near 0; a word in one document of 1,000 gets about 6.5.Taught in: Embeddings, retrieval and RAG
$pass^{k}$: “pass hat k”The chance that all $k$ independent attempts at a task succeed, averaged over tasks; estimated as $(k c) / (k n)$ from $c$ successes in $n$ trials. A reliability metric, unlike pass@k.A task solved 7 times in 10 has pass@1 = 0.7 but pass^8 = 0.Taught in: Agents and tool use
$SE = s ˉ (1 - s ˉ) / n$: “standard error of a benchmark score”The standard error of an accuracy $s ˉ$ measured on $n$ questions scored 0 or 1. A 95% interval is about $\pm 1.96 SE$ .70% on 1,000 questions: $SE \approx 1.4$ points, so $\pm 2.8$ points at 95%.Taught in: Evaluating language models
$σ (ξ_{m} - ξ_{m^{'}})$: “sigma of xi m minus xi m prime”The Bradley–Terry probability that model $m$ beats model $m^{'}$ in a pairwise vote, from their strengths $ξ$ . Chatbot Arena fits the $ξ$ by logistic regression on votes.Equal strengths give a 50% win probability; a difference of 1 gives about 73%.Taught in: Evaluating language models
$F (θ) = E [\nabla lo g p_{θ} \nabla lo g p_{θ}^{⊤}]$: “the Fisher information matrix”The expected outer product of the score, with outcomes drawn from the model; equals minus the expected Hessian of the log-likelihood. Measures how fast the model's distribution changes with its parameters.For $N (μ, σ^{2})$ in $(μ, σ)$ : $F = diag (1/ σ^{2}, 2/ σ^{2})$ .Taught in: Newton, quasi-Newton and natural gradients
$μ_{k}$: “mu k”The centroid of cluster $k$ in k-means: the mean of the points assigned to it.Taught in: Three kinds of learning signal
$p (θ ∣ D)$: “p of theta given D”The posterior: your belief about the parameters after seeing the data; proportional to likelihood times prior.With a flat prior and 4 heads in 4 flips, $p (θ ∣ D) = Beta (θ ∣ 5, 1)$ .Taught in: Bayesian inference
$p (D)$: “p of D, the evidence”The evidence or marginal likelihood $\int p (D ∣ θ) p (θ) d θ$ : the probability a model gave the data, averaged over its prior. Normalises the posterior and scores whole models.A fair-coin model gives any sequence of 10 flips evidence $0. 5^{10} \approx 9.8 \times 1 0^{- 4}$ .Taught in: Bayesian inference
$Beta (θ ∣ a, b)$: “beta of theta with a and b”A distribution over a probability $θ \in (0, 1)$ ; conjugate to Bernoulli data, so heads add to $a$ and tails to $b$ (pseudo-counts). $Beta (2, 2) \to Beta (9, 5)$ after 7 heads and 3 tails; mean $9/14$ .Taught in: Bayesian inference
$p (y^{*} ∣ D)$: “p of y star given D, the posterior predictive”The probability of a new observation, averaging the model's prediction over the posterior: $\int p (y^{*} ∣ θ) p (θ ∣ D) d θ$ .For a coin with a flat prior: $(k + 1) / (n + 2)$ , Laplace's rule of succession.Taught in: Bayesian inference
$x_{i} ∣ x_{pa (i)}$: “x i given its parents”A node's conditional distribution in a Bayes net, given the values of its parents; the joint is the product of these over all nodes. $p (d, c, s) = p (d) p (c) p (s ∣ d, c)$ .Taught in: Graphical models
$A ⊥ B ∣ C$: “A is independent of B given C”Conditional independence: once $C$ is known, learning $B$ tells you nothing more about $A$ . Also written with a double bar, $A ⊥ ⊥ B ∣ C$ .In a chain $X \to Y \to Z$ : $X ⊥ Z ∣ Y$ .Taught in: Graphical models
$ψ_{c} (x_{c})$: “psi c”A potential (factor): a non-negative compatibility score on the variables of clique $c$ in a Markov random field.Ising model: $ψ_{ij} = e^{J}$ if neighbouring spins agree, $e^{- J}$ if not.Taught in: Graphical models
$Z$: “Z, the partition function”The normalising constant of an undirected model: the sum over all configurations of the product of potentials. $p (x) = \frac{1}{Z} \prod_{c} ψ_{c} (x_{c})$ .Taught in: Graphical models
$m_{s \to t} (x_{t})$: “the message from s to t”In belief propagation, what the part of the graph behind node $s$ says about node $t$ 's value.On a tree, two sweeps of messages give every node's exact marginal.Taught in: Graphical models
$α_{t} (j)$: “alpha t of j”The forward (filtering) probability in an HMM: $p (z_{t} = j ∣ y_{1 : t})$ (some texts use the unnormalised joint $p (z_{t} = j, y_{1 : t})$ ).In the lesson's example, after hearing silence the state Asleep has $α_{1} = 0.784$ .Taught in: Latent variables, HMMs and Kalman filters
$δ_{t} (j)$: “delta t of j”Viterbi score: the probability of the best hidden path ending in state $j$ at time $t$ , jointly with the observations so far. $δ_{t} (j) = B_{j, y_{t}} max_{i} δ_{t - 1} (i) A_{ij}$ .Taught in: Latent variables, HMMs and Kalman filters
$A_{ij}, B_{j k}$: “A i j, B j k”HMM transition probability $p (z_{t} = j ∣ z_{t - 1} = i)$ and emission probability $p (y_{t} = k ∣ z_{t} = j)$ .Each row of $A$ and of $B$ sums to 1.Taught in: Latent variables, HMMs and Kalman filters
$K_{t}$: “K t, the Kalman gain”How much of the innovation (observed minus expected reading) the Kalman filter adds to its state estimate.1-D: prediction variance 4, sensor variance 1 gives $K = 0.8$ .Taught in: Latent variables, HMMs and Kalman filters
$μ_{t ∣ t - 1}, Σ_{t ∣ t - 1}$: “mu and sigma, t given t minus 1”Mean and covariance of the predicted state at time $t$ using observations up to $t - 1$ (also written $m_{t}^{-}$ , $P_{t}^{-}$ ). $Σ_{t ∣ t - 1} = F Σ_{t - 1} F^{⊤} + Q$ .Taught in: Latent variables, HMMs and Kalman filters
$ELBO (q)$: “the ELBO”The evidence lower bound $E_{q} [lo g p (x, z) - lo g q (z)]$ ; equals $lo g p (x)$ minus $KL (q ∥ p (z ∣ x))$ .Maximising it over $q$ is variational inference.Taught in: Variational inference
$q_{ϕ} (z ∣ x)$: “q phi of z given x”A variational (approximate) posterior with parameters $ϕ$ ; amortised when a network outputs $ϕ$ from $x$ .A diagonal Gaussian $N (μ_{ϕ} (x), diag σ_{ϕ}^{2} (x))$ in a VAE.Taught in: Variational inference
$z = μ + σ ⊙ ϵ$: “z equals mu plus sigma times epsilon”The reparameterisation trick: write a sample as a differentiable function of parameters and parameter-free noise $ϵ \sim N (0, I)$ , so gradients flow through it.torch.distributions.Normal(mu, s).rsample().Taught in: Variational inference
$A (θ^{'} ∣ θ)$: “A of theta prime given theta”Metropolis–Hastings acceptance probability $min (1, p ~ (θ^{'}) q (θ ∣ θ^{'}) / [p ~ (θ) q (θ^{'} ∣ θ)])$ ; the normalising constant cancels.Symmetric proposal: accept uphill moves always, downhill ones with probability $p ~ (θ^{'}) / p ~ (θ)$ .Taught in: Markov chain Monte Carlo
$R^$: “R hat”Potential scale reduction factor: compares between-chain and within-chain variance of several MCMC chains; near 1 when they agree.Current advice (Vehtari et al., 2021): use the sample only if $R^< 1.01$ .Taught in: Markov chain Monte Carlo
$H (θ, v)$: “H of theta and v, the Hamiltonian”Total energy in HMC: potential $- lo g p ~ (θ)$ plus kinetic $\frac{1}{2} v^{⊤} v$ .Leapfrog nearly conserves $H$ , so HMC proposals are usually accepted.Taught in: Markov chain Monte Carlo
$f \sim G P (m, k)$: “f is drawn from a Gaussian process with mean m and kernel k”A distribution over functions: the values at any finite set of inputs are jointly Gaussian with means $m (x_{i})$ and covariances $k (x_{i}, x_{j})$ .Zero mean and an SE kernel give smooth random functions.Taught in: Gaussian processes
$k (x, x^{'})$: “k of x and x prime, the kernel”A positive-definite covariance function: how correlated the function values at two inputs are. Choosing it is choosing a prior over functions.SE kernel: $σ_{f}^{2} exp (- ∥ x - x^{'} ∥^{2} /2 ℓ^{2})$ .Taught in: Gaussian processes
$ℓ$: “ell, the length-scale”In a stationary kernel, how far apart two inputs must be before their function values become nearly unrelated.Short $ℓ$ : wiggly functions. Long $ℓ$ : nearly straight ones.Taught in: Gaussian processes
$U ⊥ V ∣ W$: “U is independent of V given W”Conditional independence: once $W$ is known, $V$ tells you nothing more about $U$ . Group-fairness criteria are written this way.Separation (equalized odds): $R ⊥ A ∣ Y$ .Taught in: Fairness
$P (Y^= 1 ∣ A = a)$: “the probability of a positive decision in group a”A group's selection rate; demographic parity asks that it be equal across groups.Equal selection rates for two groups of loan applicants.Taught in: Fairness
$FPR = \frac{p}{1 - p} \cdot \frac{1 - PPV}{PPV} \cdot (1 - FNR)$: “Chouldechova's identity”Links a group's false-positive rate to its prevalence $p$ , positive predictive value and false-negative rate; with equal PPV and FNR, different prevalences force different FPRs.Prevalences 0.51 and 0.39 force an FPR ratio of about 1.63.Taught in: Fairness
$(ε, δ) -DP$: “epsilon delta differential privacy”For all neighbouring datasets and all output sets, $P (M (D) \in S) \leq e^{ε} P (M (D^{'}) \in S) + δ$ .Abadi et al. trained MNIST to 97% accuracy at $(8, 1 0^{- 5})$ -DP.Taught in: Privacy and differential privacy
$Δ f$: “the sensitivity of f”The largest change in a query's answer caused by changing one record. Sets the scale of privacy noise.A count has $Δ f = 1$ .Taught in: Privacy and differential privacy
$Lap (b)$: “Laplace with scale b”The Laplace distribution with density $\frac{1}{2 b} e^{- ∣ z ∣ / b}$ ; adding $Lap (Δ f / ε)$ noise gives $ε$ -DP. $ε = 1$ on a count: noise scale $b = 1$ .Taught in: Privacy and differential privacy
$σ_{*}^{2} (x) = \frac{1}{M} \sum_{m} σ_{m}^{2} + \frac{1}{M} \sum_{m} (μ_{m} - μ_{*})^{2}$: “ensemble predictive variance”A deep ensemble's variance: average predicted noise (aleatoric) plus disagreement between members (epistemic).Far from the training data the second term dominates.Taught in: Uncertainty in deep learning
$q^$: “q hat”In split conformal prediction: the $⌈(n + 1) (1 - α)⌉$ -th smallest calibration score, used as the threshold for prediction sets. $n = 50$ , $α = 0.1$ : the 46th smallest score.Taught in: Uncertainty in deep learning
$1 - α \leq P (Y_{test} \in C (X_{test})) \leq 1 - α + \frac{1}{n + 1}$: “the conformal coverage guarantee”Split-conformal prediction sets cover the true label with probability at least $1 - α$ (marginally, for exchangeable data), whatever the model.90% intervals from 200 calibration points.Taught in: Uncertainty in deep learning
$p / n$: “p over n”Parameters (or features) per training example; the interpolation threshold of least-squares models is at $p / n = 1$ . Often written $γ$ or $ψ$ .20 random features for 20 points: $p / n = 1$ , the peak of double descent.Taught in: Double descent
$p (N)^{L}$: “p of N to the L”Exact-match accuracy on an $L$ -token answer if each token is right with probability $p (N)$ independently — a sharp function of a smooth one. $0. 9^{10} \approx 0.35$ but $0.9 9^{10} \approx 0.90$ .Taught in: Scaling laws and their limits
$Ω$: “omega”The sample space: the set of all possible outcomes of an experiment.Two dice: 36 outcomes.Taught in: Probability as bookkeeping
$P (E)$: “probability of E”Probability of an event $E$ . Non-negative, $P (Ω) = 1$ , additive over disjoint events (Kolmogorov's axioms). $P (two dice sum to 7) = 1/6$ .Taught in: Probability as bookkeeping
$X \sim p$: “X is distributed as p”The random variable $X$ has distribution $p$ (a pmf or pdf). $x \sim N (0, 1)$ .Taught in: Probability as bookkeeping
$p (x, y)$: “p of x and y”Joint distribution. Sum rule: $p (x) = \sum_{y} p (x, y)$ . Product rule: $p (x, y) = p (y ∣ x) p (x)$ .Independence means $p (x, y) = p (x) p (y)$ .Taught in: Probability as bookkeeping
$p (y ∣ x)$: “p of y given x”Conditional distribution of $y$ once $x$ is known: $p (x, y) / p (x)$ .A classifier outputs $p (class ∣ x)$ .Taught in: Probability as bookkeeping
$p (h ∣ d) = \frac{p ( d ∣ h ) p ( h )}{p ( d )}$: “Bayes' rule”Posterior = likelihood × prior / evidence.A positive test for a rare condition can still mean a low posterior.Taught in: Probability as bookkeeping
$E [X]$: “expected value of X”The probability-weighted average of $X$ : $\sum_{x} x p (x)$ or $\int x p (x) d x$ . Linear.A fair die has $E [X] = 3.5$ .Taught in: Probability as bookkeeping
$E_{x \sim p} [f (x)]$: “expectation of f of x, x drawn from p”Average of $f$ over samples from $p$ ; the subscript names the distribution.The training loss is $E_{(x, y) \sim data} [ℓ]$ .Taught in: Probability as bookkeeping
$Var [X]$: “variance of X”Average squared distance from the mean, $E [(X - μ)^{2}] = E [X^{2}] - μ^{2}$ . Square root: standard deviation $σ$ .Variance of an average of $n$ i.i.d. draws: $σ^{2} / n$ .Taught in: Probability as bookkeeping
$Cov [X, Y]$: “covariance of X and Y” $E [(X - μ_{X}) (Y - μ_{Y})]$ : how two variables move together. Zero covariance does not imply independence. $Cov [X, X] = Var [X]$ .Taught in: Probability as bookkeeping
$Σ$: “capital sigma (covariance matrix)”Covariance matrix of a random vector: $E [(x - μ) (x - μ)^{⊤}]$ . Symmetric, positive semidefinite. (In an SVD, $Σ$ instead holds singular values.)Its eigenvectors are the axes of a Gaussian's density ellipses.Taught in: Probability as bookkeeping
$N (μ, σ^{2})$: “normal with mean mu and variance sigma squared”The Gaussian distribution. The second argument is the variance (some software takes the standard deviation instead). $N (0, 1)$ puts about 95% of its probability within $\pm 2$ .Taught in: Probability as bookkeeping
$N (μ, Σ)$: “multivariate normal with mean mu and covariance Sigma”The multivariate Gaussian; density $\propto exp (- \frac{1}{2} (x - μ)^{⊤} Σ^{- 1} (x - μ))$ .Contours of constant density are ellipses.Taught in: Probability as bookkeeping
$i.i.d.$: “independent and identically distributed”Each sample drawn independently from the same distribution; makes a dataset's likelihood a product.The CLT and LLN are stated for i.i.d. samples.Taught in: Probability as bookkeeping
$X ˉ_{n}$: “X bar n”The average of the first $n$ samples. By the LLN it converges to $μ$ ; by the CLT it is approximately $N (μ, σ^{2} / n)$ .The average of 100 die rolls has standard deviation about $0.17$ .Taught in: Probability as bookkeeping
$L (θ) = p (D ∣ θ)$: “the likelihood of theta”The likelihood: probability of the observed data as a function of the parameters. Not a distribution over $θ$ .For $k$ heads in $n$ flips, $L (θ) = θ^{k} (1 - θ)^{n - k}$ .Taught in: Maximum likelihood: where losses come from
$ℓ (θ) = lo g L (θ)$: “the log-likelihood”Log of the likelihood: same maximiser, turns products into sums, avoids underflow. Minus it is the NLL. $ℓ (θ) = k lo g θ + (n - k) lo g (1 - θ)$ .Taught in: Maximum likelihood: where losses come from
$θ^_{MLE}$: “theta hat M L E”Maximum-likelihood estimate: the parameters that make the observed data most probable.For a coin, $θ^_{MLE} = k / n$ .Taught in: Maximum likelihood: where losses come from
$θ^_{MAP}$: “theta hat M A P”Maximum a posteriori estimate: the peak of likelihood × prior. A Gaussian prior on weights gives an L2 penalty.With a Beta $(a, b)$ prior: $(k + a - 1) / (n + a + b - 2)$ .Taught in: Maximum likelihood: where losses come from
$p_{θ} (x_{t} ∣ x_{< t})$: “p theta of x t given x before t”A language model's next-token distribution given the preceding tokens; multiplying these over $t$ gives the probability of a whole sequence.Training minimises $- lo g p_{θ} (x_{t + 1} ∣ x_{\leq t})$ averaged over positions.Taught in: GPT and next-token prediction
$s (i) = \frac{b ( i ) - a ( i )}{m a x { a ( i ) , b ( i )}}$: “s of i”The silhouette of point $i$ : $a (i)$ is its mean distance to its own cluster, $b (i)$ to the nearest other cluster. Between $- 1$ and $1$ . $a = 1$ , $b = 4$ gives $s = 0.75$ .Taught in: k-means, properly
$p (x) = \sum_{k = 1}^{K} π_{k} N (x ∣ μ_{k}, Σ_{k})$: “p of x equals the sum over k of pi k times a Gaussian”A Gaussian mixture: $K$ Gaussians with weights $π_{k}$ that sum to 1.Two components with $π = (0.7, 0.3)$ .Taught in: Gaussian mixtures and EM
$π_{k}$: “pi k”A mixing weight: the prior probability that a point comes from component $k$ . $π_{2} = 0.3$ : 30% of points come from component 2.Taught in: Gaussian mixtures and EM
$r_{ik}$: “r i k”The responsibility: posterior probability that component $k$ generated point $i$ . Sums to 1 over $k$ . $r_{i 1} = 0.9, r_{i 2} = 0.1$ .Taught in: Gaussian mixtures and EM
$N_{k} = \sum_{i} r_{ik}$: “N k”The effective number of points in component $k$ during EM.Responsibilities $0.5, 0.5, 1$ give $N_{k} = 2$ .Taught in: Gaussian mixtures and EM
$ELBO$: “ELBO”Evidence lower bound: $\sum_{z} q (z) lo g \frac{p ( x , z )}{q ( z )} \leq lo g p (x)$ , with equality when $q$ is the posterior.Taught in: Gaussian mixtures and EM
$BIC = - 2 ℓ (θ^) + p lo g n$: “B I C”Bayesian information criterion (scikit-learn's sign convention: lower is better); $p$ is the number of free parameters.Taught in: Gaussian mixtures and EM
$p_{j ∣ i}$: “p of j given i”In SNE/t-SNE: the probability that point $i$ picks $j$ as its neighbour, from a Gaussian of width $σ_{i}$ .Taught in: t-SNE, UMAP and manifold learning
$q_{ij}$: “q i j”In t-SNE: the similarity of map points $i$ and $j$ under a Student-t (Cauchy) kernel, normalised over all pairs.Taught in: t-SNE, UMAP and manifold learning
$p^_{h} (x) = \frac{1}{nh} \sum_{i} K (\frac{x - x _{i}}{h})$: “p hat h of x”The kernel density estimator with kernel $K$ and bandwidth $h$ .Taught in: Density estimation
$h$: “h”The bandwidth of a kernel density estimate: bias grows like $h^{2}$ , variance like $1/ (nh)$ .Silverman: $h = 0.9 min (σ^, IQR /1.34) n^{- 1/5}$ .Taught in: Density estimation
$Δ^{2} = (x - μ)^{⊤} Σ^{- 1} (x - μ)$: “delta squared”Squared Mahalanobis distance; for Gaussian data it follows a chi-square distribution with $d$ degrees of freedom.Taught in: Anomaly detection
$LOF_{k} (p)$: “L O F k of p”Local outlier factor: average ratio of the neighbours' local densities to the point's own. About 1 for inliers.Taught in: Anomaly detection
$s (x, ψ) = 2^{- E [h (x)] / c (ψ)}$: “s of x”Isolation-forest anomaly score from the average path length $h$ : near 1 means anomalous, well below 0.5 normal; if every point scores about 0.5, there are no distinct anomalies.Taught in: Anomaly detection

Information theory

$lo g x$: “log x”In ML papers, the natural logarithm (base $e$ ) unless stated otherwise.Taught in: Reading the symbols
$lo g_{2} x$: “log base 2 of x”The logarithm used when information is measured in bits. $lo g_{2} 8 = 3$ bits.Taught in: Reading the symbols
$exp (x)$: “exp of x” $e^{x}$ , written this way when the exponent is long.Taught in: Reading the symbols
$IG$: “information gain”The drop in entropy from a parent node to the weighted average entropy of its children after a split.Splitting (5, 5) into (4, 1) and (1, 4) gains $1 - 0.722 = 0.278$ bits.Taught in: Decision trees
$L_{InfoNCE} = - E [lo g \frac{f ( x , c )}{\sum _{x^{'}} f ( x ^{'} , c )}]$: “the InfoNCE loss”Cross-entropy for picking the true partner of a context among negatives; CLIP applies it in both directions over a batch.Taught in: Multimodal models
$bpd$: “bits per dimension”A model's negative log-likelihood of an image in bits, divided by the number of values in it (pixels times channels). Lower is better.5,000 nats for a $32 \times 32 \times 3$ image is $5000/ (3072 ln 2) \approx 2.35$ bits per dimension.Taught in: What generative modelling is
$JSD (p ∥ q)$: “Jensen–Shannon divergence of p and q” $\frac{1}{2} D_{KL} (p ∥ m) + \frac{1}{2} D_{KL} (q ∥ m)$ with $m = (p + q) /2$ . Symmetric, between 0 and $lo g 2$ .Against a perfect discriminator, a GAN's generator minimises $2 JSD - lo g 4$ .Taught in: Generative adversarial networks
$lo g_{2} \frac{1}{p ( x )}$: “log base two of one over p of x”The surprise (Shannon information content, surprisal) of an outcome with probability $p (x)$ , in bits. Equal to $- lo g_{2} p (x)$ .An outcome with $p = \frac{1}{8}$ carries $lo g_{2} 8 = 3$ bits; a certain outcome carries 0.Taught in: Surprise, bits and entropy
$H (X)$: “the entropy of X”Average surprise of a random variable: $\sum_{x} p (x) lo g_{2} \frac{1}{p ( x )}$ bits. A lower bound on the average number of bits (yes/no questions) per outcome of any lossless code, approached by coding long blocks. Also written $H (p)$ or $H [X]$ .A fair coin has $H = 1$ bit; a coin with $p = 0.9$ has $H \approx 0.469$ bits.Taught in: Surprise, bits and entropy
$H (p)$: “H of p (binary entropy)”For a single number $p$ : the entropy of a coin that lands heads with probability $p$ , $p lo g_{2} \frac{1}{p} + (1 - p) lo g_{2} \frac{1}{1 - p}$ . For a distribution $p$ : its entropy. $H (\frac{1}{2}) = 1$ bit, the maximum for two outcomes.Taught in: Surprise, bits and entropy
$bit, nat$: “bit, nat”Units of information: a bit uses $lo g_{2}$ , a nat uses the natural log $ln$ . 1 nat $= 1/ ln 2 \approx 1.443$ bits. ML losses are usually in nats.A loss of 2.0 nats per token is $2.0/ ln 2 \approx 2.885$ bits per token.Taught in: Surprise, bits and entropy
$F_{N}$: “F sub N”Shannon's N-gram entropy: the average uncertainty (conditional entropy) of the next symbol when the previous $N - 1$ symbols are known. It decreases with $N$ toward the entropy rate of the source.Shannon (1951), 26 letters: $F_{0} = 4.70$ , $F_{1} = 4.14$ , $F_{2} = 3.56$ bits per letter.Taught in: Predicting the next letter
$H (p, q)$: “the cross-entropy of p and q”Average code length (bits) when outcomes come from $p$ but are coded with a code built for $q$ : $\sum_{x} p (x) lo g_{2} \frac{1}{q ( x )}$ . Measured on data, it is the average negative log-likelihood: the training loss of language models.Shannon's code for $(\frac{1}{2}, \frac{1}{4}, \frac{1}{8}, \frac{1}{8})$ used on a uniform 4-letter source costs $H = 2.25$ bits per letter.Taught in: Cross-entropy, KL and perplexity
$D_{KL} (p ∥ q)$: “the KL divergence of p from q”Extra bits paid for coding $p$ with $q$ 's code: $H (p, q) - H (p) = \sum_{x} p (x) lo g_{2} \frac{p ( x )}{q ( x )}$ . Never negative (Gibbs' inequality), zero only when $q = p$ , not symmetric, infinite if $q (x) = 0$ where $p (x) > 0$ . Also called relative entropy.For $p = (0.9, 0.1)$ , $q = (0.5, 0.5)$ : $D_{KL} (p ∥ q) \approx 0.531$ bits but $D_{KL} (q ∥ p) \approx 0.737$ bits.Taught in: Cross-entropy, KL and perplexity
$PPL$: “perplexity”Perplexity: $2^{cross-entropy in bits} = e^{loss in nats}$ , the inverse probability of a text normalised by its length. The effective number of equally likely choices per symbol (a weighted average branching factor). Only comparable between models with the same vocabulary.A model with a loss of 2.0 nats per token has perplexity $e^{2} \approx 7.39$ .Taught in: Cross-entropy, KL and perplexity
$D_{KL} (Q ∥ P)$: “KL divergence from prior P to posterior Q”In PAC-Bayes bounds: the price a posterior over hypotheses pays for moving away from a prior fixed before seeing the data. $R (Q) \leq R^_{S} (Q) + (D_{KL} (Q ∥ P) + ln (n / δ)) / (2 (n - 1))$ .Taught in: Rademacher complexity and generalization bounds
$- [y lo g μ + (1 - y) lo g (1 - μ)]$: “binary cross-entropy”The negative log-likelihood of a label $y \in {0, 1}$ under a Bernoulli model that predicts $μ = p (y = 1 ∣ x)$ . Its gradient with respect to the score is $μ - y$ . $y = 1$ , $μ = 0.02$ : loss $- lo g 0.02 \approx 3.9$ .Taught in: Logistic regression and softmax
$BPB$: “bits per byte”A language model's total negative log-likelihood of a text, in bits, divided by the text's length in bytes. Unlike per-token perplexity it does not depend on the tokenizer. $BPB = \frac{N _{tokens}}{N _{bytes}} lo g_{2} PPL$ .Taught in: Evaluating language models
$E$: “E”Irreducible loss: the loss of an ideal predictor, identified with the entropy of the data; no model or dataset size can go below it. $E = 1.69$ nats per token in Chinchilla's fit to its own data.Taught in: Scaling laws and their limits
$Perp (P_{i}) = 2^{H (P_{i})}$: “perplexity”The effective number of neighbours of point $i$ in t-SNE: two to the entropy (in bits) of its neighbour distribution.A uniform choice among 30 neighbours has perplexity 30.Taught in: t-SNE, UMAP and manifold learning

Optimization

$η$: “eta”The learning rate: the step size of gradient descent, chosen by hand. $θ \leftarrow θ - η \nabla_{θ} L$ Taught in: Reading the symbols
$β_{1}, β_{2}$: “beta one, beta two”In Adam, the decay rates of the running averages of the gradient and its square (defaults $0.9$ and $0.999$ ).Taught in: Reading the symbols
$min_{x} f (x)$: “the minimum over x of f of x”The smallest value $f$ takes.For $f (x) = (x - 3)^{2} + 1$ : $min_{x} f (x) = 1$ .Taught in: Reading the symbols
$arg min_{x} f (x)$: “the arg-min over x of f of x”Where the minimum is attained — the input, not the value. Strictly a set, since several inputs can tie.For $f (x) = (x - 3)^{2} + 1$ : $arg min_{x} f (x) = 3$ .Taught in: Reading the symbols
$arg max_{k} p (k ∣ x)$: “the arg-max over k”The $k$ with the largest value — for a classifier, the predicted class.Taught in: Reading the symbols
$C_{α} (T)$: “C alpha of T”Cost-complexity of a subtree $T$ : training loss plus $α$ times the number of leaves. Pruning minimises it.Larger $α$ prunes to smaller trees; $α$ is chosen by cross-validation.Taught in: Decision trees
$F_{m} = F_{m - 1} + ν h_{m}$: “F m equals F m minus one plus nu h m”One stage of boosting: add a shrunken new base learner $h_{m}$ to the model built so far.With $ν = 0.1$ , each tree moves predictions a tenth of the way it wants to.Taught in: Gradient boosting
$r_{im}$: “r i m”Pseudo-residual: the negative gradient of the loss with respect to the model's output for example $i$ at stage $m$ . For squared error it is the ordinary residual.For log loss, $r_{im} = y_{i} - p_{i}$ .Taught in: Gradient boosting
$w_{j}^{*} = - \frac{G _{j}}{H _{j} + λ}$: “w j star equals minus G j over H j plus lambda”XGBoost's optimal leaf value from a second-order expansion: sum of gradients over sum of Hessians plus the L2 penalty.For squared loss this is the leaf's summed residual divided by (number of examples + $λ$ ).Taught in: Gradient boosting
$α_{i}$: “alpha i”In SVMs and kernel methods: the dual variable (Lagrange multiplier) of training example $i$ . Nonzero only for support vectors in an SVM. $w = \sum_{i} α_{i} y_{i} x_{i}$ .Taught in: Support vector machines and kernels
$ξ_{i}$: “xi i”Slack variable: how far example $i$ falls short of its margin in a soft-margin SVM. $ξ_{i} > 1$ means example $i$ is misclassified.Taught in: Support vector machines and kernels
$max (0, 1 - y f (x))$: “hinge loss”The SVM's loss: zero beyond the margin, growing linearly inside it or on the wrong side.A point with $y f (x) = 0.4$ has hinge loss $0.6$ .Taught in: Support vector machines and kernels
$min_{G} max_{D} V (D, G)$: “min over G, max over D, of V”A two-player minimax game: the discriminator maximises the value function, the generator minimises it. The original GAN objective. $V = E_{data} [lo g D (x)] + E_{z} [lo g (1 - D (G (z)))]$ .Taught in: Generative adversarial networks
$h^_{S} \in arg min_{h \in H} R^_{S} (h)$: “h hat S is an empirical risk minimiser”Empirical risk minimisation (ERM): pick a hypothesis in the class with the smallest training error.The threshold with the fewest training mistakes.Taught in: PAC learning
$Reg_{T}$: “regret after T rounds”Online learning: the learner's total loss minus that of the best fixed expert (or action, or parameter vector) in hindsight.Exponential weights: $Reg_{T} \leq 2 T ln N$ .Taught in: Online learning and bandits
$p_{t, i} \propto e^{- η L_{t - 1, i}}$: “p t i proportional to e to the minus eta L”Exponential weights (Hedge): follow expert $i$ with probability decreasing exponentially in its cumulative loss. $η = 2 ln N / T$ balances the two terms of the regret bound.Taught in: Online learning and bandits
$∥ y - X θ ∥^{2}$: “the squared norm of y minus X theta”The sum of squared residuals (SSE, also RSS). Dividing by $n$ gives the mean squared error (MSE); both have the same minimiser.Residuals $(1, - 2, 0)$ give $1 + 4 + 0 = 5$ .Taught in: Linear regression
$θ_{k + 1} = θ_{k} - η \nabla_{θ} L (θ_{k})$: “theta k plus one equals theta k minus eta times grad L at theta k”One step of gradient descent: move against the gradient, scaled by the learning rate $η$ .Least squares: $\nabla L = \frac{2}{n} X^{⊤} (X θ - y)$ .Taught in: Gradient descent
$0 < η < 2/ λ_{m a x}$: “eta between zero and two over lambda max”The learning rates for which gradient descent converges on a quadratic loss, where $λ_{m a x}$ is the largest eigenvalue of the Hessian.Hessian eigenvalues 1 and 50: any $η < 0.04$ converges; $η = 0.05$ diverges.Taught in: Gradient descent
$B, ∣ B ∣$: “the minibatch B and its size”A random subset of training examples used to estimate the gradient in stochastic / minibatch gradient descent. $∣ B ∣ = 1$ is SGD in the original sense; $∣ B ∣ = n$ is full-batch gradient descent. $n = 1000$ and $∣ B ∣ = 32$ : about 32 updates per epoch.Taught in: Gradient descent
$(X^{⊤} X + λ I)^{- 1} X^{⊤} y$: “the ridge solution”Minimiser of $∥ y - X θ ∥^{2} + λ ∥ θ ∥_{2}^{2}$ . The matrix is invertible for every $λ > 0$ ; each singular direction is shrunk by $σ_{j}^{2} / (σ_{j}^{2} + λ)$ . $λ = 0$ recovers ordinary least squares.Taught in: Generalization, bias–variance and regularization
$λ ∥ θ ∥_{1}$: “lambda times the L1 norm of theta”The lasso penalty, the sum of absolute values of the weights. It sets some coefficients exactly to zero; there is no closed-form solution in general. $∥(3, 0, - 2) ∥_{1} = 5$ .Taught in: Generalization, bias–variance and regularization
$sign (z) max (∣ z ∣ - τ, 0)$: “soft thresholding of z at tau”Shrinks $z$ toward zero by $τ$ and sets it exactly to zero if $∣ z ∣ \leq τ$ . The lasso solution when the features are orthonormal, and the step used by coordinate-descent lasso solvers. $τ = 1$ : $2.5 \mapsto 1.5$ , $- 0.4 \mapsto 0$ .Taught in: Generalization, bias–variance and regularization
$16Ψ$: “sixteen psi (bytes)”GPU memory for the model states of mixed-precision Adam training: 2 bytes of weights, 2 of gradients and 12 of fp32 optimizer state per parameter, for $Ψ$ parameters (ZeRO's notation).A 7B model needs about 112 GB of model states before any activations.Taught in: Fine-tuning and LoRA
$\frac{p - 1}{m + p - 1}$: “p minus one over m plus p minus one”The fraction of a pipeline-parallel training step spent idle (the bubble), with $p$ pipeline stages and $m$ microbatches per step. $p = 4$ , $m = 8$ : $3/11 \approx 27%$ idle; with $m = 32$ , about 9%.Taught in: Training at scale
$MFU$: “model FLOPs utilization”Observed training throughput as a fraction of what the hardware's peak FLOP/s would allow, counting only the forward and backward pass (not recomputation).PaLM 540B reported 46.2%; Llama 3 405B 38–43%.Taught in: Training at scale
$m_{t} = μ m_{t - 1} + g_{t}$: “momentum update”SGD with momentum (PyTorch's form): a running sum of past gradients $g_{t}$ , with momentum coefficient $μ$ ; the parameters step along $- η m_{t}$ .With $μ = 0.9$ and a constant gradient, $m_{t} \to 10 g$ .Taught in: Training in practice
$m_{t}, v_{t}, m^_{t}, v^_{t}$: “m t, v t, m hat, v hat”Adam's running means of the gradient and of its elementwise square, and their bias-corrected versions $m^_{t} = m_{t} / (1 - β_{1}^{t})$ , $v^_{t} = v_{t} / (1 - β_{2}^{t})$ .Update: $θ \leftarrow θ - α m^_{t} / (v^_{t} + ϵ)$ .Taught in: Training in practice
$\nabla^{2} f (x) ⪰ μ I$: “f is mu-strongly convex”Strong convexity with constant $μ > 0$ : every eigenvalue of the Hessian is at least $μ$ , so the function curves up at least as fast as $\frac{μ}{2} ∥ x ∥^{2}$ .Ridge regression's objective is $2 λ$ -strongly convex.Taught in: Convexity
$∥ \nabla f (x) - \nabla f (y)∥ \leq L ∥ x - y ∥$: “f is L-smooth”The gradient is $L$ -Lipschitz: it cannot change faster than $L$ times the distance moved. Then $f$ lies below a parabola of curvature $L$ on top of each tangent plane (the descent lemma).For a quadratic, $L = λ_{m a x}$ of the Hessian.Taught in: Convexity
$κ = L / μ$: “kappa equals L over mu”The condition number of a smooth, strongly convex function: the ratio of its largest to smallest curvature. Gradient descent needs about $κ lo g (1/ ϵ)$ steps; accelerated methods about $κ lo g (1/ ϵ)$ .Taught in: Why SGD converges
$f^{⋆}, p^{⋆}, d^{⋆}$: “f star, p star, d star”Optimal values: $f^{⋆} = in f f$ ; in constrained problems $p^{⋆}$ is the primal optimum and $d^{⋆}$ the optimum of the dual.Weak duality says $d^{⋆} \leq p^{⋆}$ .Taught in: Lagrange multipliers, duality and KKT
$L (x, λ, ν) = f_{0} (x) + \sum_{i} λ_{i} f_{i} (x) + \sum_{j} ν_{j} h_{j} (x)$: “the Lagrangian”The objective plus a multiplier times each constraint function ( $f_{i} \leq 0$ , $h_{j} = 0$ ). Multipliers for inequalities must be $\geq 0$ .Taught in: Lagrange multipliers, duality and KKT
$q (λ, ν) = in f_{x} L (x, λ, ν)$: “the Lagrange dual function”Minimise the Lagrangian over $x$ for fixed multipliers. Always concave, and a lower bound on $p^{⋆}$ whenever $λ ⪰ 0$ . Boyd and Vandenberghe write $g$ .Taught in: Lagrange multipliers, duality and KKT
$λ_{i}^{⋆} f_{i} (x^{⋆}) = 0$: “complementary slackness”At an optimum (with zero duality gap), each inequality constraint is either tight or has a zero multiplier.In an SVM, only the support vectors have $α_{i} > 0$ .Taught in: Lagrange multipliers, duality and KKT
$- H^{- 1} \nabla f (x)$: “the Newton step”The step to the minimum of the local quadratic (second-order Taylor) model, where $H$ is the Hessian. Exact in one step on a quadratic.Taught in: Newton, quasi-Newton and natural gradients
$J^{⊤} J$: “J transpose J (the Gauss–Newton matrix)”For a least-squares loss $\frac{1}{2} ∥ r (θ) ∥^{2}$ with Jacobian $J$ of the residuals: the part of the Hessian built from first derivatives only. Always positive semidefinite.Taught in: Newton, quasi-Newton and natural gradients
$H_{k + 1} y_{k} = s_{k}$: “the secant equation”Quasi-Newton methods choose an inverse-Hessian approximation that maps the last change of gradient $y_{k}$ to the last step $s_{k}$ .Taught in: Newton, quasi-Newton and natural gradients
$F^{- 1} \nabla L$: “the natural gradient”The gradient preconditioned by the inverse Fisher matrix: steepest descent when distance is measured by KL divergence between the model's distributions.Taught in: Newton, quasi-Newton and natural gradients
$O (1/ k), O (1/ k^{2}), O (ρ^{k})$: “rates of convergence”How the error bound shrinks with the iteration count $k$ : sublinear rates $1/ k$ (gradient descent, convex) and $1/ k^{2}$ (Nesterov), or linear rates $ρ^{k}$ with $0 < ρ < 1$ (strong convexity).Taught in: Why SGD converges
$prox_{η h} (v)$: “the prox of eta h at v”The proximal operator: $ar g min_{z} h (z) + \frac{1}{2 η} ∥ z - v ∥^{2}$ — decrease $h$ while staying close to $v$ .The prox of $τ ∥ \cdot ∥_{1}$ is soft-thresholding; the prox of an indicator function is a projection.Taught in: Constraints, projections and proximal methods
$P_{C} (v)$: “the projection of v onto C”The closest point of the set $C$ to $v$ in Euclidean distance. Projected gradient descent takes a gradient step and then projects.Projecting onto ${x \geq 0}$ is $max (v, 0)$ .Taught in: Constraints, projections and proximal methods
$ι_{C} (x)$: “the indicator function of C”Zero if $x \in C$ and $+ \infty$ otherwise (also written $I_{C}$ or $δ_{C}$ ). Turns a constraint into a penalty.Taught in: Constraints, projections and proximal methods
$D_{ϕ} (z, x)$: “the Bregman divergence of phi” $ϕ (z) - ϕ (x) - \nabla ϕ (x)^{⊤} (z - x)$ : how far a convex $ϕ$ lies above its tangent. The distance used by mirror descent.With $ϕ (x) = \sum_{i} x_{i} lo g x_{i}$ on the simplex, $D_{ϕ}$ is the KL divergence.Taught in: Constraints, projections and proximal methods
$x + ε sign (\nabla_{x} L)$: “x plus epsilon times the sign of the input gradient”The fast gradient sign method (FGSM): one step to the corner of the $L_{\infty}$ ball that most increases the linearised loss.Goodfellow et al.: ε = 0.25 on MNIST fooled a softmax classifier 99.9% of the time.Taught in: Adversarial examples and robustness
$min_{θ} E [max_{δ \in S} L (θ, x + δ, y)]$: “min over theta of the expected worst-case loss”The adversarial-training (robust optimisation) objective of Madry et al.: the attacker maximises inside the budget $S$ , the defender minimises the result.The inner max is approximated with a few steps of PGD.Taught in: Adversarial examples and robustness
$Π_{S}$: “projection onto S”Projection back into an allowed set; for an $L_{\infty}$ box, clipping each coordinate.PGD: $x^{t + 1} = Π_{S} (x^{t} + a sign (\nabla_{x} L))$ .Taught in: Adversarial examples and robustness
$g / max (1, ∥ g ∥_{2} / C)$: “g clipped to norm C”Per-example gradient clipping in DP-SGD: gradients longer than $C$ are scaled down to length $C$ .With $C = 1$ , a gradient of norm 3.8 becomes a unit vector.Taught in: Privacy and differential privacy
$FLOPs$: “flops”Floating-point operations — used as a hardware-independent measure of training cost.The transformer paper estimates $3.3 \cdot 1 0^{18}$ FLOPs for its base model.Taught in: Worked reading: Attention Is All You Need
$N^{*} (C), D^{*} (C)$: “N star and D star of C”Compute-optimal model size and number of tokens for a compute budget $C$ .Chinchilla's Table 3: $N^{*} = 1 0^{9}$ goes with $D^{*} \approx 2 \times 1 0^{10}$ tokens.Taught in: Scaling laws and their limits
$μ P$: “mu P”The maximal update parametrization: rules for scaling initialization and learning rates with width so that features keep learning and hyperparameters transfer across widths.Tune a 40M-parameter model in µP, reuse its hyperparameters at 6.7B.Taught in: Width versus depth
$arg max_{θ} f (θ)$: “arg max over theta of f”The value of $θ$ at which $f$ is largest (not the largest value itself). $arg min$ likewise. $θ^_{MLE} = arg max_{θ} L (θ)$ .Taught in: Maximum likelihood: where losses come from
$J = \sum_{i = 1}^{n} ∥ x_{i} - μ_{c_{i}} ∥^{2}$: “J equals the sum over i of the squared distance from x i to mu c i”The k-means objective: within-cluster sum of squares (scikit-learn's inertia, Murphy's distortion).Two points at distance 1 and 2 from their centroids give $J = 1 + 4 = 5$ .Taught in: k-means, properly
$Q (θ)$: “script Q of theta”EM's expected complete-data log-likelihood, maximised in the M step.Taught in: Gaussian mixtures and EM
$Ncut (A, A ˉ)$: “N-cut of A and A bar”Normalised cut: cut weight divided by the volume of each side, summed.Taught in: Spectral clustering and graph Laplacians

Neural networks

$X ⋆ K$: “X cross-correlated with K”Slide the kernel $K$ over $X$ and take a dot product at every position, with no kernel flip. This is what deep-learning libraries compute and call “convolution”. $Y_{i, j} = \sum_{a, b} K_{a, b} X_{i + a, j + b}$ Taught in: Convolution
$X * K$: “X convolved with K”Convolution in the signal-processing sense: like cross-correlation but with the kernel flipped. In ML papers $*$ usually just means “a convolutional layer”. $(X * K)_{i, j} = \sum_{a, b} K_{a, b} X_{i - a, j - b}$ Taught in: Convolution
$K_{o, c, a, b}$: “K sub o, c, a, b”A convolution kernel: output channel $o$ , input channel $c$ , row offset $a$ , column offset $b$ . The tensor has shape $(C_{out}, C_{in}, k, k)$ .64 kernels of $3 \times 3 \times 3$ on an RGB image: $64 \times (27 + 1) = 1792$ parameters.Taught in: Convolution
$C_{in}, C_{out}$: “C in, C out”Numbers of input and output channels (feature maps) of a convolutional layer.An RGB image has $C_{in} = 3$ .Taught in: Convolution
$n_{out} = ⌊(n + 2 p - k) / s ⌋ + 1$: “n out equals floor of n plus 2p minus k over s, plus one”Output size of a convolution with input size $n$ , kernel $k$ , padding $p$ and stride $s$ . $n = 224, k = 7, p = 3, s = 2$ gives $112$ .Taught in: Convolution
$r_{L}$: “r sub L”Receptive field: the width, in input pixels, of the square patch of inputs that can influence a unit in layer $L$ . $r_{L} = 1 + \sum_{ℓ} (k_{ℓ} - 1) \prod_{i < ℓ} s_{i}$ .Three stacked $3 \times 3$ layers: $r = 7$ .Taught in: Convolution
$y = F (x) + x$: “y equals F of x plus x”A residual block: layers learn a correction $F$ that is added to an identity shortcut.If $F \equiv 0$ the block is the identity, and gradients pass through unchanged.Taught in: The CNN lineage: LeNet to ResNet
$h_{t}$: “h sub t”Hidden state of a recurrent network after reading inputs $x_{1}, \dots, x_{t}$ : its memory. $h_{t} = tanh (W_{h} h_{t - 1} + W_{x} x_{t} + b)$ Taught in: Recurrent networks
$W_{h}, W_{x}$: “W h, W x”Recurrent (state-to-state) and input-to-state weight matrices of an RNN, shared across all time steps.With $d_{h} = 100$ , $d_{x} = 50$ : $W_{h}$ is $100 \times 100$ and $W_{x}$ is $100 \times 50$ .Taught in: Recurrent networks
$δ_{t} = \partial L / \partial h_{t}$: “delta sub t”In backpropagation through time: the gradient of the total loss with respect to the state at step $t$ . It collects this step's output error plus everything flowing back from later steps.Taught in: Recurrent networks
$c_{t}$: “c sub t, the cell state”An LSTM's long-term memory, updated additively: $c_{t} = f_{t} ⊙ c_{t - 1} + i_{t} ⊙ c ~_{t}$ .Taught in: LSTM and GRU
$f_{t}, i_{t}, o_{t}$: “forget, input and output gates”LSTM gates, each a sigmoid layer with values in $(0, 1)$ : how much of the old cell to keep, how much of the candidate to write, and how much of the cell to expose. $f_{t} = σ (W_{x f} x_{t} + W_{h f} h_{t - 1} + b_{f})$ Taught in: LSTM and GRU
$c ~_{t}$: “c tilde sub t, the candidate”The new values an LSTM proposes to write into its cell, in $(- 1, 1)$ . PyTorch calls it $g_{t}$ .Taught in: LSTM and GRU
$σ (z)$: “sigma of z”The logistic sigmoid $1/ (1 + e^{- z})$ . Inside gated RNNs it turns a score into a gate value between 0 (closed) and 1 (open). $σ (4.6) \approx 0.99$ Taught in: LSTM and GRU
$z_{t}, r_{t}$: “z sub t and r sub t, update and reset gates”The two gates of a GRU. $z_{t}$ sets how much of the old state to keep; $r_{t}$ sets how much of it feeds the candidate. $h_{t} = z_{t} ⊙ h_{t - 1} + (1 - z_{t}) ⊙ h ~_{t}$ (Cho et al.'s convention)Taught in: LSTM and GRU
$v$: “v, the summary vector”In an encoder–decoder, the encoder's final state: the whole source sequence compressed into one fixed-length vector (Bahdanau et al. write $c$ ).Sutskever et al.'s deep LSTM used 8,000 numbers.Taught in: Seq2seq and the bottleneck
$p (y ∣ x) = \prod_{t} p (y_{t} ∣ v, y_{< t})$: “the probability of y given x is the product over t of p of y t given v and the earlier y's”How an encoder–decoder scores a translation: one next-word prediction at a time, each conditioned on the summary vector and the words already produced.Taught in: Seq2seq and the bottleneck
$max (0, x)$: “max of zero and x”The positive part $x^{+}$ — the ReLU activation. Applied element-wise to vectors.Transformer FFN: $max (0, x W_{1} + b_{1}) W_{2} + b_{2}$ Taught in: Reading the symbols
$softmax (z)_{i}$: “softmax of z, component i” $e^{z_{i}} / \sum_{j} e^{z_{j}}$ : turns scores into positive weights that sum to 1.Taught in: Reading the symbols
$t_{x} = (x - x_{a}) / w_{a}, t_{w} = lo g (w / w_{a})$: “t x, t w”Faster R-CNN's box regression targets: the offset and log-scale of a box relative to an anchor $(x_{a}, y_{a}, w_{a}, h_{a})$ .Taught in: Vision transformers, detection and segmentation
$D ~^{- 1/2} A ~ D ~^{- 1/2}$: “the symmetrically normalised adjacency”GCN's propagation matrix: entry $(i, j)$ is $1/ d ~_{i} d ~_{j}$ for neighbours (and self). Its eigenvalues lie in $(- 1, 1]$ .A GCN layer is $H^{'} = σ (D ~^{- 1/2} A ~ D ~^{- 1/2} H W)$ .Taught in: Graph neural networks
$α_{ij} = softmax_{j} (LeakyReLU (a^{⊤} [Θ x_{i} ∥Θ x_{j}]))$: “alpha i j, graph attention”GAT's attention weight from node $j$ to node $i$ , normalised over $i$ 's neighbourhood; $∥$ is concatenation.Taught in: Graph neural networks
$softmax (z / T)$: “softmax of z over T”Temperature scaling: dividing logits by $T > 0$ softens ( $T > 1$ ) or sharpens ( $T < 1$ ) the probabilities without changing the predicted class. $T$ is fitted by minimising validation log loss.Taught in: Calibration and proper scoring rules
$β$: “beta (in a beta-VAE)”The weight on a VAE's KL term. $β = 1$ is the standard VAE, $β = 0$ a plain autoencoder; larger $β$ compresses the code more. (In diffusion models $β_{t}$ is instead the noise schedule.)With a 2-D code, the lesson's $β = 4$ VAE ignores its latent entirely (posterior collapse).Taught in: Autoencoders and VAEs
$G (z), D (x)$: “G of z, D of x”A GAN's generator, which maps noise $z$ to a sample, and discriminator, which outputs the probability that $x$ is real data.The optimal discriminator is $D^{*} (x) = p_{data} (x) / (p_{data} (x) + p_{g} (x))$ .Taught in: Generative adversarial networks
$ϵ_{θ} (x_{t}, t)$: “epsilon theta of x t and t”A diffusion model's noise predictor: a network that guesses the noise that was added to produce $x_{t}$ . Trained with $L_{simple} = ∥ ϵ - ϵ_{θ} ∥^{2}$ .The score estimate is $- ϵ_{θ} (x_{t}, t) / 1 - α ˉ_{t}$ .Taught in: Diffusion models
$w$: “w, the guidance weight”In classifier-free guidance, $ϵ ~ = (1 + w) ϵ_{θ} (x_{t}, t, c) - w ϵ_{θ} (x_{t}, t, \emptyset)$ . Larger $w$ : samples follow the condition more closely but are less diverse. $w = 0$ is the plain conditional model.Taught in: Diffusion models
$s_{θ} (x, t)$: “s theta of x and t”A score network: a learned estimate of the score of the data blurred to noise level $t$ (or $σ$ ), trained by denoising score matching.Denoising target: $(x - x ~) / σ^{2}$ .Taught in: Score matching and flow matching
$v_{θ} (x, t)$: “v theta of x and t”A velocity field learned by flow matching: samples are made by integrating $d x / d t = v_{θ} (x, t)$ from noise at $t = 0$ to data at $t = 1$ .Trained by regressing onto $x_{1} - x_{0}$ at $x_{t} = (1 - t) x_{0} + t x_{1}$ .Taught in: Score matching and flow matching
$TopK (z, k)$: “top k of z”Keeps the $k$ largest entries of a vector of router logits and sets the rest to $- \infty$ (so they get zero weight after a softmax). Used to pick experts in mixture-of-experts layers.Mixtral uses $k = 2$ of 8 experts per layer; Switch Transformer uses $k = 1$ .Taught in: Mixture of experts
$G (x)_{i}$: “G of x, sub i”The gate weight that a mixture-of-experts router gives expert $i$ for input $x$ ; zero for experts that are not selected. The layer output is $\sum_{i} G (x)_{i} FFN_{i} (x)$ .Taught in: Mixture of experts
$α N \sum_{i} f_{i} P_{i}$: “alpha N sum over i of f i P i”The load-balancing auxiliary loss for mixture-of-experts routers: $f_{i}$ is the fraction of tokens sent to expert $i$ , $P_{i}$ the average router probability for expert $i$ . Equal to $α$ when routing is perfectly uniform.Four experts, all tokens sent to expert 1 with $P = (0.7, 0.1, 0.1, 0.1)$ : $4 \times 0.7 = 2.8$ (times $α$ ).Taught in: Mixture of experts
$C (w)$: “C of w”In Bengio et al. (2003): the learned feature vector (embedding) of word $w$ , a row of the $∣ V ∣ \times m$ matrix $C$ . Later papers usually call the matrix $E$ or $W_{E}$ .With $m = 100$ features and $∣ V ∣ = 17, 964$ words, $C$ has about 1.8 million entries.Taught in: From n-grams to neural language models
$softmax (z)_{k} = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}$: “softmax of z, entry k”Turns a vector of real scores into a probability distribution: exponentiate, then normalise. Preserves order and ignores a common shift of all scores. $softmax (2, 1, 0.1, - 1.2) \approx (0.64, 0.24, 0.10, 0.03)$ .Taught in: Logistic regression and softmax
$z = W x + b$: “the logits”Unnormalised class scores, one per class, fed into a softmax (or, for two classes, a sigmoid). Losses in PyTorch such as CrossEntropyLoss take logits, not probabilities.Logits $(3, 3, - 5)$ put almost all probability on the first two classes, equally.Taught in: Logistic regression and softmax
$softmax (z / T)$: “softmax at temperature T”Dividing the logits by a temperature $T$ sharpens the distribution ( $T < 1$ , toward the largest logit) or flattens it ( $T > 1$ , toward uniform).Halving $T$ is the same as doubling every logit.Taught in: Logistic regression and softmax
$p - y$: “p minus y”The gradient of softmax cross-entropy with respect to the logits: predicted probabilities minus the one-hot label. The weight gradient is $(p - y) x^{⊤}$ . $p = (0.7, 0.2, 0.1)$ , true class 2: gradient $(0.7, - 0.8, 0.1)$ .Taught in: Logistic regression and softmax
$W_{0} + \frac{α}{r} B A$: “W nought plus alpha over r, B A”A LoRA-adapted weight matrix: the frozen pretrained $W_{0}$ ( $d \times k$ ) plus a trainable low-rank change $B A$ with $B \in R^{d \times r}$ , $A \in R^{r \times k}$ , scaled by $α / r$ . $r = 8$ on a $4096 \times 4096$ matrix trains $8 (4096 + 4096) = 65, 536$ numbers instead of about 16.8 million.Taught in: Fine-tuning and LoRA
$r$: “r, the rank”In LoRA, the rank of the update $B A$ : the width of the bottleneck between $A$ and $B$ . Chosen by hand, typically much smaller than the matrix dimensions.The LoRA paper found $r = 1$ already competitive for adapting $W_{q}$ and $W_{v}$ of GPT-3 on WikiSQL.Taught in: Fine-tuning and LoRA
$w^{⊤} x + b = 0$: “w transpose x plus b equals zero”The decision boundary of a linear classifier: a hyperplane (a line in 2-D). Every linear model's boundary has this form.XOR's classes cannot be split by any such line.Taught in: The limits of linear
$ϕ (x)$: “phi of x”A feature map: a fixed, hand-chosen function turning an input into a (usually longer) feature vector. Careful — the same letter is also used for activation functions. $ϕ (x_{1}, x_{2}) = (x_{1}, x_{2}, x_{1} x_{2})$ makes XOR linearly separable.Taught in: The limits of linear
$W^{[l]} \in R^{n_{l} \times n_{l - 1}}, b^{[l]}$: “W layer l, b layer l”The weights and biases of layer $l$ . Row $j$ of $W^{[l]}$ holds the incoming weights of unit $j$ ; the bracketed superscript is a layer index, not a power.784 inputs to 128 hidden units: $W^{[1]}$ is $128 \times 784$ .Taught in: The multilayer perceptron
$z^{[l]}, a^{[l]}$: “z layer l, a layer l”Pre-activations $z^{[l]} = W^{[l]} a^{[l - 1]} + b^{[l]}$ and activations $a^{[l]} = ϕ (z^{[l]})$ of layer $l$ ; $a^{[0]} = x$ .For a batch, stack them as rows: $Z^{[l]}, A^{[l]} \in R^{n \times n_{l}}$ .Taught in: The multilayer perceptron
$n_{l}$: “n l”The width of layer $l$ : its number of units. $n_{0} = d$ is the number of input features.Taught in: The multilayer perceptron
$ϕ (z)$: “phi of z”An activation function, applied elementwise after each layer's linear map. Without it, stacked layers collapse into one affine map. $ϕ = tanh$ , $σ$ or ReLU.Taught in: The multilayer perceptron
$ReLU (z) = max (0, z)$: “rectified linear unit of z”The rectifier: passes positive inputs unchanged, zeroes negative ones. Its slope is 1 for $z > 0$ and 0 for $z < 0$ . $ReLU (- 2) = 0$ , $ReLU (3) = 3$ .Taught in: The multilayer perceptron
$tanh z$: “tan h of z”Hyperbolic tangent: an S-shaped activation from $- 1$ to $1$ , equal to $2 σ (2 z) - 1$ , with slope $1 - tanh^{2} z$ .Taught in: The multilayer perceptron
$\frac{\partial L}{\partial W ^{[l]}}$: “d L d W layer l”The gradient of the loss with respect to layer $l$ 's weights: a matrix of the same shape as $W^{[l]}$ , computed by backpropagation.For $Z = A W^{⊤} + 1 b^{⊤}$ : $\partial L / \partial W = Δ^{⊤} A$ .Taught in: Backpropagation
$δ^{[l]}, Δ^{[l]}$: “delta layer l”The error at layer $l$ : $\partial L / \partial z^{[l]}$ for one example ( $δ$ ), or $\partial L / \partial Z^{[l]}$ for a batch ( $Δ$ ). Backprop computes it from the layer above.Softmax + cross-entropy: $Δ^{[L]} = \frac{1}{n} (P - Y)$ .Taught in: Backpropagation
$Δ ⊙ ϕ^{'} (Z)$: “Delta times phi prime of Z, elementwise”The Hadamard (entrywise) product, as it appears in the backward pass: how a gradient passes through an elementwise activation. $Δ^{[1]} = (Δ^{[2]} W^{[2]}) ⊙ ϕ^{'} (Z^{[1]})$ .Taught in: Backpropagation
$σ_{W}^{2} = \frac{2}{n _{in} + n _{out}}$: “Xavier variance”Xavier (Glorot) initialisation: a weight variance chosen so that signals keep roughly the same spread forwards and backwards through a layer.He initialisation for ReLU uses $σ_{W}^{2} = 2/ n_{in}$ instead.Taught in: Training in practice
$μ_{B}, σ_{B}^{2}$: “mu B, sigma squared B”The mean and variance of a feature over the current mini-batch $B$ , used by batch normalisation.Taught in: Training in practice
$γ, β (in a norm layer)$: “gamma, beta”The learned scale and shift applied after batch or layer normalisation: $y = γ x^+ β$ .Taught in: Training in practice
$p (dropout rate)$: “p”The probability that dropout zeroes a unit during training; survivors are scaled by $1/ (1 - p)$ . $p = 0.5$ drops about half the units on each step.Taught in: Training in practice
$h^{[l + 1]} = h^{[l]} + F (h^{[l]})$: “residual connection”A residual (skip) connection: a block adds its input to its output, so gradients have an identity path backwards.Taught in: Training in practice
$x^{'}$: “x prime”In attribution: the baseline input that means “no signal” (a black image, an all-zero embedding), against which a prediction is explained.Integrated gradients with baseline $x^{'} = 0$ .Taught in: Interpretability
$ϕ_{i}$: “phi i”The attribution of feature $i$ : its share of the change in output, e.g. an integrated-gradients or Shapley value.For $f = x y + z$ at $(1, 1, 1)$ from $0$ : $ϕ_{x} = ϕ_{y} = 0.5$ , $ϕ_{z} = 1$ .Taught in: Interpretability
$IG_{i} (x) = (x_{i} - x_{i}^{'}) \int_{0}^{1} \frac{\partial f}{\partial x _{i}} (x^{'} + α (x - x^{'})) d α$: “integrated gradients of feature i”Integrated gradients: the gradient averaged along the straight path from baseline to input, times the change in the feature. The attributions sum to $f (x) - f (x^{'})$ .Approximated by a Riemann sum with 20–1000 steps.Taught in: Interpretability
$P_{d r o p}$: “P drop”The dropout rate: the probability that a unit's output is zeroed during training. $P_{d r o p} = 0.1$ for the base transformer.Taught in: Worked reading: Attention Is All You Need
$ϵ_{l s}$: “epsilon L S”The label-smoothing value: probability mass moved from the correct token to the rest of the vocabulary in the training targets. $ϵ_{l s} = 0.1$ in the transformer paper.Taught in: Worked reading: Attention Is All You Need
$N$: “N”In scaling laws: the number of model parameters (Kaplan et al. exclude embeddings; Chinchilla counts all).Chinchilla has $N = 70$ billion parameters.Taught in: Scaling laws and their limits
$C \approx 6 N D$: “C is about six N D”Training compute in FLOPs: about $2 N$ per token for the forward pass and $4 N$ for the backward pass, times $D$ tokens. $N = 1 0^{9}$ , $D = 2 \times 1 0^{10}$ gives $C \approx 1.2 \times 1 0^{20}$ FLOPs.Taught in: Scaling laws and their limits
$L (N, D) = E + A / N^{α} + B / D^{β}$: “L of N and D equals E plus A over N to the alpha plus B over D to the beta”The Chinchilla parametric loss: irreducible loss plus a model-size term plus a data term, fitted to training runs.Hoffmann et al. fitted $E = 1.69$ , $A = 406.4$ , $B = 410.7$ , $α = 0.34$ , $β = 0.28$ .Taught in: Scaling laws and their limits
$α, β$: “alpha, beta”Scaling exponents: how fast the model-size and data terms of the loss shrink.Doubling $N$ multiplies $A / N^{α}$ by $2^{- 0.34} \approx 0.79$ .Taught in: Scaling laws and their limits
$α_{N}$: “alpha N”Kaplan et al.'s exponent for loss versus non-embedding parameters, $L (N) = (N_{c} / N)^{α_{N}}$ . $α_{N} \approx 0.076$ .Taught in: Scaling laws and their limits
$EMC_{D, ϵ} (T)$: “effective model complexity”Nakkiran et al.'s effective model complexity: the largest number of samples on which a training procedure reaches training error at most $ϵ$ on average.Test error tends to peak when the EMC is close to the number of training samples.Taught in: Double descent
$Θ (x, x^{'})$: “the neural tangent kernel at x and x prime”The neural tangent kernel: $\nabla_{θ} f (x)^{⊤} \nabla_{θ} f (x^{'})$ , how a gradient step on one input moves the output at another.In the infinite-width NTK limit, $Θ$ stays constant during training.Taught in: Width versus depth
$LN (x) = γ ⊙ \frac{x - μ}{σ ^{2} + ϵ} + β$: “layer norm of x”Layer normalisation: centre and rescale one token's vector using its own mean and variance, then apply a learned gain and bias.Pre-LN blocks compute $x + Sublayer (LN (x))$ ; the 2017 Post-LN blocks compute $LN (x + Sublayer (x))$ .Taught in: The transformer block

Transformers

$z_{0} = [x_{class}; x_{p}^{1} E; \dots; x_{p}^{N} E] + E_{pos}$: “z zero equals class token, then each patch times E, plus position embeddings”The Vision Transformer's input: flattened image patches $x_{p}^{i}$ mapped by a shared matrix $E$ , a learned class token prepended, learned position embeddings added.A $224 \times 224$ image with $P = 16$ gives $N = 196$ patch tokens plus one class token.Taught in: Vision transformers, detection and segmentation
$N = H W / P^{2}$: “N equals H W over P squared”Number of $P \times P$ patches in an $H \times W$ image: the ViT's sequence length (before the class token). $22 4^{2} /1 4^{2} = 256$ for ViT-L/14 at 224 px.Taught in: Vision transformers, detection and segmentation
$I = \frac{FLOPs}{bytes}$: “arithmetic intensity: FLOPs per byte”Arithmetic intensity: how many floating-point operations a computation does per byte it moves to or from memory. Compare it with the chip's ratio $π / β$ to see whether it is compute-bound or memory-bound.A batch of $B$ tokens times a weight matrix has $I \approx B$ ; decoding one sequence has $I \approx 1$ .Taught in: The cost of attention
$π / β$: “pi over beta, the ridge point”Peak FLOP/s divided by memory bandwidth in bytes/s: the critical arithmetic intensity (ridge point of a roofline). Below it a computation is memory-bound, above it compute-bound. Here $π$ is not the circle constant.An H100: $9.89 \times 1 0^{14} /3.35 \times 1 0^{12} \approx 295$ FLOPs per byte.Taught in: The cost of attention
$24 n d^{2} + 4 n^{2} d$: “twenty-four n d squared plus four n squared d”Forward FLOPs of one GPT-style transformer layer on $n$ tokens of width $d$ : weight matmuls (linear in $n$ ) plus the $Q K^{⊤}$ and $P V$ products (quadratic in $n$ ). The second term dominates when $n > 6 d$ .For $d = 4096$ , attention FLOPs pass the rest at about $n = 24, 576$ tokens.Taught in: The cost of attention
$h_{k v}$: “h K V, the number of key-value heads”The number of key–value heads in an attention layer. Equal to the number of query heads $h$ in multi-head attention, 1 in multi-query attention, and in between for grouped-query attention.Llama 2 70B has $h = 64$ query heads and $h_{k v} = 8$ .Taught in: The KV cache
$2 L h_{k v} d_{k} n b s$: “two L h K V d k n b s”The size of the KV cache in bytes: keys and values, for $L$ layers, $h_{k v}$ key–value heads of width $d_{k}$ , $n$ cached tokens, batch $b$ , and $s$ bytes per number.Llama 2 7B in 16-bit: $2 \times 32 \times 32 \times 128 \times 2 = 524, 288$ bytes per token.Taught in: The KV cache
$c_{t}^{K V}$: “c K V sub t”In multi-head latent attention (DeepSeek-V2), the compressed latent vector for token $t$ from which all heads' keys and values are reconstructed. It is what gets cached.DeepSeek-V2 uses $d_{c} = 512$ , plus a 64-dimensional positional key: 576 cached numbers per layer per token.Taught in: The KV cache
$S_{i}$: “S sub i, the allowed set of query i”In sparse attention, the set of key positions that query $i$ may attend to: row $i$ of the attention mask. The cost of a layer is proportional to $\sum_{i} ∣ S_{i} ∣$ .Causal attention: $S_{i} = {1, \dots, i}$ . A sliding window: $S_{i} = {i - w, \dots, i}$ .Taught in: Sparse attention
$O (n n)$: “order n root n”The cost of the Sparse Transformer's strided and fixed attention patterns with stride $ℓ \approx n$ : each of $n$ queries attends to about $2 n$ keys.At $n = 16, 384$ , about $4 \times 1 0^{6}$ pairs instead of about $1.3 \times 1 0^{8}$ for causal attention.Taught in: Sparse attention
$ϕ (q)^{⊤} ϕ (k)$: “phi of q transpose phi of k”The similarity used by linear attention in place of $exp (q^{⊤} k / d_{k})$ , with a feature map $ϕ$ (for example $elu (x) + 1$ ). It lets the sums over keys be computed once, giving $O (n d^{2})$ cost.Taught in: Sparse attention
$m_{i}, ℓ_{i}$: “m sub i and ell sub i”The running maximum and running softmax denominator for query row $i$ in the online softmax used by FlashAttention. When the maximum grows from $m$ to $m^{'}$ , the old sum is rescaled by $e^{m - m^{'}}$ .Scores $2, 5, 1$ : $(m, ℓ)$ goes $(2, 1) \to (5, 1.050) \to (5, 1.068)$ .Taught in: FlashAttention and the memory hierarchy
$B_{r}, B_{c}$: “B r and B c, the block sizes”In FlashAttention, the number of rows of $Q$ (for $B_{r}$ ) and of $K$ and $V$ (for $B_{c}$ ) in one tile, chosen so the tiles fit in on-chip SRAM.Taught in: FlashAttention and the memory hierarchy
$s b h (34 + \frac{5 a s}{h})$: “s b h times thirty-four plus five a s over h”Bytes of activations a transformer layer saves for the backward pass (Korthikanti et al.), for sequence length $s$ , microbatch $b$ , hidden size $h$ , $a$ heads, 16-bit precision. The second term is the attention scores. $h = 4096$ , $s = 8192$ , $b = 1$ : the $34 s bh$ part is about 1.1 GB per layer.Taught in: Training at scale
$BLEU$: “blue”An automatic machine-translation score based on n-gram overlap with human reference translations (Papineni et al., 2002). Higher is better.The transformer paper reports 28.4 BLEU on WMT 2014 English-to-German.Taught in: Worked reading: Attention Is All You Need
$d_{f f}$: “d f f”The inner width of the transformer's position-wise feed-forward layer. $d_{f f} = 2048$ with $d_{model} = 512$ in the base model.Taught in: Worked reading: Attention Is All You Need
$w_{k} = 2 π k / p$: “w k equals two pi k over p”A Fourier frequency on the integers mod $p$ ; networks that grok modular addition represent numbers as $cos (w_{k} a)$ and $sin (w_{k} a)$ for a few key $k$ .Nanda et al.'s model uses $k \in {14, 35, 41, 42, 52}$ for $p = 113$ .Taught in: Grokking
$c_{i} = \sum_{j} α_{ij} h_{j}$: “c i equals the sum over j of alpha i j times h j”Bahdanau attention's context vector: a fresh weighted average of the encoder states for each output step $i$ .If $α_{i \cdot} = (0.1, 0.9)$ , then $c_{i} = 0.1 h_{1} + 0.9 h_{2}$ .Taught in: Attention: learning where to look
$α_{ij}$: “alpha i j”Attention weight: how much output (or query) $i$ draws on input (or key) $j$ . Each row is a softmax, so the weights are nonnegative and sum to 1.A row $(0.7, 0.2, 0.1)$ puts 70% of the attention on the first item.Taught in: Attention: learning where to look
$e_{ij} = a (s_{i - 1}, h_{j})$: “e i j equals a of s i minus one and h j”An alignment score or energy: how well decoder state $s_{i - 1}$ matches encoder annotation $h_{j}$ (Bahdanau et al.).Additive attention uses $a (s, h) = v_{a}^{⊤} tanh (W_{a} s + U_{a} h)$ .Taught in: Attention: learning where to look
$q, k, v$: “query, key, value”The three roles in attention: a query is compared with keys to produce weights, and the weights average the values.In Bahdanau attention the decoder state is the query and each encoder state is both key and value.Taught in: Attention: learning where to look
$Q = X W^{Q}, K = X W^{K}, V = X W^{V}$: “Q equals X W Q, and so on”Queries, keys and values for a whole sequence, one row per token, from three learned projection matrices.With $X \in R^{5 \times 4}$ and $W^{Q} \in R^{4 \times 3}$ , $Q$ is $5 \times 3$ .Taught in: Scaled dot-product and multi-head attention
$softmax (\frac{Q K ^{⊤}}{d _{k}}) V$: “softmax of Q K transpose over root d k, times V”Scaled dot-product attention (Vaswani et al., 2017): score every query against every key, scale, softmax each row, average the values.For $n$ queries and $m$ keys, $Q K^{⊤}$ is $n \times m$ and the output is $n \times d_{v}$ .Taught in: Scaled dot-product and multi-head attention
$d_{k}$: “d k”The length of each query and key vector (per head). Scores are divided by $d_{k}$ so their variance does not grow with it.The base Transformer has $d_{k} = 64$ .Taught in: Scaled dot-product and multi-head attention
$d_{v}$: “d v”The length of each value vector, and so of each head's output.The base Transformer uses $d_{v} = d_{k} = 64$ .Taught in: Scaled dot-product and multi-head attention
$d_{model}$: “d model”The width of every token's vector throughout the model — the dimension of the residual stream.512 in the base Transformer, 768 in GPT-2 small, 12,288 in GPT-3.Taught in: The transformer block
$h (heads)$: “h, the number of heads”How many attention functions run in parallel in one multi-head layer; usually $d_{k} = d_{model} / h$ . $h = 8$ in the base Transformer, 96 in GPT-3.Taught in: Scaled dot-product and multi-head attention
$W_{i}^{Q}, W_{i}^{K}, W_{i}^{V}, W^{O}$: “W Q i, W K i, W V i, W O”Per-head query, key and value projections ( $d_{model} \times d_{k}$ or $d_{v}$ ) and the output projection that mixes the concatenated heads back to $d_{model}$ .Together they hold $4 d_{model}^{2}$ weights, whatever the number of heads.Taught in: Scaled dot-product and multi-head attention
$MultiHead (Q, K, V) = Concat (head_{1}, \dots, head_{h}) W^{O}$: “multi-head of Q K V”Multi-head attention: run $h$ attention heads on different learned projections and combine them.Each $head_{i} = Attention (Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V})$ .Taught in: Scaled dot-product and multi-head attention
$M_{ij} \in {0, - \infty}$: “M i j, the mask”An attention mask added to the scores before the softmax: 0 where query $i$ may look at key $j$ , $- \infty$ where it may not (weight exactly 0).The causal mask sets $M_{ij} = - \infty$ for every $j > i$ .Taught in: Scaled dot-product and multi-head attention
$FFN (x) = max (0, x W_{1} + b_{1}) W_{2} + b_{2}$: “F F N of x”The position-wise feed-forward sublayer: a two-layer MLP applied to each token separately.In the base Transformer it maps 512 → 2048 → 512.Taught in: The transformer block
$N \approx 12 n_{layer} d_{model}^{2}$: “N is about twelve n layer d model squared”Non-embedding parameter count of a standard transformer (Kaplan et al., 2020): $4 d^{2}$ for attention plus $8 d^{2}$ for the feed-forward network per layer.GPT-3: $12 \times 96 \times 1228 8^{2} \approx 1.74 \times 1 0^{11}$ .Taught in: The transformer block
$PE_{(p os, 2 i)} = sin (p os /1000 0^{2 i / d_{model}})$: “P E of pos, two i”The sinusoidal positional encoding (with cosine in dimension $2 i + 1$ ), added to the token embeddings; each pair of dimensions turns at its own frequency.Pair $i = 0$ has wavelength $2 π$ positions; the last pair about $10000 \cdot 2 π$ .Taught in: Positional information
$R_{Θ, m}$: “R theta m”The rotary position embedding (RoPE) rotation for position $m$ : rotates each pair of query/key dimensions by $m θ_{i}$ , so $(R_{m} q)^{⊤} (R_{n} k)$ depends only on $n - m$ .In 2-D, $R_{m}^{⊤} R_{n} = R_{n - m}$ .Taught in: Positional information
$a_{ij}^{K} = w_{clip (j - i, - k, k)}^{K}$: “a K i j”A relative position representation (Shaw et al., 2018): a learned vector for the clipped offset between query $i$ and key $j$ , added to the key.With $k = 4$ there are $2 k + 1 = 9$ distinct vectors.Taught in: Positional information
$p_{i} \propto exp (z_{i} / τ)$: “p i proportional to e to the z i over tau”Sampling temperature: dividing the logits by $τ$ before the softmax. $τ < 1$ sharpens (towards greedy), $τ > 1$ flattens.At $τ \to 0$ sampling becomes argmax (greedy) decoding.Taught in: GPT and next-token prediction
$V^{(p)}$: “V p, the nucleus”In top- $p$ (nucleus) sampling, the smallest set of most-probable tokens whose probabilities sum to at least $p$ ; sampling is restricted to it and renormalised.With $p = 0.9$ the nucleus may hold 3 tokens when the model is confident and 50 when it is not.Taught in: GPT and next-token prediction

Reinforcement learning

$Δ_{i} = μ^{*} - μ_{i}$: “delta i”The gap of bandit arm $i$ : how much less it pays on average than the best arm.Means 0.7 and 0.55: gap 0.15.Taught in: Online learning and bandits
$μ^_{i} + 2 ln t / N_{i}$: “mu hat i plus the square root of two log t over N i”The UCB1 index: an arm's average reward plus an exploration bonus that shrinks with its pull count $N_{i}$ .Pull the arm with the largest index.Taught in: Online learning and bandits
$π (a_{t} ∣ c_{t})$: “pi of a t given c t”An agent's policy: the distribution over its next action $a_{t}$ given the context $c_{t}$ — every observation and action so far.For an LLM agent, $c_{t}$ is literally the prompt, including earlier tool outputs.Taught in: Agents and tool use
$s_{t}$: “s t”The state of the environment at time $t$ — everything the agent needs to know to choose an action.In a maze, which square the robot is on.Taught in: Reinforcement learning foundations
$a_{t}$: “a t”The action the agent takes at time $t$ .Up, down, left or right; or the next token for a language model.Taught in: Reinforcement learning foundations
$r_{t + 1}$: “r t plus one”The reward that arrives after the action at time $t$ (many papers write $r_{t}$ for the same thing). $+ 1$ for stepping onto the exit.Taught in: Reinforcement learning foundations
$(S, A, P, R, γ)$: “the MDP tuple”A Markov decision process: states, actions, transition probabilities, reward function, discount.Taught in: Reinforcement learning foundations
$P (s^{'} ∣ s, a)$: “P of s prime given s, a”Transition probability: the chance of landing in $s^{'}$ after taking $a$ in $s$ .With a slip probability of 0.2, the intended move happens with probability 0.8.Taught in: Reinforcement learning foundations
$γ$: “gamma”The discount factor, usually in $[0, 1)$ : a reward $k$ steps away is worth $γ^{k}$ of its face value. (RLHF implementations usually set $γ = 1$ ; in InstructGPT's objective, $γ$ is instead a pretraining-loss weight.)With $γ = 0.9$ , a reward 10 steps away counts about $0.35$ .Taught in: Reinforcement learning foundations
$G_{t}$: “G t”The return: the discounted sum of all rewards from time $t$ on, $G_{t} = \sum_{k} γ^{k} r_{t + k + 1} = r_{t + 1} + γ G_{t + 1}$ .Taught in: Reinforcement learning foundations
$π (a ∣ s)$: “pi of a given s”A policy: the probability of taking action $a$ in state $s$ . For a language model, $π (y ∣ x)$ is the probability of response $y$ to prompt $x$ .Taught in: Reinforcement learning foundations
$V^{π} (s)$: “V pi of s”State value: the expected return from $s$ when following policy $π$ .Taught in: Reinforcement learning foundations
$Q^{π} (s, a)$: “Q pi of s, a”Action value: the expected return from taking $a$ in $s$ , then following $π$ .Taught in: Reinforcement learning foundations
$V^{*} (s), Q^{*} (s, a)$: “V star, Q star”Optimal values: the best achievable expected return. Acting greedily on $Q^{*}$ is optimal.Taught in: Reinforcement learning foundations
$δ_{t}$: “delta t”The temporal-difference (TD) error $r_{t + 1} + γ V (s_{t + 1}) - V (s_{t})$ : how much better things went than predicted.Taught in: Reinforcement learning foundations
$ε$: “epsilon”In ε-greedy exploration, the probability of a random action; in PPO and GRPO, the clip range for the probability ratio (often 0.2).Taught in: Reinforcement learning foundations
$τ$: “tau”A trajectory (episode, rollout): the sequence $s_{0}, a_{0}, s_{1}, a_{1}, \dots$ generated by a policy.Taught in: Policy gradients and PPO
$J (θ)$: “J of theta”The expected return of the policy with parameters $θ$ — the objective policy-gradient methods climb.Taught in: Policy gradients and PPO
$\nabla_{θ} lo g π_{θ} (a ∣ s)$: “grad log pi”The direction in parameter space that makes action $a$ more likely in state $s$ ; the policy gradient weights it by how good the action turned out.For a softmax policy, $e_{a} - π_{θ}$ .Taught in: Policy gradients and PPO
$A^{π} (s, a)$: “A pi”The advantage $Q^{π} (s, a) - V^{π} (s)$ : how much better action $a$ is than the policy's average in $s$ . $A^$ is an estimate of it.Taught in: Policy gradients and PPO
$ρ_{t} (θ)$: “rho t”The probability ratio $π_{θ} (a_{t} ∣ s_{t}) / π_{θ_{old}} (a_{t} ∣ s_{t})$ used by TRPO, PPO and GRPO (the PPO paper writes $r_{t} (θ)$ ).Taught in: Policy gradients and PPO
$clip (ρ, 1 - ε, 1 + ε)$: “clip rho”Clamp the ratio to $[1 - ε, 1 + ε]$ ; inside PPO's $min$ , this removes the incentive to move an action's probability further in one update.Taught in: Policy gradients and PPO
$L^{CLIP} (θ)$: “L clip”PPO's clipped surrogate objective $E_{t} [min (ρ_{t} A^_{t}, clip (ρ_{t}, 1 - ε, 1 + ε) A^_{t})]$ .Taught in: Policy gradients and PPO
$A^_{t}^{GAE (γ, λ)}$: “GAE advantage”Generalized Advantage Estimation: $\sum_{l} (γ λ)^{l} δ_{t + l}$ ; $λ$ trades bias against variance.Taught in: Policy gradients and PPO
$r_{ϕ} (x, y)$: “r phi of x, y”A reward model: a network scoring response $y$ to prompt $x$ with one number, trained on human preferences.Taught in: RLHF: learning from preferences
$σ (r_{w} - r_{l})$: “sigma of r w minus r l”The Bradley–Terry preference probability: the chance a person prefers the answer with reward $r_{w}$ over the one with $r_{l}$ .Rewards 2.0 and 0.5 give $σ (1.5) \approx 0.82$ .Taught in: RLHF: learning from preferences
$y_{w}, y_{l}$: “y w, y l”The preferred (winning) and rejected (losing) responses in a preference pair; also written $y_{c}, y_{r}$ (chosen, rejected).Taught in: RLHF: learning from preferences
$π_{ref}$: “pi ref”The frozen reference policy — usually the supervised fine-tuned model — that RLHF, DPO and GRPO penalise drifting away from.Taught in: RLHF: learning from preferences
$β KL (π ∥ π_{ref})$: “beta times KL”The KL penalty in RLHF-style objectives; $β$ sets how expensive drift from the reference model is.Taught in: RLHF: learning from preferences
$π^{*} (y ∣ x)$: “pi star”The closed-form optimum of the KL-regularised reward objective, $π^{*} (y ∣ x) \propto π_{ref} (y ∣ x) e^{r (x, y) / β}$ : reweight the reference model by $e^{r / β}$ .Taught in: RLHF: learning from preferences
$Z (x)$: “Z of x”A partition function (normaliser) summing over all possible responses; it cancels in the DPO derivation.Taught in: RLHF: learning from preferences
$L_{DPO}$: “L DPO”The DPO loss: $- lo g σ$ of $β$ times the difference in log-probability ratios (policy over reference) of the preferred and rejected responses.Taught in: RLHF: learning from preferences
$A_{i} = \frac{r _{i} - mean ( r )}{std ( r )}$: “A i, group-relative”GRPO's advantage: a sampled answer's reward standardised within its group of $G$ answers to the same prompt.Rewards $(1, 0, 0, 0)$ give advantages $(1.73, - 0.58, - 0.58, - 0.58)$ .Taught in: RLVR and reasoning models
$pass@ k$: “pass at k”The probability that at least one of $k$ sampled answers is correct; estimated without bias as $1 - (k n - c) / (k n)$ from $n$ samples with $c$ correct.Taught in: RLVR and reasoning models
$U = V + X$: “proxy equals true value plus error”A Goodhart model: the measured proxy $U$ is the true objective $V$ plus error $X$ ; optimising $U$ hard selects for large $X$ when $X$ is heavy-tailed.Best-of-4,096 with Student-t error: proxy ≈ 93, true value ≈ 0.13.Taught in: Alignment and safety
$Q_{w^{-}}$: “Q w minus”The target network: a delayed copy of the Q-network, used only to compute bootstrap targets. DQN refreshes it every C updates; DDPG, TD3 and SAC Polyak-average it every step.DQN's target $r + γ max_{a^{'}} Q_{w^{-}} (s^{'}, a^{'})$ .Taught in: Deep Q-learning and actor–critic methods
$ϕ_{targ} \leftarrow ρ ϕ_{targ} + (1 - ρ) ϕ$: “Polyak averaging”A slowly moving target network: each step it moves a small fraction $1 - ρ$ of the way to the trained weights. Papers often write $τ = 1 - ρ$ . $ρ = 0.995$ (i.e. $τ = 0.005$ ).Taught in: Deep Q-learning and actor–critic methods
$μ_{θ} (s)$: “mu theta of s”A deterministic policy (the actor in DDPG and TD3): it outputs an action vector rather than a distribution, and stands in for $ar g max_{a} Q (s, a)$ in continuous action spaces.Taught in: Deep Q-learning and actor–critic methods
$α H (π (\cdot ∣ s))$: “alpha times the entropy of pi”The entropy bonus of maximum-entropy RL (SAC). $α$ is a temperature trading reward against randomness, not a step size.Taught in: Deep Q-learning and actor–critic methods
$N (s, a), W (s, a), P (s, a)$: “N, W, P of s, a”Tree-search edge statistics: visit count, total backed-up value (so $Q = W / N$ ), and — in AlphaZero-style search — the network's prior probability for the move.Taught in: Model-based RL and planning
$(p, v) = f_{θ} (s)$: “p, v equals f theta of s”The AlphaGo Zero / AlphaZero network: move probabilities $p$ and a value $v$ for position $s$ . MuZero adds learned representation $h$ and dynamics $g$ functions.Taught in: Model-based RL and planning
$β N (s, a)^{- 1/2}$: “beta over root N”A count-based exploration bonus: large for rarely tried state–action pairs, shrinking like one over the square root of the visit count.MBIE-EB adds it inside the Bellman equation.Taught in: Exploration
$N^_{n} (x)$: “N hat n of x”A pseudo-count: a visit count derived from a density model's probability of $x$ before and after one more observation of $x$ , so that similar states share counts.Taught in: Exploration
$r^{e} + β r^{i}$: “extrinsic plus beta intrinsic”Task (extrinsic) reward plus an intrinsic reward the agent pays itself for novelty or surprise, as in curiosity and RND.Taught in: Exploration
$π_{β}$: “pi beta”The behaviour policy that generated a fixed dataset in offline RL (also written $μ$ or $b$ ).Taught in: Imitation and offline RL
$J (π^) \leq J (π^{*}) + T^{2} ϵ$: “the behavioural cloning bound”A policy with error rate $ϵ$ on the expert's states can incur extra cost growing quadratically in the horizon $T$ ; DAgger's bound is linear, $u T ϵ_{N}$ .Taught in: Imitation and offline RL
$π^{- i}$: “pi minus i”The policies of every agent except agent $i$ . A Nash equilibrium is a joint policy in which each $π^{i}$ is a best response to $π^{- i}$ .Taught in: Multi-agent RL
$(N, S, {A^{i}}, P, {R^{i}}, γ)$: “the Markov game tuple”A Markov (stochastic) game: agents, states, each agent's actions, transitions on the joint action, each agent's reward, discount.Taught in: Multi-agent RL
$Q_{tot}$: “Q tot”A team's joint action value, built (in QMIX) as a monotonic mix of per-agent values so decentralised greedy actions agree with the joint argmax.Taught in: Multi-agent RL