Positional information
Self-attention cannot tell “dog bites man” from “man bites dog”. Why not, and how transformers are given a sense of order: sinusoids added to the input, learned position vectors, relative positions inside attention, rotations (RoPE) and linear biases (ALiBi) — and the open problem of lengths never seen in training.
50 min
After this lesson you can
- Prove that self-attention without position information is permutation-equivariant, and explain why a causal mask changes that
- Decode the sinusoidal encoding and show that the encoding of position pos + k is a rotation of the encoding of pos
- Compare absolute (sinusoidal, learned), relative (Shaw et al.), rotary (RoPE) and linear-bias (ALiBi) schemes
- Derive RoPE's relative-position property in two dimensions
- Explain why generalising to longer sequences than in training is still an open problem
In this lesson
- Attention doesn’t know where anything is
- Fix 1: add a position vector — sinusoids
- Fix 2: just learn a vector per position
- Fix 3: put relative positions inside attention
- Fix 4: rotate queries and keys — RoPE
- Fix 5: no embeddings, just a penalty for distance — ALiBi
- The open problem: lengths you never trained on
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.