Worked reading: Attention Is All You Need
A guided three-pass reading of Vaswani et al. (2017) — its claims, its architecture, Eq. (1) decoded symbol by symbol, its training recipe and results — checking every number the way a reviewer would.
55 min
After this lesson you can
- Extract the paper's claims from its abstract and locate the evidence for each
- Decode Eq. (1), multi-head attention and the positional encoding symbol by symbol and shape by shape
- Verify the paper's own arithmetic — training time, FLOP estimates, cost ratios — from the numbers it reports
- Read its ablation table and separate what it demonstrates from what it only hypothesises
- Notice and explain the 41.0 versus 41.8 BLEU discrepancy between versions
In this lesson
- Pass 1: the claims (ten minutes)
- Pass 2: the argument
- The results, and a discrepancy (Section 6.1)
- Pass 3: what the ablations show (Section 6.2)
- Reproducing it
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.