The transformer block
Attention moves information between tokens; the block around it computes, stabilises and accumulates. Embeddings, the residual stream, the feed-forward network, LayerNorm and where to put it, encoders and decoders — and where a model's parameters actually live.
50 min
After this lesson you can
- Trace a token through a transformer block, naming each sublayer and the shapes it maps between
- Explain the residual-stream view — every sublayer reads from the stream and adds its output back
- Contrast Post-LN and Pre-LN blocks and state what Xiong et al. (2020) found about learning-rate warm-up
- Place BERT, GPT and T5 in the encoder-only, decoder-only and encoder–decoder families
- Derive the ≈ 12·d²_model parameters per layer and count a real model exactly
In this lesson
- What attention can’t do on its own
- Tokens in: embeddings
- The block, piece by piece
- The residual stream
- Stacking the blocks: encoder, decoder, and three families
- Where the parameters live
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.