Training at scale
A frontier pretraining run needs more arithmetic than any one chip could do in centuries and more memory than any one chip has. Where the FLOPs and bytes go, how the data is cleaned, how data, tensor and pipeline parallelism split the work across thousands of GPUs — and the instabilities and hardware failures that come with it.
55 min
After this lesson you can
- Estimate the compute (≈ 6ND FLOPs), model-state memory (≈ 16 bytes per parameter) and activation memory of a training run
- Explain deduplication and quality filtering of pretraining data, with published evidence for each
- Describe data parallelism, ZeRO/FSDP sharding, Megatron tensor parallelism and pipeline parallelism, and what each communicates
- Compute the pipeline bubble and explain why 1F1B saves memory
- Name the common instabilities and failure modes of large runs and the mitigations reported for them
In this lesson
- The budget
- The data
- Splitting the work, 1: data parallelism
- Splitting the work, 2: tensor parallelism
- Splitting the work, 3: pipeline parallelism
- Putting them together
- When things go wrong
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.