Mixture of experts
Give the model many feed-forward "experts" but let each token use only a few of them, so the parameter count grows while the compute per token stays fixed, and learn why keeping the experts evenly busy is the hard part.
45 min
After this lesson you can
- Explain conditional computation and the difference between total and active parameters
- Write the top-k gating equations used by Switch Transformer, Mixtral and DeepSeekMoE
- Derive the load-balancing loss, explain why it works, and compute it in NumPy
- Reproduce Mixtral's 47B total / 13B active parameter counts from its configuration
- Weigh the trade-offs of MoE in memory, communication, training stability and decoding
In this lesson
- Where the parameters are
- The router and its experts
- Keeping the experts busy: the load-balancing loss
- Total versus active parameters: Mixtral, worked out
- Finer experts, and experts everyone shares: DeepSeekMoE
- What MoE costs
- Code: a top-k MoE layer with a load-balancing loss
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.