Evaluating language models
Every claim that one model is better than another rests on an evaluation, and most evaluations are noisier, leakier and more fragile than the leaderboard makes them look. Perplexity, multiple-choice benchmarks, LLM judges and human preference arenas — what each measures, how each misleads, and the statistics that tell a real difference from noise.
50 min
After this lesson you can
- Explain why per-token perplexity can't compare models with different tokenizers, and what bits per byte fixes
- Describe how multiple-choice benchmarks such as MMLU are scored, and why prompt format changes the score
- Name the documented biases of LLM judges and how arenas turn pairwise votes into ratings
- Explain contamination and saturation, and how each is detected or mitigated
- Put an error bar on a benchmark score and compare two models with a paired analysis
In this lesson
- Perplexity: the intrinsic measure
- Task benchmarks
- Open-ended answers and LLM judges
- Human preference arenas
- Contamination
- Error bars, done properly
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.