The open questions
What we still do not understand about deep learning — why huge networks generalise, what the loss landscape looks like, what the features are, how in-context learning works, why models hallucinate, and where scaling stops — each with the key papers and an honest line between what is established and what is open.
45 min
After this lesson you can
- Explain the random-label experiment and why it broke classical explanations of generalisation
- For each major open question, name what is established, what is conjectured, and the key papers
- Recognise which lessons of this curriculum each question builds on
- Read new claims at the frontier with the right level of scepticism
In this lesson
- The experiment, at laptop scale
- A map of the unknown
- 1. Why do huge networks generalise?
- 2. What does the loss landscape look like?
- 3. Lazy or rich: how are features learned?
- 4. What are the features? Superposition and interpretability
- 5. How does in-context learning work?
- 6. Reasoning, length and test-time compute
- 7. Why so much more data than a child?
- 8. Hallucination and calibration
- 9. Can a network keep learning?
- 10. Will it do what we meant?
- 11. Where does scaling stop?
- Reading the frontier
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.