Interpretability
Why look inside a trained network, how to attribute a prediction to its inputs (gradients, integrated gradients, SHAP, LIME), why those attributions can mislead, and what probing and mechanistic interpretability add — with an honest account of what counts as an explanation.
50 min
After this lesson you can
- Say what feature attribution is for, and why the plain gradient is a poor attribution
- State the integrated-gradients axioms and derive completeness from the chain rule
- Compute Shapley values by brute force and say how SHAP and LIME approximate an explanation
- Describe the sanity checks and impossibility results that limit attribution methods
- Explain what a probe measures, why it is correlational, and what mechanistic interpretability adds
In this lesson
- The gradient: the first answer, and why it fails
- Integrated gradients
- Shapley values, SHAP and LIME
- When attributions fail
- Looking inside: probes
- Mechanistic interpretability: circuits
- What counts as an explanation?
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.