Fine-tuning and LoRA
A pretrained model knows a lot but does not yet do your task. Full fine-tuning rewrites every weight and needs roughly sixteen bytes of GPU memory per parameter; parameter-efficient methods freeze the model and train a tiny add-on instead. LoRA, the most widely used of them, learns a low-rank change BA to each weight matrix. Why that works, when it doesn't, and how QLoRA squeezed a 65-billion-parameter fine-tune onto one GPU.
50 min
After this lesson you can
- Count the bytes per parameter that full fine-tuning with Adam needs, and say where each byte goes
- Explain adapters, prefix tuning and prompt tuning in one sentence each, and what each costs
- Decode LoRA's update , including why starts at zero and how merging works
- Derive the gradients of a LoRA layer and train one from scratch in NumPy
- Describe QLoRA's three ingredients and the evidence on LoRA versus full fine-tuning, including forgetting
In this lesson
- What fine-tuning is
- The memory bill
- Freeze the model, train an add-on
- LoRA: change each matrix by a low-rank amount
- Forgetting
- QLoRA: compress the frozen part too
- In real code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.