Descent
XIX.2Under the HoodPro

Autodiff from scratch

Every framework's core is a machine that differentiates programs. Build one — forward mode with dual numbers, then a reverse-mode engine in about fifty lines — train a network with it, and check it against finite differences and PyTorch.

50 min

After this lesson you can

  • Turn a program into a computational graph and compute its gradient by hand with the adjoint rule
  • Explain forward mode with dual numbers () and reverse mode with a recorded graph
  • Say when each mode is efficient using the shape of the Jacobian (JVP gives a column, VJP gives a row)
  • Implement a scalar reverse-mode autograd engine with topological sorting and gradient accumulation
  • Verify gradients with finite differences and against PyTorch

In this lesson

  1. Three ways to differentiate
  2. Programs are graphs
  3. Forward mode: dual numbers
  4. Which mode when? JVPs and VJPs
  5. Build a reverse-mode engine
  6. Train a network with it
  7. Check it against PyTorch

This lesson is part of Descent Pro

Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.

See plansSign in / Sign up

Descent Pro

Start your free trial to keep reading

This lesson, and every other lesson on the route, is part of Descent Pro. Every plan starts with a free trial, and students with a .edu email get longer. Cancel before the trial ends and you pay nothing.

Start free trialSign in / Sign up