PyTorch internals
What really happens when you write y = w * x in PyTorch — tensors as views onto storage, a dispatcher that routes every call, an autograd graph recorded as you go, optimisers that update in place, and a compiler that can capture the whole thing.
50 min
After this lesson you can
- Read a tensor's size, stride, storage offset, dtype and device, and predict views and copies
- Explain what the dispatcher does and why the autograd kernel runs before the CPU or CUDA kernel
- Inspect the autograd graph (grad_fn, next_functions, saved tensors) and explain leaves, requires_grad, .grad accumulation, no_grad and retain_graph
- Describe nn.Module parameters and an optimiser step as an in-place update under no-grad
- Explain what torch.compile does with TorchDynamo, AOTAutograd and TorchInductor
In this lesson
- Tensors are views onto storage
- The dispatcher: which code actually runs?
- Eager execution
- Autograd: the graph recorded on the forward pass
- Modules, parameters and optimisers
- torch.compile: capturing the graph
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.