Speech and audio
Sound is a long sequence of pressure samples with no word boundaries marked. How a spectrogram turns it into a picture of frequencies, how CTC trains without knowing where each letter is, and how Whisper and neural codecs fit in.
55 min
After this lesson you can
- Explain sampling and aliasing, and compute what a DFT bin measures
- Derive what the short-time Fourier transform computes and state its time–frequency trade-off
- Build a log-mel spectrogram with the same settings as Whisper's front end
- Define CTC's alignments and collapse rule, and compute p(Y|X) with the forward algorithm
- Describe Whisper's architecture and training data accurately, including its documented limitations
- Compute word error rate from an edit-distance alignment, and explain why it can exceed 100%
In this lesson
- The data: a waveform, sampled
- The Fourier transform: which frequencies are present?
- The alignment problem, and CTC
- Sequence-to-sequence recognition and Whisper
- How recognisers are evaluated: word error rate
- Generating audio: WaveNet and neural codecs
- Open problems
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.