From n-grams to neural language models
Count tables go dark as the context grows, and they can never tell that "dog" is like "cat". Bengio and colleagues fixed both in 2003 by giving every word a learned vector, which is where every modern language model's embedding table comes from.
45 min
After this lesson you can
- Explain the sparsity problem of n-gram models and measure it on real text
- Apply add-one (Laplace) smoothing and say why it is not enough
- Describe Bengio et al.'s neural probabilistic language model, its equation and its parameter count
- Explain how learned word vectors let a model give probability to word sequences it has never seen
In this lesson
- The table goes dark
- Patch one: smoothing
- The real problem: every word is a stranger
- The idea: give every word a vector
- The model, written out
- The road from here
- The code
- In the papers
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.