Predicting the next letter
Any text is a chain of next-symbol predictions. Markov counted them by hand in a poem, Shannon built fake English from them, and then he measured how predictable real English is by making people guess.
45 min
After this lesson you can
- Write the probability of a text as a product of next-symbol predictions (the chain rule) and state the Markov assumption
- Explain Shannon's series of approximations to English and generate them with a character n-gram model
- Describe Shannon's 1951 guessing experiment and how guess counts bound the entropy of English
- Estimate next-symbol probabilities from counts in NumPy and sample from them
In this lesson
- A poem, counted by hand
- Every text is a chain of predictions
- Shannon’s fake English
- The guessing game
- Is there a limit? What’s still unknown
- The code
- In the papers
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.