Multimodal models
How to put pictures and sentences in one space. CLIP's contrastive objective decoded, zero-shot classification, vision-language models that feed image tokens to a language model, and how text steers image generation.
50 min
After this lesson you can
- Explain what a joint embedding space is and why paired web data can train one
- Write CLIP's symmetric contrastive loss, derive its gradient with respect to the similarity matrix, and say what the temperature does
- Relate CLIP's objective to InfoNCE
- Classify images zero-shot with text prompts, and state accurately what CLIP's paper and model card claim and warn about
- Describe how Flamingo and LLaVA connect a vision encoder to a language model
- Say how text conditions image generation in diffusion models
In this lesson
- The data: pairs from two different worlds
- CLIP: which caption goes with which image?
- Zero-shot classification
- Feeding images to a language model
- Text to image: conditioning a generator
- Open problems
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.