Phase 4 — Production & Depth · Lesson 44 · 20 XP
Multimodal: vision, speech, documents
A multimodal model can take images alongside text in the same request — useful for reading a chart, extracting data from a photographed receipt, or interpreting a UI screenshot, without writing custom image-processing code yourself.
Speech has two separate concerns: transcription (speech-to-text) and synthesis (text-to-speech) — neither is the LLM itself, they're separate models you compose with one. Document parsing has a similar split: a text-based PDF can be read directly, but a scanned PDF is really just an image of text and needs OCR (or a layout-aware parser) before it can be chunked and embedded for RAG at all.
Exercise
Send an image — a screenshot or a photo of a receipt — to a multimodal model API and extract structured data from it, validating the result with a Pydantic model (Lesson 8, Lesson 22).
Check yourself
1. Why might a scanned PDF need OCR before it can be used in a RAG pipeline, while a text-based PDF doesn't?
2. Name one task where sending an image directly to a multimodal model beats writing custom image-processing code.
Open-weights models and quantization
Answer the check-yourself questions to unlock this