Phase 4 — Production & Depth · Lesson 43 · 20 XP
Open-weights models and quantization
Open-weights models (like the Llama or Mistral families) can be downloaded and run yourself, unlike closed models that are only reachable through an API. Ollama makes running one locally simple for experimentation; vLLM is built for higher-throughput serving in production.
Quantization reduces the numeric precision of a model's weights — e.g. from 16-bit down to 4-bit — which shrinks memory and compute requirements substantially, at some cost to output quality. Teams self-host for reasons like data never leaving their infrastructure, predictable fixed costs at high volume, or needing a model fine-tuned specifically for one narrow task.
Exercise
Install Ollama, pull a small open-weights model, and run it locally. Compare its output quality and latency on a few prompts against an API-based model you've used earlier in the course.
Check yourself
1. What does quantizing a model's weights trade away in exchange for lower memory use?
2. Give one real reason a team would choose to self-host an open-weights model instead of calling an API.
Durable workflows for long-running agents
Answer the check-yourself questions to unlock this