Phase 3 — Core AI Engineering · Lesson 31 · 20 XP
Evals I: test sets and metrics
Without an eval, you're guessing whether a prompt or pipeline change actually helped. An eval set is a collection of representative real inputs paired with an expected output or explicit acceptance criteria — the same idea as a test suite, applied to model behavior instead of code behavior.
A 3-example eval set can pass 100% by luck; a 30-example set that covers real edge cases gives you actual confidence. Metrics differ by what you're measuring: retrieval precision/recall (did the right chunk get retrieved?) is a different question from answer quality (was the final generated answer actually correct?) — you usually need both.
Exercise
Build a 15-20 example test set for your RAG pipeline: a question, plus the expected answer or expected source. Score your current pipeline against it with a simple metric, e.g. whether the correct source landed in the top 3 retrieved chunks.
Check yourself
1. Why does an eval set of 3 examples give much less confidence than one of 30, even if both pass 100%?
2. What's the difference between a metric that checks retrieval quality and one that checks the final generated answer?
Hybrid search, reranking, citations
Answer the check-yourself questions to unlock this