Phase 3 — Core AI Engineering · Lesson 39 · 40 XP
Project: RAG app with an eval suite
The first Phase 3 capstone: a RAG app over a real set of documents you choose, with an eval suite (Lessons 31-32) that actually measures retrieval quality and answer quality — not just "it seems to work when I try it."
A useful write-up names a specific metric your eval suite reports and what a low score on it tells you to go fix. If retrieval precision is low, the problem is upstream — chunking or the retrieval query. If retrieval is fine but answers are still wrong, the problem is in generation — the prompt, or the model not using the retrieved context well.
Exercise
Build a RAG app over a real set of documents you choose, with an eval suite measuring retrieval quality and answer quality, and a short write-up of the measured results — the first Phase 3 portfolio project.
Check yourself
1. Name one metric your eval suite reports, and say what a low score on it tells you to go fix — retrieval or generation.
2. If you doubled your chunk size, what would you expect to happen to your eval scores, and why?
Fine-tuning: when and how
Answer the check-yourself questions to unlock this