Phase 3 — Core AI Engineering · Lesson 32 · 20 XP
Evals II: LLM-as-judge and regression testing
Exact-match metrics don't work for open-ended answers — two correct answers can be worded completely differently. LLM-as-judge uses a model, given an explicit rubric, to score another model's output. It's more flexible than exact match, at the cost of being another model call that can itself be wrong or biased.
Known judge biases include favoring longer, more verbose answers, and favoring whichever answer appears first when comparing two. Mitigate both with a fixed, explicit rubric, a few labeled examples in the judge prompt, and randomizing answer order when comparing two outputs.
Regression testing means rerunning your full eval set every time you change a prompt, model, or pipeline step — so a change that quietly makes things worse for some inputs gets caught immediately, instead of showing up as a vague complaint weeks later.
Exercise
Write an LLM-as-judge prompt with an explicit rubric to score your RAG pipeline's answers on your Lesson 31 test set. Run it before and after a prompt change and see whether the score actually moved in the direction you expected.
Check yourself
1. Why is an LLM judge more useful than exact string match for grading open-ended answers, and what's the tradeoff?
2. Name one concrete way to reduce bias in an LLM-as-judge setup.
Evals I: test sets and metrics
Answer the check-yourself questions to unlock this