Use CALIBER
Evaluation and test sets
The practical trust loop: define evidence, run scorecards, inspect judges, and decide what still needs review before live change.
Use this page when you need the practical trust loop: assemble evidence, run scorecards, inspect judges and per-example results, and decide what still needs review before live change.
At a glance
| Task | Start here | Deep reference |
|---|---|---|
| define the evidence base | create or inspect a test set | Test sets architecture |
| score a candidate | run evaluation against that evidence | Evaluation architecture |
| inspect judge behavior | understand the scorer and custom judge surfaces | Evaluation architecture |
| connect runtime QA to engineering evaluation | compare both trust loops explicitly | QA plan |
1. What this path is for
CALIBER separates evidence from rollout. Evaluation tells you how something performed against a known set of cases. It does not release or apply a change.
2. Common tasks
| You want to... | Read this next |
|---|---|
| curate representative examples | Test sets architecture |
| run a scorecard or judge-backed evaluation | Evaluation architecture |
| connect evaluation to calibration | Calibration |
| connect evaluation to runtime review and release | Review and release flows |
3. Common failure modes
| Symptom | First thing to check |
|---|---|
| Scores are high but production still looks weak | the evidence set may not represent the live workload |
| A judge looks inconsistent | inspect custom-judge setup and agreement expectations |
| A good score did not change production | evaluation is evidence, not apply/release |