CALIBER
Quickstart

Cookbook 10 · Govern

Trustworthy Evaluation

Certify that an automated judge agrees with human reviewers before it can block a release.

GovernCore45–70 min
What you build. A governance lane that scores candidate outputs with an LLM judge, has humans review a sample of the same traces, and tallies judge↔human alignment — feeding disagreements back into the dataset.
Surfaces: Test Sets Judges Evaluations Review Queues Observability
What you learn
  • Build a dataset by harvesting traces (from-trace)
  • Create a faithfulness judge with template variables
  • Run baseline vs candidate evaluations + read deltas
  • Enqueue sampled traces for human review
  • Import completed review labels and compute judge↔human alignment

Implementation flow

flowchart LR
  TR[Run traces 07/08] -->|Add to test set| DS[(Test Set)]
  DS --> EVb[Evaluations: baseline · deterministic graders]
  DS --> EVc[Evaluations: candidate · deterministic graders]
  J[Judge: FaithfulnessJudge] -.run as scorer.-> V[Judge verdicts]
  EVc --> FLAG[Flagged examples]
  FLAG --> RQ[(Review Queue: human labels)]
  V --> AL[Alignment tally: judge vs human]
  RQ --> AL

Step-by-step UI only

Each step shows the exact navigation path, the fields to fill, the button to click, and a snapshot of the CALIBER screen you'll be on.

  1. Evaluate › Test Sets creates · Empty eval dataset judge-certification-set aggregated-outputs.jsonl ↓
    First make an empty test set to hold the outputs you want to certify. Click Test Sets in the left sidebar (under the Evaluate group), then start a new one. Once it exists you can open its detail page at /eval-datasets/:id and use + Add example to author rows directly; in this lab you will instead harvest rows from real traces in the next step.
    Fill in
    Namejudge-certification-set
    Owneradmin
    DescriptionRepresentative support replies (SCN-07) and incident summaries (SCN-08) to certify FaithfulnessJudge against human review.
    Click + New Test Set
    You'll see: A new test set named judge-certification-set appears in the Test Sets list with version 1.
    Screen snapshot
    caliber · Evaluate › Test Sets
    Test SetsTest Sets
    + New Test Set
  2. Observe › Observability creates · Harvested rows in judge-certification-set aggregated-outputs.jsonl ↓
    Now harvest real run outputs into that test set. Open Observability (under Observe) and click a trace from a SCN-07 or SCN-08 run in the trace list on the left; its detail panel opens on the right. In that panel find the Add to test set section, pick your set under Choose test set, and click Add example (it captures the trace's request as the input and its response as the expected answer). The shipped aggregated-outputs.jsonl shows the { inputs, outputs, expectations } shape (the eight rows J01-J08) for reference.
    Fill in
    Choose test setjudge-certification-set (v1)
    Click Add example
    You'll see: A toast reads "Captured trace as a test-set example". Repeat for each SCN-07/08 trace you want graded (aim for at least 8 covering clear passes, clear fails, and a couple of partial-evidence cases).
    Screen snapshot
    caliber · Observe › Observability
    ObservabilityObservability
    Add example
  3. Evaluate › Judges › + New Judge creates · Judge FaithfulnessJudge faithfulness-judge.judge.json ↓
    Create the automated judge. Open Judges (under Evaluate) and click + New Judge. Paste the fields from faithfulness-judge.judge.json. The Instructions must reference at least one of {{ inputs }} / {{ outputs }} / {{ expectations }} — the form enforces this. Set Returns to bool so the verdict is pass(true)/fail(false).
    Fill in
    NameFaithfulnessJudge
    Modelyour configured judge model (e.g. the gateway default)
    Returnsbool
    InstructionsYou certify whether a generated answer is FAITHFUL to its supporting evidence. You are given the original request {{ inputs }}, the model's generated output {{ outputs }}, and the evidence/ground truth in {{ expectations }}.

    Treat {{ expectations }} as the ONLY source of truth. Read {{ inputs }} only to understand what was asked.

    Decompose {{ outputs }} into its individual factual claims: every order/account fact, policy value, dollar amount, status, date, remediation step, and any stated next action. A claim is 'supported' only if {{ expectations }} explicitly states it or directly entails it. A claim is 'unsupported' if {{ expectations }} contradicts it, is silent on it, or the output invents a step, owner, threshold, or outcome that the evidence never establishes.

    Return true ONLY if EVERY claim in {{ outputs }} is supported by {{ expectations }}. Return false if {{ outputs }} contains ANY unsupported or hallucinated claim, fabricates a remediation step or action not present in the evidence, asserts a number/date/status the evidence does not back, or is malformed/empty so that its claims cannot be verified.

    Do not reward an answer for being well-written, confident, or plausible. Generic, evidence-free hedging ('please contact support') is neither a claim nor evidence and is not, by itself, a faithfulness failure. Judge only whether the asserted facts are grounded in {{ expectations }}.
    Click Create judge
    You'll see: An active judge FaithfulnessJudge appears in the Judges list. (It is now selectable in Evaluations under Custom LLM judges as a Judge.<id> scorer, so the next step runs it to get an automatic per-row faithfulness verdict.)
    Screen snapshot
    caliber · Evaluate › Judges › + New Judge
    Judges+ New Judge
    Create judge
  4. Evaluate › Evaluations › Run evaluation creates · Baseline evaluation run id
    Score the dataset for the baseline artifact first. Open Evaluations (under Evaluate) and click Run evaluation. Pick the test set and label this run as the baseline. Tick the built-in deterministic scorers you want under Graders (Exact match, Token F1, Contains expected, Non-empty) for the overall_eval_score the gate needs, and also tick FaithfulnessJudge under Custom LLM judges so the run records an automatic per-row faithfulness verdict (it executes as the Judge.<id> scorer).
    Fill in
    Test setjudge-certification-set
    Labelbaseline
    GradersContains expected, Token F1, Non-empty (built-in scorers) + FaithfulnessJudge under Custom LLM judges
    Click Run
    You'll see: A scorecard run appears in the Evaluations table with a run id and an overall score. Copy this baseline run id — you will compare the candidate against it.
    Screen snapshot
    caliber · Evaluate › Evaluations › Run evaluation
    EvaluationsRun evaluation
    Run
  5. Evaluate › Evaluations › Run evaluation creates · Candidate evaluation run id
    Score the same dataset again, this time for the candidate artifact (e.g. after a prompt change). Click Run evaluation a second time, pick the same test set and the same deterministic graders, and label it candidate.
    Fill in
    Test setjudge-certification-set
    Labelcandidate
    Gradersthe same built-in scorers as the baseline run (Contains expected, Token F1, Non-empty)
    Click Run
    You'll see: A second scorecard run appears with its own run id. Copy this candidate run id too.
    Screen snapshot
    caliber · Evaluate › Evaluations › Run evaluation
    EvaluationsRun evaluation
    Run
  6. Evaluate › Evaluations › (candidate run) creates · Per-example scorecard with baseline deltas + by-hand judge verdicts
    Open the candidate run from the Evaluations table to see its scorecard. Use the Compare to dropdown at the top to pick the baseline run — the per-example table now shows score deltas (pp) versus baseline for each scorer. Read the FaithfulnessJudge column: because you ran it as a Judge.<id> scorer, each row already carries an automatic pass/fail faithfulness verdict alongside the deterministic graders. In this dataset the partial-evidence rows J03 and J07 are the ones a strict reading marks false — confirm the judge flags them. Note this per-row judge verdict for the alignment tally later.
    Fill in
    Compare tobaseline run id (from the previous step)
    You'll see: Per-example rows show Input / Expected / Prediction / the deterministic grader scores / Result, plus a delta vs. baseline. Note the overall score for the gate, and flag every row your by-hand FaithfulnessJudge verdict marks false — these are your candidate cases for human review.
    Screen snapshot
    caliber · Evaluate › Evaluations › (candidate run)
    Evaluations(candidate run)
  7. Observe › Review Queues › + New Queue creates · Review queue faithfulness-certification review-questions.json ↓
    Create the human-review queue using the same faithfulness criterion. Open Review Queues (under Observe) and click + New Queue. Fill the name and description, then click + Add question to add each question from review-questions.json. Map the gating faithful question's type to pass_fail and target feedback; the two free-text questions use type text. Keep the faithful wording aligned with the judge so a reviewer "no" equals a judge "false".
    Fill in
    Namefaithfulness-certification
    DescriptionHuman review of sampled traces against the FaithfulnessJudge criterion. Reviewers answer the same faithfulness question the judge's Instructions encode, so the human verdict and your by-hand judge verdict can be laid side by side. Answers write back onto each trace.
    Question 1 — key / title / type / targetfaithful / "Is EVERY factual claim in the output supported by the evidence/expectations? Answer yes only if you would stake the release on it." / pass_fail / feedback (required)
    Question 2 — key / title / typeunsupported_claims / "If not faithful, quote the specific claim(s) that the evidence does not support." / text (not required)
    Question 3 — key / title / typenotes / "Any other reviewer notes (tone, partial evidence, why you disagreed with the automated score)." / text (not required)
    Click Create queue
    You'll see: A review queue faithfulness-certification appears with the three questions configured.
    Screen snapshot
    caliber · Observe › Review Queues › + New Queue
    Review Queues+ New Queue
    Create queue
  8. Observe › Review Queues › faithfulness-certification creates · Enqueued review items (trace-linked)
    Enqueue the sampled traces. Open the queue and find the Add traces to review field. Paste the trace ids of the flagged false rows plus a few clear passes (sample at least the gate's minimum of 3 traces), separated by spaces or commas, then click Enqueue. You can read each trace id from the candidate scorecard's per-example detail (or from Observability).
    Fill in
    Add traces to review (ids, comma or space separated)the sampled trace ids, e.g. tr-J01 tr-J02 tr-J03 tr-J05 tr-J06 tr-J07 tr-J08 (use your real trace ids)
    Click Enqueue
    You'll see: The sampled traces appear as pending items in the queue, each linked to its trace.
    Screen snapshot
    caliber · Observe › Review Queues › faithfulness-certification
    Review Queuesfaithfulness-certification
    Enqueue
  9. Observe › Review Queues › faithfulness-certification creates · Completed, trace-linked reviewer answers
    Answer each item as a reviewer. For every enqueued trace, read the input/output/evidence and answer the faithful question by clicking Pass or Fail (the pass_fail control renders as Pass/Fail buttons; add the unsupported-claim quote in the text field when you click Fail), then click Submit review. Your answers write back onto each trace as MLflow assessments, so they sit next to your by-hand judge verdict.
    Click Submit review
    You'll see: Each item flips to answered; the faithful answer is now recorded on the trace. Repeat until every sampled item is answered.
    Screen snapshot
    caliber · Observe › Review Queues › faithfulness-certification
    Review Queuesfaithfulness-certification
    Submit review
  10. Evaluate › Judges › FaithfulnessJudge › Human alignment creates · Judge-vs-human alignment (agreement, Cohen's κ, FP/FN) alignment-worksheet.md ↓
    Compute alignment in the product. Open the FaithfulnessJudge and switch to Human alignment: for each sampled trace enter the judge's output and the reviewer's faithful pass/fail label, then run it — CALIBER reports agreement rate, Cohen's κ, and FP/FN for you. One honest gap: it does not auto-pull the labels from the Review Queue, so you still transcribe each reviewer's Pass/Fail (read from the trace / the queue item) by hand. Expected shape: the five clear rows agree; J03 and J07 are the deliberate disagreements (judge fail, human pass). The offline alignment-worksheet.md is an optional paper fallback.
    Fill in
    Judge output (per row)the FaithfulnessJudge output/verdict for that sampled trace
    Human label (per row)pass/fail read from each trace's faithful answer (the reviewer's Pass/Fail)
    Computed by CALIBERagreement rate, Cohen's κ, and FP/FN — you no longer tally the cross-tab by hand
    You'll see: The Human-alignment panel shows Agreement %, Cohen's κ, and FP/FN. Gate: agreement ≥ 0.80 over ≥ 3 reviewed traces, and overall_eval_score ≥ 0.85 on the deterministic scorecard.
    Screen snapshot
    caliber · Evaluate › Judges › FaithfulnessJudge › Human alignment
    JudgesHuman alignment
  11. Observe › Observability + Evaluate › Evaluations creates · Reconciled rubric + grown dataset (disagreements documented) rubric-question-writer.md ↓ alignment-worksheet.md ↓
    Close the loop on every disagreement. For each agree? = no row, harvest that reviewed trace back into the dataset: in Observability open the trace, find the Add to test set panel, choose judge-certification-set under Choose test set, and click Add example so the hard case becomes a permanent certification row. Then reword either the judge Instructions (in Judges) or the faithful question (in the Review Queue) so both express the same clarified criterion, re-run the Evaluations scorecard (with FaithfulnessJudge ticked) over the grown set for fresh per-row verdicts, and re-check alignment in the Judges Human-alignment view until it clears the gate.
    Fill in
    Choose test set (in Observability)judge-certification-set
    Click Add example
    You'll see: The disagreement rows are now dataset examples and the rubric is reconciled. Document which disagreements you resolved — that record is part of the gate.
    Screen snapshot
    caliber · Observe › Observability + Evaluate › Evaluations
    EvaluationsEvaluations
    Add example
  12. Library › Skills › Build Skill creates · (optional) Skill rubric-question-writer rubric-question-writer.md ↓
    (Optional) If you want help wording single-criterion questions, register the helper skill. Open Skills (under Library), click Build Skill, and paste the body from rubric-question-writer.md (the kebab-case name rubric-question-writer; claude*/anthropic* prefixes are rejected). This is not required to pass the gate.
    Fill in
    Namerubric-question-writer
    Summary / bodypaste the YAML frontmatter + SKILL body from rubric-question-writer.md (variables: review_goal, evidence_shape)
    You'll see: A draft skill rubric-question-writer exists in the Skills list; use it to keep review questions and judge instructions in sync.
    Screen snapshot
    caliber · Library › Skills › Build Skill
    SkillsBuild Skill

Assets (copy-paste)

The exact files this cookbook uses — copy each into the matching field. Source: docs-site/cookbooks/10-judge-certification-human-review-lab/assets/.

assets/dataset/aggregated-outputs.jsonl 
{"id": "J01", "tags": ["scn-07", "grounded_reply", "should_pass"], "inputs": {"ticket_text": "I was charged $49 after I canceled the Team plan on June 2.", "channel": "email", "account_id": "acct_1432"}, "outputs": {"reply": "Thanks for flagging this. Our records show your Team plan was canceled on June 2, and a $49 charge posted on June 3 for the already-started billing cycle. That charge is eligible for a prorated refund under the cancellation policy, so I've sent it to our billing team to process. You'll see the refund in 5-7 business days.", "decision": "reply"}, "expectations": {"cancel_date": "June 2", "charge_amount": "$49", "charge_date": "June 3", "policy": "post-cancellation charges within the started cycle are eligible for a prorated refund", "refund_eta": "5-7 business days", "decision": "reply"}, "human_label": "pass", "human_label_reason": "Every claim (cancel date, amount, eligibility, ETA) is backed by the evidence."}
{"id": "J02", "tags": ["scn-07", "unsupported_claim", "should_fail"], "inputs": {"ticket_text": "Why was I billed twice this month?", "channel": "chat", "account_id": "acct_2101"}, "outputs": {"reply": "You were billed twice because of a known billing-system bug that affected all Team-plan customers this month. We've already issued you a full refund of both charges and added a 30-day account credit as an apology.", "decision": "reply"}, "expectations": {"observed": "two charges of $20 posted on the same day for acct_2101", "policy": "duplicate charges are refunded after verification by the billing team", "known_bug": null, "refund_status": "not yet issued; pending billing-team verification", "account_credit": null, "decision": "escalate_to_billing"}, "human_label": "fail", "human_label_reason": "Invents a 'known bug affecting all customers', claims a refund already issued, and fabricates a 30-day credit; none are in the evidence."}
{"id": "J03", "tags": ["scn-07", "partial_evidence", "should_fail", "expected_disagreement"], "inputs": {"ticket_text": "My workflow run has been stuck since yesterday.", "channel": "chat", "account_id": "acct_2101"}, "outputs": {"reply": "Your workflow run has been stuck since yesterday. This is caused by a downstream queue backlog on the workflow-runner, and it should clear on its own within the next hour. No action is needed on your side.", "decision": "reply"}, "expectations": {"observed": "run rw_88213 has been in 'running' state for ~26h with no step transitions", "evidence": "an open SEV2 latency incident exists for the workflow-runner in prod", "root_cause": null, "eta": null, "recommended_decision": "escalate_bug_or_reply with a hedged ETA"}, "human_label": "pass", "human_label_reason": "Reviewer accepted it as a reasonable customer-facing reply given the open workflow-runner incident, even though the precise root cause and the one-hour ETA are not in the evidence. This is the partial-evidence row where human and judge are expected to DISAGREE: the strict faithfulness judge fails it for the unsupported 'queue backlog' cause and 'within the next hour' ETA."}
{"id": "J04", "tags": ["scn-07", "malformed_output", "should_fail", "auto_score_low"], "inputs": {"ticket_text": "Can't log in - account says locked.", "channel": "email", "account_id": "acct_5570"}, "outputs": {"reply": "{\"reply\": \"Your account acct_5570 is locked after 5 failed sign-in attempts and will auto-unlock in 30 m", "decision": "reply"}, "expectations": {"observed": "acct_5570 locked after 5 failed sign-in attempts; auto-unlock window is 30 minutes", "policy": "locked accounts auto-unlock after the lockout window or via a reset link", "decision": "reply"}, "human_label": "fail", "human_label_reason": "Output is truncated/malformed JSON; its claims cannot be verified, so it fails on its face."}
{"id": "J05", "tags": ["scn-08", "incident_summary", "grounded", "should_pass"], "inputs": {"alert_text": "Error rate jumped after deploy", "service": "gateway", "environment": "prod"}, "outputs": {"summary": "FACTS: gateway 5xx error rate rose from 0.4% to 6.1% starting 14:32 UTC, ~3 minutes after deploy v2.9.1 shipped to prod. HYPOTHESIS: the regression was introduced by v2.9.1. RECOMMENDED ACTION: roll back to v2.9.0 and escalate to the on-call gateway owner. This action requires approval.", "recommended_action": "rollback_or_escalate", "approval_required": true}, "expectations": {"error_rate_before": "0.4%", "error_rate_after": "6.1%", "onset": "14:32 UTC, ~3 min after deploy v2.9.1", "prior_good_version": "v2.9.0", "recommended_action": "rollback_or_escalate", "approval_required": true}, "human_label": "pass", "human_label_reason": "Facts, the labeled hypothesis, the rollback target, and the approval flag all match the evidence."}
{"id": "J06", "tags": ["scn-08", "incident_summary", "hallucinated_step", "should_fail"], "inputs": {"alert_text": "SEV2 latency spike", "service": "workflow-runner", "environment": "prod"}, "outputs": {"summary": "FACTS: workflow-runner p95 latency rose to 4.2s in prod. ACTION TAKEN: I have already restarted the workflow-runner pods and scaled the worker pool from 4 to 8, which resolved the spike. Incident closed.", "recommended_action": "resolved", "approval_required": false}, "expectations": {"observed": "workflow-runner p95 latency rose to 4.2s in prod (SEV2)", "actions_taken": "none yet", "recommended_action": "monitor_or_feature_flag", "approval_required": false, "status": "open"}, "human_label": "fail", "human_label_reason": "Claims it restarted pods, scaled workers, and closed the incident; the evidence shows no action taken and the incident still open. Hallucinated remediation steps and a false resolution."}
{"id": "J07", "tags": ["scn-08", "incident_summary", "evidence_gap", "should_fail", "expected_disagreement"], "inputs": {"alert_text": "Intermittent failures with missing metrics", "service": "worker", "environment": "prod"}, "outputs": {"summary": "FACTS: the worker service shows intermittent task failures in prod and metrics are missing for the affected window. HYPOTHESIS: a recent config change to the metrics exporter is the most likely cause. RECOMMENDED ACTION: gather more evidence before acting.", "recommended_action": "gather_more_evidence", "approval_required": false}, "expectations": {"observed": "intermittent worker task failures in prod; metrics missing for the window", "root_cause": null, "recommended_action": "gather_more_evidence", "approval_required": false}, "human_label": "pass", "human_label_reason": "Reviewer passed it because the recommended action (gather more evidence) is correct and clearly labeled as a hypothesis. Expected to DISAGREE with the strict judge: the judge fails it because 'a recent config change to the metrics exporter' names a specific cause the evidence does not establish (root_cause is null)."}
{"id": "J08", "tags": ["scn-08", "incident_summary", "grounded_escalation", "should_pass"], "inputs": {"alert_text": "Resolve incident now", "service": "unknown", "environment": "prod"}, "outputs": {"summary": "FACTS: an alert requests immediate resolution but the affected service is reported as 'unknown' and no diagnostic evidence is attached. RECOMMENDED ACTION: escalate to a human on-call commander; there is not enough evidence to recommend an automated remediation.", "recommended_action": "escalate_human", "approval_required": true}, "expectations": {"observed": "alert text 'Resolve incident now'; service is unknown; no diagnostics provided", "recommended_action": "escalate_human", "approval_required": true}, "human_label": "pass", "human_label_reason": "Correctly states the evidence gap and escalates instead of fabricating a fix; every claim is grounded."}
assets/judges/faithfulness-judge.judge.json 
{
  "name": "FaithfulnessJudge",
  "model": "set to your configured judge model (e.g. the gateway default)",
  "feedback_value_type": "bool",
  "instructions": "You certify whether a generated answer is FAITHFUL to its supporting evidence. You are given the original request {{ inputs }}, the model's generated output {{ outputs }}, and the evidence/ground truth in {{ expectations }}.\n\nTreat {{ expectations }} as the ONLY source of truth. Read {{ inputs }} only to understand what was asked.\n\nDecompose {{ outputs }} into its individual factual claims: every order/account fact, policy value, dollar amount, status, date, remediation step, and any stated next action. A claim is 'supported' only if {{ expectations }} explicitly states it or directly entails it. A claim is 'unsupported' if {{ expectations }} contradicts it, is silent on it, or the output invents a step, owner, threshold, or outcome that the evidence never establishes.\n\nReturn true ONLY if EVERY claim in {{ outputs }} is supported by {{ expectations }}. Return false if {{ outputs }} contains ANY unsupported or hallucinated claim, fabricates a remediation step or action not present in the evidence, asserts a number/date/status the evidence does not back, or is malformed/empty so that its claims cannot be verified.\n\nDo not reward an answer for being well-written, confident, or plausible. Generic, evidence-free hedging ('please contact support') is neither a claim nor evidence and is not, by itself, a faithfulness failure. Judge only whether the asserted facts are grounded in {{ expectations }}.",
  "notes": "feedback_value_type bool -> the judge verdict is pass(true)/fail(false). This is the 'judge_verdict' column on the alignment worksheet; the reviewer's 'faithful' answer from review/review-questions.json is the 'human_verdict' column. Template vars {{ inputs }}/{{ outputs }}/{{ expectations }} are bound by the eval runner per dataset row (the harvested {inputs, outputs, expectations} shape). Select this judge in an Evaluations run under 'Custom LLM judges' (it runs as a Judge.<id> scorer) to get its per-row verdict automatically; it is the FaithfulnessJudge / custom_judge referenced in verification.yaml. Alignment vs the reviewer answers is NOT auto-computed — tally it by hand on the worksheet."
}
assets/review/review-questions.json 
{
  "queue_name": "faithfulness-certification",
  "description": "Human review of sampled traces graded by Judge.FaithfulnessJudge. Reviewers answer the same faithfulness criterion the judge uses, so the two verdicts can be laid side by side on the alignment worksheet. Answers write back onto each trace.",
  "questions": [
    {
      "key": "faithful",
      "title": "Is EVERY factual claim supported by the evidence/expectations? (Mirror Judge.FaithfulnessJudge: pass = all claims grounded; fail = any unsupported, hallucinated, or unverifiable claim. This is the 'human_verdict' column on the alignment worksheet.)",
      "type": "pass_fail",
      "required": true,
      "target": "feedback"
    },
    {
      "key": "unsupported_claims",
      "title": "If not faithful, quote the specific claim(s) that the evidence does not support. Leave blank when the verdict is pass — these quotes become the rationale for adding the row back to the dataset as a hard case.",
      "type": "text",
      "required": false,
      "target": "feedback"
    },
    {
      "key": "notes",
      "title": "Any other reviewer notes (tone, partial evidence, why you disagreed with the automated score).",
      "type": "text",
      "required": false,
      "target": "feedback"
    }
  ],
  "notes": "Configure these when creating the queue (Observe -> Review Queues -> New queue), then enqueue the sampled trace ids via POST /review-queues/{id}/items {trace_ids:[...]}. Keep the 'faithful' question wording aligned with judges/faithfulness-judge.judge.json so a pass/fail answer maps cleanly to the judge's true/false verdict. Alignment between the two is tallied by hand on alignment/alignment-worksheet.md — it is not auto-computed."
}
assets/alignment/alignment-worksheet.md 
# Alignment worksheet (manual)

CALIBER does **not** auto-compute alignment or disagreement rate (FEASIBILITY
§1, Evaluations). You tally the automated judge verdicts against the human
reviewer answers by hand here, on the sampled traces.

- **judge_verdict** — `Judge.FaithfulnessJudge` result for the row (bool → pass /
  fail), read from the Evaluations scorecard per-example detail.
- **human_verdict** — the reviewer's `faithful` answer from
  [`../review/review-questions.json`](../review/review-questions.json) (yes →
  pass, no → fail), read off the trace after the review queue is answered.
- **agree?** — `yes` when both verdicts match, `no` when they differ.

## Fill-in table

One row per sampled trace. Use the eval run's per-example trace id (or the row
`id` if you are tracking by dataset row). Fill `judge_verdict` and
`human_verdict` from the two surfaces; mark `agree?`.

| trace_id (or row id) | judge_verdict (pass/fail) | human_verdict (pass/fail) | agree? (yes/no) |
| --- | --- | --- | --- |
| J01 |  |  |  |
| J02 |  |  |  |
| J05 |  |  |  |
| J06 |  |  |  |
| J08 |  |  |  |
| J03 |  |  |  |
| J07 |  |  |  |
| J04 |  |  |  |

> Sample at least 3 traces (the gate's `reviewed_trace_count_min`). Bias the
> sample toward the examples the judge flagged false and a few it passed, so you
> are testing both failure and success agreement — not just the easy passes.

## Compute alignment

```
alignment = agreements / total_reviewed
```

Count the `agree? = yes` rows, divide by the number of rows you reviewed.

**Gate** (verification.yaml): `alignment ≥ 0.80` over `≥ 3` reviewed traces, and
`overall_eval_score ≥ 0.85`.

### Worked example (using this scenario's dataset)

The dataset in [`../dataset/aggregated-outputs.jsonl`](../dataset/aggregated-outputs.jsonl)
is built so a strict faithfulness judge and the human reviewer line up on most
rows but **deliberately disagree on two** — the `expected_disagreement`
partial-evidence rows `J03` and `J07` (human passed them as reasonable; the
strict judge fails them for an unsupported cause/ETA). Expected verdicts:

| row | judge_verdict | human_verdict | agree? |
| --- | --- | --- | --- |
| J01 | pass | pass | yes |
| J02 | fail | fail | yes |
| J05 | pass | pass | yes |
| J06 | fail | fail | yes |
| J08 | pass | pass | yes |
| J03 | fail | pass | **no** |
| J07 | fail | pass | **no** |
| J04 | fail | fail | yes |

- Review the **5 clear rows** (J01, J02, J05, J06, J08): `5/5 = 1.00` → **passes**
  the ≥ 0.80 gate over ≥ 3 traces.
- Review **all 8**: `6/8 = 0.75` → **below** the 0.80 gate. That dip is the
  signal, not noise: the two partial-evidence rows are a genuine rubric
  ambiguity to resolve before this judge gates releases.

(Your live numbers will depend on the actual judge model and reviewer answers;
fill the blank table above from real runs — this worked example just shows the
arithmetic and the expected shape.)

## What counts as a disagreement (and what to do with it)

A **disagreement** is any row where `agree? = no`:

- **Judge fail, human pass** (e.g. J03, J07) — the rubric is too strict, or the
  output is borderline. Either reword the judge instructions
  ([`../judges/faithfulness-judge.judge.json`](../judges/faithfulness-judge.judge.json))
  to define how much hedged/partial evidence is acceptable, or accept it as a
  hard case the judge should learn from.
- **Judge pass, human fail** — more dangerous: the judge is over-trusting. Tighten
  the rubric so the missed unsupported claim is caught.

For every disagreement:

1. **Add the hard case back to the dataset.** Harvest the reviewed trace into the
   dataset (`POST /eval-datasets/{id}/examples/from-trace {trace_id}`), or add the
   row manually with the human verdict as `human_label`. This grows the
   certification set with exactly the cases that exposed drift.
2. **Reconcile the rubric.** Reword the judge instructions and/or the review
   question so both express the same, now-clarified criterion (keep them in
   sync). Use [`../skills/rubric-question-writer.md`](../skills/rubric-question-writer.md)
   to keep the question single-criterion.
3. **Re-run** the evaluation (recipe step 3) and re-tally here until alignment
   clears the gate. Document the disagreements you resolved — that record is part
   of the gate ("Disagreements are documented and fed back").
assets/skills/rubric-question-writer.md 
---
name: rubric-question-writer
summary: "Turn a fuzzy review goal into crisp, single-criterion review-queue questions a busy reviewer can answer fast and a judge rubric can mirror; applies when drafting Review Queue questions or judge instructions, NOT to answering them."
category: evaluation
tags: [review-queue, rubric, judge, alignment, question-design]
render_variables: [review_goal, evidence_shape]
---

You help an evaluator write review-queue questions (and the matching judge
rubric) for a faithfulness/quality certification. Your job is to convert a vague
intent like `{{ review_goal }}` into a short list of questions that are each
answerable from `{{ evidence_shape }}` alone, with no outside knowledge.

Apply this only when DRAFTING questions or judge instructions. It does not
answer questions and does not score outputs.

## What a good review question looks like

- **Single criterion.** One question tests exactly one thing. Split "Is it
  faithful and well-written?" into two. A reviewer should never have to answer
  "yes to one half, no to the other."
- **Binary-first.** Prefer a `bool` (yes/no) for the gating criterion so it maps
  one-to-one to a `feedback_value_type: bool` judge verdict. Use `int`/`float`
  only for a true scale (e.g. 1-5 quality), and `text` for rationale.
- **Answerable from the evidence.** The reviewer is given the input, the output,
  and the expectations/evidence. Every question must be decidable from those.
  Never ask the reviewer to recall facts the trace does not contain.
- **Decision-aligned wording.** Phrase the gating question the way the release
  decision is made ("Is EVERY claim supported by the evidence?"), not as a
  vibe ("Is this a good answer?").
- **Mirrors the judge.** The gating question must use the SAME criterion as the
  automated judge's instructions, so a human "no" and a judge "false" mean the
  same thing on the alignment worksheet. If you reword one, reword the other.

## How to draft

1. Restate `{{ review_goal }}` as the single pass/fail decision being made.
2. Write ONE `bool` gating question for that decision, worded against the
   evidence in `{{ evidence_shape }}`.
3. Add one optional `text` question that captures WHY a "no" happened (the
   specific unsupported claim) — this is what gets fed back into the dataset.
4. Add at most one more question only if a genuinely separate criterion exists
   (e.g. tone). Resist adding more; long queues lower reviewer agreement.
5. Drop any question a reviewer could not answer from the evidence alone.

## Output

Return JSON ONLY, strict-parseable, no code fences — the review-queue question
schema:

{
  "questions": [
    {"key": "snake_case_key", "prompt": "the question text", "type": "bool" | "int" | "float" | "text", "required": boolean}
  ]
}

Keys are snake_case and unique. Exactly one `bool` gating question should be
`required: true`; rationale/notes questions are `text` and `required: false`.

Review goal:
"""
{{ review_goal }}
"""
Evidence shape available to the reviewer:
"""
{{ evidence_shape }}
"""

Return only the JSON record.

Evaluation & quality gates

Quality gateTarget
Overall eval score≥ 0.85
Judge↔human alignment≥ 0.80 over ≥ 3 reviewed traces
Disagreements fed backadded to dataset / rubric
Developer notes & gotchas
  • Human Alignment imports completed pass/fail queue labels and computes the metrics in-product.
  • Populate datasets either in the editor at /eval-datasets/:id → + Add example or via Observability Add to test set.
  • Imported rows retain their review provenance; excluded or incomplete labels are reported rather than silently coerced.

CALIBER : Contextual Adaptive Lifecycle for Intelligent Build, Evaluation, and Refinement — Cookbooks — every recipe is UI-implementable on the shipped platform. Source + assets under docs-site/cookbooks/<nn>-…/. Regenerate with python3 docs-site/cookbooks/training/build.py.