Aria: Evaluation Harness from Intent

Implementation flow

flowchart LR
  I[/"Intent: a judge for faithfulness + an eval dataset"/] --> PL[Aria Plans: decompose]
  PL --> S1[step: judge.create]
  PL --> S2[step: eval_dataset.create]
  S1 -.create in UI.-> JU[Judges › New judge]
  S2 -.create in UI.-> TS[Test Sets › New dataset]

Step-by-step UI only

Each step shows the exact navigation path, the fields to fill, the button to click, and a snapshot of the CALIBER screen you'll be on.

1. Compose › Plans creates · Aria Plans page open

   Open Aria's planner. Click Plans in the left sidebar (under the Compose group). This opens the Aria Plans page, where you hand Aria a one-line goal and it decomposes it into platform steps you approve before anything runs.

   You'll see: The Aria Plans page with a Goal textarea, an Autonomy selector, and a Decompose goal button.

   Screen snapshot

   caliber · Compose › Plans

   PlansPlans

   Read / inspect this surface — no form to fill.

2. Compose › Plans › Goal creates · Draft Aria plan intent.md ↓

   State the intent. Paste the canonical one-line intent from intent.md into the Goal field. The exact wording matters: the default planner keys off the literal domain words judge and eval dataset, so keep both in the sentence (a natural phrasing like "test set" would need an LLM planner). Set Autonomy to Stop before each change (this is the ask_each level) so Aria pauses to confirm every mutate step.

   Fill in

   Goal	Create a judge for answer faithfulness and an eval dataset to run it on.
   Autonomy	Stop before each change (ask_each)

   Click Decompose goal

   You'll see: A draft plan appears below the form.

   Screen snapshot

   caliber · Compose › Plans › Goal

   PlansGoal

   GoalCreate a judge for answer faithfulness and an eval dataset to run it on.AutonomyStop before each change (ask_each)

   Decompose goal

3. Compose › Plans › (draft plan) creates · Reviewed plan (2 steps) expected-plan.json ↓

   Review the decomposed plan. Confirm Aria produced exactly two steps from the one sentence — judge.create and eval_dataset.create — matching expected-plan.json. This decomposition is the autonomous-orchestration showcase: one intent in, the right governed steps out.

   You'll see: The draft plan lists two mutate steps: judge.create and eval_dataset.create.

   Screen snapshot

   caliber · Compose › Plans › (draft plan)

   Plans(draft plan)

   Read / inspect this surface — no form to fill.

4. Compose › Plans › (draft plan) creates · Approved plan

   Approve the plan shape. With the plan in draft status, click Approve plan. This records that you accept the orchestration before any step runs.

   Click Approve plan

   You'll see: The plan moves to approved status and an Execute button appears.

   Screen snapshot

   caliber · Compose › Plans › (draft plan)

   Plans(draft plan)

   Approve plan

5. Compose › Plans › (approved plan) creates · Plan running, paused at judge.create

   Execute the plan. Click Execute. Because autonomy is ask_each, Aria pauses at the first mutate step (judge.create) and raises a permission interaction headed "Aria needs your approval" with the prompt "Approve step 1: … (judge.create)?" and two buttons, Approve and Deny. Important caveat (verified): the shipped HeuristicPlanner leaves each step's inputs empty and the interaction answer carries only {approved, choice, value} — it never injects a judge/dataset payload. So Approving a step makes Aria run its handler with empty inputs, which fails validation; the clean path is to Deny each step (which marks it skipped) and then create each real artifact on its own page in the following steps.

   Click Execute

   You'll see: The plan starts running and pauses on the judge.create step with a pending permission interaction (an Approve / Deny prompt).

   Screen snapshot

   caliber · Compose › Plans › (approved plan)

   Plans(approved plan)

   Execute

6. Compose › Plans › (interaction prompt) creates · judge.create step resolved (skipped)

   Resolve the judge.create interaction. Because the step has no inputs to run with, click Deny — this marks the step skipped (a terminal state) and advances the plan without a failed handler. (Clicking Approve would instead try to run judge.create with empty inputs and fail validation.) You will create the real judge on the Judges page shortly.

   Click Deny

   You'll see: The judge.create step flips to skipped and the plan pauses again on the eval_dataset.create permission interaction.

   Screen snapshot

   caliber · Compose › Plans › (interaction prompt)

   Plans(interaction prompt)

   Deny

7. Compose › Plans › (interaction prompt) creates · Plan completed (both steps skipped)

   Resolve the eval_dataset.create interaction the same way — click Deny to skip it (the empty-inputs step can't run). With both steps now in a terminal state, the plan settles to completed.

   Click Deny

   You'll see: Both steps show as skipped and the plan reaches completed — the orchestration record is done. Now create the two real artifacts on their own pages.

   Screen snapshot

   caliber · Compose › Plans › (interaction prompt)

   Plans(interaction prompt)

   Deny

8. Evaluate › Judges › + New Judge creates · Judge AnswerFaithfulness answer-faithfulness.judge.json ↓

   Create the judge that Aria's plan called for. Open Judges (under Evaluate), click + New Judge, and paste the fields from answer-faithfulness.judge.json — this file IS the judge.create payload. The Instructions reference {{ inputs }} / {{ outputs }} / {{ expectations }} (at least one is required); set Returns to bool.

   Fill in

   Name	AnswerFaithfulness
   Model	your configured judge model (e.g. the gateway default)
   Returns	bool
   Instructions	You audit whether an assistant answer is faithful to its evidence. You are given the request {{ inputs }}, the model {{ outputs }}, and the reference {{ expectations }}. Return true ONLY if every factual claim in {{ outputs }} is supported by {{ expectations }} (or by evidence cited in {{ inputs }}), and the answer makes no unsupported commitments. If any claim is unsupported, contradicted, or hallucinated, return false. Judge faithfulness only — not style.

   Click Create judge

   You'll see: An active judge AnswerFaithfulness appears in the Judges list. It is authored and reusable here, and because the Evaluations page exposes a Custom LLM judges section you can also pick it as a Judge.<id> grader there — which you do a couple of steps from now.

   Screen snapshot

   caliber · Evaluate › Judges › + New Judge

   Judges+ New Judge

   NameAnswerFaithfulnessModelyour configured judge model (e.g. the gateway default)ReturnsboolInstructionsYou audit whether an assistant answer is faithful to its evidence. You are given the…

   Create judge

9. Evaluate › Test Sets › + New Test Set creates · Test set support-faithfulness-eval support-faithfulness.dataset.json ↓

   Create the eval dataset Aria's plan called for. Open Test Sets (under Evaluate), click + New Test Set, and use the spec from support-faithfulness.dataset.json — this file IS the eval_dataset.create payload Aria's plan named. The page creates an empty versioned test set (the eval_dataset.create capability does the same); you add example rows separately afterward.

   Fill in

   Name	support-faithfulness-eval
   Owner	admin
   Description	Held-out support Q/A for faithfulness scoring. Empty versioned test set; add example rows after creation, then run it in Evaluations on the deterministic graders plus the AnswerFaithfulness judge under Custom LLM judges.

   Click Create

   You'll see: An empty test set support-faithfulness-eval appears in the Test Sets list at version 1.

   Screen snapshot

   caliber · Evaluate › Test Sets › + New Test Set

   Test Sets+ New Test Set

   Namesupport-faithfulness-evalOwneradminDescriptionHeld-out support Q/A for faithfulness scoring. Empty versioned test set; add example…

   Create

10. Evaluate › Judges + Evaluate › Test Sets + Compose › Plans creates · Verified judge + test set + completed plan

    Verify the end-to-end result. Confirm three things: Judges shows AnswerFaithfulness active; Test Sets shows support-faithfulness-eval; and the Plans page shows the plan completed with both steps skipped (the orchestration record). Together these prove Aria turned one sentence into a governed two-step plan, and that you created the two real artifacts the plan called for.

    You'll see: An active judge, a real test set, and a completed two-step plan (steps skipped) — the scenario gate.

    Screen snapshot

    caliber · Evaluate › Judges + Evaluate › Test Sets + Compose › Plans

    PlansPlans

    Read / inspect this surface — no form to fill.

11. Observe › Observability + Evaluate › Evaluations creates · (follow-up) Scored run + AnswerFaithfulness judge verdicts using the Aria-built harness

    (Follow-up) Put the harness to work. Add example rows to support-faithfulness-eval — either author them in the row editor at /eval-datasets/:id → + Add example, or in Observability open a trace, use its Add to test set panel, pick the test set under Choose test set, and click Add example (or reuse a starter dataset such as cookbooks/01-prompt-regression-lab/assets/dataset/). Then go to Evaluations, click Run evaluation, pick the test set, tick the built-in Graders, and tick your AnswerFaithfulness judge under Custom LLM judges so it runs as a Judge.<id> scorer for an automatic per-row verdict. This is the same scored-run loop as SCN-01 / SCN-10.

    Fill in

    Test set	support-faithfulness-eval
    Graders	built-in scorers (Contains expected, Token F1, Non-empty) + AnswerFaithfulness under Custom LLM judges

    Click Run

    You'll see: A scorecard run grades the rows on the deterministic scorers plus the AnswerFaithfulness judge column, closing the loop from one intent to a running evaluation harness.

    Screen snapshot

    caliber · Observe › Observability + Evaluate › Evaluations

    EvaluationsEvaluations

    Test setsupport-faithfulness-evalGradersbuilt-in scorers (Contains expected, Token F1, Non-empty) + AnswerFaithfulness under…

    Run

Assets (copy-paste)

The exact files this cookbook uses — copy each into the matching field. Source: docs-site/cookbooks/12-aria-evaluation-harness/assets/.

# The intent you give Aria

## Canonical (works with the default HeuristicPlanner)

> Create a judge for answer faithfulness and an eval dataset to run it on.

Why this wording: the default planner proposes a step for each capability whose
**domain word** appears in the goal. "judge" → `judge.create`; "eval dataset" →
`eval_dataset.create`. Keep both literal needles in the sentence.

Autonomy: **ask_each** (pauses on every mutate step so you confirm each input).

## Natural-language variant (needs an LLM planner)

> Set up faithfulness scoring for our support answers, with a test set to run it on.

This reads better but relies on an LLM planner to map "test set" → eval_dataset
and "faithfulness scoring" → judge. With the default heuristic planner, use the
canonical wording.

{
  "goal": "Create a judge for answer faithfulness and an eval dataset to run it on.",
  "autonomy": "ask_each",
  "expected_steps": [
    {
      "capability_key": "judge.create",
      "tier": "mutate",
      "interaction": "confirm/refine the judge spec (judges/answer-faithfulness.judge.json)"
    },
    {
      "capability_key": "eval_dataset.create",
      "tier": "mutate",
      "interaction": "confirm/refine the dataset spec (datasets/support-faithfulness.dataset.json)"
    }
  ],
  "notes": "Step order follows registry order; there are no inter-step dependencies in the default planner. Under autonomy=ask_each each mutate step raises one interaction before it runs. Under approve_plan/auto_guarded both run automatically once the plan is approved."
}

{
  "name": "AnswerFaithfulness",
  "model": "set to your configured judge model (e.g. the gateway default)",
  "feedback_value_type": "bool",
  "instructions": "You audit whether an assistant answer is faithful to its evidence. You are given the request {{ inputs }}, the model {{ outputs }}, and the reference {{ expectations }}.\n\nReturn true ONLY if every factual claim in {{ outputs }} is supported by {{ expectations }} (or by evidence cited in {{ inputs }}), and the answer makes no unsupported commitments. If any claim is unsupported, contradicted, or hallucinated, return false. Judge faithfulness only — not style.",
  "notes": "This IS the judge.create payload — POST it to /judges (it is a POST /judges body). Aria's plan proposes the judge.create step, but the shipped interaction answer can't carry the payload (only {approved,choice,value}), so create the judge via its route; an LLM planner would fill the step inputs for full autonomy (see ARIA-AUTONOMY.md). Instructions reference {{ inputs }}/{{ outputs }}/{{ expectations }} — at least one template variable is required."
}

{
  "name": "support-faithfulness-eval",
  "description": "Held-out support Q/A for faithfulness scoring. Aria creates the empty versioned test set; add example rows after creation, then run it in Evaluations with the deterministic graders plus AnswerFaithfulness ticked under Custom LLM judges (it runs as a Judge.<id> scorer).",
  "tags": ["aria-created", "faithfulness", "support"],
  "notes": "Answer to the Aria eval_dataset.create interaction (same shape as a POST /eval-datasets body; owner is forced to the acting identity). Example rows are NOT added by Aria — add them via POST /eval-datasets/{id}/examples or harvest-from-trace, e.g. reuse cookbooks/01-prompt-regression-lab/assets/dataset/ as a starting point."
}

Evaluation & quality gates