Aria: Governance Starter Kit from Intent

Implementation flow

flowchart TD
  I[/"Intent: judge + eval dataset + review queue"/] --> PL[Aria Plans]
  PL --> J[judge.create] -.UI.-> JU[Judges]
  PL --> D[eval_dataset.create] -.UI.-> TS[Test Sets]
  PL --> Q[review_queue.create] -.UI.-> RQ[Review Queues]
  PL --> A[review_queue.add_items] -.deny.-> SK[skip]

Step-by-step UI only

Each step shows the exact navigation path, the fields to fill, the button to click, and a snapshot of the CALIBER screen you'll be on.

1. Compose › Plans

   In the left sidebar, under the Compose group, click Plans. This is Aria's goal-planner. The flagship demo: one sentence decomposes into a whole governance scaffold — a faithfulness judge, an eval dataset, and a human-review queue — in a single plan.

   You'll see: The Aria Plans page with a Goal text box, an Autonomy dropdown, and a Decompose goal button.

   Screen snapshot

   caliber · Compose › Plans

   PlansPlans

   Read / inspect this surface — no form to fill.

2. Compose › Plans › New goal creates · Aria plan (draft)

   Type Aria's one-line intent into the Goal box, using the canonical wording verbatim. The three literal needles — judge, eval dataset, and review queue — are what make the default planner propose all three creates. Then set the Autonomy dropdown to Stop before each change (the ask_each level) so Aria pauses on every step.

   Fill in

   Goal	Stand up our governance starter kit: a judge for answer faithfulness, an eval dataset to score against, and a review queue for human checks. (canonical intent — copy from intent.md)
   Autonomy	Stop before each change (= ask_each)

   Click Decompose goal

   You'll see: The button shows Decomposing…, then Aria opens the new plan's detail page. The breadcrumb reads Plans › PLAN-… and the status badge says DRAFT.

   Screen snapshot

   caliber · Compose › Plans › New goal

   PlansNew goal

   GoalStand up our governance starter kit: a judge for answer faithfulness, an eval datase…AutonomyStop before each change (= ask_each)

   Decompose goal

3. Compose › Plans › PLAN-… (draft) expected-plan.json ↓

   Read the Plan steps list and confirm Aria found all four steps. Three are the artifacts you want; the fourth (review_queue.add_items) is proposed only because it shares the "review queue" domain — you will deny it later (no traces yet). Compare against expected-plan.json.

   Fill in

   Step 1	judge.create — tier mutate
   Step 2	review_queue.create — tier mutate
   Step 3	eval_dataset.create — tier mutate
   Step 4	review_queue.add_items — tier mutate (deny this one)

   You'll see: A four-row Plan steps list (judge.create, review_queue.create, eval_dataset.create, review_queue.add_items) in registry order, each with a PENDING badge.

   Screen snapshot

   caliber · Compose › Plans › PLAN-… (draft)

   PlansPLAN-… (draft)

   Step 1judge.create — tier mutateStep 2review_queue.create — tier mutateStep 3eval_dataset.create — tier mutateStep 4review_queue.add_items — tier mutate (deny this one)

4. Compose › Plans › PLAN-… (draft)

   Approve the plan shape (you are signing off on which steps run, not running them yet). Click Approve plan in the top-right of the plan header.

   Click Approve plan

   You'll see: The status badge flips from DRAFT to APPROVED, and an Execute button appears.

   Screen snapshot

   caliber · Compose › Plans › PLAN-… (draft)

   PlansPLAN-… (draft)

   Approve plan

5. Compose › Plans › PLAN-… (approved)

   Start running the plan. Click Execute. Because autonomy is ask_each, Aria pauses at the first step and asks for your decision. Honesty note: Aria plans these steps, but the shipped planner leaves their inputs empty and the approval prompt only carries yes/no. If you Approve a step, its capability runs with empty inputs and fails validation (each create needs its required fields), which fails the whole plan. So you will Deny every prompt — that skips the step and keeps the plan healthy — and build each artifact yourself in the later steps using the spec files.

   Click Execute

   You'll see: A violet Aria needs your approval prompt for judge.create, with Approve and Deny buttons; plan status RUNNING.

   Screen snapshot

   caliber · Compose › Plans › PLAN-… (approved)

   PlansPLAN-… (approved)

   Execute

6. Compose › Plans › PLAN-… (running)

   Walk through the four interaction prompts in order and click Deny on each (judge.create, review_queue.create, eval_dataset.create, then review_queue.add_items). Denying marks each step SKIPPED so the plan completes cleanly — it stands as the orchestration record while you build the real artifacts on their own pages next. (Approving instead would fail the plan, since the planner left every step's inputs empty.)

   Fill in

   judge.create	click Deny (build it yourself in Judges)
   review_queue.create	click Deny (build it yourself in Review Queues)
   eval_dataset.create	click Deny (build it yourself in Test Sets)
   review_queue.add_items	click Deny (no traces yet)

   You'll see: Each denied step gets a SKIPPED badge and the next prompt appears. The plan reaches COMPLETED with all four steps skipped. Now create the three real artifacts.

   Screen snapshot

   caliber · Compose › Plans › PLAN-… (running)

   PlansPLAN-… (running)

   judge.createclick Deny (build it yourself in Judges)review_queue.createclick Deny (build it yourself in Review Queues)eval_dataset.createclick Deny (build it yourself in Test Sets)review_queue.add_itemsclick Deny (no traces yet)

7. Evaluate › Judges › New judge creates · Judge: AnswerFaithfulness answer-faithfulness.judge.json ↓

   Build the faithfulness judge. In the left sidebar, under Evaluate, click Judges, then + New Judge. Fill the form from answer-faithfulness.judge.json. The Instructions must reference at least one of {{ inputs }} / {{ outputs }} / {{ expectations }} (the form warns you if none are present). For Model, use your configured judge model (the gateway default).

   Fill in

   Name	AnswerFaithfulness
   Model	your configured judge model, e.g. openai:/gpt-4o-mini
   Description	Judges whether an assistant answer is faithful to its evidence.
   Returns	bool
   Instructions	You audit whether an assistant answer is faithful to its evidence. You are given the request {{ inputs }}, the model {{ outputs }}, and the reference {{ expectations }}. Return true ONLY if every factual claim in {{ outputs }} is supported by {{ expectations }} (or by evidence cited in {{ inputs }}), and the answer makes no unsupported commitments. If any claim is unsupported, contradicted, or hallucinated, return false. Judge faithfulness only — not style.

   Click Create judge

   You'll see: The panel closes and AnswerFaithfulness appears in the Judges table with status ACTIVE and Returns = bool.

   Screen snapshot

   caliber · Evaluate › Judges › New judge

   JudgesNew judge

   NameAnswerFaithfulnessModelyour configured judge model, e.g. openai:/gpt-4o-miniDescriptionJudges whether an assistant answer is faithful to its evidence.ReturnsboolInstructionsYou audit whether an assistant answer is faithful to its evidence. You are given the…

   Create judge

8. Evaluate › Test Sets › New test set creates · Test set: release-candidates-eval release-candidates.dataset.json ↓

   Build the eval dataset (Aria creates it empty; you add example rows later). In the sidebar, under Evaluate, click Test Sets, then + New Test Set. Fill from release-candidates.dataset.json. The UI requires an Owner — type your handle (the API would force it to your identity).

   Fill in

   Name	release-candidates-eval
   Owner	@you (your handle — required by the form)
   Description	Held-out release-candidate Q/A for faithfulness scoring. Add example rows after creation, then run it in Evaluations on the deterministic graders plus the AnswerFaithfulness judge under Custom LLM judges.

   Click Create

   You'll see: The panel closes and release-candidates-eval appears in the Test Sets table at v1, status ACTIVE, Not synced.

   Screen snapshot

   caliber · Evaluate › Test Sets › New test set

   Test SetsNew test set

   Namerelease-candidates-evalOwner@you (your handle — required by the form)DescriptionHeld-out release-candidate Q/A for faithfulness scoring. Add example rows after crea…

   Create

9. Observe › Review Queues › New review queue creates · Review queue: governance-review governance-review.queue.json ↓

   Build the human-review queue. In the sidebar, under Observe, click Review Queues, then + New Queue. Fill from governance-review.queue.json. Click + Add question until you have four question rows, then fill each. The tone row is type categorical — fill its options box.

   Fill in

   Name	governance-review
   Reviewers (comma-separated)	leave blank
   Description	Human spot-check of release-candidate answers: faithfulness, citation correctness, tone, and free-text notes. Pairs with the AnswerFaithfulness judge and the release-candidates-eval test set.
   Question 1 — key / title / type / target	faithful / Is the answer faithful to the evidence? / pass_fail / feedback
   Question 2 — key / title / type / target	citation_ok / Are the citations correct and sufficient? / pass_fail / feedback
   Question 3 — key / title / type / target / options	tone / Tone of the answer / categorical / feedback / options: on_brand, neutral, off_brand
   Question 4 — key / title / type / target	notes / Reviewer notes / text / feedback

   Click Create queue

   You'll see: The panel closes and governance-review appears in the Review Queues table with 4 questions and no items.

   Screen snapshot

   caliber · Observe › Review Queues › New review queue

   Review QueuesNew review queue

   Namegovernance-reviewReviewers (comma-separated)leave blankDescriptionHuman spot-check of release-candidate answers: faithfulness, citation correctness, t…Question 1 — key / title / type / targetfaithful / Is the answer faithful to the evidence? / pass_fail / feedbackQuestion 2 — key / title / type / targetcitation_ok / Are the citations correct and sufficient? / pass_fail / feedback

   Create queue

10. Evaluate › Judges (and Test Sets, and Observe › Review Queues)

    Confirm the whole governance kit exists — three real artifacts from one sentence (built by you in their pages; planned by Aria). Check each surface: Judges shows AnswerFaithfulness active; Test Sets shows release-candidates-eval; Review Queues shows governance-review with its four-question schema. Re-open the plan under Plans to see it COMPLETED with its four steps SKIPPED — the orchestration record of what Aria decomposed.

    You'll see: An active judge AnswerFaithfulness, a test set release-candidates-eval, and a review queue governance-review — the complete kit, with the Aria plan as the orchestration record.

    Screen snapshot

    caliber · Evaluate › Judges (and Test Sets, and Observe › Review Queues)

    Test SetsReview Queues)

    Read / inspect this surface — no form to fill.

11. Evaluate › Test Sets › release-candidates-eval (follow-up)

    (Optional follow-up) Make the kit runnable end-to-end. Add example rows to release-candidates-eval (author them at /eval-datasets/:id → + Add example, or reuse the rows in 01-prompt-regression-lab/assets/dataset), then go to Evaluate › Evaluations and run the dataset with the deterministic graders plus AnswerFaithfulness ticked under Custom LLM judges (it runs as a Judge.<id> scorer). Route any hard/low-confidence traces into governance-review via its Add traces to review box → Enqueue.

    You'll see: A scored evaluation run plus, optionally, traces enqueued into governance-review for human spot-checking — closing the eval + governance loop.

    Screen snapshot

    caliber · Evaluate › Test Sets › release-candidates-eval (follow-up)

    Test Setsrelease-candidates-eval (follow-up)

    Read / inspect this surface — no form to fill.

Assets (copy-paste)

The exact files this cookbook uses — copy each into the matching field. Source: docs-site/cookbooks/14-aria-governance-starter-kit/assets/.

# The intent you give Aria

## Canonical (works with the default HeuristicPlanner)

> Stand up our governance starter kit: a judge for answer faithfulness, an eval dataset to score against, and a review queue for human checks.

Why this wording: the default planner proposes a step for each capability whose
**domain word** appears in the goal. "judge" → `judge.create`; "eval dataset" →
`eval_dataset.create`; "review queue" → `review_queue.create`. Keep all three
literal needles in the sentence.

Note: "review queue" matches BOTH `review_queue.create` and
`review_queue.add_items` (they share the domain), so the planner proposes both.
There are no traces yet, so **deny** the `add_items` interaction to skip it.

Autonomy: **ask_each** (pauses on every mutate step so you confirm each input).

## Natural-language variant (needs an LLM planner)

> Get our new support answers governed end to end: score them for faithfulness, keep a held-out test set, and let a human spot-check the tricky ones.

This reads better but relies on an LLM planner to map "score for faithfulness" →
judge, "test set" → eval_dataset, and "human spot-check" → review_queue. With
the default heuristic planner, use the canonical wording.

{
  "goal": "Stand up our governance starter kit: a judge for answer faithfulness, an eval dataset to score against, and a review queue for human checks.",
  "autonomy": "ask_each",
  "expected_steps": [
    {
      "capability_key": "judge.create",
      "tier": "mutate",
      "interaction": "confirm/refine the judge spec (judges/answer-faithfulness.judge.json)"
    },
    {
      "capability_key": "review_queue.create",
      "tier": "mutate",
      "interaction": "confirm/refine the review-queue question schema (review-queues/governance-review.queue.json)"
    },
    {
      "capability_key": "eval_dataset.create",
      "tier": "mutate",
      "interaction": "confirm/refine the dataset spec (datasets/release-candidates.dataset.json)"
    },
    {
      "capability_key": "review_queue.add_items",
      "tier": "mutate",
      "interaction": "DENY this step (answer {approved:false}) — no traces to enqueue yet. It is proposed only because it shares the 'review queue' domain. Supply {queue_id, trace_ids} instead if you already have traces."
    }
  ],
  "notes": "Step order follows registry order; there are no inter-step dependencies in the default planner. The 'review queue' needle matches both review_queue.create and review_queue.add_items, so both appear. Under autonomy=ask_each each mutate step raises one interaction before it runs (deny add_items). Under approve_plan/auto_guarded the steps run automatically once approved — so for this scenario prefer ask_each so you can deny add_items."
}

{
  "name": "AnswerFaithfulness",
  "model": "set to your configured judge model (e.g. the gateway default)",
  "feedback_value_type": "bool",
  "instructions": "You audit whether an assistant answer is faithful to its evidence. You are given the request {{ inputs }}, the model {{ outputs }}, and the reference {{ expectations }}.\n\nReturn true ONLY if every factual claim in {{ outputs }} is supported by {{ expectations }} (or by evidence cited in {{ inputs }}), and the answer makes no unsupported commitments. If any claim is unsupported, contradicted, or hallucinated, return false. Judge faithfulness only — not style.",
  "tags": ["aria-created", "faithfulness", "governance"],
  "notes": "This IS the judge.create payload — POST it to /judges (a POST /judges body). Aria's plan proposes the judge.create step, but the shipped interaction answer can't carry the payload (only {approved,choice,value}), so create the judge via its route; an LLM planner would fill step inputs for full autonomy (see ARIA-AUTONOMY.md). Instructions reference {{ inputs }}/{{ outputs }}/{{ expectations }} — at least one template variable is required. Same faithfulness judge as SCN-12; here it's one piece of a larger governance kit."
}

{
  "name": "release-candidates-eval",
  "description": "Held-out release-candidate Q/A for faithfulness scoring. Aria creates the empty versioned test set; add example rows after creation, then run it in Evaluations with the deterministic graders plus AnswerFaithfulness ticked under Custom LLM judges (it runs as a Judge.<id> scorer).",
  "tags": ["aria-created", "faithfulness", "governance", "release-candidates"],
  "notes": "Answer to the Aria eval_dataset.create interaction (same shape as a POST /eval-datasets body; owner is forced to the acting identity). Example rows are NOT added by Aria — add them via POST /eval-datasets/{id}/examples or harvest-from-trace, e.g. reuse cookbooks/01-prompt-regression-lab/assets/dataset/ as a starting point."
}

{
  "name": "governance-review",
  "description": "Human spot-check of release-candidate answers: faithfulness, citation correctness, tone, and free-text notes. Pairs with the AnswerFaithfulness judge and the release-candidates-eval test set as the human layer of the governance kit.",
  "questions": [
    {
      "key": "faithful",
      "title": "Is the answer faithful to the evidence?",
      "type": "pass_fail",
      "required": true,
      "target": "feedback"
    },
    {
      "key": "citation_ok",
      "title": "Are the citations correct and sufficient?",
      "type": "pass_fail",
      "required": false,
      "target": "feedback"
    },
    {
      "key": "tone",
      "title": "Tone of the answer",
      "type": "categorical",
      "options": ["on_brand", "neutral", "off_brand"],
      "required": false,
      "target": "feedback"
    },
    {
      "key": "notes",
      "title": "Reviewer notes",
      "type": "text",
      "required": false,
      "target": "feedback"
    }
  ],
  "reviewers": [],
  "notes": "Answer to the Aria review_queue.create interaction (same shape as a POST /review-queues body). Each question item is {key, title, type in [pass_fail, categorical, numeric, text], options (for categorical), required, target in [feedback, expectation]}; only key + title are strictly required per item, and the queue requires name + a non-empty questions array. 'reviewers' is optional (a list of identities). Submitted answers write back onto each trace as MLflow assessments/expectations. Aria does NOT enqueue traces here — that is review_queue.add_items, which you DENY in this scenario (no traces yet); add traces later via POST /review-queues/{id}/items {trace_ids}."
}

Evaluation & quality gates