Cookbook 15 · Aria — Autonomous

Aria: Triage & Recalibrate Loop

One sentence → review the weak runs, then kick off an async workflow calibration.

Aria — AutonomousAria20–35 min

What you build. On an existing workflow with weak runs, Aria plans review_queue.create + add_items + the async workflow.calibrate; you create the queue, enqueue the flagged traces, trigger calibration, and watch the plan park + poll.

Surfaces: Plans Review Queues Workflows Observability

What you learn

Plan a remediation loop from one intent
Operate on existing ids (traces, workflow, agent)
Read flagged trace ids from Observability
Trigger a workflow calibration
Understand async parking + polling in Aria plans

Implementation flow

flowchart LR
  I[/"Intent: review the flagged traces + calibrate the workflow"/] --> PL[Aria Plans]
  PL --> Q[review_queue.create] -.UI.-> RQ[Review Queues]
  PL --> E[review_queue.add_items] -.real trace ids.-> ENQ[enqueue]
  PL --> C[workflow.calibrate · async] --> POLL[park → poll → done]

Step-by-step UI only

Each step shows the exact navigation path, the fields to fill, the button to click, and a snapshot of the CALIBER screen you'll be on.

Compose › Workflows (prerequisite)
Prerequisite — this loop remediates things that already exist. Before starting, build and run the Support Triage Copilot from cookbook 07 so you have a real workflow with recent runs. You will need three real ids from it: a few flagged trace ids, the workflow_id, and an agent_id bound in that workflow. Aria does not create the subject workflow or its traces.
You'll see: An existing workflow (from SCN-07) listed under Workflows, with recent runs you can open.
Screen snapshot

caliber · Compose › Workflows (prerequisite)

WorkflowsWorkflows (prerequisite)

Read / inspect this surface — no form to fill.
Observe › Observability
Collect the flagged trace ids you will enqueue. In the sidebar, under Observe, click Observability, stay on the Traces tab, and set the Filter by status dropdown to Error to surface the weak runs. Click each weak trace to open it on the right — its full trace_id appears in the page URL (?trace=…) and you can also paste an id into the Search traces box to confirm it. Note 2-3 trace ids — you will paste them later.
Fill in
Filter by status Error (the option values are All statuses / OK / Error / In progress)
What to copy 2-3 trace_id values from the weak runs (from the URL after selecting each trace)
You'll see: A filtered list of trace rows (each showing the trace name + an ERROR status badge). Selecting a row opens its detail and puts the full trace_id in the URL — keep those ids handy for the enqueue step.
Screen snapshot

caliber · Observe › Observability

ObservabilityObservability

Filter by statusError (the option values are All statuses / OK / Error / In progress)What to copy2-3 trace_id values from the weak runs (from the URL after selecting each trace)

Compose › Plans › New goal creates · Aria plan (draft)

In the sidebar, under Compose, click Plans. Type Aria's one-line intent into the Goal box, verbatim — the phrase review queue proposes both queue steps and the word workflow (calibration) proposes the calibrate step. Set Autonomy to Stop before each change (the ask_each level).

Fill in

Goal	Our workflow's recent runs look weak — set up a review queue for the flagged traces and kick off a workflow calibration. (canonical intent — copy from intent.md)
Autonomy	Stop before each change (= ask_each)

Click Decompose goal

You'll see: The button shows Decomposing…, then Aria opens the plan's detail page. Breadcrumb Plans › PLAN-…, status badge DRAFT.

Screen snapshot

caliber · Compose › Plans › New goal

PlansNew goal

GoalOur workflow's recent runs look weak — set up a review queue for the flagged traces …AutonomyStop before each change (= ask_each)

Decompose goal

Compose › Plans › PLAN-… (draft) expected-plan.json ↓
Confirm Aria decomposed the sentence into the three remediation steps. Note the third step, workflow.calibrate, is Aria's first async capability. Compare against expected-plan.json.
Fill in
Step 1 review_queue.create — tier mutate
Step 2 review_queue.add_items — tier mutate (needs EXISTING trace ids)
Step 3 workflow.calibrate — tier mutate, ASYNC (needs EXISTING workflow_id + agent_id)
You'll see: A three-row Plan steps list (review_queue.create, review_queue.add_items, workflow.calibrate), each with a PENDING badge.
Screen snapshot

caliber · Compose › Plans › PLAN-… (draft)

PlansPLAN-… (draft)

Step 1review_queue.create — tier mutateStep 2review_queue.add_items — tier mutate (needs EXISTING trace ids)Step 3workflow.calibrate — tier mutate, ASYNC (needs EXISTING workflow_id + agent_id)
Compose › Plans › PLAN-… (draft)
Approve the plan shape. Click Approve plan in the top-right of the plan header.
Click Approve plan
You'll see: The status badge flips from DRAFT to APPROVED, and an Execute button appears.
Screen snapshot

caliber · Compose › Plans › PLAN-… (draft)

PlansPLAN-… (draft)

Approve plan
Compose › Plans › PLAN-… (approved)
Start running the plan. Click Execute. Aria pauses at the first step. Honesty note: Aria plans these steps, but the shipped planner leaves their inputs empty and the approval prompt only carries yes/no. If you Approve a step it runs with empty inputs and fails validation (e.g. workflow.calibrate requires workflow_id + agent_id), which fails the whole plan. So you will Deny each prompt — that skips the step and keeps the plan healthy — and drive each step yourself via its own UI page using the spec files (with the real ids you collected).
Click Execute
You'll see: A violet Aria needs your approval prompt for review_queue.create with Approve / Deny buttons; plan status RUNNING.
Screen snapshot

caliber · Compose › Plans › PLAN-… (approved)

PlansPLAN-… (approved)

Execute

Compose › Plans › PLAN-… (running) creates · Aria plan (completed, steps skipped — orchestration record)

Walk the three prompts in order, clicking Deny on each as it appears (review_queue.create, then review_queue.add_items, then workflow.calibrate). Denying marks each step SKIPPED so the plan completes cleanly as the orchestration record; you perform the real work (create the queue, enqueue traces, start the calibration) on the dedicated pages in the next steps. The genuine async behavior you will see is on the workflow's Calibration runs list — not on a parked plan step.

Fill in

review_queue.create	click Deny (build it yourself in Review Queues)
review_queue.add_items	click Deny (enqueue yourself in the queue page)
workflow.calibrate	click Deny (start it yourself in the Workflow Calibrate panel)

You'll see: Each denied step gets a SKIPPED badge; after the third the plan reaches COMPLETED with all three steps skipped. Now build the real artifacts.

Screen snapshot

caliber · Compose › Plans › PLAN-… (running)

PlansPLAN-… (running)

review_queue.createclick Deny (build it yourself in Review Queues)review_queue.add_itemsclick Deny (enqueue yourself in the queue page)workflow.calibrateclick Deny (start it yourself in the Workflow Calibrate panel)

Observe › Review Queues › New review queue creates · Review queue: weak-runs-triage triage-review.queue.json ↓

Build the triage queue. In the sidebar, under Observe, click Review Queues, then + New Queue. Fill from triage-review.queue.json. Click + Add question until you have four rows. Note two special types: failure_mode is categorical (fill its options) and expected_answer uses target expectation (its answer becomes ground truth on the trace).

Fill in

Name	weak-runs-triage
Reviewers (comma-separated)	leave blank
Description	Human triage of weak workflow runs: was the resolution correct, what failed, how severe, and what the answer should have been.
Question 1 — key / title / type / target	`resolution_correct` / Did the workflow reach the correct resolution? / `pass_fail` / `feedback`
Question 2 — key / title / type / target / options	`failure_mode` / Primary failure mode / `categorical` / `feedback` / options: `wrong_tool, missing_evidence, hallucinated_claim, bad_escalation, none`
Question 3 — key / title / type / target	`severity` / Severity (1-5) / `numeric` / `feedback`
Question 4 — key / title / type / target	`expected_answer` / What should the answer have been? (becomes the expectation) / `text` / `expectation`

Click Create queue

You'll see: The panel closes and weak-runs-triage appears in the Review Queues table with 4 questions and no items yet.

Screen snapshot

caliber · Observe › Review Queues › New review queue

Review QueuesNew review queue

Nameweak-runs-triageReviewers (comma-separated)leave blankDescriptionHuman triage of weak workflow runs: was the resolution correct, what failed, how sev…Question 1 — key / title / type / targetresolution_correct / Did the workflow reach the correct resolution? / pass_fail / fe…Question 2 — key / title / type / target / optionsfailure_mode / Primary failure mode / categorical / feedback / options: wrong_tool, …

Create queue

Observe › Review Queues › weak-runs-triage creates · Flagged traces enqueued for triage enqueue.json ↓
Enqueue the real flagged traces (the review_queue.add_items step). Open weak-runs-triage, then paste the real trace ids you noted from Observability into the Add traces to review box (comma- or space-separated) and click Enqueue. Replace the placeholders in enqueue.json with those real ids.
Fill in
Add traces to review (ids, comma or space separated) tr-… tr-… tr-… (the REAL flagged trace ids from Observability — not placeholders)
Click Enqueue
You'll see: The Items (N) list fills with the enqueued traces, each showing a PENDING status, and the first trace opens in the review form on the right.
Screen snapshot

caliber · Observe › Review Queues › weak-runs-triage

Review Queuesweak-runs-triage

Add traces to review (ids, comma or space separated)tr-… tr-… tr-… (the REAL flagged trace ids from Observability — not placeholders)
Enqueue
Compose › Workflows › (SCN-07 workflow) › Calibrate creates · Workflow calibration job (async) calibrate.json ↓
Kick off the real workflow calibration (the workflow.calibrate step). Open Workflows, click into your SCN-07 workflow, and click the Calibrate button (top-right of the header) to reveal the calibration panel. In the panel, pick the bound Agent (this is the agent_id), leave Objective on quality, leave Epsilon and Candidates on their defaults, then click Start Calibration. Note: the button stays disabled until a deploy-gate eval dataset is available for the workflow — the panel shows that availability status, so make sure SCN-07 has its gate dataset attached. Use the real workflow + agent from calibrate.json.
Fill in
Agent the agent bound in this workflow (the real agent_id)
Objective quality
Epsilon / Candidates leave the defaults (0.02 / 3)
Click Start Calibration
You'll see: The button shows Starting..., then a new entry appears under Calibration runs (just below the panel) with a queued/running status — this is the real async refinement job, the live counterpart of the workflow.calibrate step Aria planned.
Screen snapshot

caliber · Compose › Workflows › (SCN-07 workflow) › Calibrate

WorkflowsCalibrate

Agentthe agent bound in this workflow (the real agent_id)ObjectivequalityEpsilon / Candidatesleave the defaults (0.02 / 3)
Start Calibration
Compose › Workflows › (SCN-07 workflow) › Calibration runs
Let the async job run. This is the genuinely asynchronous part of the loop. Stay on the workflow page and use your browser Refresh (the Calibration runs list re-fetches) every minute or so to watch the run advance from queued → running → completed. The refinement worker scores the candidate(s) against the workflow's gate dataset behind the scenes.
You'll see: The Calibration runs entry you started ticks from queued/running to a terminal status; on completion it shows the run's eval summary (e.g. pass-rate / score) inline.
Screen snapshot

caliber · Compose › Workflows › (SCN-07 workflow) › Calibration runs

WorkflowsCalibration runs

Read / inspect this surface — no form to fill.
Compose › Workflows › (SCN-07 workflow) › Calibration runs
Confirm the loop closed. When the calibration job reaches completed, the workflow has been recalibrated against its gate dataset and the run's results are visible inline. (The Aria plan itself is already COMPLETED with its three steps SKIPPED — it is the orchestration record; the real async work lived in this Calibration runs list, not on a parked plan step.)
You'll see: A completed calibration run under Calibration runs, plus the Aria plan showing COMPLETED (three steps skipped) as the record of what was orchestrated.
Screen snapshot

caliber · Compose › Workflows › (SCN-07 workflow) › Calibration runs

WorkflowsCalibration runs

Read / inspect this surface — no form to fill.
Observe › Review Queues / Compose › Workflows
Final verification across surfaces. Review Queues › weak-runs-triage shows the flagged traces enqueued and ready to review. Workflows › (your workflow) › Calibration runs shows the calibration job completed. The Aria plan (under Plans, status COMPLETED with skipped steps) stands as the orchestration record for the whole triage → remediation loop.
You'll see: A populated weak-runs-triage queue, a completed calibration run on the workflow, and a completed Aria plan (steps skipped) — the governance → remediation loop end to end.
Screen snapshot

caliber · Observe › Review Queues / Compose › Workflows

WorkflowsWorkflows

Read / inspect this surface — no form to fill.
Observe › Review Queues › weak-runs-triage (follow-up)
(Optional follow-up) Have reviewers answer the queue's questions — the expectation-targeted expected_answer writes ground truth back onto each trace. Then compare the recalibrated workflow against its baseline runs and feed the hard cases into its eval dataset (see SCN-01 / SCN-10) to keep tightening the loop.
You'll see: Submitted reviews write assessments + expectations back onto the traces, giving you ground-truth labels to compare the recalibrated workflow against its baseline.
Screen snapshot

caliber · Observe › Review Queues › weak-runs-triage (follow-up)

Review Queuesweak-runs-triage (follow-up)

Read / inspect this surface — no form to fill.

Assets (copy-paste)

The exact files this cookbook uses — copy each into the matching field. Source: docs-site/cookbooks/15-aria-triage-recalibrate-loop/assets/.

assets/intent.md

# The intent you give Aria

## Canonical (works with the default HeuristicPlanner)

> Our workflow's recent runs look weak — set up a review queue for the flagged traces and kick off a workflow calibration.

Why this wording: the default planner proposes a step for each capability whose
**domain word** appears in the goal. The phrase **"review queue"** maps to
**both** `review_queue.create` **and** `review_queue.add_items`; the word
**"workflow"** (calibration) maps to `workflow.calibrate`. Keep both literal
needles in the sentence.

Autonomy: **ask_each** (pauses on every mutate step so you confirm each input).

## Natural-language variant (needs an LLM planner)

> Our latest runs aren't great — get a human to look at the bad ones and then tune the pipeline.

This reads better but relies on an LLM planner to map "get a human to look at
the bad ones" → `review_queue.create` + `review_queue.add_items` and "tune the
pipeline" → `workflow.calibrate`. With the default heuristic planner, use the
canonical wording.

## Remember: this loop remediates EXISTING things

`review_queue.add_items` needs **real existing** `trace_ids` and
`workflow.calibrate` needs an **existing** `workflow_id` + `agent_id`. Aria does
not create the subject workflow or its traces — build that in **SCN-07** first,
then supply the ids when Aria pauses (see [`enqueue.json`](enqueue.json) and
[`calibrate.json`](calibrate.json)).

assets/expected-plan.json

{
  "goal": "Our workflow's recent runs look weak — set up a review queue for the flagged traces and kick off a workflow calibration.",
  "autonomy": "ask_each",
  "expected_steps": [
    {
      "capability_key": "review_queue.create",
      "tier": "mutate",
      "interaction": "confirm/refine the triage question schema (review-queues/triage-review.queue.json)"
    },
    {
      "capability_key": "review_queue.add_items",
      "tier": "mutate",
      "interaction": "supply REAL flagged trace_ids of the weak runs (enqueue.json); uses the queue_id from the previous step"
    },
    {
      "capability_key": "workflow.calibrate",
      "tier": "mutate",
      "async": true,
      "interaction": "supply a REAL existing workflow_id + agent_id (calibrate.json); answering enqueues the async job and the plan PARKS (waiting_job) — POST /aria/plans/{id}/poll until it resolves"
    }
  ],
  "notes": "All three steps share no dependencies in the default planner, but you answer them in order. review_queue.add_items + workflow.calibrate operate on EXISTING ids (traces, workflow, agent) — Aria does not create them. The final step is asynchronous: park + poll until done, then the plan completes."
}

assets/review-queues/triage-review.queue.json

{
  "name": "weak-runs-triage",
  "description": "Human triage of weak workflow runs: was the resolution correct, what failed, how severe, and what the answer should have been.",
  "reviewers": [],
  "questions": [
    {
      "key": "resolution_correct",
      "title": "Did the workflow reach the correct resolution?",
      "type": "pass_fail",
      "required": true,
      "target": "feedback"
    },
    {
      "key": "failure_mode",
      "title": "Primary failure mode",
      "type": "categorical",
      "options": ["wrong_tool", "missing_evidence", "hallucinated_claim", "bad_escalation", "none"],
      "required": true,
      "target": "feedback"
    },
    {
      "key": "severity",
      "title": "Severity (1-5)",
      "type": "numeric",
      "required": false,
      "target": "feedback"
    },
    {
      "key": "expected_answer",
      "title": "What should the answer have been? (becomes the expectation)",
      "type": "text",
      "required": false,
      "target": "expectation"
    }
  ],
  "notes": "Answer to the Aria review_queue.create interaction (same shape as POST /review-queues). 'target: expectation' writes the reviewer's answer back onto the trace as ground truth; 'target: feedback' logs it as an assessment."
}

assets/enqueue.json

{
  "queue_id": "<the queue_id returned by the review_queue.create step>",
  "trace_ids": [
    "<REAL flagged trace id 1 from Observability>",
    "<REAL flagged trace id 2 from Observability>",
    "<REAL flagged trace id 3 from Observability>"
  ],
  "assigned_to": "@reviewer",
  "notes": "Answer to the Aria review_queue.add_items interaction (same shape as POST /review-queues/{id}/items). PLACEHOLDERS — replace trace_ids with REAL ids of the weak runs (read them from Observability) and queue_id with the id from the create step. Aria does not discover traces. experiment_id is optional."
}

assets/calibrate.json

{
  "workflow_id": "<REAL existing workflow id — e.g. the SCN-07 support workflow you built>",
  "agent_id": "<REAL existing agent id bound in that workflow>",
  "notes": "Answer to the Aria workflow.calibrate interaction. This is the only ASYNC capability: answering it enqueues a real workflow-calibration job and the plan PARKS in waiting_job. POST /aria/plans/{plan_id}/poll until the refinement_job resolves (done/failed); the plan then resumes and completes. PLACEHOLDERS — replace with a real workflow_id + agent_id (build the workflow in SCN-07). Aria does not create the workflow or agent."
}

Evaluation & quality gates

Quality gate	Target
Queue + items exist	flagged traces enqueued
Calibration completes	async job resolves; plan completes

Developer notes & gotchas

This is a remediation loop — it operates on EXISTING ids (traces, workflow, agent). Build the subject workflow in cookbook 07 first.
workflow.calibrate is Aria's async capability: the plan parks (waiting_job) and you poll until it resolves.
As elsewhere in the Aria track, create the queue + enqueue + calibration via the UI (planner emits empty inputs).

Filter by status	`Error` (the option values are All statuses / OK / Error / In progress)
What to copy	2-3 `trace_id` values from the weak runs (from the URL after selecting each trace)

Step 1	`review_queue.create` — tier mutate
Step 2	`review_queue.add_items` — tier mutate (needs EXISTING trace ids)
Step 3	`workflow.calibrate` — tier mutate, ASYNC (needs EXISTING workflow_id + agent_id)

Agent	the agent bound in this workflow (the real `agent_id`)
Objective	`quality`
Epsilon / Candidates	leave the defaults (0.02 / 3)