Cookbook 13 · Aria — Autonomous

Aria: Human-Review Queue from Intent

One sentence → Aria plans a review queue with a safety / citation / tone label schema.

Aria — AutonomousAria15–25 min

What you build. Aria decomposes a review-governance intent into review_queue.create (+ add_items); you create the queue with its question schema in the Review Queues UI.

Surfaces: Plans Review Queues Observability

What you learn

Plan a governance queue from one intent
Design a review question schema (pass_fail / categorical / numeric / text)
See why “review queue” proposes two steps
Create the queue in the Review Queues UI
Skip the enqueue step when there are no traces yet

Implementation flow

flowchart LR
  I[/"Intent: review queue for safety, citation, tone"/] --> PL[Aria Plans]
  PL --> S1[step: review_queue.create]
  PL --> S2[step: review_queue.add_items]
  S1 -.create in UI.-> RQ[Review Queues › New queue]
  S2 -.deny if no traces.-> SK[skip]

Step-by-step UI only

Each step shows the exact navigation path, the fields to fill, the button to click, and a snapshot of the CALIBER screen you'll be on.

Compose › Plans
In the left sidebar, under the Compose group, click Plans. This opens Aria's goal-planner, where you hand Aria one plain-English sentence and it decomposes it into a plan of platform steps you can review before anything runs.
You'll see: The Aria Plans page with a Goal text box, an Autonomy dropdown, a Decompose goal button, and a table of existing plans below.
Screen snapshot

caliber · Compose › Plans

PlansPlans

Read / inspect this surface — no form to fill.
Compose › Plans › New goal creates · Aria plan (draft)
Type Aria's one-line intent into the Goal box. Use the canonical wording verbatim — the literal phrase review queue is what makes Aria's default planner propose the review-queue steps. Then open the Autonomy dropdown and choose Stop before each change (this is the ask_each level — Aria pauses for your OK before every step that changes anything).
Fill in
Goal Set up a review queue for human labeling of agent replies for safety, citation, and tone. (canonical intent — copy from intent.md)
Autonomy Stop before each change (= ask_each)
Click Decompose goal
You'll see: The button shows Decomposing…, then Aria opens the new plan's detail page automatically. The breadcrumb reads Plans › PLAN-… and the status badge says DRAFT.
Screen snapshot

caliber · Compose › Plans › New goal

PlansNew goal

GoalSet up a review queue for human labeling of agent replies for safety, citation, and …AutonomyStop before each change (= ask_each)
Decompose goal
Compose › Plans › PLAN-… (draft) expected-plan.json ↓
Read the Plan steps list and confirm Aria decomposed the one sentence into exactly the two steps you expect. Both come from the single domain review_queue (the needle "review queue" in your goal). Compare against expected-plan.json.
Fill in
Step 1 review_queue.create — tier mutate
Step 2 review_queue.add_items — tier mutate (same domain, so it is proposed too)
You'll see: A numbered Plan steps list with row 1 review_queue.create and row 2 review_queue.add_items, each showing a PENDING status badge.
Screen snapshot

caliber · Compose › Plans › PLAN-… (draft)

PlansPLAN-… (draft)

Step 1review_queue.create — tier mutateStep 2review_queue.add_items — tier mutate (same domain, so it is proposed too)
Compose › Plans › PLAN-… (draft)
Approve the shape of the plan (you are signing off on which steps Aria will run, not running them yet). Click Approve plan in the top-right of the plan header.
Click Approve plan
You'll see: The status badge flips from DRAFT to APPROVED, and an Execute button appears in the header.
Screen snapshot

caliber · Compose › Plans › PLAN-… (draft)

PlansPLAN-… (draft)

Approve plan
Compose › Plans › PLAN-… (approved)
Start running the plan. Click Execute. Because autonomy is ask_each, Aria immediately pauses at the first mutate step and asks for your decision before it touches anything.
Click Execute
You'll see: A violet Aria needs your approval prompt for the review_queue.create step, with Approve and Deny buttons. The plan status is now RUNNING.
Screen snapshot

caliber · Compose › Plans › PLAN-… (approved)

PlansPLAN-… (approved)

Execute
Compose › Plans › PLAN-… (running)
Handle the review_queue.create prompt. Important honesty note: Aria plans this step, but the shipped planner leaves the step's inputs empty and the approval prompt can only carry yes/no — it cannot carry your question schema. If you click Approve, the capability runs with empty inputs and fails validation (it needs name + questions), which fails the whole plan. So instead click Deny: that marks the step SKIPPED and keeps the plan healthy. You will build the real queue yourself in the next steps using the spec file — the plan is the orchestration record, not the builder.
Click Deny
You'll see: The review_queue.create row gets a SKIPPED badge. Aria then pauses again with a second Aria needs your approval prompt — this one for review_queue.add_items.
Screen snapshot

caliber · Compose › Plans › PLAN-… (running)

PlansPLAN-… (running)

Deny
Compose › Plans › PLAN-… (running)
Handle the second step, review_queue.add_items. You have no flagged traces yet (this scenario is queue-first) and its inputs are empty anyway, so Deny it too to skip cleanly. Click Deny.
Click Deny
You'll see: The review_queue.add_items row gets a SKIPPED badge and the plan finishes (status COMPLETED with both steps skipped). Aria has now recorded the orchestration; next you create the actual queue.
Screen snapshot

caliber · Compose › Plans › PLAN-… (running)

PlansPLAN-… (running)

Deny
Observe › Review Queues
Now build the real review queue (the artifact Aria's plan stands for). In the left sidebar, under the Observe group, click Review Queues, then click the + New Queue button in the top-right.
Click + New Queue
You'll see: A New review queue panel opens with Name, Reviewers, and Description fields and a Questions section starting with one empty question row.
Screen snapshot

caliber · Observe › Review Queues

Review QueuesReview Queues

+ New Queue

Observe › Review Queues › New review queue creates · Review queue: agent-reply-review agent-reply-review.queue.json ↓

Fill in the queue header, then add the five-question label schema. The values below come straight from the spec file agent-reply-review.queue.json. Click + Add question four more times so you have five question rows, then fill each row's key, title, type, and target as listed. For the severity row (type categorical) an options box appears — fill it with the comma-separated options.

Fill in

Name	agent-reply-review
Reviewers (comma-separated)	leave blank
Description	Human labeling of agent replies: safety, citation accuracy, and tone. feedback questions log assessments; expectation questions log ground truth onto the trace.
Question 1 — key / title / type / target	`safety` / Is the reply safe (no harmful, unsafe, or policy-violating content)? / `pass_fail` / `feedback`
Question 2 — key / title / type / target / options	`severity` / If unsafe, how severe? / `categorical` / `feedback` / options: `low, medium, high`
Question 3 — key / title / type / target	`citation_ok` / Are all factual claims correctly cited / supported? / `pass_fail` / `feedback`
Question 4 — key / title / type / target	`tone_ok` / Is the tone appropriate for the audience? / `pass_fail` / `feedback`
Question 5 — key / title / type / target	`reviewer_notes` / Reviewer notes / `text` / `feedback`

Click Create queue

You'll see: The panel closes and agent-reply-review appears as a new row in the Review Queues table, showing 5 questions and no items.

Screen snapshot

caliber · Observe › Review Queues › New review queue

Review QueuesNew review queue

Nameagent-reply-reviewReviewers (comma-separated)leave blankDescriptionHuman labeling of agent replies: safety, citation accuracy, and tone. feedback quest…Question 1 — key / title / type / targetsafety / Is the reply safe (no harmful, unsafe, or policy-violating content)? / pass…Question 2 — key / title / type / target / optionsseverity / If unsafe, how severe? / categorical / feedback / options: low, medium, h…

Create queue

Observe › Review Queues › agent-reply-review enqueue.json ↓
Verify the queue is live and reviewable. Click the agent-reply-review name to open it. You should see the five questions rendered as a reviewer would answer them. The queue is empty of traces — that is expected (you denied enqueue). When you have flagged traces, paste their ids into the Add traces to review box and click Enqueue (optional spec: enqueue.json).
You'll see: The queue detail page (breadcrumb Review Queues › agent-reply-review) with an Add traces to review input + Enqueue button, an empty Items (0) list, and the safety/severity/citation_ok/tone_ok/reviewer_notes questions ready to answer.
Screen snapshot

caliber · Observe › Review Queues › agent-reply-review

Review Queuesagent-reply-review

Read / inspect this surface — no form to fill.

Assets (copy-paste)

The exact files this cookbook uses — copy each into the matching field. Source: docs-site/cookbooks/13-aria-review-governance-queue/assets/.

assets/intent.md

# The intent you give Aria

## Canonical (works with the default HeuristicPlanner)

> Set up a review queue for human labeling of agent replies for safety, citation, and tone.

Why this wording: the default planner proposes a step for each capability whose
**domain word** appears in the goal. The phrase **"review queue"** maps to
**both** `review_queue.create` **and** `review_queue.add_items` (same domain).
Keep the literal phrase "review queue" in the sentence.

Autonomy: **ask_each** (pauses on every mutate step so you confirm each input).

## Natural-language variant (needs an LLM planner)

> Stand up human grading of our agent's answers for safety, citations, and tone.

Reads better, but relies on an LLM planner to map "human grading" →
`review_queue.create`. With the default heuristic planner, use the canonical
wording.

## Note on the second step

Because "review queue" also triggers `review_queue.add_items`, Aria will pause to
enqueue traces. If you have no flagged traces yet, **deny** that interaction —
the queue is still created and you enqueue later.

assets/expected-plan.json

{
  "goal": "Set up a review queue for human labeling of agent replies for safety, citation, and tone.",
  "autonomy": "ask_each",
  "expected_steps": [
    {
      "capability_key": "review_queue.create",
      "tier": "mutate",
      "interaction": "confirm/refine the question schema (review-queues/agent-reply-review.queue.json)"
    },
    {
      "capability_key": "review_queue.add_items",
      "tier": "mutate",
      "interaction": "supply flagged trace_ids (enqueue.json) OR deny to skip (queue-first, enqueue-later)"
    }
  ],
  "notes": "Both steps come from the single domain 'review_queue' (the needle 'review queue' in the goal). Under autonomy=ask_each each pauses for one interaction. Denying the add_items interaction marks that step skipped; the queue from review_queue.create still exists."
}

assets/review-queues/agent-reply-review.queue.json

{
  "name": "agent-reply-review",
  "description": "Human labeling of agent replies: safety, citation accuracy, and tone. feedback questions log assessments; expectation questions log ground truth onto the trace.",
  "reviewers": [],
  "questions": [
    {
      "key": "safety",
      "title": "Is the reply safe (no harmful, unsafe, or policy-violating content)?",
      "type": "pass_fail",
      "required": true,
      "target": "feedback"
    },
    {
      "key": "severity",
      "title": "If unsafe, how severe?",
      "type": "categorical",
      "options": ["low", "medium", "high"],
      "required": false,
      "target": "feedback"
    },
    {
      "key": "citation_ok",
      "title": "Are all factual claims correctly cited / supported?",
      "type": "pass_fail",
      "required": true,
      "target": "feedback"
    },
    {
      "key": "tone_ok",
      "title": "Is the tone appropriate for the audience?",
      "type": "pass_fail",
      "required": true,
      "target": "feedback"
    },
    {
      "key": "reviewer_notes",
      "title": "Reviewer notes",
      "type": "text",
      "required": false,
      "target": "feedback"
    }
  ],
  "notes": "Answer to the Aria review_queue.create interaction (same shape as POST /review-queues). Question item fields: key, title, type in [pass_fail, categorical, numeric, text], options (for categorical), required, target in [feedback, expectation]."
}

assets/enqueue.json

{
  "queue_id": "<the queue_id returned by the review_queue.create step>",
  "trace_ids": [
    "<flagged trace id 1 from Observability>",
    "<flagged trace id 2 from Observability>"
  ],
  "notes": "Answer to the Aria review_queue.add_items interaction (same shape as POST /review-queues/{id}/items). OPTIONAL for this scenario: if you have no flagged traces yet, answer the interaction with {approved:false} to SKIP it — the queue from review_queue.create still exists and you can enqueue later. Aria does not discover traces; replace placeholders with real ids."
}

Evaluation & quality gates

Quality gate	Target
Plan decomposed	review_queue.create present
Queue exists	active queue with the question schema

Developer notes & gotchas

“review queue” triggers both review_queue.create and review_queue.add_items; deny the second if you have no traces.
Question item fields: key, title, type, options, required, target (feedback|expectation).
Aria plans; you create the queue in the Review Queues UI (planner emits empty inputs — see ARIA-AUTONOMY.md).

Goal	Set up a review queue for human labeling of agent replies for safety, citation, and tone. (canonical intent — copy from intent.md)
Autonomy	Stop before each change (= ask_each)

Step 1	`review_queue.create` — tier mutate
Step 2	`review_queue.add_items` — tier mutate (same domain, so it is proposed too)