Cookbook 16 · Operate

Production Observability & Triage

Turn live traces into a triaged review queue + a regression test set — the production monitoring loop.

OperateCore30–45 min

What you build. An operations loop over a running workflow: monitor traces in Observability, filter to errors, drill into a failing run to find the root cause, capture the failing examples into a regression Test Set, and route them to a human triage Review Queue — so production signal becomes durable evidence instead of being lost.

Surfaces: Observability Review Queues Test Sets Evaluations

What you learn

Filter live traces by status (OK / Error / In progress) and search by name
Read a trace's node tree, inputs/outputs, latency, and feedback to find a root cause
Capture a failing trace into a regression Test Set (Add to test set → Add example)
Stand up a triage Review Queue and enqueue the flagged trace ids
Answer review questions so labels write back onto the trace, and baseline the failure rate

Implementation flow

flowchart LR
  RUN[Live workflow runs] --> OBS[Observability: filter Status = Error]
  OBS --> TR[Open failing trace: node tree + root cause]
  TR -->|Add to test set| DS[(Test Set: prod-regression)]
  TR -->|trace ids| RQ[Review Queue: prod-triage]
  RQ -->|Submit review| WB[Labels written back to trace]
  DS --> EV[Evaluations: deterministic baseline]
  EV --> FIX[Re-run after fix: rate drops]

Step-by-step UI only

Each step shows the exact navigation path, the fields to fill, the button to click, and a snapshot of the CALIBER screen you'll be on.

Prereq · a running workflow with traces creates · existing workflow runs (cookbook 07 / 09)
This is an operator playbook — it monitors a workflow that is already running and turns its failures into durable evidence. So first make sure there is something to observe: build and run a workflow that produces traces, ideally with a few failures. The easiest path is to reuse cookbook 07 (support-copilot) or cookbook 09 (recovery target) and execute a handful of inputs in its Run Monitor — including at least one that errors or is rejected — so the Observability page has real OK and Error traces to work with.
You'll see: A workflow exists and its Run Monitor shows several completed runs (some failed). You don't need anything new here — just real run history to triage.
Screen snapshot

caliber · Prereq · a running workflow with traces

Prereq · a running workflow with traces

Read / inspect this surface — no form to fill.
Observe › Observability creates · filtered Error traces
Open Observability (under Observe). It lists recent runtime traces, newest first, each with a name and a status badge. To isolate failures, open the Status filter and choose Error (the options are All statuses · OK · Error · In progress). Use the search box to scope to a single workflow by name if the list is busy.
Fill in
Status Error
Search the workflow name, e.g. support-copilot (optional)
You'll see: The list narrows to traces whose status is Error — your triage queue of raw failures. Each row shows the trace name, status, and timestamp.
Screen snapshot

caliber · Observe › Observability

ObservabilityObservability

StatusErrorSearchthe workflow name, e.g. support-copilot (optional)
Observe › Observability › trace detail creates · root cause (read from the trace)
Click a failing trace row to open its detail — the trace_id appears in the page URL as ?trace=… (you'll copy it shortly). Read the node tree: walk to the node whose status is error, and inspect its recorded inputs, outputs, duration, and any attached feedback. That node + its error message is your root cause.
You'll see: The trace tree opens with the failing node highlighted; its inputs/outputs and the raised error are visible, so you can explain why the run failed (e.g. a tool node raised, retrieval returned nothing, or the agent emitted an invalid decision). Note the trace_id from the URL.
Screen snapshot

caliber · Observe › Observability › trace detail

Observabilitytrace detail

Read / inspect this surface — no form to fill.
Evaluate › Test Sets › New test set, then Observe › Observability › Add to test set creates · test set: prod-regression (from traces) prod-regression.sample.jsonl ↓
Turn this failure into durable regression evidence. The Observability picker only attaches a trace to an existing test set — there is no create option there — so first create the set out-of-band: go to Evaluate › Test Sets, click + New Test Set, name it prod-regression, fill the required Owner with your handle, and Create. Then return to Observability, open the failing trace, click Add to test set, pick prod-regression from the Choose a test set… dropdown, and click Add example. The capture sends only the trace id: the trace's request becomes the input and its response becomes the expected — so for an error/regression trace you are capturing the failing output, and the correct gold expectation must be set by hand later (or via the from-trace API's optional expected override). Repeat for 2–3 representative failing traces so you have a small, real regression slice.
Fill in
Test Sets → + New Test Set Name prod-regression, Owner @you (required), then Create
Observability → Add to test set pick prod-regression from the Choose a test set… dropdown
Click Add example
You'll see: A confirmation that the example was added; the captured pair now lives in the prod-regression test set with the trace's request as the input and its response as the expected (the failing output for an error trace — fix the gold by hand later). You can author or edit these rows in the dataset editor at /eval-datasets/:id → + Add example; in this lab every row is captured here from a real trace.
Screen snapshot

caliber · Evaluate › Test Sets › New test set, then Observe › Observability › Add to test set

Test SetsAdd to test set

Test Sets → + New Test SetName prod-regression, Owner @you (required), then CreateObservability → Add to test setpick prod-regression from the Choose a test set… dropdown
Add example
Observe › Review Queues › + New Queue creates · review queue: prod-triage triage-review.review-questions.json ↓
Stand up a human triage queue so each failure gets classified. Click + New Queue, name it prod-triage, and with + Add question build the label schema from triage-review.review-questions.json: root_cause_known (pass_fail, required), failure_mode (categorical: prompt / tool / retrieval / data / infra), severity (categorical: low / medium / high), expected_output (text, target = expectation), reviewer_notes (text). Click Create queue.
Fill in
Queue name prod-triage
Questions 5 questions — types pass_fail / categorical / text; each with a Target of feedback or expectation
Click Create queue
You'll see: An empty prod-triage queue appears with your five questions defined. Categorical questions show their option chips; the expected_output question targets expectation so its answer becomes a trace expectation.
Screen snapshot

caliber · Observe › Review Queues › + New Queue

Review Queues+ New Queue

Queue nameprod-triageQuestions5 questions — types pass_fail / categorical / text; each with a Target of feedback o…
Create queue
Observe › Review Queues › prod-triage creates · enqueued failing traces
Enqueue the flagged traces. On the queue, in the Add traces to review (ids, comma or space separated) field, enter the failing trace ids you copied from the Observability URLs (comma or space separated), and click Enqueue.
Fill in
Trace ids the trace_ids of the Error traces from step 3 (comma or space separated)
Click Enqueue
You'll see: The queue shows N pending review items, each linked back to its source trace.
Screen snapshot

caliber · Observe › Review Queues › prod-triage

Review Queuesprod-triage

Trace idsthe trace_ids of the Error traces from step 3 (comma or space separated)
Enqueue
Observe › Review Queues › prod-triage › review creates · triaged labels (written back to the trace)
Triage an item. Open a pending item — it shows the trace's input/output alongside your questions. Answer them: is the root_cause_known? pick the failure_mode and severity, optionally fill the expected_output, add reviewer_notes, then click Submit review.
Click Submit review
You'll see: The item flips to reviewed and the answers are written back onto the trace as MLflow assessments/expectations — so the failure is now self-documenting in Observability, not just in a spreadsheet.
Screen snapshot

caliber · Observe › Review Queues › prod-triage › review

Review Queuesreview

Submit review
Evaluate › Test Sets › prod-regression creates · regression slice (durable)
Confirm the regression slice. Open Test Sets (under Evaluate) and select prod-regression — it now holds the failing examples you captured. This is the durable set you'll re-run after a fix ships to prove the regression is actually closed.
You'll see: The prod-regression test set lists your captured examples at version 1, each with the trace's request as the input and its response auto-derived as the expected (correct the gold by hand where needed).
Screen snapshot

caliber · Evaluate › Test Sets › prod-regression

Test Setsprod-regression

Read / inspect this surface — no form to fill.
Evaluate › Evaluations › Run evaluation creates · failure-rate baseline run
(Optional) Put a number on the failure. Click Run evaluation, pick Test set = prod-regression, give it a Label (e.g. prod baseline), check the deterministic grader Contains expected, and click Run. Open the run to read the current pass/fail — your operational baseline. After a fix ships, re-run the same set and confirm the failure rate drops.
Fill in
Test set prod-regression
Label prod baseline
Graders Contains expected (a deterministic scorer — all this slice needs; the Evaluations page also has a Custom LLM judges section, not used here)
Click Run
You'll see: A scorecard with the per-example pass/fail on the regression slice — a baseline you compare against after each fix. (This slice scores with the deterministic Contains expected grader; the Evaluations page can also run authored LLM judges via its Custom LLM judges section, which this scenario does not need.)
Screen snapshot

caliber · Evaluate › Evaluations › Run evaluation

EvaluationsRun evaluation

Test setprod-regressionLabelprod baselineGradersContains expected (a deterministic scorer — all this slice needs; the Evaluations pa…
Run

Assets (copy-paste)

The exact files this cookbook uses — copy each into the matching field. Source: docs-site/cookbooks/16-observability-triage/assets/.

assets/dataset/prod-regression.sample.jsonl

{"inputs": {"ticket": "Refund my $4,800 order A-1007 immediately."}, "expectations": {"decision": "escalate_support", "note": "over-limit refund must route to human approval, not auto-approve"}}
{"inputs": {"question": "What is the refund window?"}, "expectations": {"decision": "clarify", "note": "policy conflict 30 vs 14 days — must surface the conflict, not silently pick one"}}
{"inputs": {"alert": "gateway/prod p99=4200ms, error_rate=0.18 after deploy a1b9f3c"}, "expectations": {"recommended_action": "rollback a1b9f3c", "requires_approval": true}}
{"inputs": {"ticket": "How do I rotate my API key?"}, "expectations": {"decision": "reply", "note": "answerable from KB; must cite the docs and not escalate"}}

assets/review-queues/triage-review.review-questions.json

{
  "queue_name": "prod-triage",
  "description": "Human triage of failing production traces. Reviewers classify each flagged trace so the failure mode, severity, and the correct expected output are captured as structured labels written back onto the trace (assessments + an expectation).",
  "questions": [
    {
      "key": "root_cause_known",
      "title": "Is the root cause identifiable from the trace?",
      "type": "pass_fail",
      "required": true,
      "target": "feedback"
    },
    {
      "key": "failure_mode",
      "title": "Where did it fail?",
      "type": "categorical",
      "options": ["prompt", "tool", "retrieval", "data", "infra"],
      "required": true,
      "target": "feedback"
    },
    {
      "key": "severity",
      "title": "Operational severity",
      "type": "categorical",
      "options": ["low", "medium", "high"],
      "required": true,
      "target": "feedback"
    },
    {
      "key": "expected_output",
      "title": "What should the run have produced?",
      "type": "text",
      "required": false,
      "target": "expectation"
    },
    {
      "key": "reviewer_notes",
      "title": "Triage notes",
      "type": "text",
      "required": false,
      "target": "feedback"
    }
  ],
  "notes": "Question field shape matches the Review Queues 'New Queue' form: key, title, type (pass_fail | categorical | numeric | text), options (categorical only), required, target (feedback | expectation). 'expected_output' targets expectation so the reviewer's answer becomes a trace expectation; the rest are feedback. Enqueue real failing trace ids (copied from the Observability ?trace= URL); answers write back onto each trace as MLflow assessments/expectations."
}

Evaluation & quality gates

Quality gate	Target
Error traces captured	every error trace in the window lands in the test set or the triage queue
Root cause explainable	each flagged trace's failure is readable from its node tree
Triage labeled + written back	queue items answered; assessments visible on the traces
Regression baseline recorded	prod-regression scored in Evaluations (re-run after a fix)

Developer notes & gotchas

Observability Status options are All statuses / OK / Error / In progress; selecting a trace puts its id in the URL (?trace=…) — copy ids from there for the queue.
Dataset rows can be authored in the editor at /eval-datasets/:id → + Add example, or captured via Add to test set → Add example.
Review answers write back as MLflow assessments / expectations on the trace, so the trace becomes self-documenting.
This is operations, not authoring — it converts live production signal into durable evidence (a regression test set + reviewed labels) you can act on and re-check after a fix.

Status	`Error`
Search	the workflow name, e.g. `support-copilot` (optional)

Test Sets → + New Test Set	Name `prod-regression`, Owner `@you` (required), then Create
Observability → Add to test set	pick `prod-regression` from the Choose a test set… dropdown

Queue name	`prod-triage`
Questions	5 questions — types `pass_fail` / `categorical` / `text`; each with a Target of `feedback` or `expectation`

Test set	`prod-regression`
Label	`prod baseline`
Graders	Contains expected (a deterministic scorer — all this slice needs; the Evaluations page also has a Custom LLM judges section, not used here)