Self-Healing Workflows

{"id": "WF01", "tags": ["approval_rejected", "hitl_review"], "inputs": {"workflow_template": "hitl_review", "failing_node": "review", "node_input": {"response": "Refund of $4,800 approved for account A-1007."}, "node_error": "approval_rejected: reviewer rejected (amount over discretionary limit)", "recent_events": ["agent.completed", "pii_guard.passed", "review.waiting_approval", "review.rejected"]}, "expectations": {"failure_kind": "approval_rejected", "expected_root_cause": "The human_approval node 'review' was rejected by the reviewer, so the run failed; recover via run-retry -> run-approve -> run-resume."}}
{"id": "WF02", "tags": ["node_exception", "python_code", "reproducer"], "inputs": {"workflow_template": "blank+failing_node", "failing_node": "failing_node", "node_input": {"payload": {"amount": 50}}, "node_error": "ValueError: failing_node: invalid input -> missing or empty required field 'account_id'", "recent_events": ["start.completed", "failing_node.error"]}, "expectations": {"failure_kind": "node_exception", "expected_root_cause": "The python_code node 'failing_node' raised because the input payload was missing the required 'account_id' field; minimal fix validates/defaults that field instead of indexing it."}}
{"id": "WF03", "tags": ["node_exception", "python_code"], "inputs": {"workflow_template": "blank+failing_node", "failing_node": "failing_node", "node_input": {"payload": {"account_id": "A-2210", "amount": "N/A"}}, "node_error": "ValueError: failing_node: invalid input -> field 'amount' is missing or not a number", "recent_events": ["start.completed", "failing_node.error"]}, "expectations": {"failure_kind": "node_exception", "expected_root_cause": "The python_code node 'failing_node' raised because 'amount' was non-numeric ('N/A'); minimal fix coerces/validates 'amount' and returns a rejected_input result instead of throwing."}}
{"id": "WF04", "tags": ["guardrail_block", "hitl_review"], "inputs": {"workflow_template": "hitl_review", "failing_node": "pii_guard", "node_input": {"response": "Sure — your card 4111 1111 1111 1111 was charged; email me at jdoe@example.com."}, "node_error": "guardrail_block: pii_detection matched [credit_card, email]; on_failure=redact", "recent_events": ["agent.completed", "pii_guard.blocked"]}, "expectations": {"failure_kind": "guardrail_block", "expected_root_cause": "The guardrail node 'pii_guard' blocked/redacted the agent response because it leaked a credit_card and email; fix tightens the upstream agent output (do not emit raw PII) or adjusts the guardrail scope."}}
{"id": "WF05", "tags": ["approval_rejected", "hitl_review", "edge"], "inputs": {"workflow_template": "hitl_review", "failing_node": "review", "node_input": {"response": "Account A-3300 will be unlocked and credited 3 months free."}, "node_error": "approval_rejected: reviewer rejected (commitment not supported by policy)", "recent_events": ["agent.completed", "pii_guard.passed", "review.waiting_approval", "review.rejected"]}, "expectations": {"failure_kind": "approval_rejected", "expected_root_cause": "The human_approval node 'review' was rejected because the drafted reply made an unsupported commitment; recover by editing the upstream draft, then run-retry -> run-approve -> run-resume."}}

---
name: workflow-failure-triage
summary: "Triage a failed CALIBER workflow run: classify the symptom (guardrail block / approval reject / node exception / wait timeout), point to the likely node, and name the next probe in the Run Monitor. Diagnosis only — patching is a manual editor edit."
category: operations
tags: [workflow, debugging, run-monitor, recovery, triage]
render_variables: [run_status, failing_node, node_error, recent_events]
---

# Workflow Failure Triage

Use this checklist when a workflow run is in a terminal `failed` (or stuck)
state and you need to localize the cause fast in the **Run Monitor**. It maps a
symptom to the likely node type and the next concrete probe. It does NOT change
the workflow: the fix is always a manual edit of the manifest in the editor
followed by **save a new version** (there is no `propose_workflow_patch` tool),
and any assistant text here is narration — the actual recovery is done with the
run-retry / run-approve / run-resume controls.

## First, read the run state

- Open the run in **Run Monitor**. Note `{{ run_status }}` and the active node
  from the **Checkpoint** panel (active node id + state blob).
- Open the **Debugger** panel and find the last node with an `error` marker —
  that node id is your prime suspect. Read its inputs/outputs.
- Open the **Recovery** panel for the approval/event timeline and any integrity
  warnings.

## Symptom -> likely node -> next probe

| Symptom (from status / debugger) | Likely failure kind | Likely node | Next probe |
| --- | --- | --- | --- |
| Run sat at `waiting_approval`, then went `failed` after a reject; error mentions approval/rejected | **approval_rejected** | a `human_approval` node (e.g. `review`) | Recovery panel: confirm the reject event + reviewer/reason. Recovery = `run-retry` then `run-approve` -> `run-resume`. |
| A node shows a raised exception in the debugger (`KeyError`, `ValueError`, traceback) | **node_exception** | a `python_code` / `tool` node | Debugger: read that node's input; reproduce with the same input. Fix is a manual edit to that one node (validate/guard the bad field), save new version. |
| Run failed at/after a `guardrail` node; output was blocked or redacted | **guardrail_block** | a `guardrail` node (e.g. `pii_guard`) | Inspect the guardrail node's check + `on_failure`. Decide: tighten upstream output, or adjust the guardrail scope. Manual edit + new version. |
| Run never reached terminal success; stuck/expired waiting on an event or clock | **wait_timeout** | a `wait_for_event` / `wait_until` node | Recovery panel: check the awaited event + timeout. Use `resume-by-event` if the event is now available; else adjust the timeout (manual edit). |
| Run failed before the first node; "workflow_not_found" / bad input shape | **bad_request** (not a node fault) | none (run setup) | Re-check the workflow id and the input payload against the input schema before re-running. |

## Localize, don't guess

- Tie every conclusion to a node id and the actual error text from the
  debugger. If the evidence does not name a node and a reason, say so and gather
  more trace detail before proposing a change.
- Keep the fix minimal and scoped to the failing node. A smaller patch isolates
  impact and is easier to validate.

## After you localize

1. Reproduce the failing input once more to confirm it is deterministic.
2. Make the smallest manual edit in the editor that addresses the root cause;
   **save a new version**.
3. Validate: Preview + a real run of the failing input on the new version, then
   re-run a small regression slice of prior-good inputs.
4. Convert the incident into a regression case (add the reproducer to the
   scenario dataset) so it cannot silently come back.

Inputs you may be given for narration:
- run status: `{{ run_status }}`
- failing node: `{{ failing_node }}`
- node error: `{{ node_error }}`
- recent events: `{{ recent_events }}`

"""failing_node — Caliber workflow `python_code` node body (deliberate fault).

This is NOT a registered tool. Paste the body of this module into a CALIBER
**Python Code** node (`Compose → Workflows` → drag a *Python Code* node). It
needs no registration, versions with the workflow, and uses **stdlib only**.

Its purpose is to MANUFACTURE a reproducible, code-level node failure for the
workflow-debugger demo (the second failure option in this scenario's README):
on a malformed input it raises, so the run lands in `failed` with a real node
exception you can localize in the Debugger panel.

CONTRACT (how the sandbox calls this)
  The node body runs inside the runtime wrapper:
      run_python_node(input=None, context=None, inputs=None, run_input='')
  Wire the upstream port so this node receives:
      inputs["payload"]  -> a dict; MUST contain a non-empty "account_id"
                            (str) and an "amount" (number >= 0).
  It also tolerates the payload arriving as the single `input` dict (or a JSON
  string) so the node still works when the upstream emits one object.

THE FAULT (what triggers it) — before the patch
  Pre-patch, the body still validates the payload with `.get(...)` (it never
  indexes a key directly), collects every problem into an `errors` list, and
  when `STRICT = True` raises a single composed `ValueError` joining those
  reasons. So ANY of these inputs reproduces a node-level exception (-> `failed`):
    * "account_id" missing entirely        -> ValueError (missing/empty id)
    * "account_id" present but empty ""     -> ValueError (missing/empty id)
    * "amount" non-numeric, e.g. "N/A"      -> ValueError (bad amount)
  The raised message is "failing_node: invalid input -> " + "; ".join(errors).
  In the Debugger step trace this node shows the raised error on its `error`
  marker; that exception text + this node id IS the concrete evidence the
  diagnosis-summary prompt and RootCauseQuality judge must cite.

THE FIX (fixed behavior) — after the manual patch
  The patch is a MANUAL editor edit (save a new workflow version): swap the
  raising lookups for validated, defaulted access and emit a structured
  `validation_error` on the result port instead of throwing. The dividing line
  below — `STRICT = True` — is the smallest change that flips fault->handled.
  Flip it to `False` (or delete the raising branch) and re-run the SAME failing
  input to confirm the node now completes with `status="rejected_input"`
  instead of crashing the run. There is no propose_workflow_patch tool; this
  edit is the patch.

OUTPUTS (on the node's `result` / `text` ports), post-fix
  {
    "status":   "ok" | "rejected_input",
    "account_id": <validated id> | None,
    "amount":     <float>        | None,
    "errors":     [<human-readable reasons the input was rejected>]
  }
"""

# `json` is pre-injected into the node namespace by the workflow Python
# sandbox, so we deliberately do NOT `import json` here — the sandbox's
# SAFE_BUILTINS has no `__import__`, and an import would crash the node.

# The one knob the manual patch flips. True = original deliberately-faulty
# behavior (raise on malformed input -> reproducible node exception).
# Set to False as the minimal scoped fix (validate + return, never throw).
STRICT = True


def _coerce_payload(value):
    """Accept a dict directly or a JSON string; anything else -> {}."""
    if isinstance(value, dict):
        return value
    if isinstance(value, str):
        try:
            parsed = json.loads(value)
        except Exception:
            return {}
        return parsed if isinstance(parsed, dict) else {}
    return {}


def _validate(payload):
    """Return (account_id, amount, errors) without raising."""
    errors = []

    account_id = payload.get("account_id")
    if not isinstance(account_id, str) or not account_id.strip():
        errors.append("missing or empty required field 'account_id'")
        account_id = None
    else:
        account_id = account_id.strip()

    raw_amount = payload.get("amount")
    amount = None
    try:
        amount = float(raw_amount)
        if amount < 0:
            errors.append("field 'amount' must be >= 0")
            amount = None
    except Exception:
        errors.append("field 'amount' is missing or not a number")

    return account_id, amount, errors


def run_python_node(input=None, context=None, inputs=None, run_input=""):
    src = inputs if isinstance(inputs, dict) else {}

    # Resolve the payload from inputs["payload"], else the single input object.
    payload = _coerce_payload(src.get("payload"))
    if not payload:
        payload = _coerce_payload(input if input is not None else context)

    account_id, amount, errors = _validate(payload)

    if STRICT and errors:
        # --- DELIBERATE FAULT (pre-patch) -----------------------------------
        # Reproduces a real node-level exception so the run lands in `failed`.
        # The first failing reason becomes the raised message — exactly the
        # evidence the debugger surfaces and the diagnosis must cite.
        raise ValueError(
            "failing_node: invalid input -> {}".format("; ".join(errors))
        )

    # --- FIXED BEHAVIOR (post-patch: STRICT=False) --------------------------
    if errors:
        result = {
            "status": "rejected_input",
            "account_id": account_id,
            "amount": amount,
            "errors": errors,
        }
    else:
        result = {
            "status": "ok",
            "account_id": account_id,
            "amount": amount,
            "errors": [],
        }
    return {"text": json.dumps(result), "result": result}

---
name: diagnosis-summary
model_hint: any capable instruct model; this is summarization over a trace, not generation
variables: [failing_node, node_input, node_error, recent_events]
commit_message: "v1 trace-grounded workflow failure diagnosis"
---

You are a workflow recovery assistant. You read the evidence from ONE failing
workflow run — the failing node, its input, the error it produced, and the
recent run events — and you write a tight, trace-grounded diagnosis. You do not
debug or patch anything; the operator does that in the Run Monitor. Your only
job is to state the root cause from the evidence and propose the smallest
scoped fix.

You return JSON ONLY — no prose, no markdown, no code fences:
{
  "root_cause": short string: WHY the run failed, stated from the evidence,
  "evidence_node": the node id you are blaming (must equal the failing node),
  "fix_summary": short string: the single smallest change that addresses the
                 root cause (a manual editor edit + save-new-version; there is
                 no automatic patch),
  "regression_slice": array of 2-4 short input descriptions to re-run after the
                      fix to confirm no new failures (include the reproducer
                      plus a couple of prior-good cases)
}

Rules:
- Cite ONLY the concrete evidence below. The blamed `evidence_node` MUST be
  exactly `{{ failing_node }}`. Quote or paraphrase the actual `{{ node_error }}`
  text in `root_cause`; do not invent an error you were not shown.
- Do NOT speculate beyond the trace. If the evidence does not determine the
  cause, set `root_cause` to "insufficient evidence" and `fix_summary` to
  "gather more trace detail before patching" — never guess a cause.
- Classify the failure as exactly one of: an approval rejection (a
  human_approval node was rejected), a node exception (a node raised), a
  guardrail block (a guardrail node blocked), or a wait timeout (a
  wait_for_event / wait_until node expired) — and let that classification drive
  the `fix_summary`.
- Keep `fix_summary` minimal and scoped to one node/config. Prefer the smallest
  change that flips the failing node from fault to handled. Never propose a
  change to a node other than the failing one unless the evidence names it.
- Add no commitments, no timelines, and no claims about whether the fix will
  pass — that is decided by re-running, not by you.

Failing node id: {{ failing_node }}
Failing node input:
"""
{{ node_input }}
"""
Failing node error / outcome:
"""
{{ node_error }}
"""
Recent run events (most recent last):
"""
{{ recent_events }}
"""

Return only the JSON diagnosis.

{
  "name": "RootCauseQuality",
  "model": "set to your configured judge model (e.g. the gateway default)",
  "feedback_value_type": "bool",
  "instructions": "You audit a workflow-failure diagnosis for whether it is grounded in the actual run evidence and proposes a minimal, scoped fix. You are given the failing-run evidence {{ inputs }} (which includes failing_node, node_input, node_error, recent_events) and the diagnosis under test {{ outputs }} (a JSON object with root_cause, evidence_node, fix_summary, regression_slice). You may also be given labeled {{ expectations }} (failure_kind, expected_root_cause).\n\nReturn true ONLY if ALL of the following hold:\n1. {{ outputs }} is a single valid JSON object with the keys root_cause, evidence_node, fix_summary, regression_slice.\n2. outputs.evidence_node equals inputs.failing_node exactly (the diagnosis blames the node that actually failed — not a different node).\n3. outputs.root_cause cites the concrete failing evidence from {{ inputs }}: it reflects the actual inputs.node_error (e.g. the same error kind / missing field / rejected approval / blocked guardrail) and is consistent with inputs.recent_events. It must NOT introduce a cause that the evidence does not support.\n4. The diagnosis's failure classification is consistent with the evidence and, when provided, with expectations.failure_kind (approval_rejected / node_exception / guardrail_block / wait_timeout).\n5. outputs.fix_summary is a SINGLE minimal, scoped change addressing that root cause (e.g. validate/default the missing field on the failing node, or re-run the approval, or tighten the upstream output) and does not expand scope to unrelated nodes or propose a broad rewrite.\n6. outputs.regression_slice is a non-empty list that includes re-running the reproducing input plus at least one prior-good case.\n\nReturn false if the diagnosis blames the wrong node, invents an error or cause not present in {{ inputs }}, speculates beyond the trace, or proposes a non-minimal / out-of-scope fix. Base the verdict only on the evidence in {{ inputs }} (and {{ expectations }} when present), not on your own opinion of the workflow.",
  "notes": "Optional LLM judge (verification.yaml: RootCauseQuality, criteria 'diagnosis cites concrete failing evidence'). feedback_value_type bool -> pass/fail. Select it in an Evaluations run under 'Custom LLM judges' (it runs as a Judge.<id> scorer; or use the calibration path) over dataset/failing-inputs.jsonl against the diagnosis-summary output. Template vars {{ inputs }}/{{ outputs }}/{{ expectations }} are bound by the eval runner per row."
}