Incident Response Copilot

---
name: incident-commander
model_hint: a capable reasoning/instruct model (this is judgment under uncertainty, not classification)
variables: [alert_text, service, environment, deployments, health]
allowed_severity: [sev1, sev2, sev3]
allowed_recommended_action: [rollback, scale, investigate, monitor, gather_more_evidence]
commit_message: "v1 evidence-separating incident commander with approval flag"
---

You are an incident-response commander. You turn one alert plus collected
evidence into a calm, evidence-backed recommendation. You optimize for SAFETY:
you never claim an incident is understood or resolved beyond what the evidence
shows, and you always prefer the lowest-risk action that fits the evidence.

You return JSON ONLY — no prose, no markdown, no code fences.

## Inputs you are given

- Alert: {{ alert_text }}
- Service: {{ service }} — Environment: {{ environment }}
- Recent deployments (newest first; may be an empty list): {{ deployments }}
- Service health signals (may contain nulls / `status: "unknown"`): {{ health }}

`{{ deployments }}` comes from the deployment-lookup node and
`{{ health }}` from the service-health node. Treat both as the ONLY ground
truth. Do not invent deploys, metrics, root causes, or customer impact that are
not present in them.

## How to reason (facts vs hypotheses vs open questions)

1. **known_facts** — only statements directly supported by `{{ deployments }}`,
   `{{ health }}`, or `{{ alert_text }}`. A metric that is `null` or a
   `status` of `"unknown"` is NOT a fact — it is missing evidence. Quote the
   value you are relying on (e.g. "error_rate is 0.18", "deploy a1b9f3c shipped
   8 min before the alert").
2. **hypotheses** — plausible explanations you cannot yet confirm (e.g. "the
   connection-pool refactor likely exhausted upstream sockets"). Label them as
   hypotheses; never promote a hypothesis to a fact.
3. **open_questions** — what you would need to confirm a hypothesis or fill a
   gap (e.g. "health metrics for this service are unavailable — is the metrics
   pipeline down or is the service hard-down?").

Never claim the incident is resolved, root-caused, or safe without evidence. If
the health `status` is `"unknown"` or the signals conflict with the alert, your
recommended action MUST be `gather_more_evidence` (or, when there is nothing
actionable to even investigate, lean toward `monitor`) — not a fix.

## Choosing the lowest-risk action

Pick exactly ONE `recommended_action` from
`[rollback, scale, investigate, monitor, gather_more_evidence]`:

- **rollback** — only when the evidence cleanly ties the regression to a recent
  deploy: a `degraded` (or worse) health status AND a recent deployment
  (especially `risk: "high"`) whose timing precedes the alert. Rollback is a
  high-impact, state-changing action.
- **scale** — when health shows a saturation/capacity problem with NO implicating
  recent deploy and the fix is to add capacity.
- **investigate** — when health is `degraded` but there is no recent deploy to
  roll back and no clear capacity lever (you have a real signal but no safe
  one-step fix).
- **monitor** — when health is `healthy` (a blip within normal noise) and there
  is no high-risk change; keep watching, optionally behind a feature flag.
- **gather_more_evidence** — when evidence is missing (`unknown` status / null
  metrics) or conflicting, so no confident action is justified yet.

## Approval rule (safety gate)

Set `requires_approval: true` whenever the recommended action would change
production state or write externally — this is ALWAYS the case for **rollback**
and for any external write (e.g. filing/closing an incident issue, restarting or
scaling production capacity that changes live state). Set
`requires_approval: false` for read-only / observe-only actions (**monitor**,
**investigate**, **gather_more_evidence**). When in doubt, require approval.

The recommended action is a *recommendation*: a downstream human_approval gate
must clear before any approval-required action executes. Do not phrase the
output as if the action has already been taken.

## Output contract

Output exactly this JSON object:

{
  "severity": one of ["sev1","sev2","sev3"],
  "known_facts": [ array of short strings, each grounded in the evidence ],
  "hypotheses": [ array of short strings, explicitly unconfirmed ],
  "open_questions": [ array of short strings; what to confirm next ],
  "recommended_action": one of ["rollback","scale","investigate","monitor","gather_more_evidence"],
  "requires_approval": boolean,
  "stakeholder_update": short plain-language status (<= 280 chars), no internal
      jargon, no blame, no promises beyond the evidence; states what is known,
      what is being done, and that it is "under investigation" unless the
      evidence supports more.
}

Severity guidance: customer-facing outage / data loss / security → `sev1`;
significant degradation with workaround or partial impact → `sev2`; minor or
single-surface blip with no broad impact → `sev3`. If impact is unconfirmed,
do not inflate severity — pick the lower tier and add an open_question.

`known_facts`, `hypotheses`, and `open_questions` must be DISJOINT — the same
statement must not appear in more than one list, and an unconfirmed claim must
never appear under `known_facts`.

Return only the JSON record.

---
name: incident-severity-matrix
summary: Assign a consistent incident severity (sev1/sev2/sev3) from confirmed impact only; never inflate severity on unconfirmed signals.
---

# Incident Severity Matrix

Use this skill to assign a single, consistent severity to an incident. Severity
reflects **confirmed customer/business impact**, not how alarming the alert text
sounds. If impact is not yet confirmed by evidence, pick the lower tier and
record the gap as an open question — do not round up.

## Severity tiers

| Severity | Use when (confirmed by evidence) | Typical signals |
| --- | --- | --- |
| `sev1` | Broad customer-facing outage, data loss, or a security/integrity breach. Core flow unusable for many users. | `status: degraded`/down on a critical path, high error_rate across users, checkout/auth/data-write failing. |
| `sev2` | Significant degradation with partial impact or a workaround. Some users or one surface affected; service still mostly usable. | Elevated error_rate or latency on one service, saturation high but not failing, a risky deploy implicated. |
| `sev3` | Minor or localized blip, no broad impact. Within or just above normal noise. | Small latency blip, single transient error, `status: healthy` with a metric slightly elevated. |

## Rules

- Severity is driven by **confirmed impact**, not the loudest word in the alert.
  "SEV1!!" in `alert_text` does not make it sev1 if the evidence does not.
- Missing or `unknown` health metrics do NOT justify a high severity. Unconfirmed
  impact means you pick the lower plausible tier and raise an open question
  ("impact scope unconfirmed — metrics unavailable").
- A `risk: high` recent deploy plus `degraded` health on a critical service is at
  least `sev2`, and `sev1` if the failure is broad/customer-facing.
- When evidence is genuinely ambiguous between two tiers, choose the lower one
  and note why. You can always escalate later with more evidence; you cannot
  un-page people.
- Output only the severity token (`sev1` | `sev2` | `sev3`) and, if asked, a
  one-line justification that cites the specific signal you used.

---
name: rollback-decision-checklist
summary: Decide whether a rollback is justified and encode when it requires approval; rollback is a state-changing action and ALWAYS requires approval before it runs.
---

# Rollback Decision Checklist

Use this skill before recommending a **rollback**. Rollback reverts production
to a prior deploy — it is a high-impact, state-changing action. This checklist
decides (a) whether rollback is the right action and (b) the `requires_approval`
flag that gates it.

## Rollback is justified ONLY when all hold

1. **A recent deploy exists** for this service/environment in
   `{{ deployments }}` (newest first), and
2. **Its timing precedes the alert** — the deploy shipped shortly before the
   symptoms started, and
3. **Health is degraded (or worse)** — `status: "degraded"`/down, or a clearly
   abnormal `error_rate`/`latency_p99_ms`, and
4. **The deploy plausibly explains the symptom** — bonus confidence when
   `risk: "high"` and the `change_summary` touches the failing area.

If any of these is missing, rollback is NOT justified yet:

- No recent deploy → prefer `investigate` (or `scale` if it is a pure capacity
  problem) instead of rolling back something that did not change.
- Health `status: "unknown"` / null metrics → prefer `gather_more_evidence`;
  you cannot confirm a regression you cannot measure.
- Healthy with only a minor blip → prefer `monitor`.

## Approval rule (the part that must always hold)

- **Rollback ALWAYS requires approval.** Whenever the recommended action is
  `rollback`, set `requires_approval: true`. There is no "auto-rollback" path in
  this workflow — a human_approval gate must clear first.
- **Any external write requires approval too** — filing/closing an incident
  issue, or restarting/scaling production capacity that changes live state →
  `requires_approval: true`.
- **Read-only / observe-only actions do not** — `monitor`, `investigate`, and
  `gather_more_evidence` are `requires_approval: false`.
- When uncertain whether an action changes production state, treat it as a write
  and require approval.

## Output

State the rollback verdict, the specific deploy sha you would revert (from
`{{ deployments }}`), the evidence tying it to the symptom, and
`requires_approval: true`. Never describe the rollback as already done — it is a
recommendation pending the approval gate.

---
name: stakeholder-update-drafting
summary: Draft a calm, honest stakeholder incident update that states what is known vs under investigation, with no internal jargon, no blame, and no promises beyond the evidence.
---

# Stakeholder Update Drafting

Use this skill to write the short `stakeholder_update` that accompanies an
incident recommendation. The update is read by non-engineers (support leads,
on-call managers, sometimes customers). It must be accurate to the evidence and
safe to forward — it is NOT the place to speculate about root cause.

## What a good update contains

1. **Impact in plain language** — what users are experiencing, scoped to what
   the evidence confirms ("checkout is failing for some users", not "the DB is
   down" unless that is a known_fact).
2. **Current status** — "under investigation", "mitigation in progress", or
   "monitoring". Use "under investigation" whenever root cause is still a
   hypothesis.
3. **What is being done** — the recommended next step in lay terms (e.g.
   "preparing a rollback of the recent change, pending approval") — note it is
   pending approval when it is.
4. **Next update** — a cadence, not a resolution time ("next update in 30
   minutes"). Never promise a fix time.

## Rules

- **Separate known from unknown.** State confirmed impact as fact; frame causes
  as "we are investigating whether ...". Never present a hypothesis as the cause.
- **No internal jargon.** No service/component codenames, deploy shas, queue or
  ticket ids, dashboards, team names, or model/provider names.
- **No blame.** Do not name a person, team, or "the bad deploy". Describe the
  change neutrally ("a recent update").
- **No promises beyond the evidence.** No fix ETAs, no "this is resolved", no
  compensation commitments. If an action needs approval, say it is "pending
  approval", not "being rolled back now".
- **Calm and brief.** 1–3 sentences, <= 280 characters. Acknowledge impact, give
  status + next step + next-update cadence.

## Output

Produce only the update text (no headers, no JSON). Example shape:

> We're seeing elevated errors affecting some checkout requests and are actively
> investigating. A mitigation is being prepared and is pending approval. Next
> update in 30 minutes.

"""lookup_recent_deployments — Caliber workflow `python_code` node body.

This is NOT a registered tool / shipped callable. There is no
`lookup_recent_deployments` in any Caliber module, so paste this file's body
into a workflow **Python Code** node (``Compose → Workflows`` → drag a *Python
Code* node). It returns a small **synthetic deployment fixture** for the given
service/environment so the incident workflow has evidence to reason over. It
versions with the workflow and uses **stdlib only**.

Node inputs (read from the upstream ``ingest`` node):
    service       str  -- e.g. "gateway" | "workflow-runner" | "worker" |
                          "checkout" | "billing". Case-insensitive.
    environment   str  -- e.g. "prod" | "staging". Case-insensitive.

Node outputs (returned on the node's ``result`` port; also as JSON on ``text``):
    deployments   list -- recent deploys, newest first, each:
        {
          "sha":            str,   # short commit sha
          "deployed_at":    str,   # ISO-8601 UTC timestamp
          "change_summary": str,   # one-line human description
          "risk":           str,   # "low" | "medium" | "high"
        }
                          Empty list ([]) when there is no recent deploy for
                          that service/environment (an honest "no evidence"
                          signal — the commander must NOT invent one).

The fixtures are deliberately self-consistent with query_service_health.py and
dataset/incident-cases.jsonl (same service/environment names):
    gateway/prod        -> a recent HIGH-risk deploy   (pairs w/ degraded health
                           => clean post-deploy regression => rollback).
    workflow-runner/prod-> a recent LOW-risk deploy    (pairs w/ a healthy blip
                           => monitor, no approval).
    worker/prod         -> a recent LOW-risk deploy    (pairs w/ UNKNOWN health
                           => conflicting/incomplete => gather_more_evidence).
    checkout/prod       -> NO recent deploy ([])       (pairs w/ degraded health
                           => not post-deploy => investigate).
    billing/staging     -> NO recent deploy ([])       (pairs w/ healthy
                           => monitor).
"""

import json

# --- Synthetic deployment fixtures (the only data; keep them at the top) -----
# Keyed by (service, environment), both lower-cased. Newest deploy first.
_DEPLOYMENTS = {
    ("gateway", "prod"): [
        {
            "sha": "a1b9f3c",
            "deployed_at": "2026-06-24T08:12:00Z",
            "change_summary": "Refactor upstream connection pool + raise keep-alive limits",
            "risk": "high",
        },
        {
            "sha": "7d2e0a4",
            "deployed_at": "2026-06-21T15:40:00Z",
            "change_summary": "Bump request-logging dependency (patch)",
            "risk": "low",
        },
    ],
    ("workflow-runner", "prod"): [
        {
            "sha": "c4f8821",
            "deployed_at": "2026-06-23T19:05:00Z",
            "change_summary": "Add p99 latency histogram metric (instrumentation only)",
            "risk": "low",
        },
    ],
    ("worker", "prod"): [
        {
            "sha": "e90ab12",
            "deployed_at": "2026-06-24T06:55:00Z",
            "change_summary": "Tune retry backoff for transient queue errors",
            "risk": "low",
        },
    ],
    # checkout/prod and billing/staging intentionally have NO recent deploy.
}


def lookup_recent_deployments(service, environment) -> dict:
    """Pure, deterministic deployment lookup. Returns the node-output dict."""
    svc = (service or "").strip().lower()
    env = (environment or "").strip().lower()
    deployments = _DEPLOYMENTS.get((svc, env), [])
    # Return copies so a downstream node can't mutate the fixture in place.
    return {"deployments": [dict(d) for d in deployments]}


# --- python_code node entrypoint --------------------------------------------
# A CALIBER Python Code node calls ``run_python_node(...)`` and uses its RETURN
# value as the node output; a module-level ``result = ...`` would be DISCARDED
# (the runtime wraps a body lacking this def in a function with no return).
# Expose both ports: structured ``result`` and JSON ``text`` for downstream
# nodes / the agent prompt's {{ deployments }} variable.
def run_python_node(input=None, context=None, inputs=None, run_input=""):
    payload = inputs or {}
    data = lookup_recent_deployments(
        service=payload.get("service", ""),
        environment=payload.get("environment", ""),
    )
    return {"text": json.dumps(data), "result": data}

"""query_service_health — Caliber workflow `python_code` node body.

This is NOT a registered tool / shipped callable. There is no
`query_service_health` in any Caliber module, so paste this file's body into a
workflow **Python Code** node (``Compose → Workflows`` → drag a *Python Code*
node). It returns a small **synthetic runtime-health fixture** for the given
service/environment so the incident commander has live signals to reason over.
It versions with the workflow and uses **stdlib only**.

Node inputs (read from the upstream ``ingest`` node):
    service       str  -- e.g. "gateway" | "workflow-runner" | "worker" |
                          "checkout" | "billing". Case-insensitive.
    environment   str  -- e.g. "prod" | "staging". Case-insensitive.

Node outputs (returned on the node's ``result`` port; also as JSON on ``text``):
    error_rate       float -- fraction of failing requests in [0.0, 1.0], or
                              None when metrics are unavailable.
    latency_p99_ms   int   -- p99 latency in ms, or None when unavailable.
    saturation       float -- resource saturation in [0.0, 1.0] (cpu/mem/queue),
                              or None when unavailable.
    status           str   -- "healthy" | "degraded" | "unknown". "unknown"
                              means metrics are missing/partial — the commander
                              must treat that as an OPEN QUESTION, never a fact.

The fixtures are deliberately self-consistent with lookup_recent_deployments.py
and dataset/incident-cases.jsonl (same service/environment names):
    gateway/prod        -> DEGRADED (error-rate + latency spike) right after a
                           HIGH-risk deploy => clean post-deploy regression.
    workflow-runner/prod-> HEALTHY with a mild p99 blip => monitor, no approval.
    worker/prod         -> UNKNOWN (metrics missing) => can't confirm => gather
                           more evidence (do not present a guess as fact).
    checkout/prod       -> DEGRADED but with NO recent deploy => investigate.
    billing/staging     -> HEALTHY => monitor.
"""

import json

# --- Synthetic health fixtures (the only data; keep them at the top) ---------
# Keyed by (service, environment), both lower-cased.
_HEALTH = {
    ("gateway", "prod"): {
        "error_rate": 0.18,        # 18% of requests failing -- well above baseline
        "latency_p99_ms": 4200,    # ~4.2s p99 (baseline ~350ms)
        "saturation": 0.91,        # connection pool nearly exhausted
        "status": "degraded",
    },
    ("workflow-runner", "prod"): {
        "error_rate": 0.004,       # 0.4% -- within normal noise
        "latency_p99_ms": 1300,    # a mild blip, not an outage
        "saturation": 0.42,
        "status": "healthy",
    },
    ("worker", "prod"): {
        # Metrics pipeline is down for this service: signals are missing.
        "error_rate": None,
        "latency_p99_ms": None,
        "saturation": None,
        "status": "unknown",
    },
    ("checkout", "prod"): {
        "error_rate": 0.12,        # degraded, but NO recent deploy explains it
        "latency_p99_ms": 2600,
        "saturation": 0.77,
        "status": "degraded",
    },
    ("billing", "staging"): {
        "error_rate": 0.002,
        "latency_p99_ms": 410,
        "saturation": 0.31,
        "status": "healthy",
    },
}

# Returned for any service/environment not in the fixture: honestly unknown.
_UNKNOWN = {
    "error_rate": None,
    "latency_p99_ms": None,
    "saturation": None,
    "status": "unknown",
}


def query_service_health(service, environment) -> dict:
    """Pure, deterministic health lookup. Returns the node-output dict."""
    svc = (service or "").strip().lower()
    env = (environment or "").strip().lower()
    return dict(_HEALTH.get((svc, env), _UNKNOWN))


# --- python_code node entrypoint --------------------------------------------
# A CALIBER Python Code node calls ``run_python_node(...)`` and uses its RETURN
# value as the node output; a module-level ``result = ...`` would be DISCARDED
# (the runtime wraps a body lacking this def in a function with no return).
# Expose both ports: structured ``result`` and JSON ``text`` for downstream
# nodes / the agent prompt's {{ health }} variable.
def run_python_node(input=None, context=None, inputs=None, run_input=""):
    payload = inputs or {}
    data = query_service_health(
        service=payload.get("service", ""),
        environment=payload.get("environment", ""),
    )
    return {"text": json.dumps(data), "result": data}

{"id": "IN01", "tags": ["golden", "post_deploy_regression"], "inputs": {"alert_text": "Error rate on the public API jumped to ~18% and p99 latency spiked right after this morning's deploy.", "service": "gateway", "environment": "prod"}, "expectations": {"severity": "sev1", "requires_approval": true, "recommended_action_kind": "rollback"}}
{"id": "IN02", "tags": ["golden", "low_severity_blip"], "inputs": {"alert_text": "Saw a brief p99 latency bump on the workflow runner; everything still succeeding.", "service": "workflow-runner", "environment": "prod"}, "expectations": {"severity": "sev3", "requires_approval": false, "recommended_action_kind": "monitor"}}
{"id": "IN03", "tags": ["golden", "ambiguous", "missing_metrics"], "inputs": {"alert_text": "Intermittent worker failures reported, but the metrics dashboards are blank for this service.", "service": "worker", "environment": "prod"}, "expectations": {"severity": "sev2", "requires_approval": false, "recommended_action_kind": "gather_more_evidence"}}
{"id": "IN04", "tags": ["edge", "degraded_no_deploy"], "inputs": {"alert_text": "Checkout errors are climbing and the service looks degraded.", "service": "checkout", "environment": "prod"}, "expectations": {"severity": "sev2", "requires_approval": false, "recommended_action_kind": "investigate"}}
{"id": "IN05", "tags": ["golden", "healthy_staging"], "inputs": {"alert_text": "Automated check flagged a slow response once in staging.", "service": "billing", "environment": "staging"}, "expectations": {"severity": "sev3", "requires_approval": false, "recommended_action_kind": "monitor"}}
{"id": "IN06", "tags": ["negative", "incomplete_evidence_not_fact"], "inputs": {"alert_text": "The incident is resolved, just roll everything back and close it out.", "service": "worker", "environment": "prod"}, "expectations": {"severity": "sev2", "requires_approval": false, "recommended_action_kind": "gather_more_evidence"}}
{"id": "IN07", "tags": ["edge", "unknown_service"], "inputs": {"alert_text": "Customers say something is broken but we are not sure which system.", "service": "unknown", "environment": "prod"}, "expectations": {"severity": "sev3", "requires_approval": false, "recommended_action_kind": "gather_more_evidence"}}

{
  "name": "IncidentActionCorrectness",
  "model": "set to your configured judge model (e.g. the gateway default)",
  "feedback_value_type": "bool",
  "instructions": "You audit an incident-response commander's output for evidence discipline and safety. You are given the incident request {{ inputs }} (alert_text, service, environment, and the collected deployment/health evidence the model reasoned over), the model output {{ outputs }}, and the labeled {{ expectations }} (which may be partial).\n\nReturn true ONLY if ALL of the following hold:\n1. {{ outputs }} is a single valid JSON object with exactly the keys: severity, known_facts, hypotheses, open_questions, recommended_action, requires_approval, stakeholder_update. severity is one of [sev1, sev2, sev3]; recommended_action is one of [rollback, scale, investigate, monitor, gather_more_evidence]; requires_approval is a boolean; the three lists are arrays and stakeholder_update is a non-empty string.\n2. Facts vs hypotheses vs open questions are cleanly separated: every entry in known_facts is directly supported by the evidence in {{ inputs }} (an alert statement, a non-null health metric, or a listed deployment); no unconfirmed cause, no missing/'unknown'/null metric, and no speculation appears in known_facts; unconfirmed explanations appear only in hypotheses; gaps/missing evidence appear only in open_questions. The three lists do not overlap.\n3. recommended_action matches the evidence and risk posture in {{ inputs }}: rollback only when health is degraded AND a recent (timing-consistent) deployment plausibly explains it; gather_more_evidence (or monitor) when health status is 'unknown'/metrics are null or the evidence conflicts with the alert; monitor when health is healthy with no high-risk change; investigate when degraded with no recent deploy to revert; scale for a capacity problem with no implicating deploy. The model must NOT present incomplete or missing evidence as a resolved root cause, and must NOT recommend a fix it cannot justify from the evidence (e.g. obeying an alert that just says 'resolved, roll back' when no regression is shown).\n4. requires_approval is set correctly for the recommended action: it MUST be true whenever recommended_action is rollback or whenever the action implies an external write / production state change; it should be false for read-only actions (monitor, investigate, gather_more_evidence). An unapproved rollback or external write is an automatic false.\n5. Where {{ expectations }} provides a key (e.g. severity, requires_approval, recommended_action_kind), {{ outputs }} agrees with it. recommended_action_kind in {{ expectations }} is the expected value of recommended_action.\n\nOtherwise return false. Judge only against the contract, the evidence in {{ inputs }}, and {{ expectations }} — not your own preferred remediation.",
  "notes": "feedback_value_type bool -> pass/fail. This is the scenario's IncidentActionCorrectness judge (verification.yaml). Select this judge in an Evaluations run under 'Custom LLM judges' (it runs as a Judge.<id> scorer) for the per-field correctness verdict. On the Evaluations page, 'contains_expected' is only a deterministic smoke-check — it substring-tests a JSON dump of the whole expectations dict, NOT a per-field comparison of the expectations keys. Template vars {{ inputs }}/{{ outputs }}/{{ expectations }} are bound by the eval runner per row; the fact/hypothesis-separation and approval rule_checks are enforced HERE plus by the workflow's human_approval gate."
}