Release Signoff Factory

{
  "_comment": "Fill-in template. The operator pastes REAL evidence here before scoring — there is no engine that fills this. Leave a field empty until you have verified the evidence; the release-risk-summarizer (and the rubric) treat any empty required field as MISSING and force no_go. eval_run_id = the Evaluations scorecard run id (Evaluate -> Evaluations). review_queue_complete = true only when every required Review Queue is fully answered (Observe -> Review Queues). approval_branch_run_id = the fresh run id from re-running that scenario's high-risk pipeline slice (SCN-07 approval branch, SCN-08 rollback path, SCN-09 retry/resume recovery); it is REQUIRED only for SCN-07/08/09. SCN-10 is NOT required to carry approval_branch_run_id — its critical slice is human-review-queue completion, not a pipeline re-run, so SCN-10 readiness gates on eval_run_id + review_queue_complete only.",
  "release_tag": "",
  "prepared_by": "",
  "prepared_at": "",
  "scenarios": [
    {
      "scenario": "SCN-07",
      "name": "support-triage-resolution-loop",
      "critical_slice": "approval branch (human_approval gate -> approve -> resume)",
      "required": ["eval_run_id", "review_queue_complete", "approval_branch_run_id"],
      "eval_run_id": "",
      "review_queue_complete": false,
      "approval_branch_run_id": "",
      "notes": ""
    },
    {
      "scenario": "SCN-08",
      "name": "incident-response-commander",
      "critical_slice": "rollback path (re-run to confirm it still passes)",
      "required": ["eval_run_id", "review_queue_complete", "approval_branch_run_id"],
      "eval_run_id": "",
      "review_queue_complete": false,
      "approval_branch_run_id": "",
      "notes": ""
    },
    {
      "scenario": "SCN-09",
      "name": "workflow-debugger-self-healing-lab",
      "critical_slice": "retry/resume recovery path",
      "required": ["eval_run_id", "review_queue_complete", "approval_branch_run_id"],
      "eval_run_id": "",
      "review_queue_complete": false,
      "approval_branch_run_id": "",
      "notes": ""
    },
    {
      "scenario": "SCN-10",
      "name": "judge-certification-human-review-lab",
      "critical_slice": "judge certification + human-review queue completion",
      "required": ["eval_run_id", "review_queue_complete"],
      "eval_run_id": "",
      "review_queue_complete": false,
      "approval_branch_run_id": "",
      "notes": ""
    }
  ],
  "allure": {
    "_comment": "Generated EXTERNALLY via `make allure-report`, then served in-app at Settings -> Allure Report (GET /observability/allure-report). Set loaded:true only after you have confirmed the report opens in-app; the tab reads the stored URL caliber.allure.reportUrl.",
    "report_url": "",
    "loaded": false,
    "regenerated_at": ""
  }
}

{
  "_comment": "Optional Review Queue question schema for blocker triage. Create a queue via POST /review-queues with these questions, enqueue the candidate trace ids (.../items), and answer them; answers write back onto the trace as MLflow assessments. type is the feedback value type (bool|int|float|str), matching the platform's review-queue question shape. Use this to decide which surfaced issues are real release blockers and to record waivers.",
  "queue_name": "release-blocker-triage",
  "description": "Triage candidate release blockers: confirm whether each is a true release blocker, who owns it, and (if waived) when the waiver expires.",
  "questions": [
    {
      "key": "is_release_blocker",
      "type": "bool",
      "prompt": "Is this a release blocker (must be fixed or waived before go)?"
    },
    {
      "key": "owning_scenario",
      "type": "str",
      "prompt": "Which upstream scenario owns this blocker? (e.g. SCN-07, SCN-08, SCN-09, SCN-10)"
    },
    {
      "key": "severity",
      "type": "str",
      "prompt": "Severity: one of [critical, high, medium, low]."
    },
    {
      "key": "waiver_granted",
      "type": "bool",
      "prompt": "Is a waiver granted for this release? (false means it must be fixed, not waived)"
    },
    {
      "key": "waiver_expiry",
      "type": "str",
      "prompt": "If waived, the waiver expiry date (YYYY-MM-DD). Leave empty if no waiver."
    },
    {
      "key": "waiver_owner",
      "type": "str",
      "prompt": "If waived, the accountable owner for the waiver. Leave empty if no waiver."
    }
  ]
}

---
name: release-risk-summarizer
model_hint: any capable instruct model; this only narrates supplied evidence (no tools, no retrieval)
variables: [scenario_run_ids, review_summary, allure_status, blocker_list]
commit_message: "v1 release-risk summarizer (evidence-in, decision-out, no invention)"
---

You are a release-readiness summarizer for a software release control lane. You
DO NOT score releases on your own authority and you DO NOT gather evidence. You
are given evidence that an operator already collected, and you turn it into a
single structured decision. You return JSON ONLY — no prose, no markdown, no code
fences.

You are given:
- Scenario run ids (the evidence the operator collected per upstream scenario):
{{ scenario_run_ids }}
- Review summary (review-queue completion + any blocker-triage answers):
{{ review_summary }}
- Allure status (whether the in-app Allure report was regenerated and loaded):
{{ allure_status }}
- Blocker list (open blockers, each with its owning scenario):
{{ blocker_list }}

Output exactly this shape:
{
  "release_score": number between 0 and 1,
  "blocker_count": integer >= 0,
  "decision": one of ["go", "no_go"],
  "rationale": short string (<= 600 chars) citing the evidence by id/name,
  "missing_evidence": array of strings (empty if nothing is missing)
}

Hard rules — follow these exactly, in this order:
1. Use ONLY the evidence supplied above. NEVER invent, assume, or "fill in"
   passing evidence. If a required field is empty, null, or absent, treat that
   evidence as MISSING — do not guess that it passed.
2. Compute "blocker_count" as the number of entries in {{ blocker_list }} that
   are still open (not waived/closed). Count it; do not estimate it.
3. Force "decision":"no_go" if ANY of the following is true:
   - any required evidence is missing (list every missing item in
     "missing_evidence"),
   - blocker_count > 0,
   - Allure was not regenerated or did not load (allure_status is not a clear
     "loaded/visible" signal).
   When you force no_go, the rationale MUST name the specific failing condition.
4. Only return "decision":"go" when ALL of these hold simultaneously:
   "missing_evidence" is empty, blocker_count == 0, Allure is loaded, and
   release_score >= 0.90.
5. "release_score" must be consistent with the evidence: if any dimension lacks
   its evidence, the score cannot be >= 0.90. Do not output a score that
   contradicts "missing_evidence" or "blocker_count".
6. Never approve a release to be helpful, to resolve ambiguity, or because the
   evidence is "probably fine". When in doubt, return no_go and say why.

Return only the JSON record.

{
  "_comment": "Operator scoring config for the SCN-11 release rubric. There is NO engine that evaluates this — the operator scores each dimension BY HAND from the evidence in dataset/run-id-manifest.json, computes the weighted sum, counts blockers, and applies the gates. Weights mirror verification.yaml.",
  "version": "1",
  "scoring": {
    "_comment": "Per dimension, mark a status and convert to a score factor, then multiply by the weight. overall_release_score = sum(weight * factor).",
    "status_to_factor": { "pass": 1.0, "partial": 0.5, "fail": 0.0 }
  },
  "dimensions": [
    {
      "key": "component_readiness",
      "weight": 0.30,
      "satisfied_by": "Component-level evidence is green and recorded: prompt/tool/skill scorecards from the upstream scenarios pass. Concretely, every SCN-07/08/09/10 eval_run_id is filled and its scorecard passes its own gate.",
      "evidence_source": "dataset/run-id-manifest.json -> scenarios[].eval_run_id (Evaluate -> Evaluations scorecards)",
      "pass_if": "all required eval_run_id present AND each scorecard passes",
      "partial_if": "some scorecards pass, others missing or below their gate",
      "fail_if": "any required eval_run_id missing OR any scorecard fails"
    },
    {
      "key": "workflow_readiness",
      "weight": 0.30,
      "satisfied_by": "The high-risk workflow slices were re-run and still pass: SCN-07 approval branch, SCN-08 rollback path, SCN-09 retry/resume recovery, with fresh run ids. Scoped to SCN-07/08/09 only — SCN-10's critical slice is human-review-queue completion (scored under review_coverage), not a pipeline re-run, so it carries no approval_branch_run_id.",
      "evidence_source": "dataset/run-id-manifest.json -> scenarios[SCN-07/08/09].approval_branch_run_id (Workflow run monitor + Observability traces)",
      "pass_if": "every SCN-07/08/09 critical-slice re-run has a fresh run id AND completed successfully",
      "partial_if": "some SCN-07/08/09 critical slices re-run green, others not yet re-run",
      "fail_if": "any SCN-07/08/09 critical slice missing a fresh run id OR a re-run failed"
    },
    {
      "key": "review_coverage",
      "weight": 0.20,
      "satisfied_by": "Every required Review Queue from the upstream scenarios is fully answered (no unresolved items), and blocker-triage answers are recorded where used.",
      "evidence_source": "dataset/run-id-manifest.json -> scenarios[].review_queue_complete (Observe -> Review Queues); optional review/blocker-questions.json answers",
      "pass_if": "all required review_queue_complete == true",
      "partial_if": "some queues complete, others still have unresolved items",
      "fail_if": "any required review queue not complete"
    },
    {
      "key": "evidence_visibility",
      "weight": 0.20,
      "satisfied_by": "The in-app Allure report was regenerated (make allure-report) and LOADS at Settings -> Allure Report, and the run-id manifest is complete enough to trace every gate state back to a concrete run id / review answer / Allure result.",
      "evidence_source": "dataset/run-id-manifest.json -> allure {report_url, loaded}; GET /observability/allure-report",
      "pass_if": "allure.loaded == true AND allure.report_url set AND every gate state maps to a recorded run id / review answer / Allure result",
      "partial_if": "Allure loads but some evidence links are missing",
      "fail_if": "Allure not regenerated or does not load (allure.loaded != true)"
    }
  ],
  "gates": {
    "_comment": "All three must hold for a go. Mirrors verification.yaml quality_gates.",
    "overall_release_score_min": 0.90,
    "blocker_count_must_equal": 0,
    "allure_visibility_required": true
  },
  "decision_rule": "decision = go ONLY IF overall_release_score >= overall_release_score_min AND blocker_count == 0 AND allure_visible == true; otherwise no_go. Missing required evidence is a fail for its dimension (and typically forces blocker review), so a missing-evidence release cannot reach 0.90.",
  "worked_example": {
    "_comment": "Illustrative only — shows the arithmetic and how one open blocker forces no_go even when the weighted score clears the threshold.",
    "scenario": "Release candidate r2026.06.0",
    "dimension_status": {
      "component_readiness": "pass",
      "workflow_readiness": "pass",
      "review_coverage": "partial",
      "evidence_visibility": "pass"
    },
    "calculation": [
      "component_readiness: 0.30 * 1.0 = 0.30",
      "workflow_readiness:  0.30 * 1.0 = 0.30",
      "review_coverage:     0.20 * 0.5 = 0.10",
      "evidence_visibility: 0.20 * 1.0 = 0.20"
    ],
    "overall_release_score": 0.90,
    "allure_visible": true,
    "blockers": [
      {
        "id": "BLK-1",
        "summary": "SCN-07 approval-branch re-run surfaced one unresolved review item: an over-refund case was approved without the eligibility check.",
        "owning_scenario": "SCN-07",
        "status": "open"
      }
    ],
    "blocker_count": 1,
    "decision": "no_go",
    "rationale": "Weighted score is exactly 0.90 and Allure is visible, but blocker_count == 1 (BLK-1, owned by SCN-07). The blocker_count_must_equal:0 gate fails, so the release is no_go regardless of the score. Resolve BLK-1 in SCN-07 (or log a waiver with an explicit expiry + owner), re-run the SCN-07 approval branch, then re-score."
  }
}

# Release decision preparation template

This is an optional drafting aid. Record the authoritative candidate, waiver,
report-job, and final signoff in **Operate → Releases**.

- Candidate name / artifact / version:
- Required score:
- Planned action:
- Rollback target:
- Evidence references:
- Blocking criteria:
- Waivers with reason / expiry / approver:
- Decision: `go` / `no_go`
- Rationale:
- Report-job id:
- Candidate snapshot SHA-256:

Before `go`, confirm the server reports `ready`, no unwaived blocker remains,
and both the planned action and rollback target are populated.