CALIBER
Quickstart

Cookbook 05 · Foundations

Governed Tool Connectivity (MCP)

Connect an external system, discover its tools, and prove a blocked tool refuses to run.

FoundationsCore20–35 min
What you build. A governed GitHub MCP connection with a working read tool and a write tool that is policy-blocked — invoked from the Playground to show the refusal.
Surfaces: MCP Servers Observability
What you learn
  • Quick-connect an MCP server from the catalog
  • Test connection + tool discovery (schemas)
  • Invoke a read-only tool from the Playground
  • Apply a per-tool policy overlay and prove enforcement
  • Save tool test cases and calibrate them

Implementation flow

flowchart LR
  C[Catalog: GitHub] --> S[Register server]
  S --> TC[Test connection] --> D[Discover tools]
  D --> RT[search_repositories · read] --> OK[Playground invoke ✓]
  D --> WT[issue_write · write]
  WT -->|policy allowed:false| BLK[Playground invoke ✗ refused]

Step-by-step UI only

Each step shows the exact navigation path, the fields to fill, the button to click, and a snapshot of the CALIBER screen you'll be on.

  1. Library › MCP Servers creates · MCP catalog
    Open the MCP Servers page and find the quick-connect catalog of 8 templates. You will use the GitHub tile because it ships both a safe read tool (search_repositories) and a risky write tool (issue_write) — perfect for proving a policy block.
    You'll see: A catalog grid; the GitHub tile is data-testid="catalog-github" alongside PostgreSQL, pgvector, Apache AGE, MinIO, Hugging Face, Ollama, and Playwright.
    Screen snapshot
    caliber · Library › MCP Servers
    MCP ServersMCP Servers
    Read / inspect this surface — no form to fill.
  2. Library › MCP Servers › GitHub tile creates · mcp server: github-readonly
    First add api.githubcopilot.com to the CALIBER server's CALIBER_MCP_REMOTE_HOST_ALLOWLIST. Click the GitHub catalog tile (catalog-github). The Add-server dialog opens pre-filled with GitHub's official remote Streamable HTTP endpoint; set the server name and supply your personal-access token, then submit.
    Fill in
    Server namegithub-readonly
    Transportstreamable-http (pre-filled)
    Endpoint URIhttps://api.githubcopilot.com/mcp/ (pre-filled)
    GITHUB_PERSONAL_ACCESS_TOKENyour GitHub personal-access token (the env var the template expects)
    Click Register Server
    You'll see: A new server row github-readonly appears in the Servers table with a Test action.
    Screen snapshot
    caliber · Library › MCP Servers › GitHub tile
    MCP ServersGitHub tile
    Register Server
  3. Library › MCP Servers › github-readonly › Test creates · connection + discovered tools
    On the server row click Test to verify the transport/auth, then let it discover the tool catalog. Inspect a couple of discovered tools and their input/output schemas so you know the read tool's required arguments.
    Click Test
    You'll see: Connected · N tools. The discovered tool list shows search_repositories (read) and issue_write (write) with their JSON input/output schemas. Gate: connection_success_rate = 1.0.
    Screen snapshot
    caliber · Library › MCP Servers › github-readonly › Test
    MCP ServersTest
    Test
  4. Library › MCP Servers › Playground creates · successful read invocation mcp-tool-test-cases.jsonl ↓
    Open the Playground tab, select server github-readonly, click Test Connection to discover its tools, then pick the read tool search_repositories from the discovered list. Fill its parameter inputs and click Invoke Tool to confirm a real, successful result. This is your one good read-tool invocation record.
    Fill in
    Toolsearch_repositories (select from the discovered tools)
    Arguments (golden)set query = modelcontextprotocol/servers — from the golden row of mcp-tool-test-cases.jsonl
    Click Invoke Tool
    You'll see: success:true with a result payload (mentions items) and a duration_ms figure.
    Screen snapshot
    caliber · Library › MCP Servers › Playground
    MCP ServersPlayground
    Invoke Tool
  5. Library › MCP Servers › Playground › issue_write › Tool Policy creates · policy: block issue_write github-issue-write.block.policy.json ↓
    Now govern the risky write tool. Still in the Playground with github-readonly selected, pick the issue_write tool — its Tool Policy panel appears. Turn the Allow tool toggle OFF (sets allowed:false), then save. On the direct invoke path a block (allowed:false) is the only control that actually refuses execution — so use the block for a true read-only explorer. (requires_approval and side_effect_level are advisory on direct invoke; the block is what enforces.)
    Fill in
    Toolissue_write (select in Playground)
    Allow tooltoggle OFF → allowed:false (matches github-issue-write.block.policy.json; strip any _comment key — it is documentation only)
    Click Save Policy
    You'll see: The issue_write policy now shows allowed:false (a “blocked” badge appears on the tool).
    Screen snapshot
    caliber · Library › MCP Servers › Playground › issue_write › Tool Policy
    MCP ServersTool Policy
    Save Policy
  6. Library › MCP Servers › Playground creates · blocked-tool enforcement record policy-enforcement-checks.jsonl ↓
    Re-invoke the now-blocked tool to capture the enforcement record. With issue_write still selected, fill its parameters with a payload that would create an issue and click Invoke Tool — it must be refused, not executed. This proves blocked_tool_enforcement = 1.0.
    Fill in
    Toolissue_write (selected in Playground)
    Argumentsset method=create, owner=rrahimi-uci, repo=caliber-suite, title=blocked test, body=should never be created — the PE01 row of policy-enforcement-checks.jsonl
    Click Invoke Tool
    You'll see: A structured refusal: success:false, result:null, and an error stating the tool is blocked by policy — the tool never ran.
    Screen snapshot
    caliber · Library › MCP Servers › Playground
    MCP ServersPlayground
    Invoke Tool
  7. Library › MCP Servers › Playground › search_repositories › Calibration creates · mcp tool test cases mcp-tool-test-cases.jsonl ↓
    Harden the read tool with a small saved suite. In the Playground with search_repositories selected, scroll to its Calibration panel. Use Add case to add a row per case; each row has a name, an Input (JSON) object (the tool arguments), and an Assertion of type no_error, output_contains, or equals (the latter two also take a value). Click Save cases. Important about the assertion model: there is no "expects-error" assertion — when an invoke errors, the case fails before any assertion runs (a gateway error short-circuits to a fail). So the two negative rows (N01 missing the required arg, N02 a malformed qualifier) are deliberately failing cases: they exist to show up as red lines in the pass-rate, not as green output_contains:"error" matches. The four golden/edge rows (G01/G02/E01/E02) are the ones expected to pass.
    Fill in
    Cases (Add case per row)Add the 6 cases from mcp-tool-test-cases.jsonl — four expected-pass (G01/G02 golden, E01/E02 edge) plus two intentionally-failing negatives (N01/N02). Send name + input + assertion type/value only
    Click Save cases
    You'll see: Six saved test cases listed under search_repositories's Calibration panel, each showing its assertion type.
    Screen snapshot
    caliber · Library › MCP Servers › Playground › search_repositories › Calibration
    MCP ServersCalibration
    Save cases
  8. Library › MCP Servers › Playground › search_repositories › Calibration creates · calibration run (read tool)
    Run the calibration. Click Run calibration in the same panel — each saved case is invoked and scored against its assertion inline (deterministic — no LLM judge), and a pass rate comes back immediately. Expect the pass-rate to land below 100% by design: the four golden/edge cases pass, and the two negative cases (N01/N02) fail because their invokes error out before any assertion can match. That red is the intended signal — it is the suite documenting how the tool misbehaves on bad input, not a defect in your setup.
    Click Run calibration
    You'll see: An inline pass-rate report ({pass_rate, total, passed}) — four green cases and two red negatives, so passed:4 of total:6 (a pass_rate of 0.6667, intentionally under 100%) with a per-case verdict on each row. This satisfies at_least_one_read_tool_calibrated = true.
    Screen snapshot
    caliber · Library › MCP Servers › Playground › search_repositories › Calibration
    MCP ServersCalibration
    Run calibration
  9. Library › MCP Servers › Playground › History creates · playground history (invoke + block)
    Confirm the governance story is recorded on the surface that actually captures it: the Playground History panel below the invoke form. MCP invokes do not emit MLflow spans, so they will not appear in Observe › Observability — read the evidence here instead. Each invocation you ran this session is listed newest-first; find the search_repositories read (success:true, green dot) and the issue_write write (success:false, red dot — refused by the policy block). Click any row to reload its arguments and result.
    You'll see: A History list with one row per invoke: a green dot + search_repositories + a duration_ms for the successful read, and a red dot + issue_write + 0ms for the blocked write. The issue_write tool still wears its red Blocked badge, and the search_repositories Calibration panel still shows its saved pass/fail verdicts — together these are your end-to-end enforcement evidence.
    Screen snapshot
    caliber · Library › MCP Servers › Playground › History
    MCP ServersHistory
    Read / inspect this surface — no form to fill.

Assets (copy-paste)

The exact files this cookbook uses — copy each into the matching field. Source: docs-site/cookbooks/05-mcp-connectivity-governance-lab/assets/.

assets/dataset/mcp-tool-test-cases.jsonl 
{"id": "G01", "tags": ["golden"], "tool_name": "search_repositories", "name": "valid-query-returns-no-error", "input": {"query": "modelcontextprotocol/servers"}, "assertion": {"type": "no_error"}}
{"id": "G02", "tags": ["golden"], "tool_name": "search_repositories", "name": "valid-query-output-mentions-items", "input": {"query": "language:python topic:mcp"}, "assertion": {"type": "output_contains", "value": "items"}}
{"id": "E01", "tags": ["edge"], "tool_name": "search_repositories", "name": "empty-query-still-no-error", "input": {"query": ""}, "assertion": {"type": "no_error"}}
{"id": "E02", "tags": ["edge"], "tool_name": "search_repositories", "name": "narrow-zero-result-query-no-error", "input": {"query": "repo:rrahimi-uci/caliber-suite-does-not-exist-xyz"}, "assertion": {"type": "no_error"}}
{"id": "N01", "tags": ["negative"], "tool_name": "search_repositories", "name": "missing-required-query-arg-output-reports-error", "input": {}, "assertion": {"type": "output_contains", "value": "error"}}
{"id": "N02", "tags": ["negative"], "tool_name": "search_repositories", "name": "malformed-search-qualifier-output-reports-error", "input": {"query": "stars:>>not-a-number"}, "assertion": {"type": "output_contains", "value": "error"}}
assets/policy/github-issue-write.block.policy.json 
{
  "_comment": "PATCH /mcp-servers/{id}/tools/issue_write/policy. issue_write is the GitHub server's WRITE tool; in a read-only explorer it stays OFF. allowed:false blocks invocation outright — a re-invoke returns success:false with error 'Tool issue_write is blocked by policy' and never reaches the gateway. Strip this _comment before sending.",
  "allowed": false
}
assets/dataset/policy-enforcement-checks.jsonl 
{"id": "PE01", "desc": "blocked write tool refuses to invoke", "tool_name": "issue_write", "policy": {"allowed": false}, "invoke_input": {"method": "create", "owner": "rrahimi-uci", "repo": "caliber-suite", "title": "blocked test", "body": "should never be created"}, "expect": {"invoked": false, "error_kind": "policy_blocked", "error_contains": "blocked by policy", "duration_ms": 0}}
{"id": "PE02", "desc": "approval flag alone is recorded but NOT enforced on the direct invoke path (it gates workflow binding via the human-approval node, not the Playground); to actually stop a direct invoke you must set allowed:false. This row documents that gap so the read-only explorer relies on the hard block, not approval.", "tool_name": "issue_write", "policy": {"requires_approval": true, "side_effect_level": "write"}, "invoke_input": {"method": "create", "owner": "rrahimi-uci", "repo": "caliber-suite", "title": "needs approval", "body": "approval gate is a workflow-time control"}, "expect": {"invoked": true, "note": "direct invoke still executes — only allowed:false refuses on this path"}}
{"id": "PE03", "desc": "allowed read tool invokes normally", "tool_name": "search_repositories", "policy": {"allowed": true, "side_effect_level": "read"}, "invoke_input": {"query": "modelcontextprotocol/servers"}, "expect": {"invoked": true}}

Evaluation & quality gates

Quality gateTarget
Connection + discovery100% connect; schemas visible
Blocked tool enforcementblocked tool cannot execute
Read tool calibrated≥ 1 saved calibration run
Developer notes & gotchas
  • On the direct invoke path only allowed:false blocks; requires_approval gates at workflow time (a human_approval node), not on direct invoke.
  • Assertion types are no_error, output_contains, equals.
  • Keep write-capable tools blocked by default until the read path is stable.

CALIBER : Contextual Adaptive Lifecycle for Intelligent Build, Evaluation, and Refinement — Cookbooks — every recipe is UI-implementable on the shipped platform. Source + assets under docs-site/cookbooks/<nn>-…/. Regenerate with python3 docs-site/cookbooks/training/build.py.