Tools Architecture

The tools module is CALIBER's registry and validation surface for callable runtime tools. Workflow designers reference registered tools by family and version constraint rather than embedding source code directly in workflow manifests, which keeps tool identity stable and independent of any single workflow.

The module carries the following responsibilities. It registers, inspects, updates, archives, and baselines versioned tools. It exposes source reflection and test-run history for registered tools. It runs sandboxed single-invoke and suite-based tool tests. It calibrates tools against saved fixtures and aggregates the result. And it provides registry metadata to workflow compilation and runtime binding.

These responsibilities are realized across a small set of primary code paths:

• caliber/src/caliber/routes/tools.py
• caliber/src/caliber/workflows/tools.py
• caliber/src/caliber/tool_sandbox/service.py
• caliber/src/caliber/workflows/sandbox.py
• caliber/caliber-ui/src/pages/ToolRegistry.tsx
• caliber/caliber-ui/src/pages/ToolWizard.tsx
• caliber/caliber-ui/src/pages/ToolDetail.tsx

Tools are the product's principal code-execution surface, so the module draws a firm line between describing a tool and running one. The table below assigns each responsibility to its owner.

The boundaries reduce to a single design split. On one side are registry concerns: identity, versioning, lifecycle, and test history. On the other are execution concerns: import and bind, preview safety, and isolated testing. Holding these apart is what lets the compiler reason about tools cheaply without ever importing their code.

At runtime the request path and the workflow path both lead to the same registry, but they touch it differently: the API drives testing and lifecycle, while the workflow compiler and runtime resolve and bind tools. The diagram shows both paths.

flowchart LR
    UI[Tool Registry UI]:::ui
    API[routes/tools.py]:::ctrl
    DB[(CaliberToolRegistry<br/>CaliberToolTestRun)]:::store
    RES[workflows/tools.py<br/>resolver and binder]:::ctrl
    SB[tool_sandbox/service.py]:::ctrl
    WF[Workflow compiler and runtime]:::ctrl
    DEP[Workflow deployments]:::ctrl

    UI --> API
    API --> DB
    API --> SB
    API --> RES
    API --> DEP
    WF --> RES
    WF --> DB

Several structural properties follow from this layout. Tools are versioned registry entries rather than ad hoc code blobs embedded in workflow manifests. Tool resolution is metadata-driven first and import-driven second, so most reasoning happens without loading code. Archive protection checks live deployments before allowing lifecycle changes, preventing in-use tools from disappearing. And testing and calibration happen against the registered tool artifact, not against workflow graphs, which keeps tool quality measurable in isolation.

Tool state lives in two durable tables, with a third pair of tables consulted to protect tools that are still in use. The table summarizes their roles.

The CaliberToolRegistry row carries the fields that define a tool and govern how it may be run. The name and version fields establish stable family and version identity. The module_path and callable_name fields locate the Python import target. The input_schema and output_schema fields fix the typed invocation contract. The side_effect_level, requires_approval, and allow_in_preview fields capture runtime safety. The secret_refs field provides secret indirection rather than literal secrets. The test_cases field holds the saved fixture suite. The last_calibration field records the latest aggregate scored result. calibration_revision advances with a database expression and clears that result on every supported definition/fixture mutation; workers conditionally attach new evidence only at the submitted revision. And baseline_run_id pins the comparison baseline used for workspace diffs.

As with the other asset modules, lifecycle state is derived from both registry status and historical evidence rather than stored as one column, and it advances through the stages Draft, Has fixtures, Tested, Hardened, and Published.

A tool family is the set of registry rows sharing a name, each a distinct version (the pair is unique). Tools have no live alias to promote or roll back, so the family's history is a read-only inventory: GET /tools/{tool_id}/versions returns every version in the family, scoped by visibility so it never leaks another project's versions, and ordered newest-first with a version-aware sort (so 9 precedes 10 and 1.9 precedes 1.10, not the lexical order a raw string column would give).

All HTTP routes in this module are mounted under /ajax-api/2.0/mlflow/caliber and are shown relative to that prefix below. The surface in routes/tools.py groups into three areas.

The first area covers core registry management, including the archive-protected lifecycle and usage analysis:

• GET /tools
• POST /tools
• GET /tools/{tool_id}
• PATCH /tools/{tool_id}
• POST /tools/{tool_id}/archive
• GET /tools/{tool_id}/source
• GET /tools/{tool_id}/usage
• GET /tools/{tool_id}/versions

Calibration also has a durable form, because scoring up to 200 cases through the sandbox is minutes of work and a synchronous request loses the result to a proxy timeout or a closed laptop lid:

• POST /tools/{tool_id}/calibration-jobs — returns 202 with a job id
• GET /tools/{tool_id}/calibration-jobs — recent jobs, newest first
• GET /tools/{tool_id}/calibration-jobs/{job_id} — poll one job
• POST /tools/{tool_id}/calibration-jobs/{job_id}/resolve — abandon an ambiguous claim or retry it as a new lineage-linked job; a reason is required

A background drain claims a job with a conditional UPDATE, runs it off the event loop without holding a database session, and records the outcome. It attaches a result to the tool only if the executable definition, cases, and revision still match the submission; otherwise the job keeps a stale-result diagnosis without presenting it as current evidence. A crashed drain deliberately leaves the job claimed rather than re-queueing it: calibration invokes tools, so silently re-running one after an ambiguous failure is the wrong default. Bounded shutdown fences the active generation immediately and waits at most the configured grace for the tracked drain; it performs no stop-time database settlement. An interrupted claim therefore remains visibly running/ambiguous, while the retained generation fence prevents a late scorer from persisting a terminal result or failure. The Tool detail page shows this durable history. An operator can explicitly abandon the original row or create a new queued retry with the exact snapshotted definition and cases. The original becomes terminal and records the actor, reason, timestamp, resolution, and retry job ID; status-conditional settlement discards a late result instead of overwriting that decision. The synchronous POST /tools/{tool_id}/calibrate remains for small fixture sets; the Tool workspace also exposes the durable queue for longer runs.

The second area covers testing and calibration against the sandbox and fixtures:

• POST /tools/{tool_id}/test-run
• PUT /tools/{tool_id}/test-cases
• POST /tools/{tool_id}/calibrate
• POST /tools/test-runs
• GET /tools/test-runs
• GET /tools/test-runs/{test_run_id}

The third area exposes workspace metadata:

• GET /tools/{tool_id}/workspace
• POST /tools/{tool_id}/baseline

The frontend reaches these endpoints through three entry points:

• caliber/caliber-ui/src/pages/ToolRegistry.tsx
• caliber/caliber-ui/src/pages/ToolWizard.tsx
• caliber/caliber-ui/src/pages/ToolDetail.tsx

The endpoints above support three recurring flows: registering a tool, testing it in the sandbox, and calibrating it against fixtures. A fourth path, in which the workflow compiler and runtime consume the registry without an API call, runs alongside them. The sequence diagram traces all four.

sequenceDiagram
    participant U as Operator
    participant UI as Tool UI
    participant API as routes/tools.py
    participant DB as CALIBER DB
    participant SB as Tool sandbox
    participant D as Calibration drain
    participant WF as Workflow compiler/runtime

    U->>UI: Register tool metadata
    UI->>API: POST /tools
    API->>DB: Insert CaliberToolRegistry row
    API-->>UI: Return tool record

    U->>UI: Run sandbox test or suite
    UI->>API: POST /tools/{id}/test-run
    API->>SB: Execute isolated subprocess run
    SB-->>API: Return output, errors, verdict inputs
    API->>DB: Persist CaliberToolTestRun
    API-->>UI: Return durable run detail

    U->>UI: Save fixtures and calibrate inline
    UI->>API: PUT /test-cases, POST /calibrate
    API->>DB: Snapshot definition, cases, revision
    API->>SB: Execute fixture suite off event loop
    API->>DB: Conditionally attach at submitted revision

    opt Durable API client
        U->>API: POST /calibration-jobs
        API->>DB: Persist immutable snapshot, return 202
        D->>DB: Conditionally claim queued job
        D->>SB: Execute fixture suite
        D->>DB: Settle job + conditionally attach result
    end

    WF->>API: No direct call
    WF->>DB: Read tool metadata through resolver inputs
    WF->>WF: Resolve best version and bind callable

The runtime rules that govern these flows reinforce the registry/execution split. The module in routes/tools.py reflects source code for inspection and, with the default sandbox enabled, never executes it in the request process. The LocalSubprocessToolSandbox runs the code under python -I inside a temporary directory, with an empty environment, clipped output, and a hard timeout. Workflow compilation resolves tool references without importing the callable. And the workflow runtime binds and imports only the resolved tool version that has already passed resolution and safety checks.

On POSIX, timeout cleanup targets the sandbox process group so descendants cannot normally outlive the request. A constrained host that denies group signalling falls back to killing the direct child instead of converting an already-detected timeout into a 500; such a host cannot claim descendant containment from this fallback and must supply an external container/VM boundary. Windows likewise requires Job Objects or external containment for process-tree guarantees.

Controls in this module are explicit because tools are the main code-execution surface in the product, and they are layered across scopes, secrets, preview policy, archival, and isolation. The enforced controls are the following. Registration and archive actions require admin scope. Workspace baseline changes and calibration require operator scope. Secret material is never stored on the row, only secret_refs. Preview safety is encoded on the tool row and enforced in the runtime preview paths. Archive is blocked while any active workflow deployment still references the tool family, regardless of alias and not limited to prod. And testing executes within a subprocess boundary, never inside the request handler.

These controls converge on one trust boundary. The registry accepts operator-supplied Python source metadata, but the runtime executes that code only through the sandbox or controlled workflow binding, and source reflection is best-effort and read-only. Authoring a tool, in other words, is not the same as being allowed to run arbitrary code in the request process.

7.1 The execution boundary, and what it is not#

A registered tool's module is imported and invoked in a child process by default, not in the control plane. This matters more than it sounds: the import is execution too, so module-level code used to run in the API server on first bind. Source inspection, test-run, and calibration all go through the same child, so reflecting on a tool for its signature no longer executes it here either.

The allowlist is enforced on both binder paths. Routing execution into a subprocess narrows where a module runs; it does not change which modules an operator sanctioned, so containment and authorization do not trade off against each other.

What this boundary is not. The shipped sandbox is a process boundary: a separate interpreter, an empty environment, a private working directory, and POSIX resource limits. It is not a container, VM, or seccomp boundary, and the child retains ambient filesystem and network access on the host. That is appropriate for trusted tool authors in one organization and is not isolation for untrusted ones. Portable Python cannot provide OS-enforced isolation — namespaces are Linux-only and privileged, seccomp needs a native binding, containers are infrastructure — so a deployment that needs it supplies its own factory via CALIBER_TOOL_SANDBOX_BACKEND rather than forking.

Cost. Running out of process is not free: measured on an idle machine, a cold child costs roughly 0.05s for a trivial module and 0.55s for one importing the caliber package, per call. The child signals ready and waits; the parent then starts authored work and owns the exact runtime deadline, with process-group termination even if authored code mutates os._exit or monopolizes the GIL. The child's uncatchable watchdog still covers source/module resolution, import, inspection, every test/invocation, result representation, and serialization. A positive startup grace is separate from that runtime clock; with zero grace, startup consumes the caller's overall budget. The standalone ASGI service offloads synchronous execution so another request can still reach its health route. A warm worker pool would recover most of the process cost and is not implemented.

Generated exports take the same decision. A compiled standalone export binds through the same path after entering the explicit run configuration context; absent explicit configuration it resolves the environment normally. The selected sandbox backend and an intentionally empty module allowlist are therefore preserved, so a workflow does not silently change binding policy by being exported.

The module surfaces enough signal to evaluate, compare, and debug tools over time. The operational signals it exposes are the following. Durable CaliberToolTestRun history is recorded for every major testing surface. Pinned baseline support enables run comparisons in the workspace. A lifecycle summary is computed from fixtures, test history, calibration, and registry status. Source-code introspection and signature reflection are available for debugging. Usage inspection runs against workflow versions and deployments. Durable calibration jobs expose claim identity, timestamps, result/error, resolution metadata, and retry lineage; the registry revision records whether current evidence still belongs to the current definition.

To keep history listings fast, the module separates heavy payloads from cheap summaries:

• full per-case result arrays stay in results;
• counters and aggregate scores are materialized for fast history listing.

The module has clear seams for growth alongside constraints that reflect its current implementation.

The primary extension points are the following. New test-run kinds or richer calibration assertions can be added. Preview policies and approval semantics can be strengthened. Additional tool resolvers or registry backends can be introduced if the database registry is ever abstracted. And sandbox isolation can be hardened with containers or virtual machines.

The current constraints are the following. The default sandbox is still local subprocess isolation, which is suitable for development and CI but is not a hardened multi-tenant isolation boundary. Workflow manifests reference tool families and version constraints, so usage analysis is dependency-aware rather than row-id direct. Tool registration relies on importable Python modules being available in the CALIBER runtime environment. And no workflow designer ever authors tool code inline in manifests, by design.

The tools module is therefore both a catalog and an execution boundary. Its value is not merely storing tool metadata, but making tool use testable, auditable, preview-safe, and version-resolved for the workflow runtime.