CALIBER : Contextual Adaptive Lifecycle for Intelligent Build, Evaluation, and Refinement

CALIBER

Contextual Adaptive Lifecycle for Intelligent Build, Evaluation, and Refinement

An MLflow-integrated control plane for trusted agentic workflows

Embedded or standalone · Apache 2.0

Design · Verify · Calibrate · Evaluate · Publish · Observe

The problem

Agentic workflows are easy to build and hard to trust.

Prompts, tools, and multi-step agents ship faster than anyone can govern them. They drift, they're tuned by feel, and the lineage — what changed, how it scored, who approved it — lives in chat threads and notebooks instead of one system of record.

?

No eval gate

Changes ship without a measured score against a held-out set

?

No lineage

What changed, why, how it scored, who approved — unrecorded

?

No rollback

When a workflow regresses, there's no clean way back

Every shipped artifact should answer: how did it score, who approved it, and how do I revert it?

The market

The trust gap is real — and it's the size of the market.

~$50B

Market by 2030

AI-agents market, growing ~46%/yr — two firms agree (MarketsandMarkets & Grand View, 2025)

33%

Of enterprise software

will include agentic AI by 2028, up from <1% in 2024 (Gartner, 2025)

40%+

Projects canceled

of agentic-AI projects by end of 2027 — cost, unclear value, weak controls (Gartner, 2025)

21%

Have mature governance

for agentic AI — the other ~79% do not (Deloitte, 2026)

28%

Can trace to a human

of orgs across all environments; ~72% cannot (Cloud Security Alliance, 2025)

A large, fast-growing market — held back by a governance and trust gap.

Figures as published on the dates cited; each horizon is its source's own.

Why projects stall

A patchwork stack measures — almost none close the loop.

Teams stitch the lifecycle together from point tools — and the deepest gap is what almost none of them do.

Point tools

Orchestrate in one tool, trace in another, evaluate in a third, manage prompts in a fourth
Most cover only 1–2 lifecycle stages
Lineage scatters across chat threads and notebooks
Leaders are SaaS-first, self-hosting paywalled; open options are observability-led, not full-lifecycle (CALIBER competitive analysis, mid-2026)

The cost

76% of enterprise AI use cases are bought, not built — more vendors to integrate (Menlo Ventures, 2025)
They measure — surface scores and dashboards, then stop
Optimizers are widespread; a gated optimize → re-evaluate → authorize → promote path packaged as one product is not

Most tools measure quality. Almost none close the loop on it.

Introducing

One MLflow-integrated control plane for agentic workflows.

CALIBER is one ASGI control-plane codebase: mount it inside MLflow as one process, or run it beside vanilla MLflow over HTTP. The API and the interface are identical in both — the choice is a failure-domain and operations decision, not a feature decision — while CALIBER metadata and MLflow evidence retain explicit owners.

Compose

Prompts, tools, skills, MCP servers
Workflows built in a visual Studio
Knowledge bases with hybrid retrieval

Measure

Evaluation scorecards on test sets
Per-asset calibration algorithms
MLflow tracing & per-tool-call spans

Govern

Asset-specific validation, testing, apply, and publish
Runtime approvals: role, quorum, checkpoints
Server-validated identity, RBAC, audit

The platform

Build composes. The Library supplies.

Agent resources are first-class registered assets. Each workspace exposes the controls that family actually implements — immutable versions and aliases where they exist, tests and evidence where they are recorded — so you can verify an asset without pretending every family has one uniform lifecycle.

Prompts

Versioned against MLflow's Prompt Registry, with aliases and a per-prompt test workspace.

Tools

Definitions run in a bounded subprocess: separate interpreter, empty environment, private working directory, and hard time, memory, and output limits.

Skills

Named capabilities with selection logic; preview which skills resolve for an input.

MCP Servers

Registered Model Context Protocol servers — including a first-party database server — exposed as tools under command and host allowlists.

Knowledge Bases

Hybrid retrieval (BM25 + dense RRF, tri-hybrid with graph) plus retrieval calibration.

Workflows

Agentic graphs composed in Studio, preview-run, published, and traced end to end.

Build

Compose agentic workflows in Studio

The Studio editor wires prompts, tools, skills, and knowledge bases into a graph. Preview-run a draft version without publishing, then enqueue real runs governed by runtime approvals and checkpointing.

step 1

Compose

Wire nodes in the graph editor

›

step 2

Preview-run

Execute without publishing

›

step 3

Enqueue

One queue — Studio, API, service, cron

›

step 4

Trace

Per-tool-call spans in MLflow

Runs land in one queue with atomic claims for multi-replica workers; every run carries an MLflow trace, and checkpointing makes long runs resumable. Webhook and API nodes are checked against the resolved address, with internal ranges — metadata, loopback, RFC1918 — blocked by default, and they claim an effect-ledger row so an outbound call is made at most once across a restart.

Data & knowledge

The data plane behind every workflow

Knowledge bases provide hybrid retrieval with their own calibration metrics; the Object Store is CALIBER's own file UI over configured local or S3-compatible storage; test sets are the versioned datasets every scored run draws from.

Knowledge Base

BM25 + dense RRF hybrid search
Tri-hybrid with a knowledge graph
Recall@k · nDCG@k · faithfulness

Object Store

Local or S3-compatible storage
The file UI for the whole platform
CALIBER's own file store — separate from MLflow's artifact root

Test Sets

Versioned {input, expected} examples
Grown from real run traces
The substrate for every scorecard

Measure

Evaluate, then calibrate — with receipts

Evaluation runs a test set through scorers and compares runs. Prompt refinement can search for a better candidate and apply per-dimension regression checks before candidate-ready; tools use deterministic, revision-fenced fixture calibration. Moving a prompt candidate live still requires explicit operator apply.

stage 1

Dataset

Test set of examples

›

stage 2

Score

Scorers per example

›

stage 3

Calibrate

MetaPrompt · GEPA · DSPy

›

stage 4

Check

Per-dimension regression evidence

›

stage 5

Apply

Operator promotes the candidate

Prompt candidates that pass the job's regression checks land at candidate_ready and move live only through an explicit operator apply — never auto-promoted. Registry gate verdicts outside the job are advisory.

Observe & operate

Every run is a trace you can open

Observability is built on MLflow tracing: each workflow run records per-tool-call spans, and Evaluations turns those traces into scorecards and comparisons. Operating it is the other half — dependency readiness, queue and worker health, and operator-declared objectives that open a durable incident when they breach.

Tracing

Root run trace per workflow run
Per-tool-call spans with timing
Integrated with MLflow's trace backend

Evaluations

Run a dataset through scorers
Per-example results & run compare
Add a trace to a dataset in one click
Real vs simulated providers — no fabricated scores

Runtime health

Readiness probes dependencies: ready · not ready · skipped
Queue depth, oldest wait, worker heartbeats
Declared objectives open durable incidents

Aria

An embedded copilot that runs, observes, and fixes

On OpenAI and Claude, Aria runs a real tool-calling loop inside one turn: it reads live CALIBER state, executes capabilities, observes the result — including a workflow run's trace and scored evaluations — and iterates, bounded to eight tool steps.

Modes

chat talks · plan outlines
build materializes drafts
Attachments, queue, and steering

Permissioned tools

Read in every mode
manual is the default — the operator runs each gate
Sandboxed/reversible in auto_safe
Runs & publish only in auto_all

Governed

Same RBAC and permission checks
Tool calls recorded per turn
Autonomy bounded & auditable

Aria · run, observe, fix

Build and test an artifact in one turn

In build mode with auto_safe approvals, Aria drafts the artifact, validates it, and runs it in the tool sandbox — a separate interpreter with an empty environment, a private working directory, and hard time, memory, and output limits — then reports exactly what it observed.

# Aria panel · mode = build · approval = auto_safe

you › Build a tool that returns the weekday for an ISO date, then test it.

✓ list_tools — checked for a name clash

✓ validate_draft — schema + signature OK

✓ run_tool_sandbox — "2026-06-20" → "Saturday"

aria › Drafted iso_weekday, validated, sandbox-tested. It's at the tested gate — say the word and I'll approve & publish.

Govern

Asset-specific controls, one authorization boundary

Governance follows each asset's implemented lifecycle. Identity is server-validated, and Aria reuses the same account scopes, permission checks, sandbox decisions, and audit path as human-driven routes rather than a separate authorization path.

control 1

Authorize

Accounts, revocable sessions, four scopes

›

control 2

Contain

Bounded tool subprocess · internal-range egress blocks · MCP allowlists

›

control 3

Verify

Asset-specific validation & tests

›

control 4

Act

Explicit apply or publish

›

control 5

Record

Audit actor, action & entity

Validation, tests, explicit apply or publish actions, and alias rollback exist where the asset implements them. Runtime approvals honour the node's own required role and a quorum of distinct approvers, and by default whoever triggered the run cannot approve it. Credentials live in an encrypted store with versions, rotation, and revocation; an asset holds a secret:// reference and no route reads a value back.

Why different

Unified, and evidence-to-action.

CALIBER collapses the patchwork into one interface, deployed inside MLflow or beside it. CALIBER metadata and MLflow evidence keep explicit owners, while an integrated prompt-refinement path connects measured proposals to explicit human apply.

step 1

Evaluate

Held-out test set through scorers — real scores, no fabricated numbers

›

step 2

Search

Provider paths (Meta-Prompt, GEPA, DSPy) search for a better prompt candidate

›

step 3

Check

Record per-dimension regression evidence

›

step 4

Candidate-ready

Clears the gate but stays offline — never auto-promoted

›

step 5

Apply

Goes live only via an explicit operator step

Embedded or standalone. Evidence connected to action. Self-hostable — your data stays home.

Why different

Open, and governed.

Asset-specific controls govern each change, while humans and the Aria copilot share the same authorization boundary.

Same authorization boundary

Humans and the Aria copilot resolve the same server-validated identity — password accounts, revocable sessions, and four scopes. Each asset follows the lifecycle it actually implements, and the audit trail records actor, action, and entity.

Traceable to a human

Only 28% of organizations can trace an agent's actions back to a human across all environments — ~72% cannot (Cloud Security Alliance, 2025). CALIBER ties every change to an actor in one audit trail.

Human in the loop

61% of executives required a human in the loop when surveyed (KPMG, Q3 2025). Runtime approvals honour the node's required role and a quorum of distinct approvers, and by default whoever triggered the run cannot approve it.

Open, and audit-ready

EU AI Act high-risk obligations phase in from 2 August 2026; penalties reach €35M or 7% of global annual turnover (Article 99). Apache 2.0 and self-hostable — no lock-in, data and lineage stay in your own environment.

Governance binds humans and the copilot equally. Open source, no lock-in.

The vision

Agentic workflows you can measure, refine, govern, and trust.

CALIBER : Contextual Adaptive Lifecycle for Intelligent Build, Evaluation, and Refinement is open source and MLflow-integrated, using real evidence to guide each asset's supported lifecycle.

Integrated

Mounts at /caliber inside MLflow, or runs as a standalone ASGI service connected to vanilla MLflow over HTTP.

Agentic

Aria runs a permissioned tool loop on OpenAI and Claude — read state, execute, observe, iterate — inside the product.

Measured & refined

Evaluation scorecards, prompt refinement with regression checks, and deterministic tool calibration connect proposals to recorded evidence.

Governed

Server-validated identity and shared RBAC scopes, a bounded and swappable tool execution boundary, SSRF-checked workflow HTTP, and explicit operator apply or publish.

Observable

MLflow tracing with per-tool-call spans, dependency readiness, queue and worker health, and operator-declared objectives that open durable incidents.

Open

Apache 2.0 · Python 3.10–3.12 · SQLAlchemy 2.0 · React SPA · local/S3 storage · No vendor lock-in.

caliber-suite · Apache 2.0 · open source