Bring up MLflow, run CALIBER, tour the platform.

Prerequisites

You need a working Unix-like shell (macOS, Linux, or WSL) with the following on $PATH:

• Python 3.10–3.12 — 3.11 is recommended and is the canonical CI interpreter.
• Node ≥ 20 and npm — to build and serve the React SPA.
• Docker (optional) — needed for the suite's MinIO stack; local filesystem storage is also supported.
• git — to clone the repo.

1Clone & install

Clone the suite and install the plugin in editable mode with the dev and S3 extras, then install the SPA's npm dependencies:

git clone https://github.com/rrahimi-uci/caliber-suite.git
cd caliber-suite/caliber

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,s3]"

cd caliber-ui
npm install

What that gives you:

• A .venv/ with CALIBER, MLflow (≥ 3.14), Alembic, the Starlette runtime, and the dev toolchain (pytest, ruff, mypy).
• The s3 extra, which pulls in the object-storage client CALIBER uses for artifacts.
• The SPA's build dependencies (Vite, React, TanStack Query) under caliber-ui/.

2Storage & provider keys

make dev defaults MLflow artifacts to s3://mlflow/mlruns via MinIO at http://127.0.0.1:9000, so bring that up first (the suite ships a Compose stack under deploy/) or point the variables at your own endpoint. CALIBER workflow/file storage supports both local filesystem and S3-compatible backends; this walkthrough uses MinIO so the object-storage surfaces are available.

Set the provider keys CALIBER should use before launching (a suite-root .env is the usual home for these):

# Real LLM provider for Aria + LLM-backed eval/calibration
export OPENAI_API_KEY=sk-...        # or ANTHROPIC_API_KEY=...

# Object storage (defaults shown; override for your own endpoint)
export MLFLOW_S3_ENDPOINT_URL=http://127.0.0.1:9000
export MLFLOW_ARTIFACT_ROOT=s3://mlflow/mlruns
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin

3Start MLflow with the CALIBER plugin

In this embedded walkthrough CALIBER doesn't run as a separate service. It's loaded as an MLflow application: when the server launches with --app-name caliber, MLflow mounts CALIBER's SPA at /caliber/ and its REST endpoints at /ajax-api/2.0/mlflow/caliber/* on the same port, alongside MLflow's native UI at /.

From caliber/ with the venv active, in terminal 1:

make dev

That wraps scripts/run-dev.sh, which:

1. Sources .env if present (without clobbering anything already exported).
2. Applies local-dev defaults: a metadata database (SQLite at ./caliber.db unless you point CALIBER_DATABASE_URL at Postgres), the MinIO artifact root, and a dev admin identity.
3. Runs alembic upgrade head to bring the schema current.
4. Enables the workflow run queue, runtime approvals, and checkpointing.
5. Execs mlflow server --app-name caliber in the foreground.

The startup banner echoes the resolved configuration and the exact URLs to open:

4Verify the server is healthy

In a second terminal (leave make dev running in the first), hit the CALIBER health endpoint:

curl -s http://127.0.0.1:5000/ajax-api/2.0/mlflow/caliber/health | jq

The readiness endpoint reports which providers are real versus simulated — this is the same honesty signal the SPA shows in its banner. With a provider key exported, llm should read openai or anthropic and the simulated list should be empty (or list only the subsystems you left on a stub):

curl -s http://127.0.0.1:5000/ajax-api/2.0/mlflow/caliber/readiness | jq .data

You can also confirm MLflow itself answered (its native UI is at /):

curl -sI http://127.0.0.1:5000/ | head -1

5Open the SPA

In your browser, open the caliber UI URL from the startup banner:

open http://127.0.0.1:5000/caliber/      # macOS
xdg-open http://127.0.0.1:5000/caliber/  # Linux

On first load, sign in with admin / admin and immediately replace that local-only first-boot password in Administration. The loopback launcher is the component that explicitly enables this known credential; the product default leaves it disabled. The database starts empty — there is no demo seed in this build, so the lists are blank until you create artifacts, either by hand on each page or by asking Aria to build them for you. The rest of this runbook walks the SPA in sidebar order and then has Aria build a governed artifact end to end.

Dashboard · the platform at a glance

The landing page summarizes platform state — recent workflow runs, registered artifacts, and quick links into each surface. It is the jumping-off point; every count is a link into the page that owns those rows. On a fresh database it is mostly empty, which is the honest starting state for a new install.

Try this

• Open Aria from the top bar and say hello. It should greet you and explain what it can do — it does not start building anything until you express intent.
• Note the provider banner. It reflects the same /readiness payload you curled above.

Build · Workflows (Studio)

Workflows are the composition surface — the agentic graphs that wire prompts, tools, skills, and knowledge bases together. The Studio editor lets you build a workflow version, preview-run it, and publish it; the run views surface each run's MLflow trace, per-tool-call spans, runtime approvals, and checkpoints.

Try this

• Create a workflow, add a node or two in the editor, and use Preview run — it executes without persisting a published version.
• Enqueue a real run. With the run queue enabled, it lands in the queue and streams status; open the run to inspect its trace.

Libraries · Prompts, Tools, Skills, MCP Servers

The Library supplies what Build composes. Each library resource is its own versioned asset with a "pytest-for-an-asset" workspace — a status header, stage tabs, and durable test runs with baseline/diff — so you can develop and verify a prompt, tool, or skill in isolation before a workflow ever uses it.

Try this

• Create or edit a prompt and save a draft, then explicitly choose Save & promote when you intend to move the live alias.
• Open a tool's workspace and run its draft in the sandbox against a sample input — the run is recorded and diffable against a baseline.
• On a skill, use the selection preview to see which skills a given user message would pull in.

Knowledge · Knowledge Base & Object Store

These pages are the data plane — the grounding sources agents retrieve over. Knowledge bases provide hybrid retrieval; the object store is the file UI over the S3-compatible bucket.

Try this

• Upload a file in the Object Store and confirm it lands in the bucket (MinIO console or aws s3 ls).

Evaluate · Test Sets, Judges & Evaluations

The evaluate group is the scoring plane: curate datasets, define how to judge, then run and inspect scores. Test sets are the datasets every scored run draws from; judges are custom LLM rubrics; evaluations run a test set through scorers and persist per-example results.

Try this

• Create a small test set, then click Sync to MLflow and watch the row's badge flip to Synced.
• Author a Judge with a one-line rubric (it becomes available to calibration and optimization), then run a small Evaluations pass over your test set with the built-in graders and open the result for per-example scores.

Observe · Observability & Review Queues

The observe surfaces close the loop on what actually happened at runtime. Observability is the trace view over MLflow tracing; Review Queues add structured human review of those traces on top of MLflow 3.14's assessment primitives.

Try this

• Open a workflow run's trace and inspect the per-tool-call spans (and any attached source documents).
• Create a Review Queue with a pass/fail question, enqueue a trace, and review it — the answer lands back on the trace.

Platform · LLM Gateway & Settings

The platform group is configuration and discovery. The Gateway page surfaces an external MLflow AI Gateway when one is configured (it is discovery and visibility, not a CALIBER-owned gateway). Settings is the grouped, safe-to-display inventory of runtime configuration.

Try this

• Open Settings and find the assistant group — it shows the resolved engine (auto → your real provider) and model.
• Confirm the storage group reflects the object-storage endpoint you configured.

Build with Aria · the agentic copilot

Aria is the embedded copilot, available from the top bar on every page. On OpenAI and Claude it runs a real tool-calling loop: within one turn it reads live CALIBER state, executes and observes real capabilities, and iterates before it answers. It supports interaction modes (chat, build, plan), approval modes (manual, auto_safe, auto_all), context attachments, a message queue, and steering — the same affordances a code assistant gives you.

Run, observe, fix · an end-to-end turn

Put Aria in build mode with auto_safe approvals and ask it to build a small tool. A single turn typically reads the relevant registry, drafts the artifact, validates it, and runs it in the sandbox — then reports what it observed, all before handing control back to you.

Switch the approval mode to auto_all when a turn needs admitted mutations or real runs. Aria still cannot approve or publish a draft inside that synchronous turn; a human advances those lifecycle gates. Every admitted action is recorded in the message's tool-call trail, so the autonomy remains bounded and auditable.

What you just ran — the receipt

To pin it down in one place, by the end of this runbook you have:

1. Brought up MLflow with the CALIBER plugin co-resident at /caliber/, against the configured storage backend.
2. Confirmed via /health and /readiness which services are healthy and which providers are real or simulated.
3. Toured every SPA surface in sidebar order: Dashboard, Workflows, the Library, the data plane, the observe surfaces, and Platform.
4. Used Aria's agentic loop to draft, validate, sandbox-test, and (optionally) publish a governed artifact — with every action recorded.

Everything Aria did this session is visible in the message's tool-call trail and persisted in the assistant tables. Artifact changes flow through the same validate → test → approve → publish gates as a manual edit, so there is no Aria-specific bypass to audit separately.

Reset the database

There is no demo seed to re-run — state is whatever you (or Aria) created. To start from a clean schema, stop the server and drop the metadata database, then let Alembic rebuild it:

# Stop terminal 1 (Ctrl-C), then, for the default SQLite metadata DB:
rm -f caliber.db caliber.db-wal caliber.db-shm
make dev   # alembic re-creates the schema; the database is empty

If you're on Postgres, drop and recreate the CALIBER database (or its schema) instead, then run make dev so Alembic migrates the fresh database to head. Object-storage artifacts live in the bucket independently — clear the bucket separately if you want a truly clean slate.

Troubleshooting

If you hit something not on this table, capture the output of curl -s http://127.0.0.1:5000/ajax-api/2.0/mlflow/caliber/readiness — that payload tells us which providers are real versus simulated in one shot.