CALIBER Python SDK API reference

Every documented class and module-level function, with the module that defines it. Members hang off their class, so start here and follow the link.

A

B

C

E

F

G

I

J

K

L

M

N

O

P

R

S

T

W

Module caliber_sdk#

caliber-sdk — a typed Python client for the CALIBER management API.

Tested example

def quickstart(caliber: CaliberClient) -> dict[str, Any]:
    """Report who you are and which API surfaces are GA on this deployment."""
    identity = caliber.me.get()
    if identity.is_anonymous:
        # /me answers "who am I" rather than requiring a credential, so an
        # invalid token shows up here as anonymous instead of an exception.
        raise SystemExit("no usable credential — check CALIBER_TOKEN")

    capabilities = caliber.capabilities_api.get()
    return {
        "user_id": identity.user_id,
        "scopes": identity.scopes,
        "ga_surfaces": sorted(capabilities.sdk_stability.get("ga", [])),
        "queue_enabled": capabilities.workflow_runs.queue_enabled,
    }

Public exports

API_PREFIX, ENV_BASE_URL, ENV_PROJECT, ENV_TOKEN, ENV_USER, FAILURE_STATES, TERMINAL_STATES, AuthProvider, CaliberAPIError, CaliberAuthenticationError, CaliberClient, CaliberConfigError, CaliberConflictError, CaliberError, CaliberNotFoundError, CaliberPermissionError, CaliberRateLimitError, CaliberServerError, CaliberTransportError, CaliberValidationError, ErrorBody, FieldError, NoAuth, Page, RawAPI, Response, Stability, TokenAuth, Transport, TrustedHeaderAuth, WaitFailed, WaitTimeout, WorkflowRunFailed, __version__, wait_for, wait_for_terminal_state

Module constants

Module caliber_sdk.client#

The root client — what a developer constructs first.

Tested example

def quickstart(caliber: CaliberClient) -> dict[str, Any]:
    """Report who you are and which API surfaces are GA on this deployment."""
    identity = caliber.me.get()
    if identity.is_anonymous:
        # /me answers "who am I" rather than requiring a credential, so an
        # invalid token shows up here as anonymous instead of an exception.
        raise SystemExit("no usable credential — check CALIBER_TOKEN")

    capabilities = caliber.capabilities_api.get()
    return {
        "user_id": identity.user_id,
        "scopes": identity.scopes,
        "ga_surfaces": sorted(capabilities.sdk_stability.get("ga", [])),
        "queue_enabled": capabilities.workflow_runs.queue_enabled,
    }

Public exports

ENV_BASE_URL, ENV_PROJECT, ENV_TOKEN, ENV_USER, CaliberClient

Module constants

Classes#

CaliberClient#

class CaliberClient(base_url: str | None = None, *, token: str | None = None, user: str | None = None, proxy_secret: str | None = None, auth: AuthProvider | None = None, project: str | None = None, timeout: float = 30.0, max_retries: int = 2, verify: bool | str = True, http_client: httpx.Client | None = None)

A connection to one CALIBER deployment.

Usage example

def quickstart(caliber: CaliberClient) -> dict[str, Any]:
    """Report who you are and which API surfaces are GA on this deployment."""
    identity = caliber.me.get()
    if identity.is_anonymous:
        # /me answers "who am I" rather than requiring a credential, so an
        # invalid token shows up here as anonymous instead of an exception.
        raise SystemExit("no usable credential — check CALIBER_TOKEN")

    capabilities = caliber.capabilities_api.get()
    return {
        "user_id": identity.user_id,
        "scopes": identity.scopes,
        "ga_surfaces": sorted(capabilities.sdk_stability.get("ga", [])),
        "queue_enabled": capabilities.workflow_runs.queue_enabled,
    }

Constructor

__init__(base_url: str | None = None, *, token: str | None = None, user: str | None = None, proxy_secret: str | None = None, auth: AuthProvider | None = None, project: str | None = None, timeout: float = 30.0, max_retries: int = 2, verify: bool | str = True, http_client: httpx.Client | None = None) -> None#

Operate on the caliber client surface with the supplied arguments and return the server response.

Returns: None

Raises:

• CaliberConfigError

Attributes

Properties

stability() -> dict[str, list[str]]#

Tags grouped by `ga / beta / internal`.

This callable takes no public parameters.

Returns: dict[str, list[str]]

Methods

close() -> None#

Close the underlying HTTP client or transport owned by this object.

This callable takes no public parameters.

Returns: None

__enter__() -> CaliberClient#

Return this instance so it can be used inside a context manager.

This callable takes no public parameters.

Returns: CaliberClient

__exit__(*_: object) -> None#

Close any owned resources when leaving the context manager.

Returns: None

capabilities() -> Any#

Runtime feature flags and the SDK stability tiers.

The cheap half of feature detection: it answers "may I call this?" without downloading the full OpenAPI document.

This callable takes no public parameters.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

openapi() -> Any#

The management OpenAPI document, generated from the live routes.

This callable takes no public parameters.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

whoami() -> Any#

The identity and scopes CALIBER resolved for this client's credential.

The first call worth making when a script gets an unexpected 403: it distinguishes "wrong credential" from "right credential, wrong scope".

It reports identity rather than requiring it, so an invalid or revoked credential does not raise here -- it returns `user_id: "anonymous"` with no scopes. Check the value; do not rely on an exception to detect a bad token.

This callable takes no public parameters.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

health() -> Any#

Fetch the lightweight health/readiness view exposed by the deployment.

This callable takes no public parameters.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

bootstrap_csrf() -> str | None#

Fetch a CSRF token up front.

Rarely needed: the transport fetches one automatically when a write is refused for want of it. Exposed for callers who would rather pay that round trip at startup than on their first write.

This callable takes no public parameters.

Returns: str | None

__repr__() -> str#

Operate on the caliber client surface with the supplied arguments and return the server response.

This callable takes no public parameters.

Returns: str

Module caliber_sdk.auth#

Authentication strategies for the CALIBER management API.

Tested example

def issue_scoped_token(caliber: CaliberClient, *, name: str = "ci") -> dict[str, Any]:
    """Create a token limited to operator scope.

    The scope list is a *ceiling*: the effective authority is the intersection
    with what the owner holds when the request is made. Requesting more than
    you hold is refused outright rather than silently narrowed, so a token
    never claims authority it cannot exercise.
    """
    issued = caliber.auth.tokens.create(name, scopes=["caliber.operator"])
    # The plaintext exists exactly once. There is no endpoint that returns it
    # again — store it now or rotate to get a new one.
    secret = issued.token

    live = [token.token_id for token in caliber.auth.tokens.list() if token.active]
    caliber.auth.tokens.revoke(issued.token_id)
    return {"token_id": issued.token_id, "secret_len": len(secret), "live_before_revoke": live}

Public exports

AuthProvider, NoAuth, TokenAuth, TrustedHeaderAuth

Classes#

AuthProvider#

class AuthProvider()

Bases: Protocol

Supplies per-request auth headers.

A protocol rather than a base class so a caller can plug in their own -- fetching a short-lived token from a secret manager, for instance -- without subclassing anything in this package.

Properties

uses_cookie_auth() -> bool#

Whether this credential is cookie-based.

Drives CSRF: CALIBER's protection exists for browser credentials, and a Bearer client should not be forced to bootstrap a token it does not need. See :mod:caliber_sdk.csrf.

This callable takes no public parameters.

Returns: bool

Methods

headers() -> dict[str, str]#

Headers to attach to every request.

This callable takes no public parameters.

Returns: dict[str, str]

TokenAuth#

class TokenAuth(token: str)

`Authorization: Bearer <token>` — personal access or session token.

Constructor

__init__(token: str) -> None#

Operate on the token auth surface with the supplied arguments and return the server response.

Returns: None

Raises:

• CaliberConfigError

Properties

uses_cookie_auth() -> bool#

Report whether this auth strategy relies on cookie-backed authentication.

This callable takes no public parameters.

Returns: bool

Methods

headers() -> dict[str, str]#

Build the authentication headers added to outgoing HTTP requests.

This callable takes no public parameters.

Returns: dict[str, str]

__repr__() -> str#

Operate on the token auth surface with the supplied arguments and return the server response.

This callable takes no public parameters.

Returns: str

TrustedHeaderAuth#

class TrustedHeaderAuth(user: str, *, proxy_secret: str | None = None)

`X-CALIBER-User — only for deployments in trusted_header` mode.

Carries no proof of identity by itself, which is why CALIBER ignores the header entirely in the default `session` mode. Offered because local development and proxy-terminated deployments genuinely use it.

Constructor

__init__(user: str, *, proxy_secret: str | None = None) -> None#

Operate on the trusted header auth surface with the supplied arguments and return the server response.

Returns: None

Raises:

• CaliberConfigError

Properties

uses_cookie_auth() -> bool#

Report whether this auth strategy relies on cookie-backed authentication.

This callable takes no public parameters.

Returns: bool

Methods

headers() -> dict[str, str]#

Build the authentication headers added to outgoing HTTP requests.

This callable takes no public parameters.

Returns: dict[str, str]

__repr__() -> str#

Operate on the trusted header auth surface with the supplied arguments and return the server response.

This callable takes no public parameters.

Returns: str

NoAuth#

class NoAuth()

Send no credential. Useful for probing an unauthenticated endpoint.

Properties

uses_cookie_auth() -> bool#

Report whether this auth strategy relies on cookie-backed authentication.

This callable takes no public parameters.

Returns: bool

Methods

headers() -> dict[str, str]#

Build the authentication headers added to outgoing HTTP requests.

This callable takes no public parameters.

Returns: dict[str, str]

Module caliber_sdk.transport#

HTTP transport: envelopes, errors, retries, CSRF, and correlation.

Tested example

def quickstart(caliber: CaliberClient) -> dict[str, Any]:
    """Report who you are and which API surfaces are GA on this deployment."""
    identity = caliber.me.get()
    if identity.is_anonymous:
        # /me answers "who am I" rather than requiring a credential, so an
        # invalid token shows up here as anonymous instead of an exception.
        raise SystemExit("no usable credential — check CALIBER_TOKEN")

    capabilities = caliber.capabilities_api.get()
    return {
        "user_id": identity.user_id,
        "scopes": identity.scopes,
        "ga_surfaces": sorted(capabilities.sdk_stability.get("ga", [])),
        "queue_enabled": capabilities.workflow_runs.queue_enabled,
    }

Public exports

API_PREFIX, USER_AGENT, Response, Transport

Module constants

Classes#

Response#

class Response(*, data, status_code: int, headers: Mapping[str, str], request_id: str | None)

A decoded response plus the context needed to debug it.

Constructor

__init__(*, data, status_code: int, headers: Mapping[str, str], request_id: str | None) -> None#

Operate on the response surface with the supplied arguments and return the server response.

Returns: None

Attributes

Transport#

class Transport(base_url: str, *, auth: AuthProvider | None = None, project: str | None = None, timeout: float = 30.0, max_retries: int = 2, backoff_factor: float = 0.5, verify: bool | str = True, client: httpx.Client | None = None, user_agent: str | None = None)

Synchronous HTTP transport against one CALIBER deployment.

Constructor

__init__(base_url: str, *, auth: AuthProvider | None = None, project: str | None = None, timeout: float = 30.0, max_retries: int = 2, backoff_factor: float = 0.5, verify: bool | str = True, client: httpx.Client | None = None, user_agent: str | None = None) -> None#

Send a prepared request through the shared transport and decode the typed response wrapper.

Returns: None

Raises:

• CaliberConfigError

Attributes

Methods

close() -> None#

Close the underlying HTTP client or transport owned by this object.

This callable takes no public parameters.

Returns: None

__enter__() -> Transport#

Return this instance so it can be used inside a context manager.

This callable takes no public parameters.

Returns: Transport

__exit__(*_: object) -> None#

Close any owned resources when leaving the context manager.

Returns: None

url_for(path: str) -> str#

Absolute URL for an API path, with or without the prefix.

Returns: str

bootstrap_csrf() -> str | None#

Fetch and cache a CSRF token, returning it.

Idempotent and cheap to call. Returns `None` when the deployment does not issue one, which is not an error: CSRF enforcement is configurable and a Bearer client may not need it.

This callable takes no public parameters.

Returns: str | None

request(method: str, path: str, *, params: Mapping[str, Any] | None = None, json = None, headers: Mapping[str, str] | None = None, files = None, data: Mapping[str, Any] | None = None, timeout: float | None = None, _csrf_retry: bool = True) -> Response#

Perform one API call, returning the unwrapped payload.

Returns: Response

Raises:

• CaliberAPIError
• CaliberTransportError

get(path: str, **kwargs) -> Response#

Fetch one record from the transport surface identified by path.

Returns: Response

Raises:

• CaliberAPIError
• CaliberTransportError

post(path: str, **kwargs) -> Response#

Send a prepared request through the shared transport and decode the typed response wrapper.

Returns: Response

Raises:

• CaliberAPIError
• CaliberTransportError

put(path: str, **kwargs) -> Response#

Send a prepared request through the shared transport and decode the typed response wrapper.

Returns: Response

Raises:

• CaliberAPIError
• CaliberTransportError

patch(path: str, **kwargs) -> Response#

Send a prepared request through the shared transport and decode the typed response wrapper.

Returns: Response

Raises:

• CaliberAPIError
• CaliberTransportError

delete(path: str, **kwargs) -> Response#

Delete a record on the transport surface and return the server acknowledgement.

Returns: Response

Raises:

• CaliberAPIError
• CaliberTransportError

download(path: str, **kwargs) -> bytes#

Fetch raw bytes.

Separate from :meth:request because file content is not JSON: it has no envelope to unwrap and decoding it would corrupt binary data.

Returns: bytes

Raises:

• CaliberAPIError
• CaliberTransportError

stream_lines(path: str, *, params: Mapping[str, Any] | None = None, timeout: float | None = None) -> Iterator[str]#

Yield lines from a server-sent-events endpoint.

Streaming needs its own path: the ordinary request reads the whole body before returning, which for an endpoint that never ends means blocking forever. No default timeout is applied either — a stream staying open is the success case, not a hang.

Returns: Iterator[str]

Raises:

• CaliberAPIError
• CaliberTransportError

paginate(path: str, *, params: Mapping[str, Any] | None = None, limit: int = 100) -> Iterator[Any]#

Yield items across `limit/offset` pages.

CALIBER's list endpoints are offset-based today. Exposing an iterator rather than the raw pages means the eventual move to cursors does not change this signature.

Returns: Iterator[Any]

Raises:

• CaliberAPIError
• CaliberTransportError

Module caliber_sdk.errors#

Exception hierarchy for the CALIBER SDK.

Tested example

def quickstart(caliber: CaliberClient) -> dict[str, Any]:
    """Report who you are and which API surfaces are GA on this deployment."""
    identity = caliber.me.get()
    if identity.is_anonymous:
        # /me answers "who am I" rather than requiring a credential, so an
        # invalid token shows up here as anonymous instead of an exception.
        raise SystemExit("no usable credential — check CALIBER_TOKEN")

    capabilities = caliber.capabilities_api.get()
    return {
        "user_id": identity.user_id,
        "scopes": identity.scopes,
        "ga_surfaces": sorted(capabilities.sdk_stability.get("ga", [])),
        "queue_enabled": capabilities.workflow_runs.queue_enabled,
    }

Public exports

CaliberAPIError, CaliberAuthenticationError, CaliberConfigError, CaliberConflictError, CaliberError, CaliberNotFoundError, CaliberPermissionError, CaliberRateLimitError, CaliberServerError, CaliberTransportError, CaliberValidationError, error_for_response

Functions#

error_for_response(*, status_code: int, payload, method: str, url: str, request_id: str | None = None) -> CaliberAPIError#

Build the right exception for a non-2xx response.

Tolerates a body that is not the documented shape -- an HTML error page from a proxy in front of CALIBER is a realistic response, and it must produce a usable exception rather than a KeyError inside the SDK.

Returns: CaliberAPIError

Classes#

CaliberError#

class CaliberError()

Bases: Exception

Base class for everything this SDK raises.

A caller who wants "any SDK failure" catches this and nothing else.

CaliberConfigError#

class CaliberConfigError()

Bases: CaliberError

The client was constructed with an unusable configuration.

CaliberTransportError#

class CaliberTransportError()

Bases: CaliberError

The request never produced an HTTP response.

Connection refused, DNS failure, timeout. Distinct from :class:CaliberAPIError because there is no server verdict to inspect -- and because retrying is often correct here and often wrong there.

CaliberAPIError#

class CaliberAPIError(message: str, *, status_code: int, detail: str | None = None, method: str | None = None, url: str | None = None, request_id: str | None = None, payload = None)

Bases: CaliberError

The server returned a non-2xx response.

Constructor

__init__(message: str, *, status_code: int, detail: str | None = None, method: str | None = None, url: str | None = None, request_id: str | None = None, payload = None) -> None#

Operate on the caliber a p i error surface with the supplied arguments and return the server response.

Returns: None

Attributes

CaliberAuthenticationError#

class CaliberAuthenticationError()

Bases: CaliberAPIError

401 — no usable identity. The credential is missing, wrong, or revoked.

CaliberPermissionError#

class CaliberPermissionError()

Bases: CaliberAPIError

403 — authenticated, but the identity lacks the required scope.

CaliberNotFoundError#

class CaliberNotFoundError()

Bases: CaliberAPIError

404 — no such resource.

CALIBER also returns this for a resource that exists but belongs to another user, deliberately: distinguishing the two would let a caller enumerate ids.

CaliberConflictError#

class CaliberConflictError()

Bases: CaliberAPIError

409 — the request conflicts with current state (duplicate name, etc.).

CaliberValidationError#

class CaliberValidationError(message: str, *, errors: list[dict[str, Any]], **kwargs)

Bases: CaliberAPIError

400 with a structured `errors` list.

Constructor

__init__(message: str, *, errors: list[dict[str, Any]], **kwargs) -> None#

Operate on the caliber validation error surface with the supplied arguments and return the server response.

Returns: None

Attributes

CaliberRateLimitError#

class CaliberRateLimitError()

Bases: CaliberAPIError

429 — too many requests.

CaliberServerError#

class CaliberServerError()

Bases: CaliberAPIError

5xx — the server failed. Usually worth retrying; never worth assuming.

Module caliber_sdk.waiters#

Polling helpers for CALIBER's long-running operations.

Tested example

def run_and_wait(
    caliber: CaliberClient, *, workflow_id: str, alias: str = "prod"

Public exports

FAILURE_STATES, TERMINAL_STATES, WaitFailed, WaitTimeout, state_of, wait_for, wait_for_terminal_state

Functions#

wait_for(poll: Callable[[], T], *, is_done: Callable[[T], bool], timeout: float = 300.0, interval: float = 2.0, max_interval: float = 15.0, backoff: float = 1.5, sleep: Callable[[float], None] = time.sleep, now: Callable[[], float] = time.monotonic) -> T#

Poll until `is_done` or the timeout expires.

The interval grows geometrically to `max_interval. A fixed short interval is what turns a slow job into thousands of requests; a fixed long one makes a fast job feel slow. sleep and now` are injectable so tests do not have to spend real seconds proving this.

Returns: T

Raises:

• ValueError
• WaitTimeout

state_of(payload, *, keys: Sequence[str] = ('status', 'state')) -> str#

Read a status field from a payload, tolerating either spelling.

Returns: str

wait_for_terminal_state(poll: Callable[[], Any], *, terminal: frozenset[str] = TERMINAL_STATES, failure: frozenset[str] = FAILURE_STATES, raise_on_failure: bool = True, **kwargs) -> Any#

Poll until the payload's status is terminal.

`raise_on_failure` is on by default because the common script wants a failed job to stop it; a caller inspecting the outcome themselves turns it off rather than wrapping every call in a try.

Returns: Any

Raises:

• WaitFailed
• WaitTimeout

Classes#

WaitTimeout#

class WaitTimeout(message: str, *, last = None, elapsed: float = 0.0)

Bases: CaliberError

The operation did not reach a terminal state within the budget.

Constructor

__init__(message: str, *, last = None, elapsed: float = 0.0) -> None#

Operate on the wait timeout surface with the supplied arguments and return the server response.

Returns: None

Attributes

WaitFailed#

class WaitFailed(message: str, *, state: str, last = None)

Bases: CaliberError

The operation reached a terminal state that indicates failure.

Constructor

__init__(message: str, *, state: str, last = None) -> None#

Operate on the wait failed surface with the supplied arguments and return the server response.

Returns: None

Attributes

Module caliber_sdk.resources#

Resource modules — typed façades over route groups.

Public exports

AccountsAPI, AriaAPI, AuditAPI, AuthAPI, CapabilitiesAPI, CookbooksAPI, EvalDatasetsAPI, EvaluationsAPI, EventsAPI, GatewayAPI, JobsAPI, JudgesAPI, KnowledgeBasesAPI, McpServersAPI, MeAPI, ObjectStoreAPI, ObservabilityAPI, OpenApiIntegrationsAPI, ProjectFilesAPI, ProjectsAPI, PromptsAPI, RawAPI, ReleasesAPI, Resource, ReviewQueuesAPI, SecretsAPI, SettingsAPI, SkillsAPI, TokensAPI, ToolsAPI, WorkflowRunFailed, WorkflowRunsAPI, WorkflowServicesAPI, WorkflowVersionsAPI, WorkflowsAPI

Module caliber_sdk.resources.auth#

Authentication, tokens, and accounts.

Tested example

def issue_scoped_token(caliber: CaliberClient, *, name: str = "ci") -> dict[str, Any]:
    """Create a token limited to operator scope.

    The scope list is a *ceiling*: the effective authority is the intersection
    with what the owner holds when the request is made. Requesting more than
    you hold is refused outright rather than silently narrowed, so a token
    never claims authority it cannot exercise.
    """
    issued = caliber.auth.tokens.create(name, scopes=["caliber.operator"])
    # The plaintext exists exactly once. There is no endpoint that returns it
    # again — store it now or rotate to get a new one.
    secret = issued.token

    live = [token.token_id for token in caliber.auth.tokens.list() if token.active]
    caliber.auth.tokens.revoke(issued.token_id)
    return {"token_id": issued.token_id, "secret_len": len(secret), "live_before_revoke": live}

Public exports

AccountsAPI, AuthAPI, TokensAPI

Classes#

TokensAPI#

class TokensAPI()

Personal access tokens for automation.

Methods

list() -> list[PersonalAccessToken]#

Every token belonging to the caller. Never includes a secret.

This callable takes no public parameters.

Returns: list[PersonalAccessToken]

Raises:

• CaliberAPIError
• CaliberTransportError

create(name: str, *, scopes: Sequence[str] | None = None, expires_at: str | None = None) -> IssuedToken#

Issue a token. The plaintext is returned once — store it now.

`scopes` is a ceiling, not a grant: the effective authority is the intersection with what the owner holds at request time. Omit it to inherit the owner's scopes. Requesting a scope the caller does not hold is refused rather than silently narrowed.

Returns: IssuedToken

Raises:

• CaliberAPIError
• CaliberTransportError

revoke(token_id: str) -> bool#

Revoke a token. Returns whether a live token was actually revoked.

Returns: bool

Raises:

• CaliberAPIError
• CaliberTransportError

rotate(token_id: str) -> IssuedToken#

Replace a token's secret, preserving its name and scope ceiling.

One transaction on the server: the old token is revoked and the replacement issued together, so a failure cannot leave an account with two live tokens or none.

Returns: IssuedToken

Raises:

• CaliberAPIError
• CaliberTransportError

AccountsAPI#

class AccountsAPI()

User accounts. Admin-only on the server.

Methods

list() -> list[Account]#

Return the current collection of user accounts, applying any supported filters.

This callable takes no public parameters.

Returns: list[Account]

Raises:

• CaliberAPIError
• CaliberTransportError

create(user_id: str, password: str) -> Any#

Create a new record on the user accounts surface and return the server-normalized result. Validation and permission failures are surfaced through the standard CALIBER error hierarchy.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

update(user_id: str, *, password: str | None = None, disabled: bool | None = None) -> Any#

Reset a password or enable/disable an account.

Both revoke the account's sessions server-side, so they take effect immediately rather than at the next expiry.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

revoke_sessions(user_id: str) -> int#

Sign an account out everywhere. Returns how many sessions were cut.

Returns: int

Raises:

• CaliberAPIError
• CaliberTransportError

AuthAPI#

class AuthAPI(transport)

Session inspection, plus the token and account sub-resources.

Constructor

__init__(transport) -> None#

Operate on the authentication and session state surface with the supplied arguments and return the server response.

Returns: None

Attributes

Methods

session() -> SessionInfo#

How this client's identity was established.

This callable takes no public parameters.

Returns: SessionInfo

Raises:

• CaliberAPIError
• CaliberTransportError

Module caliber_sdk.resources.system#

Identity, capabilities, and settings — the deployment-level surfaces.

Tested example

def quickstart(caliber: CaliberClient) -> dict[str, Any]:
    """Report who you are and which API surfaces are GA on this deployment."""
    identity = caliber.me.get()
    if identity.is_anonymous:
        # /me answers "who am I" rather than requiring a credential, so an
        # invalid token shows up here as anonymous instead of an exception.
        raise SystemExit("no usable credential — check CALIBER_TOKEN")

    capabilities = caliber.capabilities_api.get()
    return {
        "user_id": identity.user_id,
        "scopes": identity.scopes,
        "ga_surfaces": sorted(capabilities.sdk_stability.get("ga", [])),
        "queue_enabled": capabilities.workflow_runs.queue_enabled,
    }

Public exports

CapabilitiesAPI, MeAPI, SettingsAPI

Classes#

MeAPI#

class MeAPI()

The caller's own identity.

Methods

get() -> Identity#

Resolve this client's identity and scopes.

Reports rather than requires: an invalid or revoked credential returns an anonymous identity instead of raising. Check :attr:Identity.is_anonymous; do not rely on an exception.

This callable takes no public parameters.

Returns: Identity

Raises:

• CaliberAPIError
• CaliberTransportError

CapabilitiesAPI#

class CapabilitiesAPI()

Runtime feature flags and API stability tiers.

Methods

get() -> Capabilities#

Fetch one record from the runtime capabilities surface identified by id.

This callable takes no public parameters.

Returns: Capabilities

Raises:

• CaliberAPIError
• CaliberTransportError

SettingsAPI#

class SettingsAPI()

Runtime configuration inventory and LLM credential status.

Methods

runtime() -> RuntimeSettings#

Return the current runtime settings snapshot for the deployment.

This callable takes no public parameters.

Returns: RuntimeSettings

Raises:

• CaliberAPIError
• CaliberTransportError

llm() -> LlmSetupStatus#

Which LLM credentials are configured.

Presence flags and masked fingerprints only — the endpoint does not disclose key values, deliberately.

This callable takes no public parameters.

Returns: LlmSetupStatus

Raises:

• CaliberAPIError
• CaliberTransportError

update_llm(**changes) -> Any#

Write LLM provider settings. Secrets are write-only on the server.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

Module caliber_sdk.resources.projects#

Projects and their files — the workspace-scoping surface.

Tested example

def prompt_lifecycle(
    caliber: CaliberClient, *, agent_id: str = "intake-classifier"

Public exports

ProjectFilesAPI, ProjectsAPI

Classes#

ProjectFilesAPI#

class ProjectFilesAPI()

Files inside one project.

Methods

list(project_id: str) -> tuple[list[ProjectFile], list[ProjectFolder]]#

Files and the directories containing them.

Returned as a pair rather than one flattened list: a directory is not a file, and collapsing them would make an empty folder indistinguishable from a missing one.

Returns: tuple[list[ProjectFile], list[ProjectFolder]] — see ProjectFile, ProjectFolder

Raises:

• CaliberAPIError
• CaliberTransportError

upload(project_id: str, *, filename: str, content: bytes | BinaryIO, path: str | None = None, kind: str = 'input', media_type: str | None = None) -> ProjectFile#

Upload a file. Multipart, so it does not go through the JSON path.

Returns: ProjectFile

Raises:

• CaliberAPIError
• CaliberTransportError

create_folder(project_id: str, path: str) -> ProjectFolder#

Operate on the project files and folders surface with the supplied arguments and return the server response.

Returns: ProjectFolder

Raises:

• CaliberAPIError
• CaliberTransportError

delete(project_id: str, file_id: str) -> bool#

Delete a record on the project files and folders surface and return the server acknowledgement. Validation and permission failures are surfaced through the standard CALIBER error hierarchy.

Returns: bool

Raises:

• CaliberAPIError
• CaliberTransportError

download(project_id: str, file_id: str) -> bytes#

Raw bytes. Not JSON, so it bypasses the envelope entirely.

Returns: bytes

Raises:

• CaliberAPIError
• CaliberTransportError

ProjectsAPI#

class ProjectsAPI(transport)

Projects, project access, and the file sub-resource.

Related APIs: ProjectFilesAPI

Constructor

__init__(transport) -> None#

Operate on the projects surface with the supplied arguments and return the server response.

Returns: None

Attributes

Methods

list(*, status: str | None = None) -> list[Project]#

Active projects by default; pass `status="all"` for everything.

Returns: list[Project]

Raises:

• CaliberAPIError
• CaliberTransportError

get(project_id: str) -> Project#

Fetch one record from the projects surface identified by project_id.

Returns: Project

Raises:

• CaliberAPIError
• CaliberTransportError

create(name: str, *, description: str | None = None) -> Project#

Create a new record on the projects surface and return the server-normalized result. Validation and permission failures are surfaced through the standard CALIBER error hierarchy.

Returns: Project

Raises:

• CaliberAPIError
• CaliberTransportError

update(project_id: str, *, name: str | None = None, description: str | None = None, status: str | None = None) -> Project#

Patch an existing record on the projects surface and return the updated result. Validation and permission failures are surfaced through the standard CALIBER error hierarchy.

Returns: Project

Raises:

• CaliberAPIError
• CaliberTransportError

list_members(project_id: str) -> list[ProjectMember]#

List active members and their effective project roles.

Returns: list[ProjectMember]

Raises:

• CaliberAPIError
• CaliberTransportError

add_member(project_id: str, user_id: str, *, role: str = 'viewer') -> ProjectMember#

Grant `user_id` a project role; only owners may manage members.

Returns: ProjectMember

Raises:

• CaliberAPIError
• CaliberTransportError

update_member(project_id: str, user_id: str, *, role: str | None = None, status: str | None = None) -> ProjectMember#

Change a member's role or active status.

Returns: ProjectMember

Raises:

• CaliberAPIError
• CaliberTransportError

remove_member(project_id: str, user_id: str) -> bool#

Deactivate a member; the project owner cannot be removed.

Returns: bool

Raises:

• CaliberAPIError
• CaliberTransportError

storage() -> Any#

Where project files live, and what else the deployment supports.

This callable takes no public parameters.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

Module caliber_sdk.resources.assets#

Prompts, skills, and tools — the governed asset families.

Tested example

def prompt_lifecycle(
    caliber: CaliberClient, *, agent_id: str = "intake-classifier"

Public exports

PromptsAPI, SkillsAPI, ToolsAPI

Classes#

PromptsAPI#

class PromptsAPI()

Prompt registry surfaces.

Prompts are MLflow registry objects that CALIBER governs. Versions are immutable and an alias points at one of them, so "update a prompt" is always "register a new version", never an edit in place.

Usage example

def prompt_lifecycle(
    caliber: CaliberClient, *, agent_id: str = "intake-classifier"

Related APIs: EvaluationsAPI, ReviewQueuesAPI

Methods

list() -> list[Prompt]#

Return the current collection of prompts and prompt versions, applying any supported filters.

This callable takes no public parameters.

Returns: list[Prompt]

Raises:

• CaliberAPIError
• CaliberTransportError

get(agent_id: str) -> Prompt#

Fetch one record from the prompts and prompt versions surface identified by agent_id.

Returns: Prompt

Raises:

• CaliberAPIError
• CaliberTransportError

create(name: str, template: str, *, commit_message: str | None = None) -> Any#

Register a prompt and its first version.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

versions(agent_id: str) -> Any#

Every registered version, newest first.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

register_version(agent_id: str, template: str, *, commit_message: str | None = None) -> Any#

Add a version without touching the live alias.

The alias is rotated separately by :meth:promote, so authoring is never a deployment — the property the whole refinement loop depends on.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

promote(agent_id: str, version: int, *, alias: str = 'prod') -> Any#

Point an alias at a version. This is the deployment step.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

SkillsAPI#

class SkillsAPI()

Skill registry, rendering, selection testing, and versions.

Related APIs: JudgesAPI, EvaluationsAPI

Methods

list(*, status: str | None = None, tag: str | None = None) -> list[Skill]#

Return the current collection of skills, applying any supported filters.

Returns: list[Skill]

Raises:

• CaliberAPIError
• CaliberTransportError

get(skill_id: str) -> Skill#

Fetch one record from the skills surface identified by skill_id.

Returns: Skill

Raises:

• CaliberAPIError
• CaliberTransportError

create(name: str, *, content: str, owner: str, summary: str | None = None, description: str | None = None, tags: Sequence[str] | None = None) -> Skill#

Create a skill.

`owner` is required by the server and is therefore keyword-required here rather than defaulted to the caller's identity: a skill's owner is a governance field, and quietly inferring it would make authorship an accident of which credential happened to run the script.

Returns: Skill

Raises:

• CaliberAPIError
• CaliberTransportError

update(skill_id: str, **changes) -> Skill#

Patch an existing record on the skills surface and return the updated result. Validation and permission failures are surfaced through the standard CALIBER error hierarchy.

Returns: Skill

Raises:

• CaliberAPIError
• CaliberTransportError

render(skill_id: str, *, variables: dict[str, Any] | None = None) -> SkillRender#

Substitute `{{variables}}` and report what was left unresolved.

Returns: SkillRender

Raises:

• CaliberAPIError
• CaliberTransportError

test_selection(skill_id: str, query: str) -> SkillSelection#

Would this skill be auto-selected for this query?

Returns: SkillSelection

Raises:

• CaliberAPIError
• CaliberTransportError

versions(skill_id: str) -> list[SkillVersion]#

Operate on the skills surface with the supplied arguments and return the server response.

Returns: list[SkillVersion]

Raises:

• CaliberAPIError
• CaliberTransportError

ToolsAPI#

class ToolsAPI()

Tool registry, fixtures, and calibration.

Methods

list(*, status: str | None = None) -> list[Tool]#

Return the current collection of tools and calibration cases, applying any supported filters.

Returns: list[Tool]

Raises:

• CaliberAPIError
• CaliberTransportError

get(tool_id: str) -> Tool#

Fetch one record from the tools and calibration cases surface identified by tool_id.

Returns: Tool

Raises:

• CaliberAPIError
• CaliberTransportError

register(name: str, *, version: str, module_path: str, callable_name: str, input_schema: dict[str, Any] | None = None, output_schema: dict[str, Any] | None = None, **options) -> Tool#

Operate on the tools and calibration cases surface with the supplied arguments and return the server response.

Returns: Tool

Raises:

• CaliberAPIError
• CaliberTransportError

update(tool_id: str, **changes) -> Tool#

Patch an existing record on the tools and calibration cases surface and return the updated result. Validation and permission failures are surfaced through the standard CALIBER error hierarchy.

Returns: Tool

Raises:

• CaliberAPIError
• CaliberTransportError

calibrate(tool_id: str, **options) -> CalibrationJob#

Queue a calibration run. Returns immediately with a job to poll.

Returns: CalibrationJob

Raises:

• CaliberAPIError
• CaliberTransportError

calibration_job(tool_id: str, job_id: str) -> CalibrationJob#

Operate on the tools and calibration cases surface with the supplied arguments and return the server response.

Returns: CalibrationJob

Raises:

• CaliberAPIError
• CaliberTransportError

calibration_jobs(tool_id: str) -> list[CalibrationJob]#

Operate on the tools and calibration cases surface with the supplied arguments and return the server response.

Returns: list[CalibrationJob]

Raises:

• CaliberAPIError
• CaliberTransportError

wait_for_calibration(tool_id: str, job_id: str, *, timeout: float = 600.0, **options) -> CalibrationJob#

Poll a calibration job until it stops.

Returns the terminal job rather than raising on failure: a failed calibration is a result to inspect, not an error in the call.

Returns: CalibrationJob

Raises:

• CaliberAPIError
• CaliberTransportError
• WaitFailed
• WaitTimeout

Module caliber_sdk.resources.workflows#

Workflows, versions, runs, deployments, and services.

Tested example

def run_and_wait(
    caliber: CaliberClient, *, workflow_id: str, alias: str = "prod"

Public exports

WorkflowRunFailed, WorkflowRunsAPI, WorkflowServicesAPI, WorkflowVersionsAPI, WorkflowsAPI

Classes#

WorkflowRunFailed#

class WorkflowRunFailed(run: WorkflowRun)

Bases: CaliberError

A run reached a terminal state that is not success.

Constructor

__init__(run: WorkflowRun) -> None#

Operate on the workflow run failed surface with the supplied arguments and return the server response.

Returns: None

Attributes

WorkflowVersionsAPI#

class WorkflowVersionsAPI()

Immutable manifest snapshots of one workflow.

Related APIs: WorkflowsAPI, WorkflowRunsAPI, WorkflowServicesAPI

Methods

list(workflow_id: str) -> list[WorkflowVersion]#

Return the current collection of workflow versions, applying any supported filters.

Returns: list[WorkflowVersion]

Raises:

• CaliberAPIError
• CaliberTransportError

get(version_id: str) -> WorkflowVersion#

Fetch one record from the workflow versions surface identified by version_id.

Returns: WorkflowVersion

Raises:

• CaliberAPIError
• CaliberTransportError

create(workflow_id: str, manifest: dict[str, Any]) -> WorkflowVersion#

Register a draft version. Drafts are not runnable until published.

Returns: WorkflowVersion

Raises:

• CaliberAPIError
• CaliberTransportError

validate(version_id: str) -> Any#

Validation report for a version's manifest.

Returned untyped on purpose: the report is produced by the server's validator, and a schema here would be a second definition of a contract that lives there.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

compile(version_id: str) -> Any#

Ask the server to compile the draft workflow or asset into its executable form.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

publish(version_id: str) -> WorkflowVersion#

Promote the draft or version into the published state used by operators or runtime callers.

Returns: WorkflowVersion

Raises:

• CaliberAPIError
• CaliberTransportError

WorkflowRunsAPI#

class WorkflowRunsAPI()

Executions, and waiting on them.

Usage example

def run_and_wait(
    caliber: CaliberClient, *, workflow_id: str, alias: str = "prod"

Related APIs: WorkflowsAPI, WorkflowVersionsAPI, WorkflowServicesAPI

Methods

list(workflow_id: str, *, status: str | None = None) -> list[WorkflowRun]#

Runs of one workflow.

Scoped to a workflow because the server has no unscoped run listing: `/workflow-runs` is POST-only (submission). An SDK method implying otherwise returned 405 at runtime, which is how this was found.

Returns: list[WorkflowRun]

Raises:

• CaliberAPIError
• CaliberTransportError

get(run_id: str) -> WorkflowRun#

Fetch one record from the workflow runs surface identified by run_id.

Returns: WorkflowRun

Raises:

• CaliberAPIError
• CaliberTransportError

submit(*, workflow_version_id: str | None = None, workflow_id: str | None = None, alias: str | None = None, input = None, idempotency_key: str | None = None, **options) -> WorkflowRun#

Queue a run. Returns immediately with a run to poll.

A run targets either a specific version or a workflow plus a deployment alias — the server accepts both, and forcing one here would make the alias path unreachable, which is how a deployed workflow is invoked.

`idempotency_key` is passed through because submission is the one mutating call the SDK cannot safely retry on its own.

Returns: WorkflowRun

Raises:

• CaliberAPIError
• CaliberTransportError

cancel(run_id: str) -> WorkflowRun#

Operate on the workflow runs surface with the supplied arguments and return the server response.

Returns: WorkflowRun

Raises:

• CaliberAPIError
• CaliberTransportError

wait(run_id: str, *, timeout: float = 900.0, raise_on_failure: bool = True, **options) -> WorkflowRun#

Poll until the run stops.

Raises by default, unlike calibration: a script that submitted work and got a failure almost always wants to stop, whereas a calibration score is the thing being measured. Pass `raise_on_failure=False` to inspect instead.

Returns: WorkflowRun

Raises:

• CaliberAPIError
• CaliberTransportError
• WaitFailed
• WaitTimeout
• WorkflowRunFailed

WorkflowServicesAPI#

class WorkflowServicesAPI()

Workflows published as external HTTP services.

The server splits this in two, and so does this class. Management lives under `/workflows/{id}/service — configuring and publishing is a property of the workflow. *Invocation* lives under /services/{id}` — that is the external surface, authenticated by per-service tokens rather than a user credential.

There is no unscoped service listing; a service is reached through its workflow. An earlier version of this class invented `GET /services` and returned 404 at runtime.

Related APIs: WorkflowsAPI, WorkflowVersionsAPI

Methods

get(workflow_id: str) -> WorkflowService#

Fetch one record from the workflow services surface identified by workflow_id.

Returns: WorkflowService

Raises:

• CaliberAPIError
• CaliberTransportError

publish(workflow_id: str, **options) -> WorkflowService#

Promote the draft or version into the published state used by operators or runtime callers.

Returns: WorkflowService

Raises:

• CaliberAPIError
• CaliberTransportError

unpublish(workflow_id: str) -> bool#

Remove the published state from the targeted runtime asset.

Returns: bool

Raises:

• CaliberAPIError
• CaliberTransportError

openapi(workflow_id: str) -> Any#

The per-workflow OpenAPI document the service surface publishes.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

invoke(workflow_id: str, payload = None, **options) -> Any#

Call a published service.

The external surface: in production this is authenticated by a per-service token rather than the user credential the rest of this client carries.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

WorkflowsAPI#

class WorkflowsAPI(transport)

Workflows, plus versions, runs, and services as sub-resources.

Usage example

def run_and_wait(
    caliber: CaliberClient, *, workflow_id: str, alias: str = "prod"

Related APIs: WorkflowVersionsAPI, WorkflowRunsAPI, WorkflowServicesAPI

Constructor

__init__(transport) -> None#

Operate on the workflows surface with the supplied arguments and return the server response.

Returns: None

Attributes

Methods

list(*, status: str | None = None) -> list[Workflow]#

Return the current collection of workflows, applying any supported filters.

Returns: list[Workflow]

Raises:

• CaliberAPIError
• CaliberTransportError

get(workflow_id: str) -> Workflow#

Fetch one record from the workflows surface identified by workflow_id.

Returns: Workflow

Raises:

• CaliberAPIError
• CaliberTransportError

create(name: str, *, description: str | None = None, **options) -> Workflow#

Create a new record on the workflows surface and return the server-normalized result. Validation and permission failures are surfaced through the standard CALIBER error hierarchy.

Returns: Workflow

Raises:

• CaliberAPIError
• CaliberTransportError

update(workflow_id: str, **changes) -> Workflow#

Patch an existing record on the workflows surface and return the updated result. Validation and permission failures are surfaced through the standard CALIBER error hierarchy.

Returns: Workflow

Raises:

• CaliberAPIError
• CaliberTransportError

delete(workflow_id: str) -> Any#

Delete a record on the workflows surface and return the server acknowledgement. Validation and permission failures are surfaced through the standard CALIBER error hierarchy.

Returns: Any

Raises:

• CaliberAPIError
• CaliberTransportError

Module caliber_sdk.resources.quality#

Datasets, judges, and evaluations — the evidence and scoring surfaces.

Tested example

def build_and_score(caliber: CaliberClient, *, owner: str = "@you") -> dict[str, Any]:
    """Create a dataset, add a row, define a judge, and run an evaluation."""
    dataset = caliber.datasets.create("intake-golden", owner=owner)
    caliber.datasets.add_example(
        dataset.dataset_id,
        inputs={"ticket": "I was charged twice"},
        expected={"intent": "billing"},
    )

    # Instructions must reference an evaluation variable. A judge with no
    # variable grades nothing — it returns the same verdict every time — so
    # the server rejects it rather than letting you collect meaningless scores.
    judge = caliber.judges.create(
        "valid-intent",
        instructions="Given {{ inputs }} and {{ outputs }}, return true if intent is allowed.",
        feedback_value_type="bool",
    )

    evaluation = caliber.evaluations.create(dataset.dataset_id, judge_id=judge.judge_id)
    return {
        "dataset_id": dataset.dataset_id,
        "judge_id": judge.judge_id,
        "evaluation_id": evaluation.evaluation_id,
    }

Public exports

EvalDatasetsAPI, EvaluationsAPI, JudgesAPI

Classes#

EvalDatasetsAPI#

class EvalDatasetsAPI()

Versioned evaluation datasets and their examples.

Usage example

def build_and_score(caliber: CaliberClient, *, owner: str = "@you") -> dict[str, Any]:
    """Create a dataset, add a row, define a judge, and run an evaluation."""
    dataset = caliber.datasets.create("intake-golden", owner=owner)
    caliber.datasets.add_example(
        dataset.dataset_id,
        inputs={"ticket": "I was charged twice"},
        expected={"intent": "billing"},
    )

    # Instructions must reference an evaluation variable. A judge with no
    # variable grades nothing — it returns the same verdict every time — so
    # the server rejects it rather than letting you collect meaningless scores.
    judge = caliber.judges.create(
        "valid-intent",
        instructions="Given {{ inputs }} and {{ outputs }}, return true if intent is allowed.",
        feedback_value_type="bool",
    )

    evaluation = caliber.evaluations.create(dataset.dataset_id, judge_id=judge.judge_id)
    return {
        "dataset_id": dataset.dataset_id,
        "judge_id": judge.judge_id,
        "evaluation_id": evaluation.evaluation_id,
    }

Related APIs: EvaluationsAPI, JudgesAPI, ReviewQueuesAPI

Methods

list(*, status: str | None = None) -> list[EvalDataset]#

Return the current collection of evaluation datasets, applying any supported filters.

Returns: list[EvalDataset]

Raises:

• CaliberAPIError
• CaliberTransportError

get(dataset_id: str) -> EvalDataset#

Fetch one record from the evaluation datasets surface identified by dataset_id.

Returns: EvalDataset

Raises:

• CaliberAPIError
• CaliberTransportError

create(name: str, *, owner: str, description: str | None = None, **options) -> EvalDataset#

Create a dataset.

`owner` is required by the server and kept keyword-required here for the same reason as skills: ownership is a governance field, not something to infer from whichever credential ran the script.

Returns: EvalDataset

Raises:

• CaliberAPIError
• CaliberTransportError

add_example(dataset_id: str, *, inputs, expected = None, **options) -> EvalExample#

Append one labeled example row to the targeted evaluation dataset.

Returns: EvalExample

Raises:

• CaliberAPIError
• CaliberTransportError

examples(dataset_id: str) -> list[EvalExample]#

Return the example rows currently stored for the targeted evaluation dataset.

Returns: list[EvalExample]

Raises:

• CaliberAPIError
• CaliberTransportError

add_from_trace(dataset_id: str, trace_id: str, **options) -> EvalExample#

Capture a production trace as a dataset row.

The path that turns an observed failure into evidence, which is where the refinement loop starts.

Returns: EvalExample

Raises:

• CaliberAPIError
• CaliberTransportError

JudgesAPI#

class JudgesAPI()

Model-backed graders and their human alignment.

Usage example

def build_and_score(caliber: CaliberClient, *, owner: str = "@you") -> dict[str, Any]:
    """Create a dataset, add a row, define a judge, and run an evaluation."""
    dataset = caliber.datasets.create("intake-golden", owner=owner)
    caliber.datasets.add_example(
        dataset.dataset_id,
        inputs={"ticket": "I was charged twice"},
        expected={"intent": "billing"},
    )

    # Instructions must reference an evaluation variable. A judge with no
    # variable grades nothing — it returns the same verdict every time — so
    # the server rejects it rather than letting you collect meaningless scores.
    judge = caliber.judges.create(
        "valid-intent",
        instructions="Given {{ inputs }} and {{ outputs }}, return true if intent is allowed.",
        feedback_value_type="bool",
    )

    evaluation = caliber.evaluations.create(dataset.dataset_id, judge_id=judge.judge_id)
    return {
        "dataset_id": dataset.dataset_id,
        "judge_id": judge.judge_id,
        "evaluation_id": evaluation.evaluation_id,
    }

Methods

list() -> list[Judge]#

Return the current collection of judges and alignment assets, applying any supported filters.

This callable takes no public parameters.

Returns: list[Judge]

Raises:

• CaliberAPIError
• CaliberTransportError

get(judge_id: str) -> Judge#

Fetch one record from the judges and alignment assets surface identified by judge_id.

Returns: Judge

Raises:

• CaliberAPIError
• CaliberTransportError

create(name: str, *, instructions: str, feedback_value_type: str = 'bool', model: str | None = None, **options) -> Judge#

Create a model-backed grader.

`instructions must reference at least one evaluation variable — {{ inputs }}, {{ outputs }}, {{ expectations }}, {{ conversation }}, or {{ trace }}` — or the server rejects it. The rule exists because a judge with no variable grades nothing: it would return the same verdict for every example.

`feedback_value_type defaults to bool`. A numeric judge is not interchangeable with a boolean one downstream, so scorecards read this field to know which they have.