Experiment Management in Studio

Overview

This tutorial walks through the complete Studio experiment workflow. An experiment groups immutable candidates that vary prompts, model versions, datasets, providers, or runtime parameters while preserving the evidence needed for review and recommendation.

Prepare Evaluation Inputs

Start the REST API and Studio. This tutorial uses the normal production sequence, not just seeded demo data: configure the evaluator that scores outputs, configure the runtime that invokes the model, register immutable input assets, then create and run an experiment.

An Evaluation Provider scores an output. A Runtime Connection invokes a model. They are separate configurations with separate secret references; most real experiments require both.

1. Configure Evaluation Provider

1. Open Assets → Evaluation Providers.
2. Select New Installation and choose the installed evaluator type, such as TruLens.
3. Set the evaluator model, enabled metrics, and timeout.
4. Provide credentials as a secret reference—not a raw API key—then select Validate Connection.
5. Keep the installation enabled and create it.

The provider installation is the governed evaluator configuration selected by each candidate. It defines how outputs are scored; it does not select or invoke the runtime model.

2. Configure Runtime Connection

1. Open Settings → Runtime Connections.
2. Create a connection for the provider of the managed model you will evaluate.
3. Enter the endpoint settings and an API-key secret reference, then choose organization or project scope.
4. Select Test Connection; an active connection must validate before it can be saved.
5. Keep the connection enabled.

When you select a model in a candidate, Studio shows only enabled Runtime Connections with the matching provider. Mock models do not require a Runtime Connection. Read the Runtime Connection API reference for its scope and secret-handling contract.

Configure a provider endpoint with secret references, validate it, and keep the connection enabled for compatible candidate execution.

3. Register Governed Assets

1. In Assets → Datasets, upload CSV, JSONL, or NDJSON, then freeze or activate the immutable dataset version.
2. In Assets → Prompts, create a managed prompt version and make the version active.
3. In Assets → Models, register a managed model with its provider model ID and runtime capability profile, then activate it.

Candidates can select active prompt and model versions and active or frozen dataset versions. Every candidate in an experiment must use the same immutable dataset identity and version, ensuring the comparison uses the same test cases.

See Asset Catalogs for ownership, lifecycle, and immutable-versioning details.

Experiment Inventory

The Experiments page is a paginated inventory. It shows the experiment name, status, owner, last modified time, and an action to open the detailed view. Use the status filter to focus on draft, running, completed, failed, or archived experiments.

Create Experiment

Select Create Experiment, provide a name and objective, and submit the form. New experiments begin in DRAFT. The detail page opens after creation so candidates can be registered before execution.

curl -X POST http://localhost:8000/api/v1/experiments \
  -H "content-type: application/json" \
  -d '{
    "name": "Customer Support RAG Evaluation",
    "description": "Compare governed support configurations.",
    "metadata": {"owner": "studio-demo"}
  }'

Overview

The Overview tab summarizes experiment health with candidate count, evaluation run count, success rate, and best leaderboard score. It also shows ownership, created and last-modified timestamps, execution health, metadata, and the latest outcome.

Candidates

In the Candidates tab, select Add Candidate. The backend-driven form provides governed prompt, model, dataset, and evaluation-provider versions; free-form version strings are not accepted by the Studio workflow. Select the enabled provider installation and the compatible Runtime Connection after selecting the model.

Each candidate records:

• candidate and experiment identity
• prompt, model, and dataset identity plus version
• evaluation provider
• matching Runtime Connection
• supported temperature, top-p, and max-tokens runtime parameters
• metadata and creation timestamp

Candidates are immutable comparison units. To change a governed configuration, register a new candidate instead of modifying the existing one.

Run and Monitor Evaluations

Select Start Experiment. Studio first shows the declared run plan: candidates × immutable dataset items, model invocations, and evaluation items. Confirm it to run every candidate against the same dataset.

While the experiment is running, the Overview shows Live Execution Progress: the active candidate, model items executed, evaluation items completed, and a progress bar. Progress is persisted after every model invocation, so it survives refreshes. The Evaluation Runs tab also shows each run's status, duration, executed/evaluated item counts, and a safe failure reason where applicable.

The Evaluation Runs view refreshes every 5 seconds by default; choose 10 seconds or turn refresh off for the current browser session. You can also cancel a running experiment. Completed evidence remains available and no further model or evaluator calls are started.

Inspect Item Results

Below the run list, Item Evaluation Results provides a paginated record for every completed dataset item. Select a run when the experiment has multiple candidates. Each row shows the evaluator type and version, completion time, model API latency, and the individual metric scores returned for that item.

These rows are durable evidence, not a recalculated summary. Completed item results remain visible if a later item fails or the experiment is cancelled, making partial progress reviewable instead of hiding it behind an overall run status.

Candidates Comparison

The Comparison tab is the detailed candidate review surface. Choose a Baseline Candidate and a different Comparison Candidate. Comparison remains disabled until both selections are different.

The configuration comparison displays these fields side by side:

• prompt identity and version
• model identity and version
• dataset identity and version
• evaluation provider
• temperature, top-p, and max-tokens

Changed configuration rows use neutral theme-aware highlighting. The metric table displays only metrics returned by the backend, with baseline value, comparison value, and delta. Missing metrics are shown as unavailable rather than invented in the browser.

Known directional metrics use semantic colors: green indicates improvement, red indicates regression, and neutral indicates no change or an unknown direction. Higher quality scores are better; lower cost, latency, and hallucination score are better.

Leaderboard

The Leaderboard tab displays the latest ranked snapshot, including rank, candidate, overall score, latency, cost, and ranking reason. A leaderboard is evidence for recommendation; it does not deploy or promote a candidate. Use View in Ontology to inspect the leaderboard and its candidate, run, and evaluation relationships in the graph.

REST & MCP Access

Studio uses the same read contracts that are available to REST and MCP clients:

# Experiment detail data
curl http://localhost:8000/api/v1/experiments/{experiment_id}
curl http://localhost:8000/api/v1/experiments/{experiment_id}/candidates
curl http://localhost:8000/api/v1/experiments/{experiment_id}/run-plan
curl http://localhost:8000/api/v1/experiments/{experiment_id}/runs

# Item-level scores and model latency for one evaluation run
curl "http://localhost:8000/api/v1/experiments/{experiment_id}/runs/{run_id}/evaluations?page=1&page_size=25"

# Exact candidate comparison
curl "http://localhost:8000/api/v1/experiments/{experiment_id}/comparison?baseline_candidate_id={baseline_id}&comparison_candidate_id={comparison_id}"

# Latest leaderboard
curl http://localhost:8000/api/v1/experiments/{experiment_id}/leaderboard

from ai_governance import create_mcp_server

server = create_mcp_server()

server.call_tool("experiment.candidates", {"experiment_id": "experiment-1"})
server.call_tool("experiment.runs", {"experiment_id": "experiment-1"})
server.call_tool(
    "experiment.compare_candidates",
    {
        "experiment_id": "experiment-1",
        "baseline_candidate_id": "candidate-a",
        "comparison_candidate_id": "candidate-b",
    },
)

Operational Notes

• During a run, use the live progress panel and the Evaluation Runs view rather than waiting for the final leaderboard. Both read durable progress and result records.
• A comparison requires two different candidates with completed evaluation runs. Each candidate must use the experiment's same immutable dataset identity and version.
• Candidate configuration is sourced from governed registries and should be treated as immutable evidence.
• Leaderboard recommendations are advisory and do not perform deployment or promotion.