§ Agents & EvalsDocument · REG-EVAL-2026.06

You never trust the model. You trust the evidence that it passed.

Every agent gets an eval suite: quality, consistency, cost, and latency, on cases you choose. Then Regisseur tells you the cheapest model that still passes the bar. For every agent.

Harvard Business Review (June 2026): the strongest agent teams are built from different models. See the evidence ↓

A real three-model eval group (DeepSeek, Claude Haiku 4.5, and Nemotron) all passing, with Nemotron starred as the cheapest passing model at $0.00012 per run, and a drift panel below flagging two real regressions in red. — Cheapest model that passes, and the gate that proves itThree models compared; the drift panel flags real regressions in red

§ 01Eval suites

Test every agent.

Each agent has a Testing tab. Author eval cases (an input plus the expected result or a rubric), or let the platform draft them for you. Run the suite and read four signals, not a single green checkmark.

Quality

Correct output on known inputs

Consistency

Each case sampled repeatedly. A good agent repeats its answer, not a coin-flip

Cost & latency

Tokens, $/run, p50 / p95

Judgment

A calibrated LLM judge scores open-ended output, and is itself blind-calibrated before it can gate anything

An agent's Testing tab showing three eval cases, a judge-calibration panel, and PASS runs. — The Testing tabEval cases, judge calibration, and PASS runs per agent

§ 02Model selection

The cheapest model that still passes.

Run the same suite across every model you'd consider. Regisseur folds the results into a single model × {quality, latency, cost} matrix and stars ★ the cheapest configuration that PASSes, with ties broken by lower latency.

Unpriced models show a dash, never "$0"; a recommendation is only made among models that actually pass. Premium reasoning where the case is hard, a cheap fast model where it isn't. Chosen on evidence, not on vibes.

It is how you staff a team: the associate handles what the associate does well, the specialist is pulled in when complexity demands it, and the faster operator is used when speed is worth the extra cost. Regisseur applies that logic across models, and the matrix is the evidence for each choice.

The model registry in Settings, with providers carrying default and judge badges and per-config pricing. — Model registryDefault and judge models, priced per config

§ 03Drift

Catch drift before your customers do.

Save a baseline. Every later run is rated pass / warn / fail against it across quality, rubric, cost, and latency.

A prompt tweak or model upgrade that quietly degrades quality, or balloons cost, is caught instead of shipping silently. A failing drift check can block the change.

Baseline

A saved reference run

Rated

Quality · rubric · cost · latency

Deploy gate

A regression can block the change

A drift card rating a run against its baseline across deterministic, rubric, cost, and latency, all PASS. — Drift vs baselineDeterministic, rubric, cost, and latency, all PASS

§ 04Why it matters

Different models for different agents.

A regulated vertical can run many specialized agents. Right-sizing each to the cheapest model that still passes, instead of stretching one expensive model across everything, is where quality holds and unit cost falls. It is the concrete mechanism behind the unit-cost half of an AI transformation.

The agent roster: dozens of agents, each with its own model, autonomy ceiling, run count, approval rate, and cost per run. — Right-sized, agent by agentEach agent carries its own model, ceiling, and cost/run

§ Evidence · Harvard Business Review

Harvard Business Review (Mark Purdy, June 18 2026) argues the strongest agent teams are built from different models: when every agent runs on the same model, their errors correlate and blind spots compound. Two studies he cites, on diversity-selected agent teams were 25% better at resolving software-engineering problems than agents working individually, and just two diverse agents can “match or exceed the performance of 16 homogeneous agents.”

Regisseur operationalizes model diversity, and goes one step further: it tests every agent across models and assigns the cheapest one that still passes.

HBR · Purdy · June 2026 →

§ Next step

See an eval suite pick the cheapest model that passes.

Bring one agent and a handful of real cases. We'll author the suite, run it across models, and show you the matrix with the recommendation: quality, latency, and cost side by side.

Book a walkthrough →How the pipeline stays deterministic