Evaluations

Record objective outcomes as immutable evidence, freeze reproducible snapshots, and use them to improve routing policy.

4 min readEdit this page

An evaluation answers whether a result met an objective. It is deliberately separate from telemetry, which records what ran, and from optimization, which decides how admitted evidence may change a routing policy.

ResponsibilityQuestion it answersPrimary surface
TelemetryWhat route ran, what did it cost, and what happened on each attempt?Request receipts, traces, and metrics
EvaluationDid the result satisfy the declared objective?Subjects, evaluator results, and evidence snapshots
OptimizationWhat policy transition is justified by admitted evidence?Controller state and policy candidates

An HTTP 200, a completed stream, or a cheaper route is not proof of task success. Those are observations that an evaluator may use, not the objective result itself.

Evaluation evidence

The evaluator-neutral exchange accepts immutable evaluation subjects and results from an evaluator you choose. Admitted results can be frozen into a reproducible snapshot for optimization or policy compilation. A trace does not evaluate its outcome by itself.

In alpha.33, legacy request-time adequacy learning is disabled in both policy modes. Adequacy fields remain compiler inputs; they do not enable live escalation or exploration.

Evidence lifecycle

1. Define and seal a subject

An evaluation subject identifies what is being evaluated and carries redacted evidence items. Seal a draft to calculate its canonical digest and write validated JSON:

bro eval subject seal --output subject.json subject.yaml
bro eval subject put subject.json

The digest makes the subject content-addressed. Changing the evidence creates a different subject rather than mutating the one an evaluator already scored.

2. Evaluate outside the router

Run the evaluator appropriate to the objective: a test suite, benchmark harness, rubric implementation, human review, or another process whose configuration you can identify. BitRouter does not infer task success from model text or choose an evaluator for you.

The result should name the sealed subject and preserve enough evaluator identity to distinguish materially different scoring configurations.

3. Submit the result

Submit the result through the configured authority boundary:

bro eval result submit result.json
bro eval status

Admission determines which results are eligible evidence. A stored result is not permission to rewrite a policy, and an unadmitted or malformed result must not influence optimization.

4. Freeze a snapshot

Freeze the currently admitted results into an immutable, content-addressed manifest:

bro eval snapshot freeze
bro eval snapshot get <EVIDENCE_ROOT>

The evidence root pins the exact set used for a later decision. This prevents a candidate from changing meaning when new results arrive.

5. Consume the evidence explicitly

Optimization can advance its controller from admitted evaluation history. status is read-only; run authorizes one controller step, may enable adaptive mode, and can publish a policy change without a separate approval:

bro optimize status --policy auto
bro optimize run --policy auto

Policy compilation can also pin a frozen evaluation snapshot:

bro policy compile \
  --eval-snapshot <EVIDENCE_ROOT> \
  --output candidate-policy.yaml
bro policy diff policy-lock.yaml candidate-policy.yaml

Compilation does not make the candidate active. Review it and use bro policy publish candidate-policy.yaml when it should serve traffic. The optimizer is a separate, explicitly authorized workflow and can publish its own transition during bro optimize run.

Reproducibility boundary

A defensible evaluation decision needs all of the following to stay identifiable:

  • the sealed subject and its evidence digest;
  • the evaluator and materially relevant evaluator configuration;
  • the submitted result and admission decision;
  • the frozen evidence root consumed by optimization or compilation;
  • the active and candidate policy digests being compared.

Keep private prompts, responses, and artifacts out of the subject unless the evaluator requires them. Redaction changes the evidence and therefore belongs before sealing, not as an undocumented post-processing step.

What evaluation does not establish

  • A successful transport or HTTP status does not establish task success.
  • Lower token cost does not establish acceptable quality.
  • A trace or request receipt does not become evaluation evidence merely because it is detailed.
  • One admitted result does not prove a general routing policy is better.
  • A compiled candidate is not active until it passes publication; an authorized optimizer step can publish its own transition.

Use repeated, comparable subjects and an evaluator aligned with the workload objective. Keep exploratory conclusions separate from policy changes you are prepared to serve.

How is this guide?

On this page