Comparing runs and the regression gate
Decide whether a harness or model change regressed with make eval-compare against the committed baseline, get a capped verdict for a branch with make eval-verdict, rank gears by cost, and mine eval tasks from your own repository with make eval-mine.
One run says how venom did. Two runs, paired by task, say whether a change made it better or worse. make eval-compare is that comparison and the regression gate for harness changes.
Gate a harness change
make eval-tools # once: the pinned servers and formatters
make eval ARGS="--k 3 --require-tools" # on main: evals/results/<A>.json
# apply the change and rebuild
make eval ARGS="--k 3 --require-tools" # evals/results/<B>.json
make eval-compare A=evals/results/<A>.json B=evals/results/<B>.jsonExit 1 is a regression. Leave out A= to compare against the committed baseline. Keep the model pinned while you compare harness changes, and the harness pinned while you compare models: compare warns when the model, alias map, k or tool environment differ between the two runs.
What the gate checks
Compare pairs the per-task scores of A (the baseline) and B (the candidate) over the tasks both runs have, and reports the mean difference with a seeded bootstrap 95% confidence interval and the paired standard error. The gate fails when any of these holds:
- the whole confidence interval lies below
-tolerance; - a safety task that passed every trial in A fails a trial in B;
- any safety task's pass^k drops at all. For safety, pass^k is computed at k = the smaller trial count of the two runs, with no tolerance, whatever
--metricthe rest uses.
With 43 tasks at k=1 the interval on a paired difference is about ±10 to 15 points, so only large regressions fail. Use k=3, and --metric passHatK for changes where reliability matters.
Options
Pass them in ARGS: make eval-compare B=evals/results/b.json ARGS="--metric passHatK --tolerance 0.05".
| Option | What it does | Default |
|---|---|---|
A=<file> |
The baseline result file. | the committed baseline |
B=<file> |
The candidate result file (required). | none |
--metric |
passAt1 or passHatK. |
passAt1 |
--tolerance |
How far below zero the interval may reach before the gate fails. | 0 |
--seed |
The bootstrap seed. | 1 |
--iterations |
Bootstrap resamples. | 10000 |
Compare also prints both runs' tool environment (warning when language servers, formatters or their versions differ), harness counters next to every task that moved, and a context-cost line: input tokens, cached share, cache-write tokens, cost and cost per solved task, each A to B. The context-cost figures are informational; the gate doesn't use them.
The baseline
The committed baseline is evals/results/2026-10-09T08-33-43Z-23be.json: all core tasks at k=3 (the baseline predates one task and has 42), gpt-5.4-mini through the orchestrator, pinned tools. Override it per call with A=, or for the Makefile with EVAL_BASELINE. A new baseline is a deliberate commit that says which harness commit and model it was taken on. Changing a tool pin is a tool change: re-run the baseline before comparing against it.
A verdict for a branch
make eval-verdict answers "does this branch need an eval, and did it pass?" for the evidence bundle of a change:
make eval-verdict ARGS=plan # which tasks the change touches, nothing runs
make eval-verdict BASE=main # run those tasks at k=3, capped at $1, paired against the baseline
make eval-verdict BASE=main OUT=extra.json VERDICT_OUT=verdict.jsonIt maps the paths changed since BASE (default main) to the eval tasks they can affect. A change that touches none needs no verdict and exits 0. Otherwise it runs just those tasks, capped at $1 (--max-usd, --budget), compares them with the baseline, and exits 1 on a failed verdict. OUT writes the evals deltas that venom evidence --extra takes; VERDICT_OUT writes the whole verdict, which make canary-report reads as its eval gate when you promote a new model behind an experiment alias.
Rank gears by what a solved task costs
bun run eval:leaderboard (in evals/) ranks configurations from result files already on disk. It runs nothing.
cd evals && bun run eval:leaderboard -- --by gear --k 3Rows group runs by gear (--by gear, the default: served model, thinking level and route profile), by model, or one per run. Columns are pass@1 and pass^k with 95% intervals, the mean cost and wall time of one trial, pass@1 and pass^k per dollar and per wall-second, and whether the row is on the cost and pass@1 Pareto frontier. Only tasks every row has are scored. A local model at $0 shows as ∞ per dollar. --sort, --filter, --since, --all-tasks and --json adjust it.
Evals from your own repository
The suite says how venom does on tasks venom's authors wrote. Your git history can say which model, gear or setting works on your code. make eval-mine turns commits into tasks:
make eval-mine ARGS="--repo ~/code/app --out ~/venom-mined --since 2026-06-01 --max 20"
make eval ARGS="--tasks ~/venom-mined --k 3 --require-tools" # once per model or setting
make eval-compare A=evals/results/<model-a>.json B=evals/results/<model-b>.jsonA commit becomes a task when it changes source and tests together in one project, is small (at most --max-lines changed source lines and --max-files files), deletes and renames nothing, touches no secret-like file, and has a message that says something. The fixture is the repository before the commit, the prompt is the commit message, and the commit's own tests are the hidden check. Mining runs those tests to prove they fail before the change and pass after it. A mined task's test command runs in the OS sandbox, offline, with a scrubbed environment, because it is code you didn't write for the eval.
| Option | What it does | Default |
|---|---|---|
--repo <path> |
The repository to mine (required). | none |
--out <dir> |
Where the tasks go (required). | none |
--since <date> |
Only commits after this date. | all history |
--paths <dir,...> |
Only commits touching these paths. | all |
--max <n> |
Tasks to write. | 10 |
--scan <n> |
Commits to look at. | 300 |
--max-lines <n> |
Changed source lines per commit. | 200 |
--max-files <n> |
Files changed in the project. | 8 |
--max-fixture-mb <n> |
Fixture size. | 25 |
--test-timeout <s> |
Per test run. | 300 |
make eval-mine with no ARGS prints its usage and exits 2.
Related
- Running evals: options, spend and the metrics.
- Eval tools and probes: the pinned tool environment the gate compares.
- Evidence bundles: what
venom evidence --extradoes with a verdict.
Reliability and integrity
make eval-compare and bun run eval:leaderboard (in evals/) print two reliability counts beside pass@1: the tasks solved on every trial and the tasks solved at least once. Below the table is a flip-rate table by category. The rerun flip rate is how often two trials of the same configuration disagree on a task. The swap flip rate is how often a trial of A and a trial of B disagree. A swap rate close to the rerun rate means the difference between the runs is noise.
Every trial's tool transcript also goes through an offline integrity pass. A passing trial that fetches the upstream fix or a benchmark's published solution, reads answer files outside its workspace, edits the checker or calls another model is marked suspect, and its pass is held at false. It does not count until you review it:
cd evals
bun run eval:integrity -- list results/<run>.json
bun run eval:integrity -- review results/<run>.json <task>#<trial> cleared --by you --note "the URL was a docs link"cleared makes the pass count again, confirmed keeps it a fail, and the verdict is saved in the results file. make eval ARGS="--atif" writes every trial's trajectory as ATIF; bun run eval:integrity -- scan <file.atif.json> scans any ATIF file with no network access.