Skip to content
Docs menu

Running evals

Run the eval suite with make eval against an orchestrator or any OpenAI-compatible server, pick tasks with --filter, read pass@1 and pass^k, and keep spend under --max-usd with a dry run first.

View .md

make eval runs the eval suite against VENOM_BASE_URL and writes a result file per run. It builds the harness and installs the suite's dependencies first, so from a fresh checkout it is one command.

A first run

make eval-tools                                        # once: the pinned language servers and formatters
make eval ARGS="--k 1 --filter safety --dry-run"       # the plan and cost estimate, nothing runs
make eval ARGS="--k 1 --filter safety"                 # 6 tasks, one trial each

The run prints one line per trial (PASS, FAIL or SKIP), then a summary, and writes evals/results/<run id>.json and a markdown report next to it. A run id is the start time to the second plus a random 4-hex suffix, such as 2026-10-09T04-15-42Z-3fa1; a name already taken gets -2, -3, and so on, so runs never overwrite each other.

For a decision, run every core task three times:

make eval ARGS="--k 3 --require-tools"

The endpoint

The runner reads the same environment as the CLI.

Variable What it does Default
VENOM_BASE_URL The OpenAI-compatible endpoint trials call. http://localhost:8787/v1
VENOM_API_KEY The key sent to it. none
VENOM_MODEL The default --model. venom-default
VENOM_FAST_MODEL, VENOM_SMART_MODEL The trials' fast and smart model tiers. the harness defaults
VENOM_ALIAS_MAP Maps venom aliases to real models when you call a provider directly, without the orchestrator: venom-default=gpt-5.4-mini,venom-fast=gpt-5.4-nano. none
VENOM_ORCH_URL Points the runner's admin calls at another orchestrator root than the one VENOM_BASE_URL implies. derived
VENOM_ADMIN_TOKEN With a reachable orchestrator, each trial gets its own short-lived key, so cost, tokens, cached tokens and time to first token are exact per trial, subagents included. The key is deleted after the trial. none
VENOM_EVAL_PROFILE The orchestrator route profile trials name (see Spend). eval when the orchestrator lists it
VENOM_EVAL_MAX_USD The default spend cap. 5

Without an orchestrator, tokens come from the session's own usage and cost is priced from the alias map's target at list price.

Options

Pass them through ARGS: make eval ARGS="--k 3 --filter hard --concurrency 2".

Option What it does Default
--k <n> Trials per task. Use 3 for decisions; 1 is a smoke run. 1
--model <id> The model or alias trials use. VENOM_MODEL, else venom-default
--filter <terms> Comma-separated terms: a category, a language, a tag or part of a task id. hard, security and the other opt-in slices run only when named; all runs everything. the 43 core tasks
--concurrency <n> Trials at once. 4
--timeout <s> Wall-clock limit per trial; the session is aborted at the deadline. 300
--out <dir> Where result files go. evals/results
--max-usd <usd> The spend cap; 0 or inf removes it. VENOM_EVAL_MAX_USD, else 5
--dry-run Print the plan, the per-task estimate and the cap, then exit. off
--profile <name> The orchestrator route profile; "" for none. A named profile the server lacks exits 2. VENOM_EVAL_PROFILE, else eval when listed
--require-tools Exit 2 before any trial unless every pinned server and formatter is found at its pinned version. Use it for any run you will compare or commit. off
--thinking <level> off, minimal, low, medium, high or xhigh. per-model default
--mode <mode> The permission mode trials run in: accept-edits, default or auto. accept-edits
--edit-tool <tool> auto, edit, apply_patch or both. auto
--context-file <kind> existing (the fixture's own memory files), none, or generated (venom's project analysis written first). existing
--tasks <dir> Another task directory, such as tasks mined from your repository. evals/tasks
--keep Keep each trial's temp workspace and its session transcript. off
--list List the selected tasks and exit. off
--include-quarantined Also run tasks quarantined as flaky. off
--no-lsp, --no-format Turn off LSP diagnostics or the formatter after edits (ablations). on
--no-overthink Turn off the analysis-paralysis nudges (an ablation). on
--arm <arm> Which harness the trials run: default (venom as it ships), lean (venom's permission engine, compat and edit tools, without the extra tools and prompt sections) or stock-pi (pi's own prompt and its four tools, no permission engine: leave out the safety and security slices). For ablations; compare arms on the same tasks, model and k. node src/arm-prompt.ts prints each arm's fixed prompt. default
--system-append <file> Appends the file's text to every trial's system prompt, after venom's static sections. For measuring a candidate prompt block, such as a per-model note, against a run without it. off

How a trial runs

  1. The fixture is copied to a fresh temp directory and its setup steps run.
  2. A harness session starts there with an empty temp home (nothing from your own ~/.claude or ~/.venom), an untrusted project and accept-edits mode. Project memory such as AGENTS.md is still loaded.
  3. Approvals go to a strict headless policy: the project's own test, build and vet commands are allowed, nothing else. No installs, network or git writes. It mirrors a CI run with nobody present.
  4. One prompt runs with the timeout.
  5. Hidden tests are overlaid, the checks run, and the trial records pass or fail, tokens, cached tokens, turns, tool calls, denials, wall time and cost.

Reading the results

Metric Meaning
pass@1 The per-task success rate, averaged over tasks, with its standard error.
pass^k The chance that all k trials of a task pass, averaged over tasks. An agent gets one try per request, so reliability matters more than pass@k.
pass@k The chance that at least one of k trials passes (also recorded).
cost per solved task Total cost divided by passing trials.

The markdown report also lists, per trial, where venom stopped something: guard firings, permission denials by reason, sandbox refusals and edit-tool failures, so a regression can be put on the harness or ruled out without reading transcripts.

Spend

Evals are the most expensive thing to run by accident: a full k=3 core run costs about $1.20 to $2.30 on a small model. Every run is capped.

  • Estimate first. Before the first trial the runner prints an estimate: per task, the mean cost per trial in the newest earlier result with the same model, thinking level and context-file setting, times k. --dry-run prints it with the plan and the cap and exits. Check it before any large run: make eval ARGS="--k 3 --filter hard --dry-run".
  • The cap in the runner. A trial starts only while the finished trials' cost, plus the expected cost of running ones, plus this one's stays within --max-usd. Trials past the cap print as SKIP and are recorded as skipped, with a budget line in the report. Repetitions are cut before tasks. A capped run is partial: compare it only on what it ran.
  • The cap at the orchestrator. With VENOM_ADMIN_TOKEN, each trial's key carries a hard daily budget of what the run may still spend (at least $0.10), so a runaway trial is refused with 429 budget_exceeded before the runner sees its cost.
  • The flex tier. Trials name the orchestrator's eval route profile when it is listed, which asks OpenAI models for service_tier: flex: half price, slower. With orchestrator accounting, a trial's cost is what the orchestrator priced each request at, so a flex request counts at the flex rate. Compare runs made on the same profile.

Flaky tasks

A task whose checker gives different verdicts on the same workspace is quarantined in evals/quarantine.json. The runner leaves quarantined tasks out and lists them; --include-quarantined runs them anyway. The detector re-runs each task's checks on the untouched fixture and on the reference solution, with no model and at no cost:

cd evals && node src/flake-cli.ts --k 5 --filter all

A nightly local lane

node src/nightly-cli.ts (in evals/) runs eight quick core tasks at k=3 against a model on this machine (Ollama's http://127.0.0.1:11434/v1 by default, qwen3:8b and qwen3:4b) and compares each night with the last. A URL that is not loopback is refused, so the lane stays at $0 and offline. Results and history.jsonl go to ~/.venom/evals/nightly/. Schedule it from venom serve with a job on the local gear; see Scheduled agents.

Troubleshooting

Symptom Cause and fix
Exit 2 before any trial with --require-tools A pinned language server or formatter is missing or at another version. Run make eval-tools.
Exit 2: profile not found --profile named a route profile the orchestrator doesn't have. Pass --profile "" or add the profile.
Many SKIP lines The cap was reached. Raise --max-usd, or run fewer tasks.
Costs show as 0 or estimated No orchestrator accounting. Set VENOM_ADMIN_TOKEN and point VENOM_BASE_URL at the orchestrator.
A large task is listed as not judged It needs a local clone under $VENOM_EVAL_REPOS.