---
title: Running evals
description: Run the eval suite with make eval against an orchestrator or any OpenAI-compatible server, pick tasks with --filter, read pass@1 and pass^k, and keep spend under --max-usd with a dry run first.
section: evals
order: 10
audience: [user, operator, builder, agent]
sources: [evals/AGENTS.md, evals/src/cli.ts, evals/src/budget.ts, evals/src/tasks.ts, Makefile]
features: []
---

`make eval` runs the eval suite against `VENOM_BASE_URL` and writes a result file per run. It builds the harness and installs the suite's dependencies first, so from a fresh checkout it is one command.

## A first run {#example}

```bash
make eval-tools                                        # once: the pinned language servers and formatters
make eval ARGS="--k 1 --filter safety --dry-run"       # the plan and cost estimate, nothing runs
make eval ARGS="--k 1 --filter safety"                 # 6 tasks, one trial each
```

The run prints one line per trial (`PASS`, `FAIL` or `SKIP`), then a summary, and writes `evals/results/<run id>.json` and a markdown report next to it. A run id is the start time to the second plus a random 4-hex suffix, such as `2026-10-09T04-15-42Z-3fa1`; a name already taken gets `-2`, `-3`, and so on, so runs never overwrite each other.

For a decision, run every core task three times:

```bash
make eval ARGS="--k 3 --require-tools"
```

## The endpoint {#endpoint}

The runner reads the same environment as the CLI.

| Variable | What it does | Default |
|---|---|---|
| `VENOM_BASE_URL` | The OpenAI-compatible endpoint trials call. | `http://localhost:8787/v1` |
| `VENOM_API_KEY` | The key sent to it. | none |
| `VENOM_MODEL` | The default `--model`. | `venom-default` |
| `VENOM_FAST_MODEL`, `VENOM_SMART_MODEL` | The trials' fast and smart model tiers. | the harness defaults |
| `VENOM_ALIAS_MAP` | Maps venom aliases to real models when you call a provider directly, without the orchestrator: `venom-default=gpt-5.4-mini,venom-fast=gpt-5.4-nano`. | none |
| `VENOM_ORCH_URL` | Points the runner's admin calls at another orchestrator root than the one `VENOM_BASE_URL` implies. | derived |
| `VENOM_ADMIN_TOKEN` | With a reachable orchestrator, each trial gets its own short-lived key, so cost, tokens, cached tokens and time to first token are exact per trial, subagents included. The key is deleted after the trial. | none |
| `VENOM_EVAL_PROFILE` | The orchestrator route profile trials name (see [Spend](#spend)). | `eval` when the orchestrator lists it |
| `VENOM_EVAL_MAX_USD` | The default spend cap. | `5` |

Without an orchestrator, tokens come from the session's own usage and cost is priced from the alias map's target at list price.

## Options {#options}

Pass them through `ARGS`: `make eval ARGS="--k 3 --filter hard --concurrency 2"`.

| Option | What it does | Default |
|---|---|---|
| `--k <n>` | Trials per task. Use 3 for decisions; 1 is a smoke run. | `1` |
| `--model <id>` | The model or alias trials use. | `VENOM_MODEL`, else `venom-default` |
| `--filter <terms>` | Comma-separated terms: a category, a language, a tag or part of a task id. `hard`, `security` and the other opt-in slices run only when named; `all` runs everything. | the 43 core tasks |
| `--concurrency <n>` | Trials at once. | `4` |
| `--timeout <s>` | Wall-clock limit per trial; the session is aborted at the deadline. | `300` |
| `--out <dir>` | Where result files go. | `evals/results` |
| `--max-usd <usd>` | The spend cap; `0` or `inf` removes it. | `VENOM_EVAL_MAX_USD`, else `5` |
| `--dry-run` | Print the plan, the per-task estimate and the cap, then exit. | off |
| `--profile <name>` | The orchestrator route profile; `""` for none. A named profile the server lacks exits 2. | `VENOM_EVAL_PROFILE`, else `eval` when listed |
| `--require-tools` | Exit 2 before any trial unless every pinned server and formatter is found at its pinned version. Use it for any run you will compare or commit. | off |
| `--thinking <level>` | `off`, `minimal`, `low`, `medium`, `high` or `xhigh`. | per-model default |
| `--mode <mode>` | The permission mode trials run in: `accept-edits`, `default` or `auto`. | `accept-edits` |
| `--edit-tool <tool>` | `auto`, `edit`, `apply_patch` or `both`. | `auto` |
| `--context-file <kind>` | `existing` (the fixture's own memory files), `none`, or `generated` (venom's project analysis written first). | `existing` |
| `--tasks <dir>` | Another task directory, such as tasks mined from your repository. | `evals/tasks` |
| `--keep` | Keep each trial's temp workspace and its session transcript. | off |
| `--list` | List the selected tasks and exit. | off |
| `--include-quarantined` | Also run tasks quarantined as flaky. | off |
| `--no-lsp`, `--no-format` | Turn off LSP diagnostics or the formatter after edits (ablations). | on |
| `--no-overthink` | Turn off the analysis-paralysis nudges (an ablation). | on |
| `--arm <arm>` | Which harness the trials run: `default` (venom as it ships), `lean` (venom's permission engine, compat and edit tools, without the extra tools and prompt sections) or `stock-pi` (pi's own prompt and its four tools, no permission engine: leave out the safety and security slices). For ablations; compare arms on the same tasks, model and k. `node src/arm-prompt.ts` prints each arm's fixed prompt. | `default` |
| `--system-append <file>` | Appends the file's text to every trial's system prompt, after venom's static sections. For measuring a candidate prompt block, such as a per-model note, against a run without it. | off |

## How a trial runs {#trial}

1. The fixture is copied to a fresh temp directory and its setup steps run.
2. A harness session starts there with an empty temp home (nothing from your own `~/.claude` or `~/.venom`), an untrusted project and accept-edits mode. Project memory such as `AGENTS.md` is still loaded.
3. Approvals go to a strict headless policy: the project's own test, build and vet commands are allowed, nothing else. No installs, network or git writes. It mirrors a CI run with nobody present.
4. One prompt runs with the timeout.
5. Hidden tests are overlaid, the checks run, and the trial records pass or fail, tokens, cached tokens, turns, tool calls, denials, wall time and cost.

## Reading the results {#metrics}

| Metric | Meaning |
|---|---|
| pass@1 | The per-task success rate, averaged over tasks, with its standard error. |
| pass^k | The chance that all k trials of a task pass, averaged over tasks. An agent gets one try per request, so reliability matters more than pass@k. |
| pass@k | The chance that at least one of k trials passes (also recorded). |
| cost per solved task | Total cost divided by passing trials. |

The markdown report also lists, per trial, where venom stopped something: guard firings, permission denials by reason, sandbox refusals and edit-tool failures, so a regression can be put on the harness or ruled out without reading transcripts.

## Spend {#spend}

Evals are the most expensive thing to run by accident: a full k=3 core run costs about $1.20 to $2.30 on a small model. Every run is capped.

- **Estimate first.** Before the first trial the runner prints an estimate: per task, the mean cost per trial in the newest earlier result with the same model, thinking level and context-file setting, times k. `--dry-run` prints it with the plan and the cap and exits. Check it before any large run: `make eval ARGS="--k 3 --filter hard --dry-run"`.
- **The cap in the runner.** A trial starts only while the finished trials' cost, plus the expected cost of running ones, plus this one's stays within `--max-usd`. Trials past the cap print as `SKIP` and are recorded as skipped, with a budget line in the report. Repetitions are cut before tasks. A capped run is partial: compare it only on what it ran.
- **The cap at the orchestrator.** With `VENOM_ADMIN_TOKEN`, each trial's key carries a hard daily budget of what the run may still spend (at least $0.10), so a runaway trial is refused with `429 budget_exceeded` before the runner sees its cost.
- **The flex tier.** Trials name the orchestrator's `eval` route profile when it is listed, which asks OpenAI models for `service_tier: flex`: half price, slower. With orchestrator accounting, a trial's cost is what the orchestrator priced each request at, so a flex request counts at the flex rate. Compare runs made on the same profile.

> [!TIP]
> Spend less by running fewer tasks (`--filter`) or a lower fixed k for a smoke run. The runner does not stop repetitions early when trials agree, because that biases the gate.

## Flaky tasks {#flaky}

A task whose checker gives different verdicts on the same workspace is quarantined in `evals/quarantine.json`. The runner leaves quarantined tasks out and lists them; `--include-quarantined` runs them anyway. The detector re-runs each task's checks on the untouched fixture and on the reference solution, with no model and at no cost:

```bash
cd evals && node src/flake-cli.ts --k 5 --filter all
```

## A nightly local lane {#nightly}

`node src/nightly-cli.ts` (in `evals/`) runs eight quick core tasks at k=3 against a model on this machine (Ollama's `http://127.0.0.1:11434/v1` by default, `qwen3:8b` and `qwen3:4b`) and compares each night with the last. A URL that is not loopback is refused, so the lane stays at $0 and offline. Results and `history.jsonl` go to `~/.venom/evals/nightly/`. Schedule it from `venom serve` with a job on the `local` gear; see [Scheduled agents](/docs/automation/schedules.md).

## Troubleshooting {#troubleshooting}

| Symptom | Cause and fix |
|---|---|
| Exit 2 before any trial with `--require-tools` | A pinned language server or formatter is missing or at another version. Run `make eval-tools`. |
| Exit 2: profile not found | `--profile` named a route profile the orchestrator doesn't have. Pass `--profile ""` or add the profile. |
| Many `SKIP` lines | The cap was reached. Raise `--max-usd`, or run fewer tasks. |
| Costs show as 0 or estimated | No orchestrator accounting. Set `VENOM_ADMIN_TOKEN` and point `VENOM_BASE_URL` at the orchestrator. |
| A `large` task is listed as not judged | It needs a local clone under `$VENOM_EVAL_REPOS`. |

## Related

- [Comparing runs](/docs/evals/compare.md): the paired gate and the committed baseline.
- [Eval tools and probes](/docs/evals/tools-and-probes.md): `make eval-tools`.
- [Gears and route profiles](/docs/configuration/gears.md).
