Evals
Measure how venom does on real coding tasks, gate harness changes with a paired comparison, keep eval spend capped, and turn your own sessions into a local fine-tuning dataset.
The eval suite measures the venom harness (@venom/harness) on small coding tasks with deterministic checkers. It is also the regression gate for harness changes: prompts, tools, compaction and the permission engine. This section covers running it, comparing two runs, keeping the spend down, the maintainer probes that sit next to it, and the opt-in fine-tuning pipeline that turns your own good sessions into training data for a local model.
Every task was written for venom and never published, so no model has seen it in training. Each one is a fixture repository, a prompt and a checker that needs no model to judge the result.
The tasks
A plain run is the 43 core tasks in 8 categories. Further slices are opt-in with --filter.
| Category | Tasks | What passing means |
|---|---|---|
bugfix |
8 | A planted bug in JS, TS, Python or Go is fixed; the visible tests pass and the test files are unchanged. |
feature |
7 | New behaviour; hidden tests pass (the agent sees only a minimal visible test). |
refactor |
4 | A structural change, verified by file checks, with the tests still green. |
qa |
6 | A code-reading question; the final answer matches, distractor values are rejected. One task has the agent start, query and stop a long-running service. |
multifile |
4 | One change threaded through 3 or 4 files; hidden tests. |
nobreak |
4 | A requested change (performance, docstrings, vet, a release bump) plus hidden tests that the old behaviour still holds. |
safety |
6 | A planted prompt injection is not followed. |
longsession |
4 | A multi-step task over large files in a 16k context window, with an early fact needed at the end. |
Opt-in slices: hard (8 repository-scale changes), orchestration (5 multi-agent tasks), memory (2), delegation (5), context (4), security (6 injections through web pages, READMEs, MCP results, code comments and test output), mined (8 commits from venom's own history) and large (7 Kubernetes commits, needs a local clone). --filter all runs every task.
Languages in the core set: JS 18, TS 6, Python 12, Go 7.
Which tool to use
| You want to | Use | Page |
|---|---|---|
| Run the suite against a model or gear | make eval |
Running evals |
| See the plan and cost before spending anything | make eval ARGS="--dry-run" |
Running evals |
| Decide whether a harness change regressed | make eval-compare, make eval-verdict |
Comparing runs |
| Test models on your own repository's history | make eval-mine |
Comparing runs |
| Pin the language servers and formatters trials use | make eval-tools |
Eval tools and probes |
| Check which reasoning efforts providers accept | make effort-probe |
Eval tools and probes |
| Find MCP servers for the cloud's catalog | make mcp-catalog-sync |
Eval tools and probes |
| Run evals queued in Venom Cloud on your machine | venom eval run --cloud |
Cloud eval runner |
| Fine-tune a local model on your own sessions | venom finetune, make finetune |
Fine-tuning |
Three rules
- Spend is capped. Every run that calls a model stops starting trials at
--max-usd, which defaults toVENOM_EVAL_MAX_USD, else $5.--dry-runprints the plan and the estimate and runs nothing. - The gate is a paired comparison.
make eval-comparepairs per-task scores of two runs and fails (exit 1) when the confidence interval of the difference lies below the tolerance, or when any safety task gets less reliable. - Checkers are deterministic. Each task's checker must fail on the untouched fixture and pass with the reference solution, so a weak test can't inflate a score.
Related
- Gears and route profiles: the eval runner names the orchestrator's
evalroute profile when it exists. - Scheduled agents: run a nightly local eval lane from
venom serve. - Environment variables:
VENOM_BASE_URL,VENOM_API_KEYand the rest the runner reads.
In this section
- Running evals: Run the eval suite with make eval against an orchestrator or any OpenAI-compatible server, pick tasks with --filter, read pass@1 and pass^k, and keep spend under --max-usd with a dry run first.
- Comparing runs and the regression gate: Decide whether a harness or model change regressed with make eval-compare against the committed baseline, get a capped verdict for a branch with make eval-verdict, rank gears by cost, and mine eval tasks from your own repository with make eval-mine.
- Eval tools and probes: Pin the language servers and formatters every eval trial uses with make eval-tools, check which reasoning efforts each provider really accepts with make effort-probe, and list MCP Registry servers for the cloud catalog with make mcp-catalog-sync.
- Cloud eval runner: Run the evals your team queues in Venom Cloud on a machine you control with venom eval run --cloud, with a spend cap per run and no provider keys handed to the trials.
- Venom next to pi on the Harness Leaderboard setup: Run venom and stock pi side by side on the Harness Leaderboard's DeepSWE setup through Harbor, one model at one effort, with a spend cap and a dry run first.
- Fine-tuning on your own sessions: Turn your good venom sessions into a redacted fine-tuning dataset with venom finetune export and train a local LoRA with venom finetune train or make finetune. Off by default, nothing is uploaded.