---
title: Evals
description: Measure how venom does on real coding tasks, gate harness changes with a paired comparison, keep eval spend capped, and turn your own sessions into a local fine-tuning dataset.
section: evals
order: 0
audience: [user, operator, builder, agent]
sources: [evals/AGENTS.md, Makefile, cli/src/eval-command.ts, docs/finetune.md]
features: []
---

The eval suite measures the venom harness (`@venom/harness`) on small coding tasks with deterministic checkers. It is also the regression gate for harness changes: prompts, tools, compaction and the permission engine. This section covers running it, comparing two runs, keeping the spend down, the maintainer probes that sit next to it, and the opt-in fine-tuning pipeline that turns your own good sessions into training data for a local model.

Every task was written for venom and never published, so no model has seen it in training. Each one is a fixture repository, a prompt and a checker that needs no model to judge the result.

> [!NOTE]
> The suite runs from a venom source checkout (the `evals/` directory and its `make` targets). The installed `venom` CLI has two entry points into it: `venom eval run --cloud`, which runs evals queued in Venom Cloud from such a checkout, and `venom finetune`, which needs no checkout.

## The tasks {#tasks}

A plain run is the 43 core tasks in 8 categories. Further slices are opt-in with `--filter`.

| Category | Tasks | What passing means |
|---|---|---|
| `bugfix` | 8 | A planted bug in JS, TS, Python or Go is fixed; the visible tests pass and the test files are unchanged. |
| `feature` | 7 | New behaviour; hidden tests pass (the agent sees only a minimal visible test). |
| `refactor` | 4 | A structural change, verified by file checks, with the tests still green. |
| `qa` | 6 | A code-reading question; the final answer matches, distractor values are rejected. One task has the agent start, query and stop a long-running service. |
| `multifile` | 4 | One change threaded through 3 or 4 files; hidden tests. |
| `nobreak` | 4 | A requested change (performance, docstrings, vet, a release bump) plus hidden tests that the old behaviour still holds. |
| `safety` | 6 | A planted prompt injection is not followed. |
| `longsession` | 4 | A multi-step task over large files in a 16k context window, with an early fact needed at the end. |

Opt-in slices: `hard` (8 repository-scale changes), `orchestration` (5 multi-agent tasks), `memory` (2), `delegation` (5), `context` (4), `security` (6 injections through web pages, READMEs, MCP results, code comments and test output), `mined` (8 commits from venom's own history) and `large` (7 Kubernetes commits, needs a local clone). `--filter all` runs every task.

Languages in the core set: JS 18, TS 6, Python 12, Go 7.

## Which tool to use {#tools}

| You want to | Use | Page |
|---|---|---|
| Run the suite against a model or gear | `make eval` | [Running evals](/docs/evals/running.md) |
| See the plan and cost before spending anything | `make eval ARGS="--dry-run"` | [Running evals](/docs/evals/running.md#spend) |
| Decide whether a harness change regressed | `make eval-compare`, `make eval-verdict` | [Comparing runs](/docs/evals/compare.md) |
| Test models on your own repository's history | `make eval-mine` | [Comparing runs](/docs/evals/compare.md#your-repo) |
| Pin the language servers and formatters trials use | `make eval-tools` | [Eval tools and probes](/docs/evals/tools-and-probes.md) |
| Check which reasoning efforts providers accept | `make effort-probe` | [Eval tools and probes](/docs/evals/tools-and-probes.md#effort-probe) |
| Find MCP servers for the cloud's catalog | `make mcp-catalog-sync` | [Eval tools and probes](/docs/evals/tools-and-probes.md#mcp-catalog-sync) |
| Run evals queued in Venom Cloud on your machine | `venom eval run --cloud` | [Cloud eval runner](/docs/evals/cloud-runner.md) |
| Fine-tune a local model on your own sessions | `venom finetune`, `make finetune` | [Fine-tuning](/docs/evals/finetune.md) |

## Three rules {#rules}

- **Spend is capped.** Every run that calls a model stops starting trials at `--max-usd`, which defaults to `VENOM_EVAL_MAX_USD`, else $5. `--dry-run` prints the plan and the estimate and runs nothing.
- **The gate is a paired comparison.** `make eval-compare` pairs per-task scores of two runs and fails (exit 1) when the confidence interval of the difference lies below the tolerance, or when any safety task gets less reliable.
- **Checkers are deterministic.** Each task's checker must fail on the untouched fixture and pass with the reference solution, so a weak test can't inflate a score.

## Related

- [Gears and route profiles](/docs/configuration/gears.md): the eval runner names the orchestrator's `eval` route profile when it exists.
- [Scheduled agents](/docs/automation/schedules.md): run a nightly local eval lane from `venom serve`.
- [Environment variables](/docs/reference/environment.md): `VENOM_BASE_URL`, `VENOM_API_KEY` and the rest the runner reads.


## In this section {#in-this-section}

- [Running evals](/docs/evals/running.md): Run the eval suite with make eval against an orchestrator or any OpenAI-compatible server, pick tasks with --filter, read pass@1 and pass^k, and keep spend under --max-usd with a dry run first.
- [Comparing runs and the regression gate](/docs/evals/compare.md): Decide whether a harness or model change regressed with make eval-compare against the committed baseline, get a capped verdict for a branch with make eval-verdict, rank gears by cost, and mine eval tasks from your own repository with make eval-mine.
- [Eval tools and probes](/docs/evals/tools-and-probes.md): Pin the language servers and formatters every eval trial uses with make eval-tools, check which reasoning efforts each provider really accepts with make effort-probe, and list MCP Registry servers for the cloud catalog with make mcp-catalog-sync.
- [Cloud eval runner](/docs/evals/cloud-runner.md): Run the evals your team queues in Venom Cloud on a machine you control with venom eval run --cloud, with a spend cap per run and no provider keys handed to the trials.
- [Venom next to pi on the Harness Leaderboard setup](/docs/evals/harness-leaderboard.md): Run venom and stock pi side by side on the Harness Leaderboard's DeepSWE setup through Harbor, one model at one effort, with a spend cap and a dry run first.
- [Fine-tuning on your own sessions](/docs/evals/finetune.md): Turn your good venom sessions into a redacted fine-tuning dataset with venom finetune export and train a local LoRA with venom finetune train or make finetune. Off by default, nothing is uploaded.
