Skip to content
Docs menu

Evals

Measure how venom does on real coding tasks, gate harness changes with a paired comparison, keep eval spend capped, and turn your own sessions into a local fine-tuning dataset.

View .md

The eval suite measures the venom harness (@venom/harness) on small coding tasks with deterministic checkers. It is also the regression gate for harness changes: prompts, tools, compaction and the permission engine. This section covers running it, comparing two runs, keeping the spend down, the maintainer probes that sit next to it, and the opt-in fine-tuning pipeline that turns your own good sessions into training data for a local model.

Every task was written for venom and never published, so no model has seen it in training. Each one is a fixture repository, a prompt and a checker that needs no model to judge the result.

The tasks

A plain run is the 43 core tasks in 8 categories. Further slices are opt-in with --filter.

Category Tasks What passing means
bugfix 8 A planted bug in JS, TS, Python or Go is fixed; the visible tests pass and the test files are unchanged.
feature 7 New behaviour; hidden tests pass (the agent sees only a minimal visible test).
refactor 4 A structural change, verified by file checks, with the tests still green.
qa 6 A code-reading question; the final answer matches, distractor values are rejected. One task has the agent start, query and stop a long-running service.
multifile 4 One change threaded through 3 or 4 files; hidden tests.
nobreak 4 A requested change (performance, docstrings, vet, a release bump) plus hidden tests that the old behaviour still holds.
safety 6 A planted prompt injection is not followed.
longsession 4 A multi-step task over large files in a 16k context window, with an early fact needed at the end.

Opt-in slices: hard (8 repository-scale changes), orchestration (5 multi-agent tasks), memory (2), delegation (5), context (4), security (6 injections through web pages, READMEs, MCP results, code comments and test output), mined (8 commits from venom's own history) and large (7 Kubernetes commits, needs a local clone). --filter all runs every task.

Languages in the core set: JS 18, TS 6, Python 12, Go 7.

Which tool to use

You want to Use Page
Run the suite against a model or gear make eval Running evals
See the plan and cost before spending anything make eval ARGS="--dry-run" Running evals
Decide whether a harness change regressed make eval-compare, make eval-verdict Comparing runs
Test models on your own repository's history make eval-mine Comparing runs
Pin the language servers and formatters trials use make eval-tools Eval tools and probes
Check which reasoning efforts providers accept make effort-probe Eval tools and probes
Find MCP servers for the cloud's catalog make mcp-catalog-sync Eval tools and probes
Run evals queued in Venom Cloud on your machine venom eval run --cloud Cloud eval runner
Fine-tune a local model on your own sessions venom finetune, make finetune Fine-tuning

Three rules

  • Spend is capped. Every run that calls a model stops starting trials at --max-usd, which defaults to VENOM_EVAL_MAX_USD, else $5. --dry-run prints the plan and the estimate and runs nothing.
  • The gate is a paired comparison. make eval-compare pairs per-task scores of two runs and fails (exit 1) when the confidence interval of the difference lies below the tolerance, or when any safety task gets less reliable.
  • Checkers are deterministic. Each task's checker must fail on the untouched fixture and pass with the reference solution, so a weak test can't inflate a score.

In this section

  • Running evals: Run the eval suite with make eval against an orchestrator or any OpenAI-compatible server, pick tasks with --filter, read pass@1 and pass^k, and keep spend under --max-usd with a dry run first.
  • Comparing runs and the regression gate: Decide whether a harness or model change regressed with make eval-compare against the committed baseline, get a capped verdict for a branch with make eval-verdict, rank gears by cost, and mine eval tasks from your own repository with make eval-mine.
  • Eval tools and probes: Pin the language servers and formatters every eval trial uses with make eval-tools, check which reasoning efforts each provider really accepts with make effort-probe, and list MCP Registry servers for the cloud catalog with make mcp-catalog-sync.
  • Cloud eval runner: Run the evals your team queues in Venom Cloud on a machine you control with venom eval run --cloud, with a spend cap per run and no provider keys handed to the trials.
  • Venom next to pi on the Harness Leaderboard setup: Run venom and stock pi side by side on the Harness Leaderboard's DeepSWE setup through Harbor, one model at one effort, with a spend cap and a dry run first.
  • Fine-tuning on your own sessions: Turn your good venom sessions into a redacted fine-tuning dataset with venom finetune export and train a local LoRA with venom finetune train or make finetune. Off by default, nothing is uploaded.