Skip to content
Docs menu

Venom next to pi on the Harness Leaderboard setup

Run venom and stock pi side by side on the Harness Leaderboard's DeepSWE setup through Harbor, one model at one effort, with a spend cap and a dry run first.

View .md

The Harness Leaderboard runs coding-agent harnesses on the same models and the same tasks through Harbor, and compares them on paired tasks. bun run eval:harbor reproduces one cell of that setup for venom: DeepSWE v1.1 tasks, one model at one effort, with venom and stock pi (configured the way the board configures it) on the same tasks.

It is a maintainer tool in the source checkout (evals/), and it calls a model, so it is opt-in, capped and estimated first like the rest of the eval suite.

Before you start

  • A venom source checkout with make install-evals done.
  • Docker. A native amd64 Linux host matches the board. Docker Desktop on a Mac works with the deviations listed below.
  • A running orchestrator that serves the model, and VENOM_ADMIN_TOKEN. Every trial gets its own key from the orchestrator, limited to the run's model and to what the run may still spend, and the cost is read from the orchestrator's request log. Without the token, a real run refuses to start.
  • A venom release archive for linux x64, for example from make release VERSION=… RELEASE_TARGETS=linux/amd64 CLI_TARGETS=linux/x64.
  • uv, unless you pass your own Harbor 0.24.0 with --harbor.

Run it

Dry run first. It prints the tasks, the trials, the estimate per trial and the total against the cap, and starts nothing:

cd evals
bun run eval:harbor -- --model zai/glm-5.3-flash --effort low --subset smoke --dry-run

Then the run:

bun run eval:harbor -- --model zai/glm-5.3-flash --effort low \
  --tasks tengo-destructuring-bindings,httpx-streaming-json-iteration --max-usd 2.5 \
  --venom-archive ../dist/release/<v>/venom-<v>-linux-x64.tar.gz
Flag Default What
--model required the orchestrator model id both harnesses use
--effort low the reasoning effort for both (low, medium, high, …)
--subset / --tasks / --limit smoke the board's smoke, pilot or full subset, or named DeepSWE tasks, cut to the first n
--harnesses venom,pi either or both
--k 1 attempts per task and harness
--max-usd VENOM_EVAL_MAX_USD, else 5 the hard cap; trials that would cross it are skipped and recorded as skipped
--concurrency 2 trials at once
--venom-archive, --venom-sha256 the release venom runs from, and its expected hash
--base-url, --container-base-url VENOM_BASE_URL; the same with host.docker.internal the orchestrator for the runner, and as the containers reach it
--context-window, --max-output-tokens, --input-modalities 1000000, 131072, text,image pi's model entry, as the board's HFPi agent writes it
--dry-run print the plan and the estimate, start nothing

VENOM_HARBOR points at a Harbor binary and VENOM_HARBOR_CACHE moves the cache (default evals/.cache/harbor, which holds Harbor's venv, the task tarball and each run's job dirs).

What it compares

  • pi: pi-coding-agent 1.2.0 installed with npm ci from the board's own lockfile, with the model entry the board's HFPi agent writes.
  • venom: the release archive you give it, run headless (venom --print --mode json), with every model tier pinned to the run's model and the effort sent as reasoning_effort.
  • Cost: measured at the orchestrator, outside both harnesses, at list prices.
  • Scoring: the board's rules. A trial passes on reward 1. It is infra, not a failure, when the agent never got to run or the provider failed. The result has the resolve rate per harness with a 95% interval, the cost per trial and per solved task, and a paired gap.

Results go to evals/results/harbor/<run id>.json and .md. A run with fewer than 30 paired tasks says in the result that it is a smoke run, not a measurement. The board's own cells use 111 tasks with 3 attempts each and still report ties within five points.

Deviations on Docker Desktop

Each run records its deviations from the board's setup in the result:

  • The tasks ask for no network. Docker Desktop's kernel can't enforce that the way Harbor does it, so the runner gives them a public network.
  • Files go into the containers as a tar stream, because docker cp breaks the image's file system on Docker Desktop's containerd store.
  • The task images are amd64. On an Apple Silicon Mac they run emulated, so times are not comparable with the board's. Tasks whose verifier needs AVX (MongoDB in eicrud-keyset-pagination-cursor) can't be graded there.

The first smoke run

The first run (2026-11-10, evals/results/harbor/ven289-smoke.md) put 8 DeepSWE tasks through both harnesses once each, on zai/glm-5.3-flash at low effort, for $1.37. Docker Desktop stopped partway through, so only two tasks were graded for both. That is enough to show the setup works end to end. It is not enough to say which harness is better.

It did surface one thing about venom: --thinking alone sends no reasoning effort for a model venom has no thinking settings for. The runner sets VENOM_THINKING=<model>=<effort> so venom gets the same effort as pi.

A full DeepSWE comparison (111 tasks, 3 attempts each, both harnesses) costs about $85 at low effort and needs a native amd64 Linux host.