Skip to content
Docs menu

Cloud eval runner

Run the evals your team queues in Venom Cloud on a machine you control with venom eval run --cloud, with a spend cap per run and no provider keys handed to the trials.

View .md

Venom Cloud's console can queue eval runs (golden tasks, gear A against gear B, a library prompt or skill under test), but the cloud has no sandbox and runs no trial itself. venom eval run --cloud turns a machine you control into the runner that does.

Start a runner

venom cloud login                                        # once: the key the runner uses
cd ~/src/venom                                           # a venom checkout with evals/ installed
venom eval run --cloud --name ci-box --max-usd 10

The runner polls the cloud for runs queued to it, so nothing listens on the machine. It says hello, claims a queued run every 5 seconds, runs the eval suite once per variant (gear A, B, and so on), posts progress at least every 15 seconds, and posts the result: per task, each trial's pass, cost, wall time, error and failed checks. The run then shows in the console's Evals page.

Flags

Flag What it does Default
--cloud Required: run the cloud's queue. none
--name <runner> The runner's name in the console. this host's name
--max-usd <usd> The most one run may spend on this machine. The cloud never queues a bigger run to it, and a bigger one is refused here too. VENOM_EVAL_MAX_USD, else 5
--repos <dir> Where golden tasks' repository names resolve. the working directory
--concurrency <n> Trials at once. 4
--poll-ms <ms> How often to ask for a queued run (at least 1000). Not in the usage text. the interval the cloud suggests, else 5000
--once Run at most one queued run, then exit. off

venom eval with no arguments, help or --help prints the usage.

Requirements

  • A Venom Cloud login: venom cloud login, or VENOM_CLOUD_URL and VENOM_CLOUD_KEY in the environment. They go to the runner in its environment, never on its command line, where any local user could read them.
  • An eval suite: VENOM_EVALS_DIR, else the nearest evals/ directory above where you run it, installed once with make install-evals build-harness.

Spend and credentials

  • Each variant runs with --max-usd set to what the run has left, split over the variants still to run.
  • Trials call the cloud with the run's own key, which the cloud minted with a hard budget of the run's cap and revokes when the run ends. When the cloud answers progress with stop (capped, cancelled or lost), the trials in flight get 60 seconds to end, then are killed.
  • The trials' environment is the runner's minus every provider key, token and VENOM_* setting, plus the run's own endpoint and key. A model the cloud doesn't list fails with "No API key found" instead of quietly calling a provider with a key from your machine.

Golden tasks and variants

  • A golden task's repository is a git URL, cloned bare once into ~/.venom/evals/repos/ and fetched for each run with the machine's own git credentials, or a name under --repos. Its ref is resolved to a commit, the fixture is that commit, and its check is a command.
  • A variant's gear gives the model (its first concrete main model), the thinking level, the route profile and the fast model.
  • A library prompt under test wraps every task prompt the way /org:<name> <prompt> would; a library skill under test is installed in every trial's home.

Errors

Message Fix
not logged in to Venom Cloud (exit 2) Run venom cloud login, or set VENOM_CLOUD_URL and VENOM_CLOUD_KEY.
no eval suite found (exit 2) Run it inside a venom checkout, or set VENOM_EVALS_DIR.
... is not installed yet (exit 2) Run make install-evals build-harness in the checkout.
unknown option or needs a value (exit 2) Check the flags above.