Evals and experiments
Run the eval suite or your own golden tasks against your gears on a runner you control, and A/B test two gears or two prompt versions on live sessions from the Venom Cloud console.
Two ways to tell whether a change made things better:
- Evals run fixed tasks (venom's built-in eval suite or your team's golden tasks) with each gear, on a runner machine you control, and compare the pass rates. The cloud runs no trial itself.
- Experiments split real sessions between two gears, or two versions of a library prompt, and compare cost and quality signals.
| Page | What it is for | Read | Change |
|---|---|---|---|
/evals |
suggestions, the new-run form with a cost estimate, run history, golden tasks, runners | routing:read (owner, admin, developer, viewer) |
org:manage to queue runs (they spend), edit golden tasks and remove runners |
/evals/runs/{id} |
one run: status (refreshed while it runs), figures, verdicts, per-task results (?changed=1 shows only tasks whose result changed) |
routing:read |
org:manage to cancel |
/evals/golden/new |
a new golden task; ?share=<token> drafts one from a shared session |
org:manage |
|
/evals/golden/{id} |
edit or delete a golden task | routing:read |
org:manage |
/experiments |
the running gear experiment, the start form, past experiments, prompt tests, the default gear | routing:read |
org:manage |
/experiments/{id} |
one gear experiment's results | routing:read |
org:manage |
/experiments/prompts/{id} |
one prompt A/B test's results | routing:read |
org:manage |
Compare two gears with an eval run
Start a runner on a machine with the repositories and credentials the tasks need.
/evalsshows the same commands:venom cloud login venom eval run --cloud --name ci-box --max-usd 10On
/evals, in the new-run form, pick Gear A and Gear B, the task set (built-in categories such asbugfixandfeature, or your golden tasks), Trials per task (k) and a Cap (USD). The form shows a live cost estimate.Queue it. "Queued for ci-box. It starts at the runner's next poll."
Open the run at
/evals/runs/{id}for the verdict per pair of gears.
Evals
The run form
| Field | Default | Notes |
|---|---|---|
| Gear A, B, … | the org's first two gears | each gear's main model is shown under it |
| Task set | built-in (categories bugfix and feature ticked), or golden tasks when there is no suite |
opt-in categories are marked |
| Trials per task (k) | 3 | 1 to 10; pass^k needs k ≥ 2 |
| Cap (USD) | none | the run's key stops at it and the runner stops starting trials |
| Runner | the first runner | offline runners pick the run up when they start; a runner may set its own maximum |
When a run is claimed the cloud mints a key just for it, with hard daily and monthly budgets equal to the cap, valid for 12 hours. The key is revoked when the run finishes, is cancelled, or is lost (no progress from the runner). Cancel says "Cancelled. The run's key is revoked, so nothing more is billed to it."
Verdicts
| Verdict | Meaning |
|---|---|
| Better | B passed significantly more tasks than A |
| Worse | B passed significantly fewer |
| No clear difference | the difference is within noise |
| Nothing to compare | not enough finished trials |
Golden tasks
A golden task is a real task from your team, with a check that tells whether it was done.
| Field | Notes |
|---|---|
| Name | how it shows in runs and results |
| Prompt | what the agent is asked, as a person on the team asked it |
| Repository | a git URL the runner clones with its own credentials, or a folder name under the runner's --repos directory. No passwords or tokens in the URL |
| Ref | the commit, tag or branch the task starts from (before the fix) |
| Setup | optional commands run in the fresh checkout before the agent starts, one per line |
| Check command | run after the agent finishes; exit 0 means the task is done |
| Timeout (seconds) | the agent's time for one trial, 30 to 3600 |
Save as golden task on a session's page drafts one from that session.
Suggestions and library tests
After someone changes a gear on the Gears page, /evals suggests running the golden tasks with the gear before and
after; Run queues it and Dismiss hides it. A library prompt's page has a Test card that runs the golden
tasks with version A (baseline) and version B (the change) of that prompt. Removing a runner cancels its queued and
running runs.
Gear experiments
One gear experiment runs at a time per org. The start form on /experiments:
| Field | Default | Notes |
|---|---|---|
| Gear A (control), Gear B (candidate) | two of the org's gears; while it runs, the Gears page refuses a change that drops either | |
| On B (%) | 50 | 1 to 99: the share of new sessions on B |
| Who takes part | every client key | or the keys of chosen members, or of chosen key groups |
| Run for (days) | none | 1 to 90 |
| Or stop at (sessions) | none | with neither, it runs until stopped |
| Members may opt out | on | members opt out in venom with /gear experiment off |
| Name | optional |
A new venom session that starts in gear A (or with no gear), without --model, joins. Its arm comes from a hash of
the experiment id and session id, so a resumed session keeps its arm. Only counts are recorded, never content.
/experiments/{id} shows per arm: sessions, cost and tokens per session, latency p50 and p95, error, abandon and tool
error rates, stuck shifts per session and the scheduled quality checks on each arm's main model. Below 30 finished
sessions per arm there is no verdict. The verdict reads like "eco is 23% cheaper, no significant quality difference
(n=412 vs 398)".
| Control | Effect |
|---|---|
| Stop | ends the experiment |
| Make B the default (promote) | that arm's gear becomes the org's default gear and the experiment ends as promoted |
| Delete | removes an ended experiment and its sessions |
| Default gear | the gear new sessions start in unless the user picks one; or none |
Prompt A/B tests
From a library prompt's page, Test on live sessions starts a test of two versions:
| Field | Default | Notes |
|---|---|---|
| Prompt | the page's prompt | must have a published version |
| Version A (control) | the published version | |
| Version B (candidate) | a version number or the current draft | |
| On B (%), Who takes part, Run for (days), Or stop at (sessions), Name | as for gear experiments |
Both versions' text is copied when the test starts, so later edits never change it. One test runs per prompt.
Promoting an arm publishes its text as the next version (nothing is published when it already is the published
version) and ends the test. Results are at /experiments/prompts/{id} and listed on /experiments.
Troubleshooting
| Message | Cause and fix |
|---|---|
| "Your role cannot run experiments (org:manage)." | owners and admins start, stop and promote experiments |
409 experiment_running |
another gear experiment (or a test of this prompt) is running; stop it first |
409 not_published |
the prompt has no published version yet |
400 same_text |
both versions have the same text and parameters |
409 no_draft |
version B is "draft" but the prompt has none |
| A queued run never starts | its runner is offline; start venom eval run --cloud on the runner machine |
Related
- Red team and quality checks: scheduled model checks feed the experiment results.
- Team library and starter packs: the prompts you test.
- Venom Cloud