Model quality, scorecard and red team
Schedule quality checks on the models your gears use, catch drift and swap degraded models, read the agent scorecard per gear or key group, and run the red-team suite from the console.
Three pages answer "are our models and agents still doing well?":
| Page | What it is for | Read | Change |
|---|---|---|---|
/quality |
scheduled quality checks: budget, flags, scoreboard, suites, auto-swap, run history | routing:read (owner, admin, developer, viewer) |
org:manage (owner, admin) |
/quality/suites/{id} |
the suite editor, with a Test button per case | routing:read |
org:manage |
/quality/runs/{id} |
one run's per-case results: answer, grade, latency | routing:read |
|
/scorecard |
how each gear's sessions went over 7, 30 or 90 days | usage:read |
org:manage for the outcome-report switch |
/redteam |
the red-team suite: score per category, every case, the trend | audit:read (owner, admin, developer) |
org:manage to run |
Turn on quality checks
Quality checks spend the org's money on a schedule, so they are off until an owner or admin turns them on with a monthly cap.
- Open
/quality. The Checks and budget card shows the status, this month's spend, an upper-bound estimate for a month and the runs per month. - Under Suites, add a starter suite (Basic reasoning
reasoning, Tool callingtoolsor JSON outputjson) and pick the models it runs against. - Set Monthly cap (USD), required and above $0, and turn checks on. "Quality checks are on, capped at $N a month."
A case starts only while the month's checks plus its estimate stay within the cap; a reached org spend cap stops runs too. Checks run with the org's internal key and are labelled, so they count toward spend but are left out of traffic figures.
Suites and cases
| Suite field | Default | Notes |
|---|---|---|
| Name, Description | ||
| Schedule | daily |
hourly (744 runs a month per model) or daily (31) |
| Models | each picked model runs every case on the schedule; the same ids as the Gears page | |
| Judge model | none | grades llm_judge cases; its calls count toward the cap |
| Run on schedule | on | paused suites keep their runs and scores |
Each case has a prompt, an optional system message, tools and max_tokens, and one grader:
| Grader | Passes when |
|---|---|
exact |
the answer equals the expected text (optionally ignoring case) |
contains |
the answer contains every expected string |
regex |
the answer matches the pattern |
json_schema |
the answer is JSON valid against the schema |
number |
the answer is the expected number, within an optional tolerance |
tool_call |
the model calls the named tool, optionally with the given arguments |
llm_judge |
the judge model says it passes |
Test in the editor runs one case against a model once, before you save.
Drift and auto-swap
The Scoreboard lists each model and suite with its score, a sparkline, latency p50, cost per run and status. A model is flagged degraded when:
- its pass rate over the last 3 runs is significantly lower (one-sided test at about p = 0.01) than over the 20 runs before, and at least 10 points lower, with enough graded cases on both sides. The flag clears when the recent pass rate is back within 5 points of the frozen baseline;
- or every one of the last 3 runs' median latency is above the baseline's 90th percentile and their median is at least 1.5 times the baseline median. It clears within 1.2 times.
The nav link's badge counts degraded models. Auto-swap (off for every role by default) lets you pick gear roles
that list two or more models: for those, a degraded model is left out of the role's list in /v1/gears, so venom
uses the next model, until the model recovers. Every swap and swap-back is in the audit log.
Scorecard
/scorecard compares groups of sessions with the org median. Query parameters: ?by=gear (default), ?by=team (by key group), or
?by=member (owners and admins only, and the page never ranks people), and ?days=7, 30 (default) or 90.
| Column | Meaning |
|---|---|
| Completed | sessions that ended cleanly, of those with a known end |
| Abandoned | sessions closed mid-run or after a failed run |
| Rework | sessions followed by another on the same branch within 24 hours, or that edited files back |
| Tool errors | failed tool calls of all tool calls venom reported |
| Stuck shifts | times the stuck detector shifted up a gear, per reported session |
| Escalations, Escalations worked | work re-run on a stronger tier, and how often the re-run passed |
| Cost / session, Cost / completed | everything the sessions cost, subagents included |
| First output | median time to the first answer or tool call of a run |
| Quality | pass rate of the gear's main models in the quality checks above |
Groups with fewer than 5 sessions get no comparison. The data comes from request logs plus session outcome reports: counts venom sends when a session ends (outcome, turns, tool calls and errors, never text). Reports are on by default; the switch on this page turns them off for the org.
Red team
/redteam runs known attacks against your guardrails: prompt injection, jailbreaks, data exfiltration, secret
leakage, policy bypass and tool abuse. It shows the latest run's score, a line per category, every case with the
guardrail code that caught it (linked to its /g/<code> page) or the catalog entry that would, and the trend with
each run's cost.
Run checks now runs the free cases (no model calls) on the org's guardrails, optionally for one key group. Cases that need a model run from the CLI, through your routing and capped:
venom redteam run --max-usd 1Runs are kept and audited as redteam.run. See Red team for the corpus and the CLI.
Troubleshooting
| Message | Cause and fix |
|---|---|
400 cap_required |
set a monthly cap above $0 before turning checks on |
| "Your role cannot change quality checks (org:manage)." | ask an owner or admin |
| "No gear role lists two or more models yet." | add a fallback model to a role on the Gears page, then allow auto-swap |
| The scorecard has no member view | grouping by member needs org:manage |
| Checks stopped mid-month | the monthly cap or the org's spend cap was reached |
Related
- Evals and experiments: pass rates on tasks, and live A/B tests.
- Guardrails and filters in the console: what the red team tests.
- Red team:
venom redteam runand the corpus.