Failures and jobs
Find out what made agent sessions and requests fail, grouped by cause with example sessions, and inspect the relayed jobs the cloud dispatched to daemons.
Two pages for when things go wrong: Failures (/failures) groups failed sessions and requests by cause; Jobs (/jobs) lists the relayed jobs behind requests, one per try. Both need logs:read (owner, admin, developer, viewer).
Failures: /failures
Open Failures to see what made venom's sessions and requests fail over the last 7, 30 (default) or 90 days. It reads error codes and venom's session reports first (tool errors, loop guards, failed session summaries), so most failures are explained without a model.
/failures?days=7&gear=cheapThe page shows, top to bottom:
- A finding: one sentence about the biggest cause and any spike.
- Figures: failures, sessions affected, number of causes, and "Unexplained" (failed with no known cause).
- Failures per day by cause: stacked bars, the top causes plus "Other causes", with spike days marked.
- One card per cause: its count over time, the kinds inside it, the newest example sessions (each opens in the replay debugger), the day it spiked, and the settings changes in the two days before that day (or a note to check providers and daemons when nothing changed).
- The classifier, for failures nothing structured explains.
When the window holds more than 20,000 failed requests, the newest 20,000 are counted and the page says so. A Gear filter narrows everything to one gear; a link opens the agent scorecard.
Causes
| Cause | Counts |
|---|---|
| Tool errors | A tool call failed: a command exited non-zero, a file was missing, an edit did not apply, a permission was denied. |
| Loops and stuck runs | venom's loop guards fired, or the stuck detector shifted gears. |
| Refusals | The model declined to answer. |
| Content filters | A prompt or response filter, the secret scanner or a provider's content filter stopped a request. |
| Timeouts | A 408 or 504, a dispatch timeout, or a daemon that never acknowledged. |
| Provider errors | A 5xx, a 429, a broken stream or a lost daemon. |
| Budget blocks | A spend guard, the org's cap or a key's budget refused the request. |
| Allowlist blocks | Model access, a route profile or a URL allowlist refused the request. |
| Context overflow | The conversation no longer fit the model's context window, or the request was too large. |
| Other | An error the structured signals do not name. |
The classifier
For failed or abandoned sessions with no structured cause, a model can read the session summary's goal and result (redacted by venom before upload, cut to 1,200 characters) and name one cause. It is off by default.
| Setting | Default | Range |
|---|---|---|
| Classifier | off | on or off |
| Gear | the org's default gear | one of your organisation's gears, or a venom built-in unless your organisation allows only its own; its classifier role is used, else its summary role, workers or main model. A gear that cannot run now is listed greyed out with the reason |
| Daily cap | 50 | 1 to 500 sessions per UTC day |
"Classify now" runs it at once instead of waiting for the next cron run; sessions it can't label stay "other". Changing the settings or running it needs org:manage (owner, admin) and is audited as failures.settings.
| Message | Meaning |
|---|---|
| "Today's cap is used up" | The daily cap was reached; the rest wait for tomorrow (UTC). |
| "Nothing to classify." | No unexplained failed session is waiting. |
| "Your role cannot change this (org:manage)." | Only owners and admins change the classifier. |
Jobs: /jobs
A relayed request becomes one or more jobs: the first attempt, failovers to the next daemon, hedges. Jobs lists the newest 500, metadata only, kept 7 days.
| Filter | Values |
|---|---|
| State (pills) | all (default), queued, dispatched, running, succeeded, failed, cancelled |
| Age | last hour, last day, last 7 days |
| Daemon | click a daemon name in the table |
| Key | click a key name in the table |
Columns: created, state, model, daemon (with a link to its page), key, route, and time to first byte and total. Click a created time to open the request's trace at /requests/{id} (see Requests).
Direct requests (the client sent to the daemon with a ticket) have no job. Find them in Requests with served:direct.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
| Failures is empty | No failed requests, tool-error or loop reports, or failed session summaries in the window. Session summaries arrive only when session history collection is on (see Session history). |
| Many "Unexplained" failures | Turn the classifier on, or open the example sessions. |
| "No relayed jobs yet" | All requests went direct or were served by the cloud's own provider keys. |
| "Showing the newest 500" | Narrow by state, daemon or age to see older jobs. |
Related
- Sessions, live feed and activity: the replay debugger the examples open in.
- Requests and request traces
- Alerts: get told when errors burst.