Skip to content
Docs menu

Alerts and notifications

Run saved log searches every 5 minutes with a signed webhook, and read the notification inbox for budget thresholds, spend forecasts, spend anomalies and key rotation.

View .md

Venom Cloud has two kinds of alert:

Page What fires Who reads Who changes
Alerts (/alerts) a saved log query matching at least N requests in a 5-minute run audit:read (owner, admin, developer) org:manage (owner, admin)
Notifications (/notifications) spend thresholds, forecasts, spend jumps, key rotation and model lifecycle reminders usage:read (every role) billing:manage or org:manage

Venom Cloud sends no email. Alerts go to the console and to webhooks you run.

Alerts: /alerts

An alert is a saved search. Every 5 minutes the cron runs it over the requests logged since its last run. A run that matches at least the threshold records a hit and, if the org set a webhook, posts it there.

Create one

Either turn on a built-in suggestion with one click, or fill in the form:

Field Rule
Name 1 to 100 characters
Query a plain search that parses clean: no time terms (the evaluator sets the window) and no pipes. Content terms (prompt:, response:, content:) work and need content:read. Examples: has:tools tool:bash, status:5xx key:ci*, prompt:"BEGIN OPENSSH"
Threshold matches in one run that make a hit; a whole number from 1 to 1,000,000, default 1

From Requests, Share, Create alert opens this form with the current search, its time range and pipes dropped.

Built-in suggestions

Name Query Threshold
Secret in a prompt has:content plus the secrets detector: a prompt holding something shaped like an API key, token or private key (sk-…, ghp_…, AKIA…, -----BEGIN … PRIVATE KEY-----) 1
Prompt blocked by a filter filter_action:block (a prompt filter returned 403 prompt_blocked) 1
Shell tool use tool:(bash OR shell OR *exec* OR *terminal* OR *command*) 1
Large prompt prompt_chars:>200k 1
Error burst status:5xx 50

The page

  • Alerts table: name, query (opens it in Requests), threshold, hits in the last 24 h, last run, and state: on, paused, or the last run's error. Pause, resume or delete each one.
  • Hits: newest first; each count opens exactly those requests in Requests. Hits newer than your last visit are unread; the sidebar's Alerts link shows the count, and opening the page marks them seen. Hits are kept 90 days.
  • Webhook: the org's URL, its signing secret's last four characters, Test, Remove.

How a run works

  • Each run reads the requests logged in (last checked, now − 60 s], at most 24 hours. The minute's lag lets sealed content land.
  • The watermark moves only when a run succeeds, so no request counts twice; a failed run keeps its error and retries next time. Pausing stops runs; resuming starts from now.
  • Alerts that read bodies (content terms, the secrets detector) open at most 500 candidates a run, newest first; the hit says truncated when there were more. Each such run writes one content.alert_evaluated audit row, never the text. Alerts on derived fields (tool:, prompt_chars:) read no bodies.
  • At most 50 alerts per org. Changes are audited: alert.created, alert.updated, alert.deleted, alert.webhook_set, alert.webhook_removed.

The webhook

The org's own http or https URL (on the hosted cloud, https to a public address only). One POST per hit, 5 s timeout, no redirects, no retries; the outcome shows on the hit. The signing secret (vwh_…) is made on the first save or on rotate, shown once, and stored sealed.

Headers:

Header Value
venom-event alert.hit, or alert.test from the Test button
venom-delivery the hit id
venom-signature t=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>.<body>">, keyed by the secret

Body (at most 50 request ids; never a prompt, response or matched text):

{
  "type": "alert.hit",
  "id": "42",
  "created_at": "2026-11-03T09:15:00.000Z",
  "org": { "id": "…" },
  "alert": { "id": "…", "name": "Shell tool use", "query": "tool:(bash OR shell)", "threshold": 1, "detector": null },
  "count": 3,
  "truncated": false,
  "window": { "since": "…", "until": "…" },
  "request_ids": ["req_…"],
  "links": { "alert": "https://example.com/alerts?hit=42", "explorer": "https://example.com/requests?q=id:…", "requests": ["https://example.com/requests/req_…"] }
}

Check the signature like a Stripe webhook: recompute, compare in constant time, refuse a t older than 5 minutes.

import { createHmac, timingSafeEqual } from "node:crypto";

export function verify(body: string, header: string, secret: string): boolean {
  const parts = Object.fromEntries(header.split(",").map((p) => p.split("=", 2)));
  const t = Number(parts.t);
  if (!t || Math.abs(Date.now() / 1000 - t) > 300) return false;
  const want = createHmac("sha256", secret).update(`${t}.${body}`).digest("hex");
  const got = String(parts.v1 ?? "");
  return got.length === want.length && timingSafeEqual(Buffer.from(got), Buffer.from(want));
}

Links use the deployment's VENOM_CLOUD_ORIGIN, else the cron's own origin.

Notifications: /notifications

The inbox behind the top bar's bell, newest first. Items newer than your last visit are unread; opening the page marks them seen.

Kind When Its button
Threshold spend passed a threshold of a budget or of the monthly spend cap; once per threshold and window Open budgets, or Open usage for the cap
Reached a threshold of 100 % or more the same
Forecast at the current rate, a budget or the cap runs out before its day or month ends; once a window the same
Spend anomaly a key, member, model or gear spent far more in an hour than its usual for that hour of day opens its evidence page
Key rotation a provider key is due for rotation, or was retired Rotate the key, or Open the provider
Lifecycle a model deprecation or an SLA event Open model lifecycle, or Open SLAs

Alert settings

Thresholds and forecasts apply to every budget and to the monthly spend cap; a changed limit or a new day or month starts over. Spend jumps need no budget. Checked every 5 minutes. Changing them needs billing:manage or org:manage.

Setting Default Rule
Spend alerts on off writes nothing to the inbox or to webhooks for spend
Thresholds (% of the limit) 50, 80, 100 comma separated, 1 to 200, at most 6
Alert when the forecast crosses a limit on
Alert on sudden spend jumps on
Ignore jumps under (USD an hour) 10 above the usual spend; keeps small keys quiet
Sensitivity Medium: 6 spreads and 3× the usual Low: 8 spreads and 5×; High: 4 spreads and 2×

Where they go

Notifications always land in this inbox. To get them elsewhere, add an org webhook under Settings, Webhooks (/settings/webhooks) that takes spend.threshold, spend.forecast or spend.anomaly: signed with HMAC-SHA256, retried, with a delivery log and a test button. For mail or chat, point a webhook at a relay you run (Slack, Teams or your mail gateway).

A spend anomaly: /notifications/anomalies/{id}

The evidence for one spend jump and what to do about it:

  • Figures: spent in the hour, the usual for that hour, the threshold, how many times the usual.
  • Last 48 hours: the spend against the usual hour-of-day curve.
  • Who spent it: the top keys, each with Pause or Resume. Pausing needs keys:write or billing:manage.
  • Top requests: each opens its trace (needs logs:read).
  • Also jumped: other keys, members, models or gears that jumped in the same hour.
  • How this was detected: the method and the sensitivity used.

Troubleshooting

Symptom Cause and fix
400 invalid_query with alert_window or alert_pipeline Remove time terms and pipes from an alert's query.
409 too_many_alerts The org has 50 alerts; delete one.
An alert shows an error badge Its last run failed; hover for the reason. It retries on the next run.
A hit shows a webhook error The POST failed or timed out after 5 s. Hits are not retried; use Test after fixing the endpoint.
No spend notifications Spend alerts are off, no budget or cap is set (thresholds and forecasts need one), or the jump was under the USD floor.