---
title: Alerts and notifications
description: Run saved log searches every 5 minutes with a signed webhook, and read the notification inbox for budget thresholds, spend forecasts, spend anomalies and key rotation.
section: cloud
order: 44
audience: [user, operator, builder]
sources: [services/nodejs-cloud/app/(console)/(shell)/alerts, services/nodejs-cloud/app/(console)/(shell)/notifications, services/nodejs-cloud/lib/alerts/rules.ts, services/nodejs-cloud/lib/alerts/index.ts, services/nodejs-cloud/lib/budget/alerts-model.ts, services/nodejs-cloud/lib/budget/anomaly-model.ts, docs/cloud-log-query.md]
features: [cloud.console-alerts, cloud.console-notifications]
---

Venom Cloud has two kinds of alert:

| Page | What fires | Who reads | Who changes |
|---|---|---|---|
| Alerts (`/alerts`) | a saved [log query](/docs/cloud/console-log-query.md) matching at least N requests in a 5-minute run | `audit:read` (owner, admin, developer) | `org:manage` (owner, admin) |
| Notifications (`/notifications`) | spend thresholds, forecasts, spend jumps, key rotation and model lifecycle reminders | `usage:read` (every role) | `billing:manage` or `org:manage` |

Venom Cloud sends no email. Alerts go to the console and to webhooks you run.

## Alerts: `/alerts` {#alerts}

An alert is a saved search. Every 5 minutes the cron runs it over the requests logged since its last run. A run that matches at least the threshold records a hit and, if the org set a webhook, posts it there.

### Create one {#create}

Either turn on a built-in suggestion with one click, or fill in the form:

| Field | Rule |
|---|---|
| Name | 1 to 100 characters |
| Query | a plain search that parses clean: no time terms (the evaluator sets the window) and no pipes. Content terms (`prompt:`, `response:`, `content:`) work and need `content:read`. Examples: `has:tools tool:bash`, `status:5xx key:ci*`, `prompt:"BEGIN OPENSSH"` |
| Threshold | matches in one run that make a hit; a whole number from 1 to 1,000,000, default 1 |

From [Requests](/docs/cloud/console-requests.md), Share, Create alert opens this form with the current search, its time range and pipes dropped.

### Built-in suggestions {#builtins}

| Name | Query | Threshold |
|---|---|---|
| Secret in a prompt | `has:content` plus the secrets detector: a prompt holding something shaped like an API key, token or private key (`sk-…`, `ghp_…`, `AKIA…`, `-----BEGIN … PRIVATE KEY-----`) | 1 |
| Prompt blocked by a filter | `filter_action:block` (a prompt filter returned `403 prompt_blocked`) | 1 |
| Shell tool use | `tool:(bash OR shell OR *exec* OR *terminal* OR *command*)` | 1 |
| Large prompt | `prompt_chars:>200k` | 1 |
| Error burst | `status:5xx` | 50 |

### The page {#alerts-page}

- **Alerts table**: name, query (opens it in Requests), threshold, hits in the last 24 h, last run, and state: on, paused, or the last run's error. Pause, resume or delete each one.
- **Hits**: newest first; each count opens exactly those requests in Requests. Hits newer than your last visit are unread; the sidebar's Alerts link shows the count, and opening the page marks them seen. Hits are kept 90 days.
- **Webhook**: the org's URL, its signing secret's last four characters, Test, Remove.

### How a run works {#runs}

- Each run reads the requests logged in `(last checked, now − 60 s]`, at most 24 hours. The minute's lag lets sealed content land.
- The watermark moves only when a run succeeds, so no request counts twice; a failed run keeps its error and retries next time. Pausing stops runs; resuming starts from now.
- Alerts that read bodies (content terms, the secrets detector) open at most 500 candidates a run, newest first; the hit says `truncated` when there were more. Each such run writes one `content.alert_evaluated` audit row, never the text. Alerts on derived fields (`tool:`, `prompt_chars:`) read no bodies.
- At most 50 alerts per org. Changes are audited: `alert.created`, `alert.updated`, `alert.deleted`, `alert.webhook_set`, `alert.webhook_removed`.

### The webhook {#webhook}

The org's own http or https URL (on the hosted cloud, https to a public address only). One POST per hit, 5 s timeout, no redirects, no retries; the outcome shows on the hit. The signing secret (`vwh_…`) is made on the first save or on rotate, shown once, and stored sealed.

Headers:

| Header | Value |
|---|---|
| `venom-event` | `alert.hit`, or `alert.test` from the Test button |
| `venom-delivery` | the hit id |
| `venom-signature` | `t=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>.<body>">`, keyed by the secret |

Body (at most 50 request ids; never a prompt, response or matched text):

```json
{
  "type": "alert.hit",
  "id": "42",
  "created_at": "2026-11-03T09:15:00.000Z",
  "org": { "id": "…" },
  "alert": { "id": "…", "name": "Shell tool use", "query": "tool:(bash OR shell)", "threshold": 1, "detector": null },
  "count": 3,
  "truncated": false,
  "window": { "since": "…", "until": "…" },
  "request_ids": ["req_…"],
  "links": { "alert": "https://example.com/alerts?hit=42", "explorer": "https://example.com/requests?q=id:…", "requests": ["https://example.com/requests/req_…"] }
}
```

Check the signature like a Stripe webhook: recompute, compare in constant time, refuse a `t` older than 5 minutes.

```ts
import { createHmac, timingSafeEqual } from "node:crypto";

export function verify(body: string, header: string, secret: string): boolean {
  const parts = Object.fromEntries(header.split(",").map((p) => p.split("=", 2)));
  const t = Number(parts.t);
  if (!t || Math.abs(Date.now() / 1000 - t) > 300) return false;
  const want = createHmac("sha256", secret).update(`${t}.${body}`).digest("hex");
  const got = String(parts.v1 ?? "");
  return got.length === want.length && timingSafeEqual(Buffer.from(got), Buffer.from(want));
}
```

Links use the deployment's `VENOM_CLOUD_ORIGIN`, else the cron's own origin.

## Notifications: `/notifications` {#notifications}

The inbox behind the top bar's bell, newest first. Items newer than your last visit are unread; opening the page marks them seen.

| Kind | When | Its button |
|---|---|---|
| Threshold | spend passed a threshold of a budget or of the monthly spend cap; once per threshold and window | Open budgets, or Open usage for the cap |
| Reached | a threshold of 100 % or more | the same |
| Forecast | at the current rate, a budget or the cap runs out before its day or month ends; once a window | the same |
| Spend anomaly | a key, member, model or gear spent far more in an hour than its usual for that hour of day | opens its evidence page |
| Key rotation | a provider key is due for rotation, or was retired | Rotate the key, or Open the provider |
| Lifecycle | a model deprecation or an SLA event | Open model lifecycle, or Open SLAs |

### Alert settings {#settings}

Thresholds and forecasts apply to every budget and to the monthly spend cap; a changed limit or a new day or month starts over. Spend jumps need no budget. Checked every 5 minutes. Changing them needs `billing:manage` or `org:manage`.

| Setting | Default | Rule |
|---|---|---|
| Spend alerts | on | off writes nothing to the inbox or to webhooks for spend |
| Thresholds (% of the limit) | 50, 80, 100 | comma separated, 1 to 200, at most 6 |
| Alert when the forecast crosses a limit | on | |
| Alert on sudden spend jumps | on | |
| Ignore jumps under (USD an hour) | 10 | above the usual spend; keeps small keys quiet |
| Sensitivity | Medium: 6 spreads and 3× the usual | Low: 8 spreads and 5×; High: 4 spreads and 2× |

### Where they go {#delivery}

Notifications always land in this inbox. To get them elsewhere, add an org webhook under Settings, Webhooks (`/settings/webhooks`) that takes `spend.threshold`, `spend.forecast` or `spend.anomaly`: signed with HMAC-SHA256, retried, with a delivery log and a test button. For mail or chat, point a webhook at a relay you run (Slack, Teams or your mail gateway).

### A spend anomaly: `/notifications/anomalies/{id}` {#anomaly}

The evidence for one spend jump and what to do about it:

- **Figures**: spent in the hour, the usual for that hour, the threshold, how many times the usual.
- **Last 48 hours**: the spend against the usual hour-of-day curve.
- **Who spent it**: the top keys, each with Pause or Resume. Pausing needs `keys:write` or `billing:manage`.
- **Top requests**: each opens its trace (needs `logs:read`).
- **Also jumped**: other keys, members, models or gears that jumped in the same hour.
- **How this was detected**: the method and the sensitivity used.

## Troubleshooting {#troubleshooting}

| Symptom | Cause and fix |
|---|---|
| `400 invalid_query` with `alert_window` or `alert_pipeline` | Remove time terms and pipes from an alert's query. |
| `409 too_many_alerts` | The org has 50 alerts; delete one. |
| An alert shows an error badge | Its last run failed; hover for the reason. It retries on the next run. |
| A hit shows a webhook error | The POST failed or timed out after 5 s. Hits are not retried; use Test after fixing the endpoint. |
| No spend notifications | Spend alerts are off, no budget or cap is set (thresholds and forecasts need one), or the jump was under the USD floor. |

## Related

- [Log query language](/docs/cloud/console-log-query.md)
- [Requests and request traces](/docs/cloud/console-requests.md)
- [Audit log](/docs/cloud/console-audit.md): the `alert.*` and `content.alert_evaluated` rows.
