---
title: The loop
description: "An error spike opens an incident, Venom ranks what changed, rolls back the flag that did it through Venom Flags (if you allow it), and files a Tasks issue for an agent with the stack and the suspects."
section: incidents
order: 13
audience: [user, operator, agent]
sources: [services/nodejs-cloud/lib/incidents/implicate.ts, services/nodejs-cloud/lib/incidents/loop.ts, services/nodejs-cloud/lib/incidents/actions.ts, services/nodejs-cloud/lib/incidents/loop-http.ts, services/nodejs-cloud/components/incidents/CausePanel.tsx, services/nodejs-cloud/components/incidents/Settings.tsx, services/nodejs-cloud/migrations/0158_incident_loop.sql, docs/adr/0055-venom-incidents.md]
features: []
---

A bad flag change goes out. Errors climb. Within a minute or two you have an incident, the flag that caused it is off
again, and an agent has a Tasks issue with the stack and the commits to look at. This page is how that works and how to
turn the risky part on.

## What happens, in order

1. An [alert rule](/docs/incidents/alerts.md) with the action **Open incident** fires on an error issue (a spike or a threshold). The
   incident is declared at the rule's severity and linked to the issue.
2. Venom ranks what changed in the two hours before the spike: flag changes in the same project and environment (read
   from the Flags audit log) and releases. The timeline gets a **Likely cause** entry with the top three.
3. If the environment allows it and the top candidate is a flag with enough confidence, the change is reverted.
4. A Tasks issue is filed in the team you mapped, with the error issue, a compact stack, the suspect, the rollback result
   and the pull requests and commits Tasks has linked around the release. It is delegated to the triage agent you chose.
5. When the incident resolves, a [postmortem](/docs/incidents/postmortems.md) draft is written to Notes.

Every step is a timeline entry, with its actor (`incidents:auto` for the automatic ones) and the channel `auto`. Running
the rule again does nothing twice: one incident per issue while it is open, one rollback, one Tasks issue.

## How the ranking works

A flag change scores on two things.

- **Timing.** It landed shortly before the spike. The score falls from 1 at the spike to 0 two hours earlier.
- **Concentration.** Among the issue's sampled events that carry the flag, how many were served one value, compared with
  that value's share of all the flag's evaluations. If every error saw `true` and `true` is served to 5% of
  evaluations, the flag is almost certainly it. If the errors split the way the traffic does, it is not.

The confidence is 30% timing and 70% concentration. A change with no sampled error carrying its flag is capped at 35%:
worth a look, never enough to act on. Releases score on timing, on the share of sampled errors on that release, and on
whether the issue first appeared after it was deployed. A release is shown but never rolled back; there is nothing in
Venom Cloud to revert a deploy.

You can read the ranking at any time: the **Likely cause** panel on the incident page, `GET
/incidents/api/v1/incidents/{ref}/implicate`, or the `incidents_incident_implicate` MCP tool.

## Automatic rollback

It is **off by default**. In **Incidents, Settings, Close the loop** (an owner or admin), switch **Roll back by itself** on
for an environment and pick the confidence it needs (50% to 90%, 70% by default). With it off, the timeline says so and
the incident page shows a **Roll back** button on the flag candidate, and agents can call `incidents_incident_rollback`.

A rollback is a normal Venom Flags change:

- It is `FlagsService.revert` of the change that was implicated, so it is audited in Flags under the actor
  `incidents:auto` (or the person or agent who pressed the button), and the original entry is marked reverted.
- It is itself revertible from the Flags audit log.
- If the flag moved on since (so the revert is refused) and it is a permanent flag, Venom turns it off in that
  environment instead. A temporary flag is left alone and the timeline says it could not be rolled back.
- **Approvals are respected.** In an environment that requires approvals, Venom files a request to turn the flag off
  and stops. A reviewer approves it in Venom Flags.

## The Tasks issue

In the same settings page, give the project a **Tasks team key** and a **triage agent** (`mcp:…`, `key:…`, `daemon:…`
or `session:…`, the ids Tasks delegates to). Empty means the first team and no delegate. The issue is created by the
system actor, and if you run an intake runner for that team, the usual intake path can take it to a pull request.

The body has what an engineer needs to start: counts and users, the stack with in-app frames first, the flags served,
the ranked suspect with its reasons, what the rollback did, suspect commits, and a reproduction hint.

## For agents

| Tool | Does |
|---|---|
| `incidents_incident_implicate` | The ranking, with evidence |
| `incidents_incident_rollback` | The rollback, with `ref` for an incident's candidate or with `project`, `flag`, `auditId` and `env` for any Flags change (a short postmortem is written when no incident is behind it) |

See [Build on Venom Incidents](/docs/developers/incidents.md). Over REST the same actions are `GET /incidents/api/v1/incidents/{ref}/implicate`,
`POST …/rollback`, `POST /incidents/api/v1/rollback` and `GET`/`PUT /incidents/api/v1/settings`.
