---
title: Postmortems
description: "A blameless postmortem draft in Notes for every resolved incident, its action items as Tasks issues, an eval case from the regression, and a draft PR with a failing test."
section: incidents
order: 14
audience: [user, operator, agent]
sources: [services/nodejs-cloud/lib/incidents/postmortem.ts, services/nodejs-cloud/lib/incidents/postmortem-run.ts, services/nodejs-cloud/lib/incidents/eval-case.ts, services/nodejs-cloud/lib/incidents/eval-case-run.ts, services/nodejs-cloud/components/incidents/EvalCasePanel.tsx, services/nodejs-cloud/migrations/0209_incident_eval_cases.sql, services/nodejs-cloud/lib/incidents/reproducer.ts, services/nodejs-cloud/lib/incidents/reproducer-run.ts, services/nodejs-cloud/components/incidents/ReproducerPanel.tsx, services/nodejs-cloud/migrations/0212_incident_reproducers.sql, docs/adr/0076-incident-reproducer-pr.md, services/nodejs-cloud/components/incidents/CausePanel.tsx, services/nodejs-cloud/components/globe/templates.ts, services/nodejs-cloud/migrations/0158_incident_loop.sql, docs/adr/0055-venom-incidents.md]
features: []
---

When an incident resolves, a draft appears in Notes, in the org vault under `Incidents/Postmortems/`, named by date and
number: `2026-11-08 INC-12 checkout-500s.md`. It is linked from the incident (under **Linked**, and on the timeline), and
both people and agents can read and edit it like any note.

## What is in it

The facts come from the record, with no model:

- **Summary.** The incident's summary, or one line built from its severity and project.
- **Impact.** Severity, how long it took to detect, mitigate and resolve, and for each linked error issue its event
  count, the users affected (an estimate) and its first and last seen.
- **Timeline (UTC).** The incident's entries in order: status updates, notes, alerts, flag changes, releases, the
  rollback.
- **Root cause and contributing factors.** The ranked suspect from [the loop](/docs/incidents/the-loop.md) and what was done about it.
- **What went well**, **Action items** and **Links.**

The note is blameless by its wording: it describes changes and systems. Anything that looks like a secret is redacted
before it is written.

## The AI draft

The sweep cron writes the postmortem within a minute of the resolve. For that one it asks your org's own gear (through an
internal key, purpose `incidents`) to draft the prose sections: summary, contributing factors, what went well and action
items. The incident's text is sent fenced as data, and the model is told to invent nothing. It is capped at 20 drafts
per org per day. Over the cap, or with no model reachable, you get the facts-only draft. A postmortem written by hand
(**Write now**, or `incidents_incident_postmortem`) spends nothing unless you ask for `ai`.

## Action items become Tasks issues

Under **Action items** each unchecked `- [ ]` line is an item. On the incident page, **File action items** creates a Tasks
issue for each one that has none (in the project's mapped team), and marks the line with its key: `- [ ] Add a canary
(ENG-14)`. The same is `incidents_incident_postmortem` with `fileActions: true`. The AI's items are suggestions: they only
become issues when someone asks.

## After a rollback with no incident

A flag rolled back through `incidents_incident_rollback` without an incident writes a short note, `… revert <project>-<flag>.md`,
with the flag, why, and empty contributing-factors and action-items sections.

## An eval case from the regression

A regression that reached production is a test your gates did not have. **Eval case** on the incident page turns it into a
draft task for the [eval suite](https://github.com/elishahbiz/venom/blob/main/evals/AGENTS.md), in the same shape the suite's
own mined tasks use, for you to review:

1. Link the change that ended it: the fix or revert **commit** (under **Linked**), or the incident's Tasks issue. When
   GitHub is connected to Tasks, the merged pull requests and pushed commits on that issue count too.
2. Press **Draft**. Venom picks the change (one you linked by hand first, then a merged pull request, then a pushed commit;
   a fix before a revert) and writes `task.json` (category `mined`, tagged `candidate`, `untrusted`, with a TODO check)
   and `REVIEW.md`: where it came from, the release that shipped the bug when one is linked, the `node src/mine.ts
   --commits <fix>:<origin>` command that builds the fixture from your clone, and what is left before it is a task.
3. Download both files (or copy `task.json`), run the command, write the failing check, and prove it with
   `validate-tasks --solve`. None of that calls a model.

Drafting again replaces the draft; there is one per incident. The prompt is the incident's own words with secrets
redacted. Nothing here runs an eval, calls a model or calls GitHub: the draft is built from what the cloud already
stores, and a paid run is yours to start (dry-run first). When nothing is linked, the panel says so and stores nothing.

Over REST: `GET` and `POST /incidents/api/v1/incidents/<ref>/eval-case`. Over MCP: `incidents_eval_case_get` and
`incidents_eval_case_draft`.

## A reproducer PR with a failing test

**Write a reproducer** on the incident page asks your org's own model for a test that fails because of the bug, and
opens it as a **draft pull request** on your repository. Your CI runs it: a red check confirms the test reproduces the
incident. Nothing is checked out and nothing runs in the cloud; the files go through GitHub's API.

### Setup (once, an owner or admin)

1. In GitHub, create a **fine-grained personal access token** (Settings, Developer settings, Personal access tokens,
   Fine-grained tokens). Under Repository access, pick only the repositories incidents happen in. Under Repository
   permissions set **Contents: Read and write** and **Pull requests: Read and write** (Metadata: Read is added for you).
   Nothing else is needed.
2. In Venom, open **Incidents → Settings → GitHub write token**, paste the token and, optionally, a default repository
   (`owner/name`) for incidents that name none. Save. The token is sealed in the vault, shown afterwards only by its
   last four characters, and never sent to a model or written to a log. **Remove** forgets it.
3. For the CI result to come back, connect the GitHub webhook in Tasks (Settings → GitHub; see
   [Tasks and GitHub](/docs/cloud/tasks-github.md)) with the **Check suites** (or Check runs) and **Pull requests** events.

Until a token is saved, the button says who adds one and nothing calls GitHub. Only an owner or admin, signed in, sets
or removes it; an agent cannot.

### What a run does

1. Picks the repository: `repo` if you pass one, else the linked fix or revert commit's repository, else the default.
2. Reads, through the GitHub API: the linked change's diff (the fix or revert commit, or the commit the linked release
   shipped), the current version of the files it touched, and up to two existing tests nearest to them, so the new
   test copies their framework and naming.
3. Sends that and the incident's title, summary and error titles, with secrets redacted, to your org's model through
   its own gear and budgets (the internal key "Venom Incidents reproducers"). The incident text is fenced as data.
4. Checks every file the model proposes: at most 3, each a new file (no existing file is changed), a test file by name
   (`*.test.ts`, `test_*.py`, `*_test.go`, `*Test.java`, a code file under `test/` …), inside the repository, nothing
   under `.github/` or any hidden directory, nothing vendored, at most 64 KiB. One bad file refuses the whole answer,
   before anything is written.
5. Commits the files on top of your default branch's head to a **new branch `venom/reproducer-<ref>`** (`-2`, `-3` … when
   an earlier one is still there) and opens a draft PR. The body explains the incident, links back to it, lists the
   files, says CI should fail until the bug is fixed, and ends with `Refs <Tasks key>` for the incident's Tasks issue, so
   the Tasks webhook links the PR there. The default branch is never written.

The panel then shows the PR and its CI: **CI failing (reproducer confirmed)**, **CI passing (test doesn't reproduce)**
(or the fix already landed, and the test guards against a regression), **CI running**, or **No CI reported** when
nothing came back within 30 minutes. A red result is also written to the timeline.

Limits: one run at a time per incident; 10 runs per org per UTC day, and $5 of model spend per day for these runs.

Over REST: `GET` and `POST /incidents/api/v1/incidents/<ref>/reproducer` (`{"repo": "owner/name"}` optional; 409
`no_github_token` without a token), and `GET`, `PUT`, `DELETE /incidents/api/v1/github-token`. Over MCP:
`incidents_reproducer_get`, `incidents_reproducer_write` and `incidents_github_token_get` (no tool sets the token). The
decision is ADR 0076 (`docs/adr/0076-incident-reproducer-pr.md` in the source).

## Settings and the Notes template {#settings-and-the-globe-template}

The **Postmortem** template in the template starters of Notes has the same sections, for incidents you write up by hand. A
project can switch automatic postmortems off with `postmortems: false` in the loop's settings
(`PUT /incidents/api/v1/settings`).

Not done yet: a postmortem for a flag revert Venom Incidents did not make.
