---
title: Venom Experiments
description: A/B tests on Venom Flags flags with sound statistics (CUPED, always-valid sequential results, sample ratio checks, guardrails), run by people or agents, with a decision note left in Notes.
section: cloud
order: 249
audience: [builder, agent, user]
sources: [services/nodejs-cloud/lib/analytics, services/nodejs-cloud/lib/stats, services/nodejs-cloud/app/experiments, services/nodejs-cloud/sdk/flags, docs/adr/0054-venom-experiments.md]
features: [cloud.experiments, cloud.page-exp]
---

Venom Experiments runs controlled experiments on your own app. An experiment sits on a [Venom Flags](/docs/cloud/flags.md) flag:
the flag's rollout decides who sees what, your app sends the events that say what happened ([sending
events](/docs/cloud/experiments-events.md)), and Experiments does the statistics. An agent can propose an experiment, start it,
read the results each day and ship the winner; everything it does to the flag is in the flag's history and can be
reverted there.

## The idea in one paragraph

An experiment never assigns anyone itself. Starting it puts a percentage rollout, marked with the experiment's id, on a
flag's fallthrough (or on one rule). The Flags evaluator then answers with a reason that says `inExperiment`, the SDK
posts an **exposure** for each unit the first time it sees it, and from then on the unit counts in the arm it was
served. Because the SDK, the server and the analysis share one evaluator and one bucketing, they agree on who saw
what.

## Metrics

A metric belongs to a Flags environment and reads the events your app tracks:

| Type | What each unit contributes |
|---|---|
| `count` | the number of its events after exposure |
| `sum` | the `value` of its events, summed |
| `conversion` | 1 if it had at least one event, else 0 |
| `retention` | 1 if it had an event exactly `retentionDays` after exposure (only units old enough are counted) |
| `ratio` | the ratio of two other metrics (revenue per order), with the delta method for the variance |

Each metric has a direction (which way is good) and, for `count` and `sum`, an optional **winsorize** cap so a few
extreme units do not decide the result. A unit with no events counts as zero. Events from the day of the exposure on
count; earlier days are the CUPED covariate.

## An experiment

An experiment names its flag, its control and treatment variations, the share of units in the treatments (50 % by
default), a primary metric, optional secondary metrics and **guardrails**, and how to analyse:

- **Sequential** (the default). Results use the mixture sequential probability ratio test: the p-value and the interval
  are valid however often you look and whenever you stop, so you can read them every day.
- **Fixed horizon.** Results wait for the planned sample per arm; before that they show estimates and no p-value.

The plan comes from the last 14 days of your traffic: the sample per arm that detects a relative change (`mde`, 5 % by
default) at the chosen power, and the days it takes. For a sound baseline, track an event for every visit or session:
the busiest event of the environment is taken as the population. Without history the plan is a stated default of
5,000 units per arm.

**CUPED** (on by default) adjusts each unit's outcome by the same metric in the 14 days before exposure. It does not
change the estimate and narrows the interval when past behaviour predicts the outcome.

Secondary metrics are corrected together with Holm's method. A **guardrail** is a metric that must not get worse by
more than a tolerated share of control's mean (2 % by default), tested one-sided and sequentially. A breach emits the
`experiments.guardrail_breached` org webhook event once. With `autoRollback` (off by default) it also serves control
through a Flags change made by the actor `experiments:guardrail`, which is audited and revertible.

### Sample ratio mismatch

Every analysis compares the units in each arm with the planned split by a chi-square test. When p is below 0.001
something in assignment or logging is broken (an SDK that exposes only in one arm, a redirect that drops users). The
numbers are then hidden behind a warning and the recommendation is "investigate".

### Changing the split

`ramp` changes the treatments' share through Flags. A new split means units can move arms, so the analysis starts over:
results count only the units exposed after the change. Ramp early and rarely.

## The life of an experiment

`draft` → `running` → `stopped` or `decided`.

- **Propose** creates the draft and plans the sample. Nothing is served.
- **Start** makes the Flags change. The flag must be on in the environment (or pass `turnOn`). In an environment that
  requires approvals the change becomes a request; the experiment starts when it is applied.
- **Stop** serves control to everyone. A stopped experiment can still be decided.
- **Decide** is `ship` (serve one variation to everyone), `rollback` (serve control) or `iterate` (serve control and
  note that another round follows). It applies the Flags change, writes a **decision note** into the organisation's
  Notes vault under `Experiments/` (hypothesis, setup, results table, the recommendation at the time, links to the
  flag's audit entry) and emits `experiments.decided`.

Only one experiment runs on a flag, in an environment and on a rule, at a time.

Starting, ramping and stopping are org webhook events too: `experiments.started` (also when an approved request starts
it), `experiments.ramped` (by hand, or a guarded rollout's next step, with the share before and after) and
`experiments.stopped`. Each carries the experiment, flag, environment, the treatments' share and who did it. Subscribe
in Settings → Webhooks ([Webhooks and events](/docs/developers/webhooks)).

## Insights

Trends, funnels and retention over the same events, saved insights, and live chart blocks for Notes pages are on the Insights page; see [Insights and dashboards](/docs/cloud/experiments-insights.md).

## Guarded rollouts

A guarded rollout is an experiment without a hypothesis: a flag change that ramps through steps while guardrail metrics
hold, and rolls back by itself when one breaks. Tick **Guarded rollout** in the new-experiment flow, or propose with
`kind: "rollout"` over REST or MCP.

- **Steps** are a share of units and the least time to stay there. The default is 1% → 5% → 25% → 50% (a day each); 100%
  follows the last step. Pass your own `steps: [{ share, minHours }]` (shares in thousandths of a percent, increasing).
- **At least one guardrail** is required (a metric and how much worse than control it may get). The analysis is
  sequential, so the guardrails can be read at any time without inflating false alarms.
- **The sweep** (every few minutes) moves each running rollout. When the step's minimum time has passed and the guardrail
  tests have enough units and show no breach, it ramps to the next share. After the last step it ships the treatment to
  everyone. Each move is a Flags change by the system actor `experiments:rollout`: audited, revertible, and a request in
  an environment that requires approval (the rollout waits for it; a turned-down request pauses the rollout).
- **A breach** serves control at once (even from a pause), marks the rollout `Rolled back`, raises
  `experiments.guardrail_breached` and `experiments.decided`, writes the Notes decision note and files a Venom Tasks
  issue with the evidence when the organisation has Tasks.
- **Pause and resume** hold the ramp (a button on the experiment page, `POST …/pause`, `…/resume`, or the
  `exp_experiment_rollout` tool). Resuming restarts the step's minimum time. A rollout moves by its steps, so a manual ramp
  is refused.

The experiment page shows the step ladder, where it is, what it is waiting for and each guardrail's state. Rollouts are
off until you start one; nothing about existing experiments changes. Canaries of deploys that are not behind a flag are
a separate matter.

## The app

Experiments is an app in the workspace rail (press G then X). It lives at `/exp` (the console's `/experiments` page is
the separate gear A/B test page) and needs the Experiments add-on, which includes Venom Flags. While billing is off, or
the add-on has no price, every organisation has it.

- **Experiments** lists the project's experiments in the environment in view: status, the primary metric's lift with
  its interval drawn against zero, days running, and a badge when the sample ratio check failed. Press `/` to filter,
  `j` and `k` to move, Enter to open and `c` for a new one. The status tabs count each state.
- **New experiment** (`c`, or ⌘K, "New experiment") picks a flag and which variations are the arms, takes a name and a
  hypothesis, a primary metric, secondary metrics and guardrails (with how much regression you tolerate), the analysis
  (sequential or fixed horizon), the treatments' share, the smallest change worth detecting and the power. The **plan**
  at the bottom updates as you change them: the units each arm needs and how many days recent traffic takes to deliver
  them. Saving makes a draft; nothing is served until you start it.
- **An experiment's page** (`/exp/{id}`) starts with the verdict: the recommendation (ship, roll back, keep running, inconclusive,
  investigate) in words, readiness and, for a fixed-horizon test, how far it is toward its planned sample. A **sample
  ratio mismatch** shows as a red banner and hides the numbers. Below: the arms with their planned and observed traffic;
  the results table with each metric's lift, interval bar, evidence and verdict, and a **CUPED** switch that recomputes
  the numbers; the **cumulative lift** chart; guardrails (holding or breached); the flag's recent history from Venom
  Flags' audit log; and the experiment's own log.
- **Start, Ramp, Stop and Decide** are buttons on the page. Each is a Venom Flags change made as you, so it is in the
  flag's history and can be reverted there; in an environment that requires approvals it becomes a request and the page
  says so. A ramp warns that the analysis starts over. **Decide** offers each treatment to ship, roll back and iterate,
  with the recommended one first, takes a reason, and writes the decision note into Notes. A decided experiment links to
  that note and to the flag's history. Press `r` to recompute and Esc to go back to the list.
- **Metrics** (`/exp/metrics`, `⇧M`) is the catalog of the environment: create and edit count, sum, conversion, ratio and retention
  metrics, with the direction and an optional cap per user. What a metric measures can't change while a running or
  stopped experiment uses it, and a metric an experiment uses can't be deleted.
- **Events** (`/exp/events`, `⇧E`) lists the event names the environment received in the last 14 days with a volume bar for each and
  when it was last seen, and the newest raw rows of all events or of one name, so you can check that `track()` arrives
  before you build a metric on it.
- **Connect an agent** (`/exp/agents`) links to the Connect page (`/connect`), and to the Experiments-only endpoint under its Advanced section.

Running experiments also appear on **Home** with their lift, and a guardrail breach (or a decision in the last three
days) shows in the bell.

## Where to go next

- [Sending events and exposures](/docs/cloud/experiments-events.md) from your app.
- [The MCP server](/docs/developers/experiments.md#tools) for agents.
- [The REST API](/docs/developers/experiments.md#rest).
