---
title: "Admin API: daemons, fleet and reliability"
description: Manage your org's daemons, public nodes, GPU autoscaling, prewarming, GPU grants between key groups, failover drills, model quality checks, SLAs and the public status page over /api/admin/v1.
section: cloud
order: 330
audience: [operator, builder, agent]
sources: [services/nodejs-cloud/app/api/admin/v1/daemons, services/nodejs-cloud/app/api/admin/v1/nodes, services/nodejs-cloud/app/api/admin/v1/autoscale, services/nodejs-cloud/app/api/admin/v1/prewarm, services/nodejs-cloud/app/api/admin/v1/gpu-grants, services/nodejs-cloud/app/api/admin/v1/failover-drill, services/nodejs-cloud/app/api/admin/v1/jobs, services/nodejs-cloud/app/api/admin/v1/autopilot, services/nodejs-cloud/app/api/admin/v1/serve-sessions, services/nodejs-cloud/app/api/admin/v1/sla, services/nodejs-cloud/app/api/admin/v1/status-page, services/nodejs-cloud/app/api/admin/v1/quality, services/nodejs-cloud/app/api/admin/v1/scorecard, services/nodejs-cloud/lib/models/admin.ts, services/nodejs-cloud/lib/autoscale, services/nodejs-cloud/lib/sla, services/nodejs-cloud/lib/status-pages, docs/cloud-mesh.md]
features: [cloud.api-admin-daemons, cloud.api-admin-nodes, cloud.api-admin-autoscale, cloud.api-admin-prewarm, cloud.api-admin-gpu-grants, cloud.api-admin-failover-drill, cloud.api-admin-jobs, cloud.api-admin-autopilot, cloud.api-admin-serve-sessions, cloud.api-admin-sla, cloud.api-admin-status-page, cloud.api-admin-quality, cloud.api-admin-scorecard]
---

A daemon is a Go orchestrator in mesh mode that serves your org's requests from your own machines. These routes
list and control daemons and the jobs they run, choose public nodes you trust, start and stop GPU machines on
demand, load models before people need them, share GPUs between key groups, rehearse outages, watch model quality, and
publish SLAs and a status page. The daemon's own protocol (`/api/mesh/v1`) is described in `docs/cloud-mesh.md` §6.

Every route takes an admin credential ([who can use the admin API](/docs/cloud/api-overview.md#admin-credentials)). The
operator token reads every org (`?tenant=<org id>` on reads, `tenant_id` in write bodies); a key group admin key
(`vak_…`) sees its own key group's daemons, jobs and grants. Errors follow [the common shape](/docs/cloud/api-overview.md#errors).
A key group holds API keys, daemons, policies and GPU grants, not people; the API keeps the name `team`, so
`team_id`, `owner_team_id` and `consumer_team_id` are key group ids.

## A first call {#first-call}

```bash
export VENOM_CLOUD=https://cloud.usevenom.dev
export ADMIN="Authorization: Bearer $VENOM_CLOUD_ADMIN_TOKEN"

# Every daemon in scope, with its state, lease and last errors
curl -s "$VENOM_CLOUD/api/admin/v1/daemons" -H "$ADMIN"

# Drain one before maintenance (no new jobs; running ones finish)
curl -s -X PATCH "$VENOM_CLOUD/api/admin/v1/daemons/<daemon id>" -H "$ADMIN" \
  -H "Content-Type: application/json" -d '{"draining": true}'

# The jobs that failed in the last hour
curl -s "$VENOM_CLOUD/api/admin/v1/jobs?state=failed&since=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)" -H "$ADMIN"
```

```python
import os, requests

base = "https://cloud.usevenom.dev/api/admin/v1"
h = {"Authorization": f"Bearer {os.environ['VENOM_CLOUD_ADMIN_TOKEN']}"}
offline = [d for d in requests.get(f"{base}/daemons", headers=h).json()["data"] if d["state"] == "offline"]
```

## Daemons {#daemons}

| Route | Permission | What it does |
|---|---|---|
| `GET /api/admin/v1/daemons` | `daemons:read` | `{data: [...]}`: every daemon in scope with its name, key group (`team`), labels, models, region, version, `state` (`enrolled`, `active`, `draining`, `offline`, `revoked`, `retired`), `lease_valid` and its last errors. Hidden (deleted) daemons are left out |
| `GET /api/admin/v1/daemons/{id}` | `daemons:read` | One daemon, deleted ones included |
| `PATCH /api/admin/v1/daemons/{id}` | `daemons:manage` | Any of: `name` (at most 128 printable characters, unique in the org, else `409 daemon_name_taken`), `labels` (at most 64 keys of 63 characters, string or boolean values of at most 200), `draining: true\|false` (an admin drain outlasts the daemon's own heartbeats until you set `false`), `revoked: true` (terminal: a revoked daemon can never come back, `409 conflict` on later changes). Audited `daemon.update` or `daemon.revoke` |
| `DELETE /api/admin/v1/daemons/{id}` | `daemons:manage` | Revoke and hide it from listings. Audited `daemon.delete` |
| `GET /api/admin/v1/daemons/{id}/shares` | `daemons:read` | Which other key groups of the org may use this daemon: `[{team_id, models, weight, created_at}]` |
| `PUT /api/admin/v1/daemons/{id}/shares` | `daemons:manage` | Replace the shares with an array of `{team_id, models?, weight?}` (`weight` above 0, default 1; `models` null for all). Key groups must be in the daemon's org. Audited `share.put` |
| `GET /api/admin/v1/daemons/{id}/timeline` | `daemons:read` | The last 24 hours: hourly requests, errors, TTFT and duration medians, the last errors, and admin events on the daemon |
| `GET /api/admin/v1/daemons/{id}/metrics` | `daemons:read` | `?hours=` (default 24): 5-minute series (GPU use, VRAM, memory, in flight, tokens per second), the last heartbeat's metrics, public-mesh credits by day and the prewarm state |
| `GET /api/admin/v1/daemons/metrics` | `daemons:read` | `?hours=` (default 24): the fleet grid, hourly per daemon in scope, plus public-mesh credits today and over 30 days |
| `GET /api/admin/v1/daemons/{id}/qos` | `daemons:read` | For a public daemon: claimed against witnessed quality of service, divergence, trust state and the last verdicts |
| `POST /api/admin/v1/daemons/{id}/trust` | cloud admin token only | `{"action": "bar" \| "clear"}`: bar a public daemon from the public mesh, or clear the bar |

New daemons enroll with a join token (`POST /api/admin/v1/join-tokens`, on [Admin API: orgs, members and keys](/docs/cloud/api-admin-access.md#join-tokens)).

## Public nodes {#nodes}

The public mesh lets anyone host a daemon. Your org's `trusted` routing mode uses only the public nodes on your
allowlist.

| Route | Permission | What it does |
|---|---|---|
| `GET /api/admin/v1/nodes` | `routing:read` | The public node catalogue: region, models, hardware, price, witnessed QoS, trust state and age (never endpoints), each marked if on your allowlist |
| `GET /api/admin/v1/nodes/{id}` | `routing:read` | One public node: its catalogue entry, 24 hours of hourly witnessed QoS and its verdicts. A private daemon answers `404` |
| `POST /api/admin/v1/nodes/trusted` | `routing:write` | `{daemon_id, note?}`: add a node to the allowlist. Audited `routing.trusted_added` |
| `DELETE /api/admin/v1/nodes/trusted/{id}` | `routing:write` | Take a node off it. Audited `routing.trusted_removed` |

## Jobs and serve sessions {#jobs}

| Route | Permission | What it returns |
|---|---|---|
| `GET /api/admin/v1/jobs` | `logs:read` | The newest 500 relayed jobs in scope: id, request id, key, session, path, model, stream, `state`, daemon, attempt, route and fallback reasons, timestamps (created, dispatched, first byte, finished) with their hybrid logical clock stamps, status and error code. Filters: `?state=` (`queued`, `dispatched`, `running`, `succeeded`, `failed`, `cancelled`), `?daemon=<id>`, `?key=<id>`, `?since=<RFC 3339>` |
| `GET /api/admin/v1/serve-sessions` | `logs:read` | `venom serve` sessions bound to a daemon through the cloud (newest 500): session id, remote id, `state`, epoch, daemon, key, last event. `?state=bound\|daemon_lost\|closed\|all` (default: `bound` and `daemon_lost`) |

## GPU autoscaling {#autoscale}

Autoscaling starts daemons on a GPU cloud (RunPod or Lambda) when demand for a pool's models builds, and stops
them when idle. The cloud API key is checked, then sealed in the vault.

| Route | Permission | What it does |
|---|---|---|
| `GET /api/admin/v1/autoscale` | `daemons:read` | The org's GPU clouds, pools, machines, demand and spend |
| `POST /api/admin/v1/autoscale/clouds` | `org:manage` | `{provider, api_key}`: check the key against the GPU cloud, seal it, keep the GPU listing |
| `POST /api/admin/v1/autoscale/clouds/{id}/refresh` | `org:manage` | List GPU types and prices again |
| `DELETE /api/admin/v1/autoscale/clouds/{id}` | `org:manage` | Forget the key; refused while a pool uses it |
| `POST /api/admin/v1/autoscale/pools` | `org:manage` | `{name, cloud_id, …config}`: a new pool (fields below) |
| `PATCH /api/admin/v1/autoscale/pools/{id}` | `org:manage` | Change a pool, or turn it off |
| `DELETE /api/admin/v1/autoscale/pools/{id}` | `org:manage` | Delete a pool with no machine left |
| `POST /api/admin/v1/autoscale/pools/{id}/scale` | `org:manage` | `{n}`: hold N machines for an hour, reconciled at once |
| `POST /api/admin/v1/autoscale/machines/{id}/drain` | `org:manage` | No new requests; terminated once idle |
| `POST /api/admin/v1/autoscale/machines/{id}/terminate` | `org:manage` | Terminate now at the GPU cloud |

Pool configuration:

| Field | Meaning |
|---|---|
| `gpu`, `gpu_count` | the GPU type from the cloud's listing and how many per machine |
| `regions` | RunPod data center ids (empty: any); for Lambda the first is the region |
| `image`, `volume_gb`, `ssh_key_name` | the machine image, disk size and SSH key (Lambda needs one) |
| `models` | Ollama models each machine pulls at boot; their demand scales the pool |
| `venom_version` | the daemon release to install; empty for the latest |
| `min`, `max` | machine bounds |
| `monthly_cap_usd` | the pool's monthly spend cap |
| `scale_up` | `{wait_p90_s, queue_depth, minutes}`: add a machine when the dispatch wait p90 or the queue depth stays at or above the value for every minute of `minutes` |
| `scale_down` | `{idle_minutes}` |
| `schedule` | `{days, from_hour, to_hour, tz, keep}`: keep N machines in a window (days 0 = Sunday) |
| `join_timeout_minutes` | a machine that has not joined by then is terminated and an alert sent |

## Prewarming {#prewarm}

Prewarming asks your live daemons to load a gear's models before the working day, so the first request does not
wait for a model load. Daemons pick the request up with their next heartbeat answer.

| Route | Permission | What it does |
|---|---|---|
| `GET /api/admin/v1/prewarm` | `daemons:read` | The schedule, its next run and last result, and the gears it may name |
| `PUT /api/admin/v1/prewarm` | `daemons:manage` | `{enabled?, gear?, days?, at?\|at_minute?, tz?, keep_alive_min?}` (`keep_alive_min` default 120). Audited `prewarm.schedule` |
| `DELETE /api/admin/v1/prewarm` | `daemons:manage` | Remove the schedule. Audited `prewarm.removed` |
| `POST /api/admin/v1/prewarm/run` | `daemons:manage` | `{gear?, daemon_id?}`: prewarm now, the schedule's gear (else the default gear) on every live daemon or one. Audited `prewarm.run` |

A session can also prewarm its own models with a client key: `POST /v1/prewarm` (see the
[Inference API](/docs/cloud/api-inference.md#prewarm)).

## GPU grants between key groups {#gpu-grants}

A grant lets another key group of the org use the owner key group's daemons, with an optional daily token quota.
Usage is booked to the receiving key group (its keys, its budgets) and shown to both. A key group that is neither
party gets `404`.

| Route | Permission | What it does |
|---|---|---|
| `GET /api/admin/v1/gpu-grants` | `daemons:read` | The grants the caller is a party to, with today's and 30 days' usage. `?direction=given\|received` |
| `POST /api/admin/v1/gpu-grants` | `daemons:manage` over the owner key group | `{owner_team_id, consumer_team_id, models?, daily_token_quota?, expires_at?, weight?, note?}`. Audited |
| `GET /api/admin/v1/gpu-grants/{id}` | `daemons:read`, either party | One grant with 30 days of usage by day, daemon and model |
| `PATCH /api/admin/v1/gpu-grants/{id}` | `daemons:manage`, the owner key group | `{daily_token_quota?, models?, expires_at?, weight?, note?}`. Audited |
| `DELETE /api/admin/v1/gpu-grants/{id}` | the owner key group, or the receiving key group giving it back | Revoke it. Audited |

## Local autopilot {#autopilot}

The autopilot finds gear roles a local model on your daemons could take over, checks them on sampled traffic in
shadow (a judge model compares answers), and applies the switch when you say so.

| Route | Permission | What it does |
|---|---|---|
| `GET /api/admin/v1/autopilot` | `routing:read` | Candidates, shadow settings and samples, open applies |
| `PUT /api/admin/v1/autopilot` | `org:manage` | `{shadow_enabled?, weekly_samples?, gpu_usd_per_hour?, judge_model?}`. Defaults: shadow off, 100 samples a week, GPU at $0 an hour, no judge |
| `POST /api/admin/v1/autopilot/apply` | `org:manage` | `{gear, role, model, local_model?}`: the role goes to the local model, daemons first, with the paid models behind it. Audited |
| `POST /api/admin/v1/autopilot/revert` | `org:manage` | `{id}`: an applied role back as it was, as long as nobody changed it since. Audited |

## Failover drill {#failover-drill}

Rehearse an outage without touching real traffic: the dry run says where each route would fall back; the probe
sends one tiny pinned request to each fallback.

| Route | Permission | What it does |
|---|---|---|
| `GET /api/admin/v1/failover-drill` | `routing:read` | The outages worth drilling for your org |
| `POST /api/admin/v1/failover-drill` | `routing:read` | `{outage: {kind, id?}}`: the dry-run plan. Nothing is sent and nothing changes. `kind` is `provider`, `credential`, `target`, `daemon`, `own_daemons`, `public` or `cloud` |
| `POST /api/admin/v1/failover-drill/probe` | `org:manage` | `{outage, confirm: true}`: the live probe, capped, booked with source `failover_drill`. Audited `failover_drill.probe` |

## Model quality checks {#quality}

Scheduled suites of test cases run against the models your gears use, graded by a judge, so you notice when a
model gets worse. Off until you turn it on with a monthly cap above $0: nothing runs or is spent before.

| Route | Permission | What it does |
|---|---|---|
| `GET /api/admin/v1/quality` | `routing:read` | Settings, the month's spend and estimate, suites, scoreboard, flags, runs and swaps |
| `PUT /api/admin/v1/quality` | `org:manage` | `{enabled?, monthly_cap_usd?, autoswap?}`. `autoswap` lets a flagged model be swapped out of a gear and back |
| `POST /api/admin/v1/quality/suites` | `org:manage` | `{starter, models?, schedule?}` adds a starter suite; any other body is a whole suite. `201` |
| `GET /api/admin/v1/quality/suites/{id}` | `routing:read` | A suite with its cases |
| `PUT /api/admin/v1/quality/suites/{id}` | `org:manage` | Change a suite; a partial body keeps the rest |
| `DELETE /api/admin/v1/quality/suites/{id}` | `org:manage` | Remove it |
| `GET /api/admin/v1/quality/runs/{id}` | `routing:read` | A run and its graded cases |

`POST /api/admin/v1/quality/test` (`org:manage`, `{model, case, judge_model?, suite_id?}`) runs one case now as a
real, booked request; checks must be on and the month's cap not reached. Another org's suite or run answers `404`.

## Agent scorecard {#scorecard}

| Route | Permission | What it returns |
|---|---|---|
| `GET /api/admin/v1/scorecard` | `usage:read`; `by=member` needs `org:manage` | How agent sessions went: `?days=7\|30\|90`, `?by=gear\|team\|member` (default `gear`; `team` groups by key group). Read only |

## SLA targets and service credits {#sla}

An SLA target is a promise on availability, TTFT p95 or error rate, for one of your resold customers or for one
route. Each month is measured; a miss drafts a service credit that you approve (it goes on the customer's next
open invoice as a negative line) or dismiss. See [Admin API: customers and MSP](/docs/cloud/api-admin-msp.md).

| Route | Permission | What it does |
|---|---|---|
| `GET /api/admin/v1/sla` | `billing:read` | Targets with the month's measured report and drafted credits. `?month=YYYY-MM` |
| `POST /api/admin/v1/sla` | `billing:manage` | A target (fields below). Audited `sla.target_created` |
| `GET /api/admin/v1/sla/{id}` | `billing:read` | One target with its last months' reports and credits |
| `PATCH /api/admin/v1/sla/{id}` | `billing:manage` | Change any field except the scope and its customer or route |
| `DELETE /api/admin/v1/sla/{id}` | `billing:manage` | Remove it with its reports and draft credits; credits already on invoices stay |
| `POST /api/admin/v1/sla/credits/{id}/{action}` | `billing:manage` | `{action}` is `approve` or `dismiss` |

Target fields: `name`, `scope` (`customer` or `route`), `customer_id` or `route`, `availability_pct`,
`ttft_p95_ms`, `error_rate_pct`, `min_requests` (a month with fewer requests is not judged), `credit_pct` (the
credit as a percent of the month's charge, default 0), `deep_below_pct` and `deep_credit_pct` (a bigger credit
below a second threshold), `alerts`, `enabled`. An archived customer answers `409 archived`.

## Public status page {#status-page}

A public page at your own slug with live component status (one per model), incidents and a changelog.

| Route | Permission | What it does |
|---|---|---|
| `GET /api/admin/v1/status-page` | `org:read` | The page's settings, incidents, changelog, and the gear and model changes you may publish |
| `PUT /api/admin/v1/status-page` | `org:manage` | Create or change it: `{enabled?, slug, title, description?, components?, auto_incidents?, support_url?}`. `components` is `[{model, label}]` |
| `DELETE /api/admin/v1/status-page` | `org:manage` | Remove the page |
| `POST /api/admin/v1/status-page/incidents` | `org:manage` | `{title, impact, status?, models?, body, published?}`: an incident with its first update. `impact`: `minor`, `major`, `critical`; `status`: `investigating`, `identified`, `monitoring`, `resolved` |
| `PATCH /api/admin/v1/status-page/incidents/{id}` | `org:manage` | `{body?, status?, title?, impact?, models?, published?}`: a `body` (with `status`) posts an update; `status: resolved` resolves it |
| `DELETE /api/admin/v1/status-page/incidents/{id}` | `org:manage` | Remove an incident |
| `POST /api/admin/v1/status-page/changelog` | `org:manage` | `{audit_id?, title, body?}`: an entry, from one of your gear or model changes in the audit log or written freely |
| `DELETE /api/admin/v1/status-page/changelog/{id}` | `org:manage` | Take an entry off |

Every change is audited (`status_page.*`, `status_incident.*`).

## Errors {#errors}

| Status | `code` | When |
|---|---|---|
| 400 | `invalid_request` | a bad field (the message names it), `state` not one of the listed values |
| 403 | `forbidden` | the role lacks the permission; `trust` without the cloud admin token |
| 404 | `not_found` | a daemon, grant, suite or target that is not yours |
| 409 | `conflict`, `daemon_name_taken`, `archived`, `limit` | a revoked daemon, a name in use, an archived customer, too many SLA targets |
| 409 | `vault_not_configured` | a GPU cloud key on a deployment without a vault key (`VENOM_VAULT_KEYS`) |

## Related {#related}

- [Venom Cloud APIs overview](/docs/cloud/api-overview.md)
- [Admin API: usage, budgets and spend](/docs/cloud/api-admin-spend.md)
- [Admin API: customers and MSP](/docs/cloud/api-admin-msp.md)
- [Orchestrator operations](/docs/orchestrator/operations.md): the daemon side of mesh mode
