Admin API: daemons, fleet and reliability
Manage your org's daemons, public nodes, GPU autoscaling, prewarming, GPU grants between key groups, failover drills, model quality checks, SLAs and the public status page over /api/admin/v1.
A daemon is a Go orchestrator in mesh mode that serves your org's requests from your own machines. These routes
list and control daemons and the jobs they run, choose public nodes you trust, start and stop GPU machines on
demand, load models before people need them, share GPUs between key groups, rehearse outages, watch model quality, and
publish SLAs and a status page. The daemon's own protocol (/api/mesh/v1) is described in docs/cloud-mesh.md §6.
Every route takes an admin credential (who can use the admin API). The
operator token reads every org (?tenant=<org id> on reads, tenant_id in write bodies); a key group admin key
(vak_…) sees its own key group's daemons, jobs and grants. Errors follow the common shape.
A key group holds API keys, daemons, policies and GPU grants, not people; the API keeps the name team, so
team_id, owner_team_id and consumer_team_id are key group ids.
A first call
export VENOM_CLOUD=https://cloud.usevenom.dev
export ADMIN="Authorization: Bearer $VENOM_CLOUD_ADMIN_TOKEN"
# Every daemon in scope, with its state, lease and last errors
curl -s "$VENOM_CLOUD/api/admin/v1/daemons" -H "$ADMIN"
# Drain one before maintenance (no new jobs; running ones finish)
curl -s -X PATCH "$VENOM_CLOUD/api/admin/v1/daemons/<daemon id>" -H "$ADMIN" \
-H "Content-Type: application/json" -d '{"draining": true}'
# The jobs that failed in the last hour
curl -s "$VENOM_CLOUD/api/admin/v1/jobs?state=failed&since=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)" -H "$ADMIN"import os, requests
base = "https://cloud.usevenom.dev/api/admin/v1"
h = {"Authorization": f"Bearer {os.environ['VENOM_CLOUD_ADMIN_TOKEN']}"}
offline = [d for d in requests.get(f"{base}/daemons", headers=h).json()["data"] if d["state"] == "offline"]Daemons
| Route | Permission | What it does |
|---|---|---|
GET /api/admin/v1/daemons |
daemons:read |
{data: [...]}: every daemon in scope with its name, key group (team), labels, models, region, version, state (enrolled, active, draining, offline, revoked, retired), lease_valid and its last errors. Hidden (deleted) daemons are left out |
GET /api/admin/v1/daemons/{id} |
daemons:read |
One daemon, deleted ones included |
PATCH /api/admin/v1/daemons/{id} |
daemons:manage |
Any of: name (at most 128 printable characters, unique in the org, else 409 daemon_name_taken), labels (at most 64 keys of 63 characters, string or boolean values of at most 200), draining: true|false (an admin drain outlasts the daemon's own heartbeats until you set false), revoked: true (terminal: a revoked daemon can never come back, 409 conflict on later changes). Audited daemon.update or daemon.revoke |
DELETE /api/admin/v1/daemons/{id} |
daemons:manage |
Revoke and hide it from listings. Audited daemon.delete |
GET /api/admin/v1/daemons/{id}/shares |
daemons:read |
Which other key groups of the org may use this daemon: [{team_id, models, weight, created_at}] |
PUT /api/admin/v1/daemons/{id}/shares |
daemons:manage |
Replace the shares with an array of {team_id, models?, weight?} (weight above 0, default 1; models null for all). Key groups must be in the daemon's org. Audited share.put |
GET /api/admin/v1/daemons/{id}/timeline |
daemons:read |
The last 24 hours: hourly requests, errors, TTFT and duration medians, the last errors, and admin events on the daemon |
GET /api/admin/v1/daemons/{id}/metrics |
daemons:read |
?hours= (default 24): 5-minute series (GPU use, VRAM, memory, in flight, tokens per second), the last heartbeat's metrics, public-mesh credits by day and the prewarm state |
GET /api/admin/v1/daemons/metrics |
daemons:read |
?hours= (default 24): the fleet grid, hourly per daemon in scope, plus public-mesh credits today and over 30 days |
GET /api/admin/v1/daemons/{id}/qos |
daemons:read |
For a public daemon: claimed against witnessed quality of service, divergence, trust state and the last verdicts |
POST /api/admin/v1/daemons/{id}/trust |
cloud admin token only | {"action": "bar" | "clear"}: bar a public daemon from the public mesh, or clear the bar |
New daemons enroll with a join token (POST /api/admin/v1/join-tokens, on Admin API: orgs, members and keys).
Public nodes
The public mesh lets anyone host a daemon. Your org's trusted routing mode uses only the public nodes on your
allowlist.
| Route | Permission | What it does |
|---|---|---|
GET /api/admin/v1/nodes |
routing:read |
The public node catalogue: region, models, hardware, price, witnessed QoS, trust state and age (never endpoints), each marked if on your allowlist |
GET /api/admin/v1/nodes/{id} |
routing:read |
One public node: its catalogue entry, 24 hours of hourly witnessed QoS and its verdicts. A private daemon answers 404 |
POST /api/admin/v1/nodes/trusted |
routing:write |
{daemon_id, note?}: add a node to the allowlist. Audited routing.trusted_added |
DELETE /api/admin/v1/nodes/trusted/{id} |
routing:write |
Take a node off it. Audited routing.trusted_removed |
Jobs and serve sessions
| Route | Permission | What it returns |
|---|---|---|
GET /api/admin/v1/jobs |
logs:read |
The newest 500 relayed jobs in scope: id, request id, key, session, path, model, stream, state, daemon, attempt, route and fallback reasons, timestamps (created, dispatched, first byte, finished) with their hybrid logical clock stamps, status and error code. Filters: ?state= (queued, dispatched, running, succeeded, failed, cancelled), ?daemon=<id>, ?key=<id>, ?since=<RFC 3339> |
GET /api/admin/v1/serve-sessions |
logs:read |
venom serve sessions bound to a daemon through the cloud (newest 500): session id, remote id, state, epoch, daemon, key, last event. ?state=bound|daemon_lost|closed|all (default: bound and daemon_lost) |
GPU autoscaling
Autoscaling starts daemons on a GPU cloud (RunPod or Lambda) when demand for a pool's models builds, and stops them when idle. The cloud API key is checked, then sealed in the vault.
| Route | Permission | What it does |
|---|---|---|
GET /api/admin/v1/autoscale |
daemons:read |
The org's GPU clouds, pools, machines, demand and spend |
POST /api/admin/v1/autoscale/clouds |
org:manage |
{provider, api_key}: check the key against the GPU cloud, seal it, keep the GPU listing |
POST /api/admin/v1/autoscale/clouds/{id}/refresh |
org:manage |
List GPU types and prices again |
DELETE /api/admin/v1/autoscale/clouds/{id} |
org:manage |
Forget the key; refused while a pool uses it |
POST /api/admin/v1/autoscale/pools |
org:manage |
{name, cloud_id, …config}: a new pool (fields below) |
PATCH /api/admin/v1/autoscale/pools/{id} |
org:manage |
Change a pool, or turn it off |
DELETE /api/admin/v1/autoscale/pools/{id} |
org:manage |
Delete a pool with no machine left |
POST /api/admin/v1/autoscale/pools/{id}/scale |
org:manage |
{n}: hold N machines for an hour, reconciled at once |
POST /api/admin/v1/autoscale/machines/{id}/drain |
org:manage |
No new requests; terminated once idle |
POST /api/admin/v1/autoscale/machines/{id}/terminate |
org:manage |
Terminate now at the GPU cloud |
Pool configuration:
| Field | Meaning |
|---|---|
gpu, gpu_count |
the GPU type from the cloud's listing and how many per machine |
regions |
RunPod data center ids (empty: any); for Lambda the first is the region |
image, volume_gb, ssh_key_name |
the machine image, disk size and SSH key (Lambda needs one) |
models |
Ollama models each machine pulls at boot; their demand scales the pool |
venom_version |
the daemon release to install; empty for the latest |
min, max |
machine bounds |
monthly_cap_usd |
the pool's monthly spend cap |
scale_up |
{wait_p90_s, queue_depth, minutes}: add a machine when the dispatch wait p90 or the queue depth stays at or above the value for every minute of minutes |
scale_down |
{idle_minutes} |
schedule |
{days, from_hour, to_hour, tz, keep}: keep N machines in a window (days 0 = Sunday) |
join_timeout_minutes |
a machine that has not joined by then is terminated and an alert sent |
Prewarming
Prewarming asks your live daemons to load a gear's models before the working day, so the first request does not wait for a model load. Daemons pick the request up with their next heartbeat answer.
| Route | Permission | What it does |
|---|---|---|
GET /api/admin/v1/prewarm |
daemons:read |
The schedule, its next run and last result, and the gears it may name |
PUT /api/admin/v1/prewarm |
daemons:manage |
{enabled?, gear?, days?, at?|at_minute?, tz?, keep_alive_min?} (keep_alive_min default 120). Audited prewarm.schedule |
DELETE /api/admin/v1/prewarm |
daemons:manage |
Remove the schedule. Audited prewarm.removed |
POST /api/admin/v1/prewarm/run |
daemons:manage |
{gear?, daemon_id?}: prewarm now, the schedule's gear (else the default gear) on every live daemon or one. Audited prewarm.run |
A session can also prewarm its own models with a client key: POST /v1/prewarm (see the
Inference API).
GPU grants between key groups
A grant lets another key group of the org use the owner key group's daemons, with an optional daily token quota.
Usage is booked to the receiving key group (its keys, its budgets) and shown to both. A key group that is neither
party gets 404.
| Route | Permission | What it does |
|---|---|---|
GET /api/admin/v1/gpu-grants |
daemons:read |
The grants the caller is a party to, with today's and 30 days' usage. ?direction=given|received |
POST /api/admin/v1/gpu-grants |
daemons:manage over the owner key group |
{owner_team_id, consumer_team_id, models?, daily_token_quota?, expires_at?, weight?, note?}. Audited |
GET /api/admin/v1/gpu-grants/{id} |
daemons:read, either party |
One grant with 30 days of usage by day, daemon and model |
PATCH /api/admin/v1/gpu-grants/{id} |
daemons:manage, the owner key group |
{daily_token_quota?, models?, expires_at?, weight?, note?}. Audited |
DELETE /api/admin/v1/gpu-grants/{id} |
the owner key group, or the receiving key group giving it back | Revoke it. Audited |
Local autopilot
The autopilot finds gear roles a local model on your daemons could take over, checks them on sampled traffic in shadow (a judge model compares answers), and applies the switch when you say so.
| Route | Permission | What it does |
|---|---|---|
GET /api/admin/v1/autopilot |
routing:read |
Candidates, shadow settings and samples, open applies |
PUT /api/admin/v1/autopilot |
org:manage |
{shadow_enabled?, weekly_samples?, gpu_usd_per_hour?, judge_model?}. Defaults: shadow off, 100 samples a week, GPU at $0 an hour, no judge |
POST /api/admin/v1/autopilot/apply |
org:manage |
{gear, role, model, local_model?}: the role goes to the local model, daemons first, with the paid models behind it. Audited |
POST /api/admin/v1/autopilot/revert |
org:manage |
{id}: an applied role back as it was, as long as nobody changed it since. Audited |
Failover drill
Rehearse an outage without touching real traffic: the dry run says where each route would fall back; the probe sends one tiny pinned request to each fallback.
| Route | Permission | What it does |
|---|---|---|
GET /api/admin/v1/failover-drill |
routing:read |
The outages worth drilling for your org |
POST /api/admin/v1/failover-drill |
routing:read |
{outage: {kind, id?}}: the dry-run plan. Nothing is sent and nothing changes. kind is provider, credential, target, daemon, own_daemons, public or cloud |
POST /api/admin/v1/failover-drill/probe |
org:manage |
{outage, confirm: true}: the live probe, capped, booked with source failover_drill. Audited failover_drill.probe |
Model quality checks
Scheduled suites of test cases run against the models your gears use, graded by a judge, so you notice when a model gets worse. Off until you turn it on with a monthly cap above $0: nothing runs or is spent before.
| Route | Permission | What it does |
|---|---|---|
GET /api/admin/v1/quality |
routing:read |
Settings, the month's spend and estimate, suites, scoreboard, flags, runs and swaps |
PUT /api/admin/v1/quality |
org:manage |
{enabled?, monthly_cap_usd?, autoswap?}. autoswap lets a flagged model be swapped out of a gear and back |
POST /api/admin/v1/quality/suites |
org:manage |
{starter, models?, schedule?} adds a starter suite; any other body is a whole suite. 201 |
GET /api/admin/v1/quality/suites/{id} |
routing:read |
A suite with its cases |
PUT /api/admin/v1/quality/suites/{id} |
org:manage |
Change a suite; a partial body keeps the rest |
DELETE /api/admin/v1/quality/suites/{id} |
org:manage |
Remove it |
GET /api/admin/v1/quality/runs/{id} |
routing:read |
A run and its graded cases |
POST /api/admin/v1/quality/test (org:manage, {model, case, judge_model?, suite_id?}) runs one case now as a
real, booked request; checks must be on and the month's cap not reached. Another org's suite or run answers 404.
Agent scorecard
| Route | Permission | What it returns |
|---|---|---|
GET /api/admin/v1/scorecard |
usage:read; by=member needs org:manage |
How agent sessions went: ?days=7|30|90, ?by=gear|team|member (default gear; team groups by key group). Read only |
SLA targets and service credits
An SLA target is a promise on availability, TTFT p95 or error rate, for one of your resold customers or for one route. Each month is measured; a miss drafts a service credit that you approve (it goes on the customer's next open invoice as a negative line) or dismiss. See Admin API: customers and MSP.
| Route | Permission | What it does |
|---|---|---|
GET /api/admin/v1/sla |
billing:read |
Targets with the month's measured report and drafted credits. ?month=YYYY-MM |
POST /api/admin/v1/sla |
billing:manage |
A target (fields below). Audited sla.target_created |
GET /api/admin/v1/sla/{id} |
billing:read |
One target with its last months' reports and credits |
PATCH /api/admin/v1/sla/{id} |
billing:manage |
Change any field except the scope and its customer or route |
DELETE /api/admin/v1/sla/{id} |
billing:manage |
Remove it with its reports and draft credits; credits already on invoices stay |
POST /api/admin/v1/sla/credits/{id}/{action} |
billing:manage |
{action} is approve or dismiss |
Target fields: name, scope (customer or route), customer_id or route, availability_pct,
ttft_p95_ms, error_rate_pct, min_requests (a month with fewer requests is not judged), credit_pct (the
credit as a percent of the month's charge, default 0), deep_below_pct and deep_credit_pct (a bigger credit
below a second threshold), alerts, enabled. An archived customer answers 409 archived.
Public status page
A public page at your own slug with live component status (one per model), incidents and a changelog.
| Route | Permission | What it does |
|---|---|---|
GET /api/admin/v1/status-page |
org:read |
The page's settings, incidents, changelog, and the gear and model changes you may publish |
PUT /api/admin/v1/status-page |
org:manage |
Create or change it: {enabled?, slug, title, description?, components?, auto_incidents?, support_url?}. components is [{model, label}] |
DELETE /api/admin/v1/status-page |
org:manage |
Remove the page |
POST /api/admin/v1/status-page/incidents |
org:manage |
{title, impact, status?, models?, body, published?}: an incident with its first update. impact: minor, major, critical; status: investigating, identified, monitoring, resolved |
PATCH /api/admin/v1/status-page/incidents/{id} |
org:manage |
{body?, status?, title?, impact?, models?, published?}: a body (with status) posts an update; status: resolved resolves it |
DELETE /api/admin/v1/status-page/incidents/{id} |
org:manage |
Remove an incident |
POST /api/admin/v1/status-page/changelog |
org:manage |
{audit_id?, title, body?}: an entry, from one of your gear or model changes in the audit log or written freely |
DELETE /api/admin/v1/status-page/changelog/{id} |
org:manage |
Take an entry off |
Every change is audited (status_page.*, status_incident.*).
Errors
| Status | code |
When |
|---|---|---|
| 400 | invalid_request |
a bad field (the message names it), state not one of the listed values |
| 403 | forbidden |
the role lacks the permission; trust without the cloud admin token |
| 404 | not_found |
a daemon, grant, suite or target that is not yours |
| 409 | conflict, daemon_name_taken, archived, limit |
a revoked daemon, a name in use, an archived customer, too many SLA targets |
| 409 | vault_not_configured |
a GPU cloud key on a deployment without a vault key (VENOM_VAULT_KEYS) |
Related
- Venom Cloud APIs overview
- Admin API: usage, budgets and spend
- Admin API: customers and MSP
- Orchestrator operations: the daemon side of mesh mode