Skip to content
Docs menu

Admin API: daemons, fleet and reliability

Manage your org's daemons, public nodes, GPU autoscaling, prewarming, GPU grants between key groups, failover drills, model quality checks, SLAs and the public status page over /api/admin/v1.

View .md

A daemon is a Go orchestrator in mesh mode that serves your org's requests from your own machines. These routes list and control daemons and the jobs they run, choose public nodes you trust, start and stop GPU machines on demand, load models before people need them, share GPUs between key groups, rehearse outages, watch model quality, and publish SLAs and a status page. The daemon's own protocol (/api/mesh/v1) is described in docs/cloud-mesh.md §6.

Every route takes an admin credential (who can use the admin API). The operator token reads every org (?tenant=<org id> on reads, tenant_id in write bodies); a key group admin key (vak_…) sees its own key group's daemons, jobs and grants. Errors follow the common shape. A key group holds API keys, daemons, policies and GPU grants, not people; the API keeps the name team, so team_id, owner_team_id and consumer_team_id are key group ids.

A first call

export VENOM_CLOUD=https://cloud.usevenom.dev
export ADMIN="Authorization: Bearer $VENOM_CLOUD_ADMIN_TOKEN"

# Every daemon in scope, with its state, lease and last errors
curl -s "$VENOM_CLOUD/api/admin/v1/daemons" -H "$ADMIN"

# Drain one before maintenance (no new jobs; running ones finish)
curl -s -X PATCH "$VENOM_CLOUD/api/admin/v1/daemons/<daemon id>" -H "$ADMIN" \
  -H "Content-Type: application/json" -d '{"draining": true}'

# The jobs that failed in the last hour
curl -s "$VENOM_CLOUD/api/admin/v1/jobs?state=failed&since=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)" -H "$ADMIN"
import os, requests

base = "https://cloud.usevenom.dev/api/admin/v1"
h = {"Authorization": f"Bearer {os.environ['VENOM_CLOUD_ADMIN_TOKEN']}"}
offline = [d for d in requests.get(f"{base}/daemons", headers=h).json()["data"] if d["state"] == "offline"]

Daemons

Route Permission What it does
GET /api/admin/v1/daemons daemons:read {data: [...]}: every daemon in scope with its name, key group (team), labels, models, region, version, state (enrolled, active, draining, offline, revoked, retired), lease_valid and its last errors. Hidden (deleted) daemons are left out
GET /api/admin/v1/daemons/{id} daemons:read One daemon, deleted ones included
PATCH /api/admin/v1/daemons/{id} daemons:manage Any of: name (at most 128 printable characters, unique in the org, else 409 daemon_name_taken), labels (at most 64 keys of 63 characters, string or boolean values of at most 200), draining: true|false (an admin drain outlasts the daemon's own heartbeats until you set false), revoked: true (terminal: a revoked daemon can never come back, 409 conflict on later changes). Audited daemon.update or daemon.revoke
DELETE /api/admin/v1/daemons/{id} daemons:manage Revoke and hide it from listings. Audited daemon.delete
GET /api/admin/v1/daemons/{id}/shares daemons:read Which other key groups of the org may use this daemon: [{team_id, models, weight, created_at}]
PUT /api/admin/v1/daemons/{id}/shares daemons:manage Replace the shares with an array of {team_id, models?, weight?} (weight above 0, default 1; models null for all). Key groups must be in the daemon's org. Audited share.put
GET /api/admin/v1/daemons/{id}/timeline daemons:read The last 24 hours: hourly requests, errors, TTFT and duration medians, the last errors, and admin events on the daemon
GET /api/admin/v1/daemons/{id}/metrics daemons:read ?hours= (default 24): 5-minute series (GPU use, VRAM, memory, in flight, tokens per second), the last heartbeat's metrics, public-mesh credits by day and the prewarm state
GET /api/admin/v1/daemons/metrics daemons:read ?hours= (default 24): the fleet grid, hourly per daemon in scope, plus public-mesh credits today and over 30 days
GET /api/admin/v1/daemons/{id}/qos daemons:read For a public daemon: claimed against witnessed quality of service, divergence, trust state and the last verdicts
POST /api/admin/v1/daemons/{id}/trust cloud admin token only {"action": "bar" | "clear"}: bar a public daemon from the public mesh, or clear the bar

New daemons enroll with a join token (POST /api/admin/v1/join-tokens, on Admin API: orgs, members and keys).

Public nodes

The public mesh lets anyone host a daemon. Your org's trusted routing mode uses only the public nodes on your allowlist.

Route Permission What it does
GET /api/admin/v1/nodes routing:read The public node catalogue: region, models, hardware, price, witnessed QoS, trust state and age (never endpoints), each marked if on your allowlist
GET /api/admin/v1/nodes/{id} routing:read One public node: its catalogue entry, 24 hours of hourly witnessed QoS and its verdicts. A private daemon answers 404
POST /api/admin/v1/nodes/trusted routing:write {daemon_id, note?}: add a node to the allowlist. Audited routing.trusted_added
DELETE /api/admin/v1/nodes/trusted/{id} routing:write Take a node off it. Audited routing.trusted_removed

Jobs and serve sessions

Route Permission What it returns
GET /api/admin/v1/jobs logs:read The newest 500 relayed jobs in scope: id, request id, key, session, path, model, stream, state, daemon, attempt, route and fallback reasons, timestamps (created, dispatched, first byte, finished) with their hybrid logical clock stamps, status and error code. Filters: ?state= (queued, dispatched, running, succeeded, failed, cancelled), ?daemon=<id>, ?key=<id>, ?since=<RFC 3339>
GET /api/admin/v1/serve-sessions logs:read venom serve sessions bound to a daemon through the cloud (newest 500): session id, remote id, state, epoch, daemon, key, last event. ?state=bound|daemon_lost|closed|all (default: bound and daemon_lost)

GPU autoscaling

Autoscaling starts daemons on a GPU cloud (RunPod or Lambda) when demand for a pool's models builds, and stops them when idle. The cloud API key is checked, then sealed in the vault.

Route Permission What it does
GET /api/admin/v1/autoscale daemons:read The org's GPU clouds, pools, machines, demand and spend
POST /api/admin/v1/autoscale/clouds org:manage {provider, api_key}: check the key against the GPU cloud, seal it, keep the GPU listing
POST /api/admin/v1/autoscale/clouds/{id}/refresh org:manage List GPU types and prices again
DELETE /api/admin/v1/autoscale/clouds/{id} org:manage Forget the key; refused while a pool uses it
POST /api/admin/v1/autoscale/pools org:manage {name, cloud_id, …config}: a new pool (fields below)
PATCH /api/admin/v1/autoscale/pools/{id} org:manage Change a pool, or turn it off
DELETE /api/admin/v1/autoscale/pools/{id} org:manage Delete a pool with no machine left
POST /api/admin/v1/autoscale/pools/{id}/scale org:manage {n}: hold N machines for an hour, reconciled at once
POST /api/admin/v1/autoscale/machines/{id}/drain org:manage No new requests; terminated once idle
POST /api/admin/v1/autoscale/machines/{id}/terminate org:manage Terminate now at the GPU cloud

Pool configuration:

Field Meaning
gpu, gpu_count the GPU type from the cloud's listing and how many per machine
regions RunPod data center ids (empty: any); for Lambda the first is the region
image, volume_gb, ssh_key_name the machine image, disk size and SSH key (Lambda needs one)
models Ollama models each machine pulls at boot; their demand scales the pool
venom_version the daemon release to install; empty for the latest
min, max machine bounds
monthly_cap_usd the pool's monthly spend cap
scale_up {wait_p90_s, queue_depth, minutes}: add a machine when the dispatch wait p90 or the queue depth stays at or above the value for every minute of minutes
scale_down {idle_minutes}
schedule {days, from_hour, to_hour, tz, keep}: keep N machines in a window (days 0 = Sunday)
join_timeout_minutes a machine that has not joined by then is terminated and an alert sent

Prewarming

Prewarming asks your live daemons to load a gear's models before the working day, so the first request does not wait for a model load. Daemons pick the request up with their next heartbeat answer.

Route Permission What it does
GET /api/admin/v1/prewarm daemons:read The schedule, its next run and last result, and the gears it may name
PUT /api/admin/v1/prewarm daemons:manage {enabled?, gear?, days?, at?|at_minute?, tz?, keep_alive_min?} (keep_alive_min default 120). Audited prewarm.schedule
DELETE /api/admin/v1/prewarm daemons:manage Remove the schedule. Audited prewarm.removed
POST /api/admin/v1/prewarm/run daemons:manage {gear?, daemon_id?}: prewarm now, the schedule's gear (else the default gear) on every live daemon or one. Audited prewarm.run

A session can also prewarm its own models with a client key: POST /v1/prewarm (see the Inference API).

GPU grants between key groups

A grant lets another key group of the org use the owner key group's daemons, with an optional daily token quota. Usage is booked to the receiving key group (its keys, its budgets) and shown to both. A key group that is neither party gets 404.

Route Permission What it does
GET /api/admin/v1/gpu-grants daemons:read The grants the caller is a party to, with today's and 30 days' usage. ?direction=given|received
POST /api/admin/v1/gpu-grants daemons:manage over the owner key group {owner_team_id, consumer_team_id, models?, daily_token_quota?, expires_at?, weight?, note?}. Audited
GET /api/admin/v1/gpu-grants/{id} daemons:read, either party One grant with 30 days of usage by day, daemon and model
PATCH /api/admin/v1/gpu-grants/{id} daemons:manage, the owner key group {daily_token_quota?, models?, expires_at?, weight?, note?}. Audited
DELETE /api/admin/v1/gpu-grants/{id} the owner key group, or the receiving key group giving it back Revoke it. Audited

Local autopilot

The autopilot finds gear roles a local model on your daemons could take over, checks them on sampled traffic in shadow (a judge model compares answers), and applies the switch when you say so.

Route Permission What it does
GET /api/admin/v1/autopilot routing:read Candidates, shadow settings and samples, open applies
PUT /api/admin/v1/autopilot org:manage {shadow_enabled?, weekly_samples?, gpu_usd_per_hour?, judge_model?}. Defaults: shadow off, 100 samples a week, GPU at $0 an hour, no judge
POST /api/admin/v1/autopilot/apply org:manage {gear, role, model, local_model?}: the role goes to the local model, daemons first, with the paid models behind it. Audited
POST /api/admin/v1/autopilot/revert org:manage {id}: an applied role back as it was, as long as nobody changed it since. Audited

Failover drill

Rehearse an outage without touching real traffic: the dry run says where each route would fall back; the probe sends one tiny pinned request to each fallback.

Route Permission What it does
GET /api/admin/v1/failover-drill routing:read The outages worth drilling for your org
POST /api/admin/v1/failover-drill routing:read {outage: {kind, id?}}: the dry-run plan. Nothing is sent and nothing changes. kind is provider, credential, target, daemon, own_daemons, public or cloud
POST /api/admin/v1/failover-drill/probe org:manage {outage, confirm: true}: the live probe, capped, booked with source failover_drill. Audited failover_drill.probe

Model quality checks

Scheduled suites of test cases run against the models your gears use, graded by a judge, so you notice when a model gets worse. Off until you turn it on with a monthly cap above $0: nothing runs or is spent before.

Route Permission What it does
GET /api/admin/v1/quality routing:read Settings, the month's spend and estimate, suites, scoreboard, flags, runs and swaps
PUT /api/admin/v1/quality org:manage {enabled?, monthly_cap_usd?, autoswap?}. autoswap lets a flagged model be swapped out of a gear and back
POST /api/admin/v1/quality/suites org:manage {starter, models?, schedule?} adds a starter suite; any other body is a whole suite. 201
GET /api/admin/v1/quality/suites/{id} routing:read A suite with its cases
PUT /api/admin/v1/quality/suites/{id} org:manage Change a suite; a partial body keeps the rest
DELETE /api/admin/v1/quality/suites/{id} org:manage Remove it
GET /api/admin/v1/quality/runs/{id} routing:read A run and its graded cases

POST /api/admin/v1/quality/test (org:manage, {model, case, judge_model?, suite_id?}) runs one case now as a real, booked request; checks must be on and the month's cap not reached. Another org's suite or run answers 404.

Agent scorecard

Route Permission What it returns
GET /api/admin/v1/scorecard usage:read; by=member needs org:manage How agent sessions went: ?days=7|30|90, ?by=gear|team|member (default gear; team groups by key group). Read only

SLA targets and service credits

An SLA target is a promise on availability, TTFT p95 or error rate, for one of your resold customers or for one route. Each month is measured; a miss drafts a service credit that you approve (it goes on the customer's next open invoice as a negative line) or dismiss. See Admin API: customers and MSP.

Route Permission What it does
GET /api/admin/v1/sla billing:read Targets with the month's measured report and drafted credits. ?month=YYYY-MM
POST /api/admin/v1/sla billing:manage A target (fields below). Audited sla.target_created
GET /api/admin/v1/sla/{id} billing:read One target with its last months' reports and credits
PATCH /api/admin/v1/sla/{id} billing:manage Change any field except the scope and its customer or route
DELETE /api/admin/v1/sla/{id} billing:manage Remove it with its reports and draft credits; credits already on invoices stay
POST /api/admin/v1/sla/credits/{id}/{action} billing:manage {action} is approve or dismiss

Target fields: name, scope (customer or route), customer_id or route, availability_pct, ttft_p95_ms, error_rate_pct, min_requests (a month with fewer requests is not judged), credit_pct (the credit as a percent of the month's charge, default 0), deep_below_pct and deep_credit_pct (a bigger credit below a second threshold), alerts, enabled. An archived customer answers 409 archived.

Public status page

A public page at your own slug with live component status (one per model), incidents and a changelog.

Route Permission What it does
GET /api/admin/v1/status-page org:read The page's settings, incidents, changelog, and the gear and model changes you may publish
PUT /api/admin/v1/status-page org:manage Create or change it: {enabled?, slug, title, description?, components?, auto_incidents?, support_url?}. components is [{model, label}]
DELETE /api/admin/v1/status-page org:manage Remove the page
POST /api/admin/v1/status-page/incidents org:manage {title, impact, status?, models?, body, published?}: an incident with its first update. impact: minor, major, critical; status: investigating, identified, monitoring, resolved
PATCH /api/admin/v1/status-page/incidents/{id} org:manage {body?, status?, title?, impact?, models?, published?}: a body (with status) posts an update; status: resolved resolves it
DELETE /api/admin/v1/status-page/incidents/{id} org:manage Remove an incident
POST /api/admin/v1/status-page/changelog org:manage {audit_id?, title, body?}: an entry, from one of your gear or model changes in the audit log or written freely
DELETE /api/admin/v1/status-page/changelog/{id} org:manage Take an entry off

Every change is audited (status_page.*, status_incident.*).

Errors

Status code When
400 invalid_request a bad field (the message names it), state not one of the listed values
403 forbidden the role lacks the permission; trust without the cloud admin token
404 not_found a daemon, grant, suite or target that is not yours
409 conflict, daemon_name_taken, archived, limit a revoked daemon, a name in use, an archived customer, too many SLA targets
409 vault_not_configured a GPU cloud key on a deployment without a vault key (VENOM_VAULT_KEYS)