Skip to content
Docs menu

Daemons, nodes, fleet, autoscaling and model lifecycle

Connect and run your own daemons, trust public nodes, see your org's agents, rent GPUs on demand and keep gears ahead of retiring models.

View .md

A daemon is the Go orchestrator in mesh mode on a machine you own: it dials out to Venom Cloud, advertises its models and serves requests routed to it, while its provider keys stay on the machine. These pages connect and run daemons, pick public nodes you trust, show the agents your org is running, rent GPU machines when requests wait, and warn before a model your gears use is retired.

Page Permission to view What
/daemons daemons:read Your daemons as cards or a table, where requests go, prewarming.
/daemons/new daemons:read (connect: daemons:enroll) Connect a daemon: a join token, the command to run, a live wait for the first heartbeat.
/daemons/{id} daemons:read (change: daemons:manage) One daemon: status, controls, 24 h health and host metrics, models, prewarm, shares.
/join-tokens daemons:read Unused join tokens, to see and revoke.
/nodes routing:read The public node marketplace: nodes anyone can host, scored on what the cloud witnessed.
/nodes/{id} routing:read One public node: verification, quality score, uptime, models, prices, audit record, Trust.
/fleet logs:read The venom agents your machines are running, waiting ones first, and the session guard.
/autoscaling daemons:read (change: org:manage) GPU clouds, pools, machines, demand and the month's spend.
/lifecycle providers:read (switch: org:manage) Retiring models, price changes and prepaid provider credits.

Roles: owners and admins have every permission here; developers can enroll and manage daemons and change routing; viewers read.

Connect a daemon

  1. Open /daemons/new and pick the key group (private key groups first; a key group holds API keys, daemons, policies and GPU grants, not people). The page creates a join token (vjt_…) and shows the command for how you run the daemon, with this cloud's URL filled in.

  2. Run it on the machine. The page offers several ways (the one-command make daemon, Docker, a release binary, the desktop app, plain env vars). With a release binary:

    # join once; paste the token when it waits (keeps it out of shell history)
    VENOM_MESH_NAME=gpu-box-1 venom-orchestrator mesh join https://cloud.usevenom.dev -
    # then run in mesh mode
    VENOM_CLOUD_URL=https://cloud.usevenom.dev VENOM_MESH_NAME=gpu-box-1 venom-orchestrator

    For systemd, Kubernetes or your own compose file, set VENOM_CLOUD_URL, VENOM_MESH_JOIN_TOKEN (first start only) and optionally VENOM_MESH_NAME, VENOM_MESH_GEO, VENOM_MESH_ENDPOINTS, VENOM_MESH_GPUS and VENOM_MESH_MAX_JOBS on the orchestrator.

  3. The page waits for the first heartbeat and then links to the daemon.

The token is used once at enrollment; afterwards the daemon keeps its own identity. /join-tokens lists tokens not used yet, with prefix, key group, uses left and expiry, and Revoke behind a confirm. Creating a token on a plan with a daemon limit can answer 403 plan_limit_reached.

Your daemons

/daemons opens on a grid, one card per daemon: health and state in words, region, version, uptime, GPU utilisation now with a 24 h sparkline, VRAM and memory bars, jobs running and queued, tokens/s, temperature and power when the GPU reports them, the models Ollama holds warm and how many are cold, a "prewarming" badge, and for a public-pool daemon the credits earned today and in 30 days. A missing figure shows "–" with the reason on hover. ?view=table switches to the table, with filters and bulk Drain and Resume.

Above the fleet, Where requests go sets the organisation's default routing: daemons then cloud, daemons only, or cloud only.

/daemons/{id} adds Drain, Resume, Revoke (type the name to confirm), Rename and labels; a 24 h health timeline (requests, errors and TTFT per hour, the last errors, admin events); the host over 24 hours in 5-minute points; models warm or cold; its prewarm; endpoints; shares with other key groups; and for a public-pool daemon its credits and claimed against witnessed quality of service.

All metrics come from the daemon's heartbeats; it sends nothing else. Metrics are kept 7 days.

Prewarming

A daemon can load a gear's local models before the first turn needs them:

Trigger What
A session starts on a gear The CLI asks the cloud to warm the gear's models on the key group's live daemons that advertise them. Automatic.
Schedule (the Prewarm card on /daemons) One per organisation: weekdays, a time and an IANA time zone, a gear (default: the org's default gear), and how long models stay loaded (5 to 720 minutes, default 120). Runs once within an hour after its time.
Prewarm now On /daemons (every daemon) or a daemon's page (that one).

A prewarm never evicts a model in use: an Ollama model loads only when it fits beside what is loaded; otherwise it is skipped with the figures. vLLM, SGLang and llama.cpp targets are always warm.

Public nodes

/nodes lists nodes anyone can host on the public mesh, with what the cloud witnessed of them, scored on the server and filtered in the browser. It never shows an endpoint or anyone's private daemon. /nodes/{id} shows verification, the quality score and its parts, 24 hours of witnessed quality of service, uptime against the SLO over 24 hours, 7 and 30 days with the error budget's burn, models and prices (with provenance registry ids), the audit verdicts of ended epochs, and Trust, which adds the node to your organisation's allowlist. Region is self-reported. Trusted nodes serve only when your routing mode uses them; the page says so when it doesn't.

Fleet

/fleet shows every venom agent your organisation's machines publish (every 15 seconds), grouped by machine, with agents waiting on a person first: state (running, waiting, stuck, looping, idle, exited), task, cost, tokens and file leases. Rows older than the time-to-live disappear. A machine whose user set serve.public_url links to its phone inbox. The page also shows the organisation's session guard: its limits and the sessions it paused. It refreshes every 15 seconds while visible.

Autoscaling

Venom Cloud can rent GPU machines from your own RunPod or Lambda Cloud account when requests for your models wait, and return them when idle, within a monthly budget. Each machine boots the released daemon with Ollama and joins with a single-use join token. Nothing calls a GPU cloud until you connect one.

  1. Connect a GPU cloud with your API key. The key is checked by listing what can be rented, sealed in the vault, and shown by its last four characters. Refresh prices lists again.
  2. Create a pool with the guided form. The page turns it into one sentence, such as "Adds a machine when requests for qwen3:8b wait over 5 s for 3 min; removes it after 15 min idle; at most 2 machines, $300/month."
Pool field Notes
GPU type and count From the cloud's listing, with prices.
Regions Lambda: the region. RunPod: optional data centres.
Image RunPod only.
SSH key name Required by Lambda.
Models Ollama models to pull.
Daemon release Empty: latest.
min / max Machines, at most 20.
monthly_cap_usd No machine is added at or past it; autoscale.cap_reached is sent once a month.
Scale up wait_p90_s (dispatch wait p90 in seconds) or queue_depth, held over minutes. Any refused request also counts.
Schedule Optional: days, hours and time zone during which keep machines stay up.
scale_down.idle_minutes A joined machine with no job for this long is drained.
join_timeout_minutes A machine that hasn't joined by then is terminated.

Per pool the page shows machines (state requested, booting, joined, draining, terminated, uptime, $/h, cost), this month's spend against the cap, and demand charts. Scale to N now holds a target for an hour; Drain and Terminate act on one machine. Each asks for confirmation. Spend is an estimate from the listed hourly price; your cloud's bill is the truth. Ollama is the only engine pools run today.

Model lifecycle

/lifecycle ("Model lifecycle", under Usage & cost) lists models your gears, routing rows and allowlists use that are retired, retiring within the warning window, or gone from a provider's list, from a curated table shipped with the cloud plus your own catalog checks. The table can be late; a provider's own notice is the source of truth.

Section What you can do
Retiring models See each model's state (retired, soon, gone), where it is used and a replacement. Switch rewrites every gear that names the model (audited, undoable); routing rows and allowlists are linked to change where they live.
Price changes Models used in the last 30 days whose effective price changed, with the monthly impact.
Credits (/lifecycle#credits) Prepaid balances or free quotas per provider key: amount, currency, start, optional expiry. Add, edit, remove (providers:write). Refresh reads the balance from DeepSeek, OpenRouter or Moonshot with the key's own secret, only when clicked.
Settings Warning window (warn_days, default 30) and "use credits first" (credits_first_days, default 0 = off): keys with expiring credits move ahead of the org's other keys.

Warnings arrive as bell notifications and the model.retiring, price.changed and credits.expiring webhook events. venom doctor also warns about gears naming a retiring model.

Troubleshooting

Symptom Cause and fix
/daemons/new keeps waiting The daemon can't reach the cloud or the token was used or revoked. Check the daemon's log, then make a new token.
403 plan_limit_reached on a token The plan's daemon limit. Upgrade under Settings, Billing, or revoke a daemon.
A daemon shows offline Its lease lapsed. Check the machine; the daemon.offline webhook can tell you.
Pool machine ends join_timeout It never enrolled: check the image, models and the cloud's console; raise join_timeout_minutes for large models.
Refresh on a credit answers 409 currency_mismatch The credit's currency differs from the provider's answer. Edit the credit.