Every request goeswhere its cache is warm.
An agent re-sends a long, growing prompt on every turn. The router keeps each session on the GPU that already holds that prompt, moves it only when the move pays for itself, and fails over before the first byte. It does this in microseconds, and every claim on this page comes with how we measured it.
Two places, one idea
One router on your machine, one across your machines
Both pick a target per request with the same goal: the fewest milliseconds to the first token, without piling every request onto one backend.
The orchestrator
in the daemonA Go gateway in front of your vLLM, SGLang, llama.cpp and Ollama servers and your provider keys. It resolves a model alias to an ordered list of targets and adds 49 µs at p50.
- Fallback to the next target before the first byte, circuit breakers per target
- Session pins and a radix tree of served prompts for prefix-cache hits
- Retry budgets, so its retries don't stack on the SDK's
- Priority classes for busy GPU slots, interactive work weighted 8:3:1 over background jobs
- Streams relayed as written, never buffered
Venom Cloud
across daemonsA pure function over the daemons that are online for your team, their leases, load and measured TTFT. It picks one in 7 µs at p50 for 10 daemons, then the request goes direct or through the relay.
- Rendezvous hashing with bounded loads keeps a session on its warm daemon
- Power of two choices, so stale load reports don't send everyone to one daemon
- A score in milliseconds of TTFT, with a bonus worth the re-prefill a move would cost
- Hedged short requests, under 5% of a key group's traffic
- Hybrid logical clocks, so a request's trace stays in order
Under the hood
The algorithms, and what each one bought
Each card names the technique, says what it does for your requests, and gives the result we measured. Where a number comes from a simulation, the card says so.
Rendezvous hashing with bounded loads
A session stays where its prompt cache is warm2
Each session ranks the daemons by a 64-bit rendezvous hash. It goes to the first one under its load cap, the bound from Mirrokni, Thorup and Zadimoghaddam with a floor for idle slots. A daemon that leaves moves only its own sessions, and a full one spills along each session's own order.
61-64% → 100%of requests on the session's warm daemon at 10% load; 64-68% → 81-86% at 50%, with no added queueing
Simulated: 10 daemons × 4 slots, Zipf-popular sessions, 50k requests.
Power of two choices
No stampede onto the daemon that looked idle5
Load reports can be 10 s old, and picking the least loaded daemon from a stale snapshot sends every request to the same one. The router samples two daemons and keeps the session's owner only if it costs no more than both.
400 → 21requests on the busiest of 20 daemons in a 400-request burst (a fair share is 20)
Simulated burst between two heartbeats.
Cost in milliseconds of TTFT
Moving a long session has to pay for its re-prefill6
Every candidate gets a score in expected milliseconds to the first token: network, queue, error rate and breaker state. The owner's bonus is the cold prefill a move would cost, from each daemon's own running average of prefill time per token, so a 50k-token session doesn't leave a warm GPU to save a second of queueing.
Radix tree of served prompts
A new session goes to the backend that already holds its prefix7
On a route with several self-hosted backends, the orchestrator keeps a bounded radix tree of the prompts each one served, in 512-byte blocks. A new session lands on the backend with the longest match, unless that backend is busier than the rest.
75.5% → 79.7%prefix-cache hit rate, for +0.62 µs and one allocation per request
Simulated on 3 fake vLLM targets with tight caches.
Hedged requests
Short requests don't wait on a slow daemon8
A short non-streamed request, an embedding or a systemone call goes to a second daemon once the first passes its own p95 time to first token. The first answer wins, the other is cancelled, and you are billed once. Hedges stay under 5% of a key group's traffic.
Retry budgets and backpressure
Retries that can't turn an outage into a storm9,10
Same-upstream retries draw on a budget of 20% of first attempts, and a retry the OpenAI SDK is already making isn't retried again underneath it. A connection reset counts as backpressure, not a dead upstream, so a cold start no longer trips the breaker.
30 → 13upstream calls for 10 requests to a failing upstream; breaker trips on a cold start went from 4 of 14 to 0
Orchestrator tests and load tests on a shared M2 Pro.
Streaming without buffering
The first token reaches you when the model writes it11,12
Bytes go to the client as the upstream writes them, and every stream tells proxies such as nginx not to hold events back. Request bodies are rewritten by scanning and splicing the JSON in place, not by decoding and re-encoding it.
5.05 s → 0.01 sto the first event behind nginx with gzip; a 525 KB body rewritten in 285 µs instead of 1.60 ms
make smoke against the prod image, and make bench on an M2 Pro.
Weighted priority classes
Interactive requests get most of a busy backend's slots13
When a backend is at its concurrency limit, waiting requests are served by class: interactive, then merge checks, then background work, by smooth weighted round robin at 8:3:1, first come first served within a class.
Hybrid logical clocks
A request's trace is in the right order even when clocks drift14
Every step of a job, on the cloud and on each daemon, carries a hybrid logical clock stamp, so a failover or a resumed session reads in causal order on the request's timeline.
6,202 → 0of 46,373 causal pairs out of order, wall clock against the stamps
Simulated: 600 jobs, daemon clocks off by up to ±5 s.
Space-Saving and HyperLogLog++
Usage questions answered in milliseconds15
Top spenders, top models and distinct sessions come from mergeable sketches kept in hourly and daily rollups, not from scanning the request log. Money totals stay exact.
1,629 ms → 155 msat 1M rows, distinct sessions within 0.28% and every top-10 entry found
Postgres, 31 days of rows, scripts/bench-usage-sketches.ts.
CUSUM over audits
Public nodes that swap in a cheaper model get flagged16
Venom replays a sample of a public node's requests on a reference and scores how far the answers drift. A cumulative sum over those scores alarms on small, steady drift that no single audit would fail, and suspicion raises the audit rate.
13 auditsmedian to flag Q6_K served in place of Q8_0; the honest node never alarmed in 3,000
Qwen3-4B on llama.cpp, an RTX 5070 Ti, bootstrapped audit sequences.
Where a request may go
You decide which tiers a route may use
Your apps call one URL. The cloud checks the key, its limits and budget, then sends the request down the tiers your routing mode permits, in this order.
- venom CLI
- pyOpenAI SDK
- Any HTTP client
cloud.usevenom.dev/v1
- key auth
- RPM / TPM
- budget
- routing mode
- trace
- 1Your daemonsprivate
GPU boxes running the orchestrator in mesh mode. They dial out to the cloud, so no inbound ports, and their provider keys never leave them.
- 2Trusted public nodestrusted
Hosts from the public catalogue that you allowlist by name, ranked by trust and by QoS the cloud measures itself.
- 3Your provider keysBYOK
OpenAI, Z.ai, Qwen (DashScope) and TypeSafe keys sealed in your org's vault. Several per provider, tried by priority, with failover on 401, 429 and 5xx.
- 4Venom's keysmetered
When your plan allows it, the cloud can serve on its own provider keys and bill the usage. Nothing to set up.
Private
privateOnly your own daemons serve the route. A model that would reach a cloud provider is left out, so the request stays on your hardware or fails.
Trusted nodes
trustedBrowse the public node catalogue and allowlist the hosts you want, by name. Nothing outside that list serves the route.
Managed
managedOne endpoint, and Venom picks the node, public daemon or provider for each request by price, latency and trust.
Set a mode for the whole org or per route alias, and combine up to three in the order you want them tried.
Measured
The numbers, with their conditions
From make bench, load tests against a fake vLLM upstream, live runs and the cloud's router benchmarks. They show what the router costs and what each change bought. They are not head-to-head runs against other gateways.
added per request, p50
Auth, alias resolution, quirks, usage accounting and session affinity on the path. Streaming adds +65 µs p50, and nothing is buffered.
make bench on an Apple M2 Pro with a local fake upstream.
concurrent streams, 0 errors
+0.4 ms TTFT at p50, throughput 0.997× of the upstream alone, 163 MB peak memory.
Fake vLLM upstream on a shared M2 Pro.
cached on a new session's first request
Sessions share a small pool of OpenAI cache keys, so a new session finds the common prompt prefix already cached.
Live, eight new sessions with the same 5k-token prefix.
prompt reused on llama.cpp
The orchestrator leaves slot choice to builds with the host-RAM prompt cache instead of pinning, so interleaved sessions keep their cache.
llama.cpp b11459, measured through the orchestrator.
prefix-cache hit rate, simulated
A new session goes to the backend that holds the longest prefix of its prompt, unless that backend is busy.
Simulation on 3 fake vLLM targets with tight caches.
for Venom Cloud to pick a daemon, p50
11 µs at p95 across 10 eligible daemons, 65 µs across 100. Routing is not where a request's time goes.
Wall clock on Node 24.
The full comparison, with every speedup
Sources
- 1.make bench, Apple M2 Pro, a local fake upstream, a ~5 KB chat request with a tool; the added time includes the extra loopback hop. Re-runs on a loaded machine measured +59 to +63 µs. services/go-orchestrator/AGENTS.md "chat body with a tool; non-stream: one JSON response;".
- 2.Bounded-load session owner, simulated (npm run sim:bounded): 10 daemons × 4 slots, Poisson arrivals, 1,000 Zipf-popular sessions (s = 0.8 and 1.1), 50k requests. A simulation, not a production measurement. docs/cloud-mesh.md "(`bun run sim:bounded`, `sim/bounded-load.ts`). The setup".
- 3.Venom Cloud's route() wall-clock time on Node 24 for 10 eligible daemons (65 µs p50 for 100). docs/cloud-mesh.md "Only the `route()` timings are wall-clock measurements", docs/cloud-mesh.md "`route()` itself takes".
- 4.Load test against a fake vLLM upstream (300 ms TTFT, 100 tokens/s) on a shared M2 Pro; numbers are at 1,000 concurrent streams. services/go-orchestrator/AGENTS.md "tokens/s per stream,", services/go-orchestrator/AGENTS.md "**Streaming levels**".
- 5.Power-of-two-choices pick, simulated: a burst of 400 sessionless requests between two heartbeats on 20 daemons (asserted in sim/chaos.test.ts). docs/cloud-mesh.md "sessionless requests between two heartbeats put all".
- 6.Venom Cloud's score: expected TTFT in ms (network, queue, error penalties) less an affinity bonus worth the prefill a move would cost, up to 20 s, from each daemon's own EWMA of prefill time per uncached token. docs/cloud-mesh.md:795-831.
- 7.Longest-prefix routing for new sessions, simulated on 3 fake vLLM targets with tight caches and 8 prompt families. A simulation, not a production measurement. CHANGELOG.md "unless that target is busy against the others (SGLang's balance".
- 8.Venom Cloud's hedging: short non-streamed requests, embeddings and systemone only; sent to the runner-up after the chosen daemon's p95 TTFT; under 5% of a key group's requests a minute; the loser is cancelled and the request billed once. Tested in test/db/hedge.test.ts. docs/cloud-mesh.md:856-895.
- 9.Retry budget test: 10 requests to an upstream that always answers with a 503 error, max_attempts 3 (server.TestRetryBudgetCapsRetries). services/go-orchestrator/AGENTS.md "`server.TestRetryBudgetCapsRetries` (".
- 10.Load test: a fresh orchestrator hit by 1,000 streams at once, 14 interleaved trials, before and after connection resets were treated as backpressure. services/go-orchestrator/AGENTS.md "**Failure behaviour under load**".
- 11.make smoke STAGES=prod: nginx 1.30.5 with gzip in front of the prod image, a ~5 s stream; time to the first event with and without the orchestrator's X-Accel-Buffering header. services/go-orchestrator/AGENTS.md "`make smoke STAGES=prod` checks it end to end".
- 12.Request rewrite on a ~525 KB body (a long agent session): a JSON scan and splice against a decode and re-encode, make bench, Apple M2 Pro. services/go-orchestrator/AGENTS.md "Request rewriting on a".
- 13.The orchestrator's priority classes: interactive, blocking-merge and background requests share an upstream's slots by smooth weighted round robin, 8:3:1. services/go-orchestrator/AGENTS.md:1668-1671.
- 14.Hybrid logical clocks, simulated (sim/hlc-skew.ts): 600 jobs, 30% failovers, 20% session resumes, three function instances, daemon clocks off by up to ±5 s. docs/cloud-mesh.md "restore before the old daemon's last event".
- 15.Usage sketches (Space-Saving, HyperLogLog++) against exact GROUP BYs, 1M request rows over 31 days in Postgres (scripts/bench-usage-sketches.ts). services/nodejs-cloud/AGENTS.md "- **Measured** (`node scripts/bench-usage-sketches.ts`".
- 16.Public-mesh audits of Qwen3-4B on llama.cpp, Q8_0 registered, on an RTX 5070 Ti; CUSUM with k = 4, h = 10,000, median over 400 bootstrapped sequences of the measured audits. docs/public-mesh-hardening.md "### Real quantisations (llama.cpp, Qwen3-4B)".
- 17.OpenAI prompt_cache_key pooled across sessions, measured live: eight new sessions with the same 5k-token prefix. CHANGELOG.md "live, eight new sessions with the same".
- 18.llama.cpp b11459 with the host-RAM prompt cache, interleaved sessions, measured through the orchestrator. CHANGELOG.md "pinning each session to a slot with `id_slot`".
Put the router in front of your GPUs
The daemon runs it on your machine for free. Venom Cloud runs it across every daemon your team connects.