Your GPUs, one API.One click to start.
The daemon puts an OpenAI-compatible endpoint in front of the models on your machine. Install it, click to start, and its dashboard opens in your browser. Connect it to Venom Cloud when you want your team to reach it.
macOS, Linux and Windows (beta). Every download
How it works
Three steps, no terminal needed
- 1
Install
Download the app or run the one-line installer. It checks the SHA-256 and needs no admin rights.
- 2
It opens your dashboard
The daemon starts on 127.0.0.1 and opens its dashboard in your browser, already signed in. No tokens to copy.
- 3
Paste a join token
In Venom Cloud, Daemons, Connect a daemon gives you a join token. Paste it into Connect to Venom Cloud and the daemon joins your org. Skip this step to use it on its own.
Install
One line, or one download
Each download holds the launcher, the orchestrator, the dashboard and the Node runtime the dashboard runs on. The scripts fetch only from the public releases repository, check the SHA-256, add a launch entry, then start the daemon and open its dashboard.
macOS and Linux
curl -fsSL https://raw.githubusercontent.com/elishahbiz/venom-releases/main/install-daemon.sh | sh
Installs to ~/.venom and adds Venom Daemon to Applications (macOS) or your app launcher (Linux).
Windows, in PowerShell Beta
irm https://raw.githubusercontent.com/elishahbiz/venom-releases/main/install-daemon.ps1 | iex
Adds a Venom Daemon shortcut to the Start Menu. On Windows, stopping the daemon ends it at once.
The macOS app
Unzip it and double-click Venom Daemon. The app is not notarized by Apple, so the first time, right-click it and choose Open, or allow it in System Settings, Privacy & Security, Open Anyway. The one-line installer doesn't need this step.
Every download
An archive unpacks to venom-daemon/; run bin/venom-daemon (on Windows, bin\venom-daemon.exe.
- DownloadmacOS (Apple silicon) appVenom-Daemon-0.2.0-macos-arm64.zip
- DownloadmacOS (Apple silicon) archivevenom-daemon-0.2.0-darwin-arm64.tar.gz
- DownloadmacOS (Intel) appVenom-Daemon-0.2.0-macos-x64.zip
- DownloadmacOS (Intel) archivevenom-daemon-0.2.0-darwin-x64.tar.gz
- DownloadLinux x64 archivevenom-daemon-0.2.0-linux-x64.tar.gz
- DownloadLinux arm64 archivevenom-daemon-0.2.0-linux-arm64.tar.gz
- DownloadWindows x64 archiveBetavenom-daemon-0.2.0-windows-x64.zip
Checksums: SHA256SUMS. Release notes and every asset: the releases page.
Day to day
venom-daemon # start, or open the dashboard venom-daemon status venom-daemon logs venom-daemon stop
Remove it
venom-daemon uninstall # keeps your data venom-daemon uninstall --purge # leaves Venom Cloud, deletes the data
Who it's for
Two ways people run it
Privacy-first orgs on their own GPUs
Run the models next to your data. Your team reaches them through Venom Cloud with keys, budgets and logs, and routes marked private only ever reach your own daemons.
GPU owners joining the mesh
The same daemon can serve the public mesh. The cloud checks the work with measured latency, audits and canaries. Payouts to hosts come in a later phase; for now, public hosting starts with a waitlist.
What it runs
Everything on your machine, nothing in a container
The Venom orchestrator
An OpenAI-compatible gateway on 127.0.0.1: model aliases, fallback, circuit breaking and session-sticky cache affinity, streaming with no buffering, usage and keys in SQLite.
Your backends
Ollama, vLLM, llama.cpp and SGLang. If Ollama isn't installed, the dashboard links to its official download; then it pulls the models that fit your hardware.
A local dashboard
Local models, routes, requests, spend and the Venom Cloud connection, on 127.0.0.1. Start at login is off until you turn it on.
Want a coding agent on top? The venom CLI talks to the daemon's /v1.
The router inside
What the daemon does with each request
The daemon is the Venom router on your machine. Before a request reaches Ollama, vLLM or llama.cpp, it picks the backend, holds its place in the queue and relays the stream as it is written.
Radix tree of served prompts
A new session goes to the backend that already holds its prefix5
On a route with several self-hosted backends, the orchestrator keeps a bounded radix tree of the prompts each one served, in 512-byte blocks. A new session lands on the backend with the longest match, unless that backend is busier than the rest.
75.5% → 79.7%prefix-cache hit rate, for +0.62 µs and one allocation per request
Simulated on 3 fake vLLM targets with tight caches.
Retry budgets and backpressure
Retries that can't turn an outage into a storm12,13
Same-upstream retries draw on a budget of 20% of first attempts, and a retry the OpenAI SDK is already making isn't retried again underneath it. A connection reset counts as backpressure, not a dead upstream, so a cold start no longer trips the breaker.
30 → 13upstream calls for 10 requests to a failing upstream; breaker trips on a cold start went from 4 of 14 to 0
Orchestrator tests and load tests on a shared M2 Pro.
Streaming without buffering
The first token reaches you when the model writes it14,15
Bytes go to the client as the upstream writes them, and every stream tells proxies such as nginx not to hold events back. Request bodies are rewritten by scanning and splicing the JSON in place, not by decoding and re-encoding it.
5.05 s → 0.01 sto the first event behind nginx with gzip; a 525 KB body rewritten in 285 µs instead of 1.60 ms
make smoke against the prod image, and make bench on an M2 Pro.
Weighted priority classes
Interactive requests get most of a busy backend's slots16
When a backend is at its concurrency limit, waiting requests are served by class: interactive, then merge checks, then background work, by smooth weighted round robin at 8:3:1, first come first served within a class.
Compared
More than a model server on its own
Keep running Ollama or vLLM. The daemon sits in front of them and your provider keys, and adds routing, fallback, cache affinity and per-key limits for 49 µs a request at p50.
- Venom daemon
- Chat, responses, completions, embeddings and models, with streaming
- Ollama on its own
- ~A subset of the OpenAI API
- vLLM on its own
- Completions, chat and more
Several backends behind one endpoint1,2,4
- Venom daemon
- vLLM, SGLang, llama.cpp, Ollama and cloud providers, by model alias
- Ollama on its own
- Its own models, plus Ollama's hosted cloud models
- vLLM on its own
- One server; the vLLM project ships separate routers for many
Fallback and circuit breaking1,2,4
- Venom daemon
- Fallback before the first byte, circuit breakers per target
- Ollama on its own
- Not documented
- vLLM on its own
- ~In the separate vllm-project/router
Prefix-cache affinity1,5,2,3,4
- Venom daemon
- Session-sticky routing, and longest-prefix routing for new sessions
- Ollama on its own
- Not documented
- vLLM on its own
- ~Automatic prefix caching on an instance; session and cache-aware routing in the separate routers
Client keys, limits and budgets1,2
- Venom daemon
- Per-key rate and token limits and spend budgets, usage and cost in SQLite
- Ollama on its own
- The local API needs no authentication
- vLLM on its own
- Not checked
Listens on localhost by default1,2
- Venom daemon
- 127.0.0.1
- Ollama on its own
- 127.0.0.1:11434
- vLLM on its own
- Not checked
Reachable by your team without an inbound port6,2,3
- Venom daemon
- Optional: it dials out to Venom Cloud after you paste a join token
- Ollama on its own
- Not documented
- vLLM on its own
- Not documented
| Capability | Venom daemon | Ollama on its own | vLLM on its own |
|---|---|---|---|
| OpenAI-compatible API1,2,3 | Chat, responses, completions, embeddings and models, with streaming | ~A subset of the OpenAI API | Completions, chat and more |
| Several backends behind one endpoint1,2,4 | vLLM, SGLang, llama.cpp, Ollama and cloud providers, by model alias | Its own models, plus Ollama's hosted cloud models | One server; the vLLM project ships separate routers for many |
| Fallback and circuit breaking1,2,4 | Fallback before the first byte, circuit breakers per target | Not documented | ~In the separate vllm-project/router |
| Prefix-cache affinity1,5,2,3,4 | Session-sticky routing, and longest-prefix routing for new sessions | Not documented | ~Automatic prefix caching on an instance; session and cache-aware routing in the separate routers |
| Client keys, limits and budgets1,2 | Per-key rate and token limits and spend budgets, usage and cost in SQLite | The local API needs no authentication | Not checked |
| Listens on localhost by default1,2 | 127.0.0.1 | 127.0.0.1:11434 | Not checked |
| Reachable by your team without an inbound port6,2,3 | Optional: it dials out to Venom Cloud after you paste a join token | Not documented | Not documented |
added per request, p50
Auth, alias resolution, quirks, usage accounting and session affinity on the path. Streaming adds +65 µs p50, and nothing is buffered.
make bench on an Apple M2 Pro with a local fake upstream.
concurrent streams, 0 errors
+0.4 ms TTFT at p50, throughput 0.997× of the upstream alone, 163 MB peak memory.
Fake vLLM upstream on a shared M2 Pro.
cached on a new session's first request
Sessions share a small pool of OpenAI cache keys, so a new session finds the common prompt prefix already cached.
Live, eight new sessions with the same 5k-token prefix.
prompt reused on llama.cpp
The orchestrator leaves slot choice to builds with the host-RAM prompt cache instead of pinning, so interleaved sessions keep their cache.
llama.cpp b11459, measured through the orchestrator.
prefix-cache hit rate, simulated
A new session goes to the backend that holds the longest prefix of its prompt, unless that backend is busy.
Simulation on 3 fake vLLM targets with tight caches.
for Venom Cloud to pick a daemon, p50
11 µs at p95 across 10 eligible daemons, 65 µs across 100. Routing is not where a request's time goes.
Wall clock on Node 24.
The full comparison, with every speedup
Sources
- 1.The orchestrator: model aliases over vLLM, SGLang, llama.cpp, Ollama and cloud providers, fallback before the first byte, circuit breakers, session-sticky routing for KV-cache reuse, per-key limits and budgets, usage and keys in SQLite. README.md:87.
- 2.Ollama docs: "supports a subset of the OpenAI API"; binds 127.0.0.1:11434 by default; the local API "does not require authentication"; hosted cloud models are also offered. docs.ollama.com (openai-compatibility, faq, api/authentication). Checked 2026-10-18.
- 3.vLLM docs: an OpenAI-compatible server (Completions, Chat and more) and Automatic Prefix Caching, which reuses the KV cache of queries that share a prefix. docs.vllm.ai (openai_compatible_server, automatic_prefix_caching). Checked 2026-10-18.
- 4.The vLLM project's separate routers: production-stack (session-ID routing) and vllm-project/router (cache-aware load balancing, circuit breakers, retries). github.com/vllm-project/production-stack, github.com/vllm-project/router. Checked 2026-10-18.
- 5.Longest-prefix routing for new sessions, simulated on 3 fake vLLM targets with tight caches and 8 prompt families. A simulation, not a production measurement. CHANGELOG.md "unless that target is busy against the others (SGLang's balance".
- 6.Venom Cloud's routing: direct (a 307 redirect with a single-use ticket) or relayed through the cloud; daemons only dial out. docs/cloud-mesh.md:36-44.
- 7.make bench, Apple M2 Pro, a local fake upstream, a ~5 KB chat request with a tool; the added time includes the extra loopback hop. Re-runs on a loaded machine measured +59 to +63 µs. services/go-orchestrator/AGENTS.md "chat body with a tool; non-stream: one JSON response;".
- 8.Load test against a fake vLLM upstream (300 ms TTFT, 100 tokens/s) on a shared M2 Pro; numbers are at 1,000 concurrent streams. services/go-orchestrator/AGENTS.md "tokens/s per stream,", services/go-orchestrator/AGENTS.md "**Streaming levels**".
- 9.OpenAI prompt_cache_key pooled across sessions, measured live: eight new sessions with the same 5k-token prefix. CHANGELOG.md "live, eight new sessions with the same".
- 10.llama.cpp b11459 with the host-RAM prompt cache, interleaved sessions, measured through the orchestrator. CHANGELOG.md "pinning each session to a slot with `id_slot`".
- 11.Venom Cloud's route() wall-clock time on Node 24 for 10 eligible daemons (65 µs p50 for 100). docs/cloud-mesh.md "Only the `route()` timings are wall-clock measurements", docs/cloud-mesh.md "`route()` itself takes".
- 12.Retry budget test: 10 requests to an upstream that always answers with a 503 error, max_attempts 3 (server.TestRetryBudgetCapsRetries). services/go-orchestrator/AGENTS.md "`server.TestRetryBudgetCapsRetries` (".
- 13.Load test: a fresh orchestrator hit by 1,000 streams at once, 14 interleaved trials, before and after connection resets were treated as backpressure. services/go-orchestrator/AGENTS.md "**Failure behaviour under load**".
- 14.make smoke STAGES=prod: nginx 1.30.5 with gzip in front of the prod image, a ~5 s stream; time to the first event with and without the orchestrator's X-Accel-Buffering header. services/go-orchestrator/AGENTS.md "`make smoke STAGES=prod` checks it end to end".
- 15.Request rewrite on a ~525 KB body (a long agent session): a JSON scan and splice against a decode and re-encode, make bench, Apple M2 Pro. services/go-orchestrator/AGENTS.md "Request rewriting on a".
- 16.The orchestrator's priority classes: interactive, blocking-merge and background requests share an upstream's slots by smooth weighted round robin, 8:3:1. services/go-orchestrator/AGENTS.md:1668-1671.
Hosting on the public mesh pays nothing yet: payouts to hosts are a later phase.
Security
Dial-out only. Keys stay local.
- Listens on 127.0.0.1 only; nothing on your network can reach it
- Connects out to Venom Cloud only after you paste a join token, and never opens an inbound port
- Provider keys and model credentials stay on the machine
- Its own tokens are generated on first run and stored readable by you only
- Disconnect in the dashboard drains running jobs and deletes the cloud credential
Every release publishes SHA256SUMS, and the installers refuse a file that doesn't match.
The daemon works on its own. Venom Cloud is optional.
With a local model, nothing leaves your machine.
FAQ
Questions people ask first
Do I need Docker?
Do I need Venom Cloud?
Where do models come from?
Can I earn money by hosting?
Why does macOS warn me about the app?
What does Windows beta mean?
Serve your GPUs in one click
Install the daemon, open its dashboard, and connect it to your team when you are ready.
No Docker. No inbound ports. Your keys stay on your machine.