Inference API
Call models through Venom Cloud with any OpenAI SDK, stream answers, run background responses, list models, resume a dropped stream and route mesh-aware clients to a daemon.
These routes are the OpenAI-compatible part of Venom Cloud: the same paths and bodies as the self-hosted
orchestrator, served at https://cloud.usevenom.dev/v1. Each request is routed to one of your daemons or to the
cloud's own providers, under the key's models, limits, budgets and your org's filters. Every route here takes a
client key (Authorization: Bearer vck_…, see Credentials).
Quick start
export VENOM_CLOUD=https://cloud.usevenom.dev
export VENOM_KEY=vck_...
# A streamed chat completion
curl -sN "$VENOM_CLOUD/v1/chat/completions" \
-H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
-H "x-venom-session: support-ticket-4812" \
-d '{"model": "venom-auto", "stream": true,
"messages": [{"role": "user", "content": "Summarise this ticket in one line: ..."}]}'import OpenAI from "openai";
const client = new OpenAI({ baseURL: "https://cloud.usevenom.dev/v1", apiKey: process.env.VENOM_KEY });
const stream = await client.chat.completions.create(
{ model: "venom-auto", stream: true, messages: [{ role: "user", content: "Summarise this ticket: ..." }] },
{ headers: { "x-venom-session": "support-ticket-4812", "x-venom-project": "support-bot" } },
);
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");import os
from openai import OpenAI
client = OpenAI(base_url="https://cloud.usevenom.dev/v1", api_key=os.environ["VENOM_KEY"])
emb = client.embeddings.create(model="text-embedding-3-small", input=["invoice overdue", "payment received"])
print(len(emb.data[0].embedding))venom-auto picks a cheap, balanced or premium model per request from your org's tier map; any model id that
GET /v1/models lists works too.
Routes
| Route | What it does |
|---|---|
POST /v1/chat/completions |
Chat completions, streamed ("stream": true, SSE) or not. Tools, vision and JSON mode pass through to the serving model. |
POST /v1/completions |
Legacy text completions. |
POST /v1/responses |
The Responses API. With "background": true it starts a response you read later (Background responses). |
GET /v1/responses/{id} |
A background response as it stands; ?stream=true streams its events, &starting_after=N from after sequence number N. |
POST /v1/responses/{id}/cancel |
Stops a queued or running background response. Idempotent: an ended one comes back as it ended. |
DELETE /v1/responses/{id} |
Deletes a background response: {"id", "object": "response", "deleted": true}. A running one is cancelled and billed first. |
POST /v1/embeddings |
Embeddings. Never streamed. |
POST /v1/rerank |
Rerank documents against a query (Rerank). Never streamed. |
POST /v1/audio/speech, GET /v1/audio/voices, POST /v1/audio/transcriptions, POST /v1/audio/translations |
Text to speech, voices and transcription on your provider keys (Voice API). |
POST /v1/systemone |
Typed decisions in TypeSafe's System One wire format (the venom-decide alias). Never streamed: "stream": true is 400 invalid_decision_request. |
GET /v1/models |
The models this key can use right now (Models). |
GET /v1/models/{id} |
One of them; ids with / work as is. Unknown or not served now: 404 model_not_found. |
GET /v1/deprecations |
The org's gears that name a model its provider retires soon or has retired (Deprecations). |
GET /v1/routing |
How this key's requests were routed: per model, the daemons that served, shares, errors, latency, fallbacks (Routing report). |
POST /v1/prewarm |
Ask the daemons of the key's key group to load a session's models ahead of the first request (Prewarm). |
POST /v1/mesh/route |
For mesh-aware clients: pick a daemon and get a session ticket to call it directly (Mesh routing). |
GET /v1/mesh/jobs/{job_id}/stream |
Resume a relayed stream that broke (Resuming a stream). |
GET /v1/mesh/daemons |
The daemons this key can use: id, name, region, models, state, load. Never their endpoints. |
The generated reference with every route's method, auth and a curl line is at /docs/api/cloud. The request
bodies are OpenAI's; the self-hosted Chat and Responses page documents
the same surface.
How a request is served
Every inference request goes through the same steps before any model sees it: the key (expiry, pause, model
allowlist), the plan's request cap and the org's spend cap, the session guard, rate limits (RPM, TPM), the org's
prompt filters, venom-auto, spend guards, model guardrails, then routing. A request refused at any step costs
nothing.
Routing then chooses one of three paths. The response header x-venom-served-by and x-venom-mesh-mode say
which one ran.
| Path | When | What the client sees |
|---|---|---|
| Relay | x-venom-mesh-mode: relay, or auto when no daemon endpoint is reachable and closer; always when the org has response filters in scope and for batch urgency |
the cloud streams the daemon's answer byte for byte; x-venom-mesh-mode: relay |
| Direct | x-venom-mesh-mode: direct, or auto when a daemon's own endpoint is closer than the relay |
307 to <daemon endpoint><path>?venom_ticket=vtk.…, a single-use ticket valid 60 seconds; the client re-sends the same body there |
| Cloud providers | no daemon serves the model, or the org's routing prefers the cloud | the cloud calls the provider itself; x-venom-served-by: cloud, x-venom-daemon: cloud |
x-venom-mesh-modetakesauto(default),directorrelay; anything else is400 invalid_request.directwith no reachable endpoint is503 no_direct_path. Declare private networks the client can reach withx-venom-networks: tailnet:<name>,lan:<label>.- openai-python and fetch-based clients follow the
307. A client that does not follow redirects should sendx-venom-mesh-mode: relay. - Relayed bodies are at most 4 MiB (
413 request_too_large); larger ones need the direct path. - A daemon that fails before the first byte (429 or 5xx) is retried on the next candidate. After the first byte nothing is retried: a broken stream ends with an SSE error event you can resume.
- Requests of one
x-venom-sessionstick to one daemon, so its prompt cache stays warm. Pin a daemon withx-venom-daemon: <name or id>.
All request and response headers are listed in Request and response headers.
Streaming
"stream": true on chat completions, completions and responses gives the provider's SSE stream unchanged.
Nothing is sent before the daemon's first event, so an error before then is a normal HTTP error with a JSON
body. After the first byte the status is already 200, so a failure arrives as a final event:
data: {"error": {"message": "The relayed stream reached the function's time limit. Resume with GET /v1/mesh/jobs/{job_id}/stream?after=<last_seq>.", "type": "api_error", "code": "stream_time_limit", "job_id": "0193…", "last_seq": 412}}Event code |
Meaning | What to do |
|---|---|---|
daemon_lost |
the daemon died mid-stream | resume within 120 seconds |
stream_time_limit |
the stream reached the cloud function's time limit (800 seconds on the hosted tier) | resume |
daemon_error |
the daemon reported an error after the first byte | retry the request |
mesh_unavailable |
the relay lost its database | retry later |
If the client hangs up, the job is cancelled (unless an identical request joined it, below).
Resuming a stream
GET /v1/mesh/jobs/{job_id}/stream?after=<seq> replays a relayed job's events after sequence number seq, then
follows it live. Take job_id from the response's x-venom-job header (or the error event) and seq from the
error event's last_seq.
curl -sN "$VENOM_CLOUD/v1/mesh/jobs/0193a4c2-.../stream?after=412" -H "Authorization: Bearer $VENOM_KEY"| Parameter | Type | Default | Rule |
|---|---|---|---|
after |
integer | 0 |
the last seq you have; 0 replays from the start. Not an integer ≥ 0: 400 invalid_request |
- Only the key that made the request can resume it; any other key gets
404 not_found. - It works while the job runs and for 120 seconds after it ends; later it is
410 job_gone. - The org's response filters apply to the resumed events as they did to the first stream.
- Plain OpenAI SDKs can't resume on their own; the venom harness and your own SSE reader can.
Retries, idempotency and deduplication
Idempotency-Key(at most 255 characters, else400 invalid_request): the same key and API key within 24 hours re-attaches to a running relayed job and replays it from the start. A finished one answers409 idempotency_conflictwith itsx-venom-job(content is not kept, so it can't be replayed).- Automatic deduplication on the relay path: a deterministic request (
temperature: 0set explicitly), any/v1/embeddingsor/v1/rerankrequest, from the same API key with the same body while an identical one is still running, joins that job instead of starting a new one. The answer carriesx-venom-dedup: joinedand is billed once. Sendx-venom-dedup: offto opt out; anIdempotency-Keyalso turns it off. Direct requests are deduplicated by the daemon's orchestrator instead.
Background responses
POST /v1/responses with "background": true returns at once and keeps running. Background responses always
run on the cloud's own providers (a daemon's response would leave the cloud no way to answer the calls that
follow). A model no cloud provider serves for this key is 400 background_unsupported with
param: "background", and so is any background request while a response filter is in scope for it.
import time
from openai import OpenAI
client = OpenAI(base_url="https://cloud.usevenom.dev/v1", api_key=os.environ["VENOM_KEY"])
r = client.responses.create(model="gpt-5-mini", input="Draft a payment reminder for invoice 1042.", background=True)
while r.status in ("queued", "in_progress"):
time.sleep(2)
r = client.responses.retrieve(r.id)
print(r.output_text)- With a provider that speaks the Responses API natively, the request goes through as sent, and
GET,cancelandDELETEgo to that provider with the same key. Leaving a stream does not stop the response. - With a chat-only provider the cloud answers
status: "queued"with its own id and runs the request as streamed chat completions after the reply. It must finish within the function's limit less 30 seconds (770 seconds on the hosted tier), else it fails withbackground_timeout. GET /v1/responses/{id}?stream=true&starting_after=Nstreams its events. A stream that would outlive the function closes early: reconnect with the last sequence number you saw.- Only the key's key group can read, cancel or delete it; any other id is
404 not_found. - Billing: the budget is reserved at the start and the response is billed once when it is seen to end (your
GET, a stream, a cancel, a delete, or the cloud's own poll). One still unresolved after 6 hours is billed at the estimate. Background requests show in usage and the log explorer asis:background. - Content: a native response's row holds no content and is kept 7 days after it is billed; an emulated response's content is deleted 24 hours after it ends.
Rerank
The body is the self-hosted orchestrator's (Rerank) and is
checked before anything is spent: a bad one is 400 invalid_rerank_request.
curl -s "$VENOM_CLOUD/v1/rerank" -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
-d '{"model": "rerank-2", "query": "overdue invoice", "top_n": 2,
"documents": ["Invoice 1042 is 30 days overdue", "Welcome to the team", {"text": "Payment reminder sent"}]}'| Field | Type | Default | Rule |
|---|---|---|---|
model |
string | required | a rerank model the key may use |
query |
string | required | not empty |
documents |
array | required | 1 to 1,000 entries, each a string or {"text": "..."} |
top_n (or top_k) |
integer | all | at least 1 |
return_documents |
boolean | false |
include each document's text in the results |
max_tokens_per_doc |
number | provider default | positive |
instruct |
string | none | an instruction for models that take one |
The answer's results are best first: {index, relevance_score, document?}.
Models
GET /v1/models answers {"object": "list", "data": [...]}: one entry per model id that at least one live daemon
the key may use serves, or that the cloud's own providers serve for the key. A model with neither is left out, so
the list is what can be served now. Entries keep the orchestrator's fields and add a mesh object:
| Field | Value |
|---|---|
id, object: "model", owned_by |
as OpenAI |
context_window |
the smallest value among the serving daemons |
capabilities.vision |
true only if every serving daemon says so |
max_inflight |
the sum over the serving daemons; left out if any is unbounded |
cloud_targets |
the union of the serving daemons' cloud targets |
mesh.daemons |
how many daemons serve it (0 when only the cloud does) |
mesh.regions |
their regions |
mesh.local_only |
true when no serving daemon has cloud targets and the cloud does not serve it |
mesh.cloud |
true when the cloud's own providers serve it |
venom-auto is listed when the key may use it and at least one of the org's tier models is listed. The list is
filtered by the key's allowed_models and denied_models.
Deprecations
GET /v1/deprecations lists the models the org's gears name that are retiring (within the org's warning window,
30 days by default) or already gone. venom doctor calls it once and warns.
{"object": "list", "warn_days": 30, "default_gear": "eco", "gears": 4,
"data": [{"model": "gpt-4o-2024-05-13", "retires": "2026-12-01", "days_left": 28, "state": "soon",
"replacement": "gpt-4.1", "gears": ["deep"]}]}state is soon, retired or gone. Admins see every use and switch gears in one step on the console's Model
lifecycle page.
Routing report
GET /v1/routing is the cloud's version of the orchestrator's venom.routing view, for this key only: per model
alias, the requests, the daemons that served them (targets[].upstream is the daemon's name, model the id it
served), shares, errors, tokens, cost, latency and TTFT percentiles, and recent fallbacks with their reason.
| Parameter | Type | Default | Rule |
|---|---|---|---|
window |
duration | 24h |
<n>m, <n>h or <n>d, from 1 minute to 31 days |
recent |
integer | 10 |
recent fallbacks per alias, 0 to 100 |
The cloud's own quality probes are left out.
Prewarm
POST /v1/prewarm asks the live daemons of the key's key group that advertise the given models to load them with their next
heartbeat answer. Nothing is loaded by the call itself. The venom harness sends it with a session's gear models.
curl -s "$VENOM_CLOUD/v1/prewarm" -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
-d '{"models": ["qwen3-coder:30b"], "keep_alive_s": 1800}'| Field | Type | Default | Rule |
|---|---|---|---|
models |
string[] | required | 1 to 8 model ids, each at most 200 characters |
keep_alive_s |
integer | 3600 |
how long to keep them loaded, at most 43,200 (12 hours) |
The answer lists each model as {"model", "state": "requested", "daemons": n}, or {"model", "state": "not_served", "detail"} when no live daemon of the key group advertises it or the key may not call it.
Mesh routing for mesh-aware clients
POST /v1/mesh/route returns a daemon and a session ticket, so a client can send a whole session straight to the
daemon. It never sees a prompt.
curl -s "$VENOM_CLOUD/v1/mesh/route" -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
-d '{"model": "qwen3-coder:30b", "session": "s-123", "mode": "auto",
"needs": {"vision": false, "tools": true, "min_context": 64000}, "networks": ["tailnet:acme.ts.net"]}'| Field | Type | Default | Meaning |
|---|---|---|---|
model |
string | required | the model |
session |
string | none | the session id, for affinity and the ticket |
needs |
object | {} |
vision, tools (booleans), min_context (tokens) |
networks |
string[] | [] |
networks the client can reach, for private endpoints |
mode |
string | auto |
auto, direct or relay |
{"object": "venom.mesh.route", "mode": "direct",
"daemon": {"id": "0193…", "name": "gpu-sg-1", "region": "ap-southeast-1"},
"endpoint": "https://gpu-sg-1.example.com/v1", "ticket": "vtk.…", "expires_at": "2026-11-03T10:10:00Z",
"alternates": [{"daemon": {"id": "…", "name": "…", "region": "…"}, "endpoint": "…", "ticket": "vtk.…"}]}- Send
Authorization: Bearer vtk.…toendpoint. A session ticket lasts 10 minutes; refresh at 80% ofexpires_at. At most 2alternatesare given for your own failover. mode: "relay"(orserved_by: "cloud") means use the cloud's/v1:endpointis the cloud,ticketandexpires_atare null. An org whose prompt or response filters could act on the session always getsrelay, whatever you asked for, so filters can't be bypassed.- The key's model access, guardrails, the org's spend cap and its spend guards are checked here. A key whose hard
budget is already spent gets
429 budget_exceeded. - The daemon reports usage itself; the client reports nothing back.
GET /v1/mesh/daemons answers {"object": "list", "data": [{"id", "name", "region", "models", "state", "load": {"inflight", "queued", "slots"}}]} with state one of active, suspect (no heartbeat for 20 seconds),
offline or draining. Daemons shared with the key's key group list only the models shared with it.
Errors
The common codes are in Errors. These are specific to inference:
| Status | code |
Meaning |
|---|---|---|
| 400 | invalid_request |
not a JSON object, no model, a bad x-venom-mesh-mode, an Idempotency-Key over 255 characters |
| 400 | invalid_rerank_request, invalid_decision_request |
a bad rerank body; a streamed /v1/systemone |
| 400 | background_unsupported |
no cloud provider for a background response, or a response filter in scope |
| 403 | model_not_allowed |
the key, its role or a guardrail does not allow the model (reason says which) |
| 403 | guardrail_blocked |
an org, key group or member guardrail refused the request |
| 403 | profile_forbidden |
x-venom-profile admits nothing the org's routing allows for the model |
| 403 | unpriced_model |
the model has no price, so a budget can't limit it; set one on the console's Pricing page |
| 404 | model_not_found |
no daemon or provider key serves the model |
| 409 | idempotency_conflict |
that Idempotency-Key already finished |
| 410 | job_gone |
a resume more than 120 seconds after the job ended |
| 429 | session_paused |
the org's session guard paused this session (x-should-retry: false) |
| 502 | daemon_error |
every candidate failed before the first byte |
| 503 | no_daemon_available, no_direct_path |
nothing can serve now (reasons says why); no direct endpoint is reachable |
| 504 | dispatch_timeout |
no daemon accepted the job in time |
Related
- Venom Cloud APIs overview: credentials, errors, limits and headers
- Agent and CLI API: gears, usage, policy and the CLI's reports
- Chat and Responses: the same surface, self-hosted
- Cloud quickstart