Orchestrator API
The Go orchestrator's OpenAI-compatible and admin HTTP API (78 endpoints): routes, headers, bodies and errors.
Generated from the HTTP API section of docs/architecture.md, the orchestrator's normative contract. The OpenAI-compatible routes take Authorization: Bearer <client key> and work with the official openai SDKs; the admin API takes Authorization: Bearer $VENOM_ADMIN_TOKEN. Each endpoint below has a stable anchor, #ep-<method>-<path>.
OpenAI-compatible surface (Authorization: Bearer <client key>)
GET /v1/models— OpenAI list format; contains every route alias (e.g.venom-default) plus the concreteupstream/modelids that can be served: each upstream's listed models and its route targets, except a route target whose model the upstream doesn't have (see "Model availability" below), which is hidden so a client never picks an id that can only fail. Each entry may carry a non-standardcontext_window(tokens); for an alias it is the smallest known window among the targets that can serve right now (healthy or not yet checked, breaker not open, model available), or among all enabled targets when none can. It follows upstream health, so it can change between calls: with Ollama down,venom-defaultadvertises OpenAI's window rather than Ollama's. An alias is listed unless every one of its targets is on a disabled upstream (glmwith neitherQWEN_API_KEYnorZAI_API_KEYset): nothing could answer it, so a client choosing by this list (the CLI's gears, Venom Cloud's gear picker for a daemon's chats) never starts on it, and a request for it still gets503 no_upstreamnaming the missing keys. When some of a listed alias's targets lack their model it carries a non-standardunavailable: [{upstream, model, reason, fix}](omitted otherwise), and when some of its enabled targets are on an upstream that isn't local (neither a self-hosted kind nor a loopback/private/single-label address) a non-standardcloud_targets: ["upstream/model", …](omitted otherwise).venom doctorreads both: the second keeps it from reporting a loopback orchestrator as keeping prompts on the machine when its routes fall back to a cloud. Every alias also carries a non-standardcapabilities: {"vision": bool}: true only when every target it can reach accepts image input (the intersection over the same targetscontext_windowuses: those that can serve now, else every enabled one; a target whose capability is unknown counts as text-only). Anupstream/modelentry carries it when the orchestrator knows: the config'scapabilities.vision, Ollama's/api/showcapabilities, or a known vision family by id (gpt-4o/4.1/5, o1/o3/o4-mini, claude, gemini, qwen-vl, llava, …;quirks.KnownVision); otherwise it is omitted. The CLI turns image input on from it (venom --chromescreenshots). Because the alias flag follows health, a client can holdvision: truefrom while a text-only target was down; a request that carries an image (animage_urlorinput_imagepart) therefore skips the targets known to be text-only (configuredvision: false, or Ollama's capabilities) whenever another target remains. A route may also name animages:route (glm→glm-vision,qwen-coder→qwen-vision): a request that carries an image is then served by that route's targets instead of its own, and the alias reports the images route's vision, so a text-only model's alias still takes screenshots. While no target of the images route can serve, the alias reports its own targets' vision, and an image request goes to those of its own targets not known to be text-only, else gets400 image_input_unavailable. The request log'srouteis the images route when it served. The list object also carries a non-standard top-levelprofiles: the configured route profiles' names, sorted ([]when none), so a client can tell whether anx-venom-profileit would send exists (venom's gears are unavailable when theirs is missing). The model list itself ignoresx-venom-profile: aliases are listed with their configured targets' windows and vision, and one a profile overrides with[]is still listed. Each entry may also carry a non-standardmax_inflight: how many requests to that id the orchestrator runs at once. Anupstream/modelentry has its upstream'smax_inflight; an alias has the sum over the distinct upstreams of its enabled targets (an upstream counted once however many of its models the route names). It is omitted when unbounded (any of those upstreams sets none) or when the alias has no enabled target. The CLI sizes its scheduler's model pools from it (ADR 0009 §2.1), so its agents queue in the client rather than at the upstream's semaphore. The CLI sizes its context from the list and refreshes it at session start. A key whoseallowed_modelsnames only an alias still gets both annotations on it, by decision (2026-10-09): every answer already names the target that served it (x-venom-upstream,x-venom-model), a 503model_unavailablenames each target and its reason, andcloud_targetstells the key holder where their prompts can go, which they are owed more than the operator's routing is owed secrecy. What a scoped key does not get is the upstreams' concrete ids as list entries.GET /v1/models/{model}— one entry of that list (the SDKs'models.retrieve; the/of anupstream/modelid may be sent raw or as%2F), with the same visibility (a key'sallowed_models, hidden unavailable targets); otherwise404 {"error":{"code":"model_not_found"}}.POST /v1/chat/completions— streaming (SSE) and non-streaming.POST /v1/completions,POST /v1/embeddings— routed the same way; bodies are passed through after model rewriting. Embeddings:input(a string or an array; token arrays where the upstream takes them),dimensionsand any extension field (input_type) pass through.encoding_format: "base64"(the OpenAI SDKs' default when the caller names none) is passed on; Ollama is asked for floats, and the answer of every kind butopenaiandvllmis encoded here when it comes back as floats (float32 little-endian, base64, as OpenAI encodes it), so openai-node's default call works against any upstream. Kindbedrockserves embeddings through InvokeModel: Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0, one input per request; an array of more is a400 unsupported_request) and Cohere Embed (cohere.embed-*, a batch;input_typedefaultsearch_document), usage fromX-Amzn-Bedrock-Input-Token-Count. A usage object with onlytotal_tokens(Voyage, Jina) is read as prompt tokens.POST /v1/rerank— rerank documents against a query (the Cohere / Jina shape). Request:{"model", "query": string, "documents": [string | {"text": string}], "top_n"? (or Voyage's "top_k"), "return_documents"?: bool, "max_tokens_per_doc"?, "instruct"?}(1–1000 documents). Answer:{"id": the request id, "object": "list", "model": the served model, "results": [{"index", "relevance_score", "document"?: {"text"}}], "usage": {"prompt_tokens", "total_tokens", "search_units"?}}, results best first and cut totop_n; the documents come back only withreturn_documents: true, from the request (an upstream is never asked to echo them). Routed like any other endpoint: aliases (venom-rerank),upstream/model, fallback, breakers, retries,strategy: cheapest, client-key auth, RPM/TPM, budgets and the request log (tokens in, 0 out, the price table'sinputper 1M tokens plussearch_unitper search). Each upstream gets it in its dialect (rerank:on the upstream, else by kind):jina(POST {base_url}/rerank: vLLM, llama.cpp, SGLang, Jina, Together; the default),cohere(base_urlhttps://api.cohere.com/v2;meta.billed_unitsread),voyage(top_k,data),tei(Hugging Face text-embeddings-inference:POST {origin}/rerank{query, texts}, its/v1dropped),dashscope(Model Studio's nativePOST {origin}/api/v1/services/rerank/text-rerank/text-rerank, the default for kinddashscope:qwen3-rerank,gte-rerank-v2), and kindbedrockthrough InvokeModel (cohere.rerank-v3-5:0,amazon.rerank-v1:0, billed per query of up to 100 documents). A reranker that reports no tokens (TEI, Bedrock) is logged at the query-and-documents estimate (bytes/4). Statuses:400 invalid_rerank_request(no query, no documents or over 1000,top_n< 1, a document that is neither a string nor{text}), a reranker's own error relayed in the OpenAI error shape ({"message"}and Model Studio's{"code","message"}become{"error":{…}}), the rest as for chat. Never streamed. An alias whose every target model has "rerank" in its name, and any suchupstream/model, carriesendpoints: ["/v1/rerank"]in/v1/models.POST /v1/systemone— typed decisions in TypeSafe's System One wire format [R285]:{"state": string|object|array, "model": "venom-decide", "questions": {"<id>": {"type": "noul"|"choice"|"score", "instructions": …, "criteria"?: …}}}in, the upstream's{"model": "jev-1.13.0", "answers": {"<id>": …}, "usage": {"input_tokens", "output_tokens"}}back unchanged (x-venom-upstream,x-venom-modeladded). Never streamed. Same client-key auth, RPM/TPM limits, budgets and request log as the other/v1routes.modelis resolved like any other: thevenom-decidealias (→typesafe/jev-latest), or anupstream/modelid such astypesafe/jev-1.13.0. Only decision models serve it (an upstream of kindtypesafe, or any kind withcapabilities.decisions: true), and they serve nothing else. Body cap: 2 MiB (proxy.MaxDecisionBodyBytes, ormax_body_bytesif smaller). Statuses:400 invalid_decision_request(nostate,questionsnot a non-empty object, orstream: true),400 unsupported_endpoint(the model is a chat model, or a chat endpoint was asked of a decision model),404 model_not_found,413 request_too_large,401/429from the client key's auth and limits,503 no_upstreamnaming the missing credential (typesafe: TYPESAFE_API_KEY is not set). Upstream answers: 2xx relayed unchanged with usage logged and priced (input tokens only,config/prices.yaml);422(TypeSafe's validation error) relayed unchanged;429and529(overloaded) retried with backoff inside the retry budget, then relayed with theirretry-afterheaders;401/403from TypeSafe →502 upstream_auth_failed(the orchestrator's key is wrong, not the client's). A decision alias and itsupstream/modelids carry a non-standardendpoints: ["/v1/systemone"]in/v1/models(chat clients skip them), and a decision alias with no enabled target is not listed at all (like any alias). An alias whose every target model has "embed" in its name (venom-embed), and any suchupstream/model, carriesendpoints: ["/v1/embeddings"]the same way, one whose every target is a reranker["/v1/rerank"], and one whose every target model has "whisper" or "transcribe" in it carriesendpoints: ["/v1/audio/transcriptions"].POST /v1/audio/transcriptions— OpenAI's speech to text, amultipart/form-databody (file,model, optionallanguage,prompt,response_format,temperature, …).modelis resolved like any other (alias with fallback,upstream/model), with the same client-key auth, limits, budgets, breaker and request log. The body is spooled to a temp file (TMPDIR), capped atmax_body_bytes, and deleted when the request ends; the orchestrator reads only themodelpart (in any position, the SDKs sendfilefirst), and each attempt streams the file to the target with only that value replaced by the target's model, so fallback resends the whole form. Every other part goes through byte for byte; the reply (JSON, text, or SSE for a streamed transcription) is relayed unchanged. Statuses:400 invalid_multipart(not multipart/form-data with a boundary, or malformed beforemodel),400 missing_model,404 model_not_found,413 request_too_large; an Anthropic target can't serve it (unsupported_endpoint). The upstream must answerGET /v1/modelsfor its health check (vLLM and speaches do, whisper.cpp'swhisper-serverdoesn't). The reply'susage {"type": "duration", "seconds"}is logged asaudio_msand priced at the model'sper_second. A target on a kind with a native speech-to-text adapter (Voice, below) gets the form translated to its provider's API and the answer translated back to theresponse_formatasked for.POST /v1/audio/translations— OpenAI's speech to English text: the same multipart body, spooling, routing and statuses as transcriptions, relayed to{base_url}/audio/translations. A native adapter that doesn't translate is skipped (400 unsupported_endpointwhen no target can).Voice (ADR 0044,
internal/voice,internal/voiceapi). The contract every voice adapter, client and console page builds on:POST /v1/audio/speech— OpenAI's text to speech, JSON:{model, input, voice?: "<id>" | {"id": "<id>"}, response_format?: mp3 (default) | opus | aac | flac | wav | pcm, speed?: 0.25–4.0, instructions?, stream_format?: "audio" (default) | "sse", stream?: bool, venom_options?: {...}}.modelresolves like any other (alias with fallback, profile,upstream/model) and only targets that are text-to-speech models are tried (capabilities.tts, a voice kind's default, or a known id:tts-1*,gpt-*-tts, Kokoro, Piper, Orpheus, CosyVoice, MiniMaxspeech-*, ElevenLabseleven_*, …). The answer is the audio,200with the format'sContent-Type(audio/mpeg,audio/ogg,audio/aac,audio/flac,audio/wav), written and flushed as the provider sends it: never buffered, and a target is given up for the next only before its first byte (503, 429, no answer → next target; the provider's other 4xx relayed as JSON).stream: trueasks an adapter whose provider has a separate streaming endpoint to use it; the client-side shape is the same either way.stream_format: "sse"answerstext/event-streamwith OpenAI's events,data: {"type": "speech.audio.delta", "audio": "<base64>"}per chunk anddata: {"type": "speech.audio.done", "usage": {…}}last: passed through from an upstream that produces them (OpenAI's gpt-4o-mini-tts), built by the orchestrator around any other (itsdoneusage is{"input_characters": n}). pcm is raw signed 16-bit little-endian mono samples, 24 kHz unless the adapter (or the request'svenom_options.sample_rate, where the provider honours it) says otherwise, and the answer always names its rate:Content-Type: audio/pcm;rate=24000;channels=1andx-venom-sample-rate: 24000(the header also onwav, and on an SSE answer of either). The realtime voice runtime (ADR 0043) reads it from there.venom_optionsis venom's one extension object (extra_bodyin the SDKs):language(BCP-47 hint, for providers that take one),sample_rate(Hz, pcm and wav where the provider lets you pick),provider(an object merged last, at the top level, into the native request body the adapter sends: any provider-specific option, e.g.{"voice_settings": {"stability": 0.4}}for ElevenLabs). To an OpenAI-compatible upstream the client's body goes as sent, withmodelreplaced,streamandvenom_optionsremoved andvenom_options.provider's keys added. Headersx-venom-upstream,x-venom-model. Usage: the input's characters (Unicode code points), logged asaudio_charsand priced at the model'sper_1m_chars; key budgets reserve the worst target's cost by characters (a budgeted key can't reach an unpriced cloud TTS model:403 unpriced_model). Statuses:400 invalid_json,missing_model,missing_input,invalid_voice,invalid_response_format,invalid_speed,invalid_stream_format,model_not_tts(no target speaks);403 model_not_allowed/profile_forbidden;404 model_not_found;413 request_too_large(body over 1 MiB);429 budget_exceeded/upstream_rate_limited;502 upstream_error. A stream that breaks after the first byte is logged502 upstream_broken(499 client_closedwhen the client left).GET /v1/audio/voices?model=<alias | upstream/model>—{"object": "list", "data": [{"id", "name", "languages"?: [...], "gender"?, "preview_url"?, "provider": "<upstream name>", "model": "<upstream>/<model>"}]}: the union of the voices of every enabled TTS target of the model, in target order. Voice ids are the provider's (an alias falling back to another provider needs a voice that one knows). Kind openai answers OpenAI's built-in voices (alloy, ash, ballad, cedar, coral, echo, fable, marin, nova, onyx, sage, shimmer, verse); other OpenAI-compatible servers are askedGET {base_url}/audio/voices(Kokoro-FastAPI) thenGET {base_url}/audio/speech/voices(Speaches), and a server with neither lists none. A target whose list fails is left out and named inx-venom-voices-failed; all failing is502.preview_urlis the provider's CDN: the orchestrator never fetches it. Statuses:400 missing_model,400 model_not_tts,404 model_not_found. Not logged.- Adapters: one per provider wire format, registered by upstream kind
(
voice.Register; ADR 0044). A kind without one is served by the OpenAI-compatible adapter (openai, vllm, sglang, llamacpp, ollama, generic; Speaches, Kokoro-FastAPI, LocalAI): zero translation. A provider kind declares its traits withconfig.RegisterKind: audio only (its models serve/v1/audio/*only; chat to one is400 unsupported_endpoint),tts/sttdefaults, its auth header, and no model list (models must be listed). Kind names:elevenlabs,minimax(shared with chat),dashscope(Qwen-TTS, CosyVoice, Paraformer),byteplus-speech,fishaudio,mistral(Voxtral, shared with chat); local serverscosyvoice,qwen3-tts,indextts,sparktts,fishspeech. - Native cloud adapters (checked against each provider's docs 2026-11-03;
all three audio only, TTS + STT, models must be listed). A format a provider
can't produce, or a missing required voice, is
400with codeunsupported_by_providerbefore any call. A native transcription answersjson,text,verbose_json,srtorvtt(built from the provider's timed transcript); none of the three translates.elevenlabs— base_urlhttps://api.elevenlabs.io/v1, key inxi-api-key. SpeechPOST /text-to-speech/{voice}(…/streamwhenstream: true),output_formatmp3_44100_128, opus_48000_64,pcm_<rate>/wav_<rate>(8000–48000; no aac, flac),speed→voice_settings.speed,language→language_code; voice required. VoicesGET /v2/voices?page_size=100. ScribePOST /speech-to-text(multipart,model_id, word timestamps grouped into sentences,audio_duration_secsbilled). HealthGET /models.fishaudio— base_urlhttps://api.fish.audio,Authorization: Bearer. SpeechPOST /v1/ttswith the model in themodelheader (s1, s2-pro, s2.1-pro, drama-3),formatmp3/opus/wav/pcm (wav/pcm at 8, 16, 24, 32 or 44.1 kHz; no aac, flac), voice →reference_id(optional),speed→prosody.speed,language. Voices: your own models (GET /model?self=true), else the 50 most used public ones. ASRPOST /v1/asr(multipartaudio, model header transcribe-1 / transcribe-1-pro,durationbilled).byteplus-speech— base_urlhttps://voice.ap-southeast-1.bytepluses.com, key inX-Api-Key(a Seed Speech console key). SpeechPOST /api/v3/tts/unidirectional(model asX-Api-Resource-Id: seed-tts-2.0, seed-tts-1.0, seed-icl-2.0; the fixedX-Api-App-Key), mp3/opus/pcm; its stream of{code, data: base64}objects is decoded to raw audio as it comes, and an error code in the first object is a refusal (4xx relayed).speed→speech_rate(−50..100),language→additions.explicit_language; voice (a speaker id) required. Voices: a static subset of the official TTS 2.0 list (no list API). Speech to text: the flash recognizerPOST /api/v3/auc/bigmodel/recognize/flash(modelseed-asr-fastorvolc.seedasr.auc_turbo; wav, mp3 or ogg files, base64 in JSON, encoded as it is sent). Streaming ASR and bidirectional TTS are WebSocket only (the realtime path, ADR 0043); the standard recognizer takes URLs only and isn't offered.
- Native adapters on kinds that also chat (checked against each provider's docs 2026-11-03):
dashscope(Qwen TTS:qwen3-tts-flash):POST {api}/services/aigc/ multimodal-generation/generation({api}is the…/compatible-mode/v1base_url with/api/v1in its place),{model, input: {text, voice, language_type}},Authorization: Bearer, alwaysX-DashScope-SSE: enable. Each event'soutput.audio.datais base64 of the next piece of one streamed WAV (24 kHz mono s16le; verified live);wavrelays it,pcmrelays it without the header; other formats are400 unsupported_response_format.voicedefaults toCherry;venom_options.language→language_type;speedandinstructionsare not sent. Voices: the documented static list of 45 system voices. Error events map to statuses (InvalidParameter*400,Throttling*429,InvalidApiKey401, else 502). CosyVoice and Paraformer / qwen-asr are WebSocket APIs: realtime (ADR 0043), not here.minimax(T2A v2:speech-2.8-hd,speech-2.8-turbo, legacy 2.6 / 02 / 01):POST {base_url}/t2a_v2(https://api.minimax.io/v1),{model, text, voice_setting: {voice_id, speed}, audio_setting: {format, sample_rate, channel: 1}, language_boost},Authorization: Bearer. Withoutstreamone JSON answer with hexdata.audio; withstream: trueorstream_format: "sse"the event stream (hex per event,exclude_aggregated_audio: true). MiniMax answers errors as HTTP 200 withbase_resp.status_code: 1002/1039 → 429, 1004 → 401, 1008 → 402, 2013/1026/1027 → 400, 1000/1001/1024/1033 → 500, else 502. Formats mp3, pcm, flac, wav, opus (aac400);voicerequired (400 missing_voice);speedclamped to 0.5–2;venom_options.language→language_boost,sample_rateone of 8000–44100 (24000 for pcm/wav by default). Voices:POST {base_url}/get_voice {"voice_type": "system"}. The same upstream serves MiniMax chat; list the speech models beside the chat ones.mistral(Voxtral):POST /v1/audio/transcriptionsis OpenAI's multipart shape, so no adapter: an upstream of kindmistral(orgeneric) athttps://api.mistral.ai/v1withvoxtral-mini-latestpasses through; itsusage.prompt_audio_secondsis logged asaudio_ms. In every native adaptervenom_options.provideris merged last into the provider's body: an object value is merged one level deep into an object already there ({"voice_setting": {"emotion": "calm"}}keepsvoice_id), anything else replaces it.
- Local TTS servers (wire formats read from each project's source on main, 2026-11-03; kinds
registered with
KindTraits.Local, so they cost $0 whereverbase_urlpoints; audio only, TTS, models must be listed, no key needed (anapi_key_envgoes out asAuthorization: Bearer)). Voice cloning:venom_options.reference_audiois adata:audio/…;base64,…URL, anhttp(s)URL (handed to an upstream that fetches it itself, refused for one that doesn't: the orchestrator never fetches it), or a file path read from the upstream'sreference_audio_dir(refused when unset;.., absolute paths elsewhere and symlinks leaving the directory are400). The clip must be WAV, MP3, FLAC or Ogg by its magic bytes and at most 10 MiB (a data: URL is also bound by the 1 MiB request body);venom_options.reference_textis its transcript.cosyvoice— CosyVoice's FastAPI server (runtime/python/fastapi/server.py, port 50000; base_url is its root). Multipart form fields to/inference_sft{tts_text, spk_id}(a voice),/inference_instruct(+instruct_text, CosyVoice 1),/inference_zero_shot{tts_text, prompt_text, prompt_wav}(a clip with its transcript),/inference_cross_lingual(a clip alone),/inference_instruct2{tts_text, instruct_text, prompt_wav}(a clip withinstructions). CosyVoice 3 (cosyvoice3in the model id) getsYou are a helpful assistant.<|endofprompt|>ahead of the transcript or instruction;<|endofprompt|>is appended to instructions. The answer is headerless s16le mono PCM at 22050 Hz (model ids with300M: CosyVoice 1) or 24000 (2 and 3);wavgets a streamed header in front, other formats400. No speed, voice list or health route.venom_options.providerkeys become form fields.qwen3-tts— Qwen3-TTS open weights served by vLLM-Omni (vllm serve <model> --omni; the Qwen3-TTS repo has no HTTP server), base_urlhttp://host:8091/v1.POST /audio/speechwith OpenAI's fields pluslanguage(fromvenom_options.language: Chinese, English, …),sample_rate,stream: true+stream_format: "audio"forstream(pcm/wav only;ssepassed through),task_type: "Base"+ref_audio+ref_text(orx_vector_only_mode) when a clip is given,task_type: "VoiceDesign"for aVoiceDesignmodel (instructions required).instructionsis the model's instruct. Rate fromx-audio-sample-rate, else 24000. VoicesGET /audio/voices({voices: [...], uploaded_voices: [{id, name}]}).indextts— IndexTTS2 / 2.5 served by vLLM-Omni (the index-tts repo's README route; base_url…/v1). The samePOST /audio/speechwithref_audiofrom the clip (required unlessvoicenames a voice uploaded to the server),instructions→extra_params: {emo_text, use_emo_text: true},venom_options.language→extra_params.lang(zh, en, ja, es, ar);emo_vector,emo_alpha,emo_audiothroughvenom_options.provider.extra_params. 22050 Hz. Voices as qwen3-tts.sparktts— Spark-TTS on Triton Inference Server (runtime/triton_trtllm; the repo has no REST server).POST /v2/models/{model}/inferwith Triton v2 JSON tensorsreference_wav(FP32 [1, N]),reference_wav_len,reference_text,target_text; the answer's FP32waveformis converted to s16le, 16 kHz, pcm or wav (decoded whole: Triton's JSON is one object). A clip is required: a 16 kHz 16-bit or float WAV, first channel, at most 60 s. The shipped model only clones;gender,pitch,speedinvenom_options.providerbecome extra BYTES inputs for a repository that declares them.fishspeech— the self-hosted Fish Speech / OpenAudio API server (tools/api_server.py, port 8080; base_url its root).POST /v1/ttsJSON{text, format: wav|pcm|mp3|opus, reference_id (voice), references: [{audio: base64, text}] (the clip), streaming}; streaming is wav only, so a streamed pcm request streams wav with the header taken off. 44.1 kHz (opus 48 kHz); no speed or language field. VoicesGET /v1/references/list; healthGET /v1/health; optional bearer key. Not Fish Audio's cloud API (fishaudio: model header, prosody, sample_rate).- Ollama needs no adapter (text to speech announced in v0.40.11, Linux and Windows in v0.40.13):
kind
ollamapasses/v1/audio/speechthrough (Ollama makes wav or pcm only, 24 kHz,speed1, noinstructions; its/v1/audio/transcriptionstakes WAV). llama.cpp'sllama-serverhasPOST /v1/audio/transcriptions(passes through) but speaks onPOST /tts(float32 pcm), not/v1/audio/speech, so it isn't a TTS target yet.
- Capabilities:
capabilities.tts/capabilities.stt(per upstream or inmodel_info) inorchestrator.yaml;/v1/modelsreportscapabilities: {tts, stt}on speech and transcription models and aliases, andendpoints: ["/v1/audio/speech"]on a speech model or an alias whose every target speaks (venom-speak). - Venom Cloud serves the same four paths on its own providers (the org's
keys, then the platform's), never relayed to daemons; its
/v1/modelslists a voice model withendpoints["/v1/audio/speech", "/v1/audio/voices"]or["/v1/audio/transcriptions", "/v1/audio/translations"]. Uploads are capped at 4.5 MB there (docs/cloud-mesh.md§7.7.9).
Realtime transcription (ADR 0043,
internal/realtime, the adapters ininternal/voice; design in docs/voice-agents.md §5.1).GET /v1/realtime?intent=transcription[&model=<alias>]is a WebSocket (subprotocolrealtimewhen offered) speaking OpenAI's realtime transcription events in front of any streaming speech-to-text backend. Auth as on every/v1route:Authorization: Bearer <key>, or OpenAI's browser-style subprotocolopenai-insecure-api-key.<key>(what the SDKs'OpenAIRealtimeWebSocketsends); a key in the query string is400 key_in_url. Before the upgrade:400 unsupported_intent(any intent buttranscription),426 websocket_required, and, whenmodelis in the URL, the routing refusals (403 model_not_allowed/profile_forbidden,404 model_not_found,400 model_not_realtimewhen no target streams,429 budget_exceeded). Key RPM limits, profiles and key budgets apply; a budgeted key's session reserves a minute of its priciest target's audio.- Session. The server sends
session.createdat once (before routing;transcriptionnull). The client's first message must besession.updatewithin 5 s (else close 4408; anything else, 4400):{type: "session.update", session: {type: "transcription", audio: {input: {format: {type: "audio/pcm", rate: 8000 | 16000 | 24000 (default)} | {type: "audio/pcmu"} | {type: "audio/pcma"}, transcription: {model, language?, languages?, prompt?, keywords?}, turn_detection: null | {type: "server_vad", silence_duration_ms?, threshold?}}}}, venom?: {partials: "replace"}}(venomalso accepted insidesession). Withoutmodelin the URL the session routes ontranscription.model. The orchestrator then opens the upstream session, falling back across the alias's targets (breaker,max_inflight; a target that fails its dial or its own session start is skipped and counted), and answerssession.updatedwith the session as accepted (transcription.modelis the alias asked for) plusvenom: {upstream, model}. Audio the client sends before that waits in the socket. Latersession.updates geterrorsession_locked. - Client events:
input_audio_buffer.append {audio: base64}(at most 1 MiB decoded, elseerroraudio_chunk_too_large),input_audio_buffer.commit(ends the item;errorinput_audio_buffer_commit_emptywhen nothing was appended),input_audio_buffer.clear. - Server events (each with an
event_id):input_audio_buffer.committed {item_id, previous_item_id}(for client commits and for items the backend's VAD ended),input_audio_buffer.cleared,input_audio_buffer.speech_started {item_id, audio_start_ms}/speech_stopped {item_id, audio_end_ms}(backends that run VAD),conversation.item.input_audio_transcription.delta {item_id, delta}(append-only: for a backend whose partials are revisable, only text it marks stable),venom.transcription.partial {item_id, text, stable_len}(whenvenom.partialsisreplace: the whole current hypothesis;stable_lenis the prefix, in Unicode code points, that won't change),conversation.item.input_audio_transcription.completed {item_id, content_index, transcript, venom?: {confidence?, words?: [{text, start_ms, end_ms}], language?}}for every item,…failed {item_id, error}, anderror {error: {type, code, message, retryable}}. Items are nameditem_<request id>_<n>and complete in order. - Close codes: 1000 normal; 1008 policy (model not allowed, budget); 1011 the upstream failed after
session.updated(errorwithretryable: truefirst; the client reconnects and the orchestrator may pick another target); 1013 no upstream could start the session (retryable); 4400 bad first message or model; 4408 nosession.updatewithin 5 s. - Never buffered. Each append is decoded, converted (G.711 / PCM16 at 8, 16, 24 kHz,
internal/audio) and handed to the upstream writer at once; each upstream event is translated and written at once. Buffers are reused (no allocation per frame). An upstream that stops taking audio never stalls the client: past 1 s of queued audio frames are dropped and counted (dropped_msin the orchestrator's log line for the session), and a commit it can't take within 1 s ends the session with 1011. - Formats. Each adapter lists the formats its backend takes; the client's format is used when it is one of them, else the audio is converted to the adapter's first.
- Log and price. One request-log row per session, written at close (status 200 for a normal close,
else the close's HTTP equivalent: 400 for 4400, 408 for 4408, 403 or 429 for 1008, 502 for 1011, 503 for
1013, and 499 when the client vanished):
stream: true,latency_msthe session's length,ttft_msthe first transcript,audio_msthe audio the backend billed (when it reports it) or the audio sent to it, priced at the model'sper_second(0 for a local upstream). /v1/models: an alias whose targets all stream carriesendpoints: ["/v1/realtime?intent=transcription"](both that and/v1/audio/transcriptionswhen its models also take multipart transcriptions) andcapabilities.stt.- Kinds:
funasr(FunASR'sfunasr-wss-server-2pass,base_url: ws://host:10095, no key; audio only, no model list, its health check a WebSocket open and close);openai({base_url}/realtime?intent= transcription, the session passed through with the model rewritten:gpt-live-transcribeandgpt-realtime-whispertake no server VAD, soturn_detectiongoes as null;gpt-live-transcribeandgpt-transcribegetlanguagesandkeywords, the otherslanguageand the keywords inprompt; G.711 passes through, PCM goes at 24 kHz; items are re-ordered when completions arrive out of order);dashscope(Model Studio, the WebSocket root derived frombase_url:qwen3-asr-flash-realtimeover/api-ws/v1/realtime, OpenAI-style events with anevent_idon each and a revisablestash;paraformer-realtime-v2,paraformer-realtime-8k-v2(Beijing only) andfun-asr-realtimeover/api-ws/v1/inferencerun-task sessions, a commit finishing the task);elevenlabs(scribe_v2_realtime,{base_url}/speech-to-text/realtime,commit_strategymanual or vad, µ-law passed through asulaw_8000; no clear, so a cleared item is committed and its transcript dropped);whisperlive(WhisperLive,base_url: ws://host:9090, no key, 16 kHz int16; it has no flush, so a commit sendsreset_transcriptand the acknowledgement ends the item with the trailing segment as last seen; health is a WebSocket open and close). Compose profileasrruns a FunASR server on 127.0.0.1:10095. Venom Cloud does not relay WebSockets: realtime goes to a local orchestrator or a reachable daemon.
- Session. The server sends
Venom Line's media bridge (ADR 0061 §7,
internal/commsmedia, VEN-277).GET /v1/comms/mediais a WebSocket a phone provider streams a live call to; it is not behind client auth (the provider holds no key) and exists only withVENOM_COMMS_MEDIA_SECRET. The credential is the cloud's ticket,vml1.<base64url(JSON {c: call, o: org, n: nonce, e: unix expiry, p: sinch|twilio|fake, r: 8000|16000})>.<base64url(HMAC-SHA256(secret, "vml1." + payload))>, valid 120 s and single use (nonces kept until expiry), in?ticket=(Sinch's stream URL, checked before the upgrade:403 bad_ticket), or in the first message (Sinch'sconnectheadersvenom-ticket, Twilio'sstartcustomParameters.ticket; close 4408 without one, 4403 when bad). Also404 comms_media_not_configured,426 websocket_required,503 too_many_calls(VENOM_COMMS_MAX_CALLS).- Setup. The bridge posts the ticket to
{VENOM_COMMS_CLOUD_URL}/api/integrations/comms/line/sessionwithX-Venom-Line-Proof: base64url(HMAC-SHA256(secret, "session." + ticket))and gets the line's settings and a session token (vls_…, held only by the bridge); thenPOST {VENOM_COMMS_SERVE_URL}/v1/callswithtransport: "bridge",output_rate(the provider's rate),context(the org's instructions asline_instructions) andline(opening, voice, tools,max_call_s,max_usd), and opens the returned media socket. Sinch then gets{"command":"answer"}. - Media. Sinch [R2090]: binary PCM16 LE at the connectStream rate (16 kHz asked) both ways,
clear; Twilio Media Streams [R1708]: base64 µ-law 8 kHzmedia,mark,clear,dtmf,stop. Caller audio goes to the agent as PCM16 16 kHz (µ-law decoded and resampled withinternal/audio), agent audio comes back as PCM16 at the provider's rate and is re-encoded. Nothing is buffered: each frame is converted and written as it arrives, in per-call buffers (no allocation per frame in the bridge's code). Marks are Twilio's own; for Sinch the bridge answers them from a playout clock (audio sent, played in real time). Agentclear(barge-in) goes straight to the provider. - Tools and transfer. The agent's
tool {id, name, arguments}andtransfer {reason}go to…/comms/line/toolswith the session token; the answers come back astool_result/transfer_result. - After the call the bridge reads
GET /v1/calls/{id}and/transcriptfrom venom serve and posts the outcome, the caller-heard transcript, cost and latencies (end of speech to first audio and barge-in p50/p95 from the voice loop, the bridge's own per-frame µs in and out) to…/comms/line/report. - The hosted media edge that relays to a daemon over its mesh connection is not built: an org runs its own
orchestrator with a public
wss://ingress (docs/public/comms/venom-line.md).
- Setup. The bridge posts the ticket to
POST /v1/images/generations— OpenAI's image API:{model, prompt, n?: 1–10, size?: "WxH" | "auto", response_format?: "url" | "b64_json", …}→{created, data: [{url | b64_json, revised_prompt?}], usage?}.modelresolves like any other (alias with fallback across targets,upstream/model); a target is a kindsdapibackend (AUTOMATIC1111 / Forge / SD.Next / stable-diffusion.cpp server,POST {base_url}/sdapi/v1/txt2img, which also takesnegative_prompt,steps,seed,cfg_scale,sampler_name) or an OpenAI-compatible one ({base_url}/images/generations: OpenAI, LocalAI). Backends are always asked for base64;url(the default) is a link into the orchestrator's own file store,/v1/files/{id}/content?exp=&sig=, that works without a key for an hour (the file is the key's, purposeimage_generation, kept 24 h). Errors:400(missing_prompt,missing_model,invalid_n,invalid_response_format, a bad size),403 model_not_allowed,404 model_not_found, a backend's own 4xx relayed,502 upstream_errorwhen every target failed. One request-log row each./v1/modelsmarks image models and aliasesendpoints: ["/v1/images/generations"].Batches and files (OpenAI's Batch API;
internal/batch). Files and batches belong to the client key that made them (another key gets 404; open mode: one shared owner).POST /v1/files—multipart/form-datawithfileandpurpose(batch,user_data,assistants,vision,evals,fine-tune), at most 200 MB, streamed to$VENOM_DATA_DIR/files/→ the file object{id: "file-…", object: "file", bytes, created_at, filename, purpose, status: "processed"}.GET /v1/files?purpose=&limit=&after=(a list object, newest first),GET /v1/files/{id},GET /v1/files/{id}/content(the bytes),DELETE /v1/files/{id}(409 file_in_usewhile an unfinished batch reads it). Batch results are files of purposebatch_output.POST /v1/batches{input_file_id, endpoint, completion_window: "24h", metadata?}→ the batch object (id: "batch_…",status,request_counts,output_file_id,error_file_id, the*_attimes,errors), as OpenAI's.endpointis/v1/chat/completions,/v1/completions,/v1/embeddingsor/v1/responses; the input file must have purposebatch. Each input line is{custom_id, method: "POST", url: <endpoint>, body}; validation (uniquecustom_id, the batch's endpoint, amodel, nostream, ≤ 50,000 lines) fails the batch with line-numberederrors.GET /v1/batches/{id},GET /v1/batches?limit=&after=,POST /v1/batches/{id}/cancel(cancelling, thencancelled;409 batch_not_cancellableonce it ended). Output lines are{id, custom_id, response: {status_code, request_id, body}, error: null}: 2xx in the output file, other statuses in the error file, and a line that never got a response (cancelled, expired, its key deleted) as{…, response: null, error: {code, message}}. A batch past its 24 h window endsexpiredwith what finished.- Execution: one batch at a time, oldest first. Each line is served in-process as the batch's key
through the same handler as
/v1(routing, fallback, budgets, the request log), withx-venom-priority: background(the background queue at a full upstream, flex on OpenAI) andx-venom-session: <batch id>, so the Sessions view sums the batch's cost. Up to 4 lines at once while no foreground request is in flight, 1 otherwise; while idle, a line for an alias whose first target is in the cloud goes to the alias's first servable local target. A 429 or 5xx is retried up to 5 times with backoff. On an upstream withnative_batch: true(kindopenai), a batch whose every line resolves there is uploaded to the provider's own Batch API instead (/files,/batches, polled every 30 s, cancel forwarded); its output and error files are copied back and each line is logged at half the list price. State (files, batches, per-line results) is in SQLite, so a restart resumes a batch where it stopped.native_batchalso takes kinddashscope(Model Studio's OpenAI-compatible batches) and kindanthropic(Message Batches, chat completions only: lines translated like a synchronous request, results back as chat completions). Native batches are polled alongside the batch running here.
Provider tiers and batch requests (
x-venom-service-tier,x-venom-urgency).x-venom-service-tier: auto | default | flex | priorityasks for that tier where the body has noservice_tier, over a route profile's and the background lane's (default= standard, outright). OpenAI: sent as is (flex only on flex models; a flex 429 is sent again at once on the default tier). Anthropic: priority and auto →auto, default →standard_only, flex → none. Other kinds: nothing. Any other value →400 invalid_service_tier. The answer carriesx-venom-service-tier: <tier sent>. Flex and priority are priced at theirflex:/priority:rates; the request log keepsservice_tierandsaved_usd(standard price minus billed), summed byGET /admin/v1/spend→tiers: [{tier, requests, cost_usd, saved_usd}].x-venom-urgency: batch(orx-venom-service-tier: batch) on/v1/chat/completions,/v1/completions,/v1/embeddings,/v1/responses: the request becomes a batch request. A model whose first target isnative_batchis collected with the key's other requests for it (batch_routing.window, default 30 s, ormax_lines, default 1000) into one provider batch at half price; anything else (and a request under a route profile, or from a key with a hard spend limit) runs at once on the background lane. Answer:202{id: "breq_…", object: "batch_request", endpoint, model, status: queued | in_progress | completed | failed | cancelled | expired, mode: provider_batch | background | "", batch_id, created_at, completed_at, response: {status_code, request_id, body} | null, error: {code, message} | null}withLocation: /v1/batch_requests/{id}andx-venom-batch-request: <id>.x-venom-batch-wait: <seconds>(≤batch_routing.max_wait, default 10 m) waits and answers like the synchronous call (or 202 when it isn't back).stream: trueon chat completions:200 text/event-streamat once, SSE comments every 15 s, then the result aschat.completion.chunkevents and[DONE](other endpoints stream on the background lane,x-venom-urgency: backgroundon the answer).x-venom-callback-urlPOSTs the finished object there, only to a host listed inbatch_routing.callback_hosts(else400 callback_host_not_allowed).x-venom-urgency: backgroundserves now on the background lane;nowchanges nothing; anything else → 400.GET /v1/batch_requests?limit=&after=(newest first, without responses),GET /v1/batch_requests/{id}?wait=<seconds>(long-poll up tomax_wait),POST /v1/batch_requests/{id}/cancel(whilequeued;409 batch_request_not_cancellableafter). Owned by the key that made them; kept 7 days after they finish.
POST /v1/search— venom's own: web search over the configured backends (search:inorchestrator.yaml, aliasvenom-search), so search keys stay on the server. Body:{"query": string (required, ≤ 1000 bytes), "model"?: "venom-search" | a target's name (forces that backend), "count"?: 1–10 (default 8; more is capped at 10), "time_range"?: "day"|"week"|"month"|"year", "allowed_domains"?: [host], "blocked_domains"?: [host], "language"?, "answer"?: bool}(answer: truetries the backends that write a grounded answer first: openai, dashscope, tavily). Answer:{"object": "search.result", "query", "model", "backend": the target whose results lead, "results": [{"title", "url", "snippet", "published"?, "source": target}], "answer"?: string, "citations"?: [{"url", "title"}], "attempts": [{"target", "reason", "results", "ms"}], "usage": {"calls", "tokens_in", "tokens_out"}, "cost_usd"}. Targets are tried in order; one without its key, base URL or upstream is skipped with no request; one that fails or finds nothing hands over (reason:5xx,429,auth,timeout,bot_check,error,empty,breaker_openafter 3 failures in a row, for 60 s); one with fewer thanmerge_below(4) results isthinand merged once with the next one's. Results are deduplicated (by canonical URL: host withoutwww., path without a trailing slash or index file, tracking parameters such asutm_*,fbclid,gclid,srsltiddropped; then a near-duplicate snippet, 80% of the words shared; then a repeated title on the same host without a snippet), trimmed (skip links, navigation and cookie-banner text, a leading date, the title repeated in the snippet, the site name at the end of a title; tracking parameters and#:~:text=fragments are removed from the returned URL;publishedis a date), cleaned (control characters, one line, title ≤ 200, snippet ≤ 300 chars cut at a word) and domain-filtered, in the backend's order, so the same backend answer is always the same bytes. Client-key auth andallowed_models(the key must listvenom-search, or the forced target), RPM, budgets (pre-charged at the costliest target's fee plus 20k/1k tokens of its model), the request log (route: venom-search,upstream= the leading target,attempt_log,cost_usd= per-call fees plus the grounded backends' tokens at list price) and the spend ledger are the chat path's. Headers:x-venom-upstream,x-venom-model(the target),x-venom-fallback(the reasons of the targets passed over). Errors:400 invalid_search_request,403 model_not_allowed,404 model_not_found(unknown target),429 budget_exceeded,502 search_failed(every target failed; the last error named),503 no_upstream(none enabled, with why).GET /v1/search/backendslists the targets:{name, backend, enabled, reason?, call_usd, answer, model?}(no keys, no addresses).backend needs per call (list price, 2026-10-11) answer searxngbase_url(SEARXNG_URL)$0 (self-hosted) no brave/tavily/exaapi_key_env$0.005 / $0.008 / $0.005–0.007 tavily duckduckgonothing $0 (HTML scraping, bot checks) no openaiupstream:a kind openai upstream,model$0.01 per search + tokens (gpt-5.4-mini ≈ $0.017 a query) yes, with url_citationsdashscopeupstream:a kind dashscope upstream,model(qwen-plus; qwen3.8 refuses)tokens + search fee (≤ $0.01, unpublished) yes zaiupstream:a kind zai upstream,model= engine (search_pro)$0.01 no call_usdon a target overrides its fee; a target on a local base URL costs 0.POST /mcp— the same search as an MCP server (Streamable HTTP transport, JSON responses, no SSE stream:GETanswers 405; protocol versions 2025-11-25, 2025-06-18, 2025-03-26), so Claude Code, Cursor, Zed and other MCP clients use venom's search and keys with one config entry. Client-key bearer auth as on/v1. Methods:initialize,ping,tools/list,tools/call; notifications get202. Tools:web_search(query,count,answer,fetch: also the text of the top 0–3 pages,time_range,allowed_domains,blocked_domains; text content plusstructuredContentwithresults,answer,citations,pages; the text isResults for "<query>":then per resultN. title (date), the URL and the snippet on their own lines, with no backend name, timing or cost, so identical results are identical text in the agent's context) andfetch_page(url,max_chars≤ 40,000: a public page as text; scripts, navigation and other chrome dropped). Page fetches refuse loopback, private, link-local, CGNAT and other non-public addresses at connect time (after DNS, redirects included), credentials in URLs and non-text types. venom's own harness calls/v1/searchdirectly, not MCP. Claude Code (.mcp.json):{ "mcpServers": { "venom-search": { "type": "http", "url": "http://localhost:8787/mcp", "headers": { "Authorization": "Bearer ${VENOM_API_KEY}" } } } }GET /v1/usage[?session=<id>&sessions=5]— venom's own: the calling key's spend and budgets, for the CLI's/usageandvenom usage. It takes no key parameter; the key is the one that authenticated the request (in open mode, the requests logged without a key), so a client can never read another key's figures. Body (object: "venom.usage"):now;key{id, name, team};budget{daily, monthly}, each{spent_usd, reserved_usd, soft_usd, hard_usd, state: ok|soft|exceeded|none, resets_at}(UTC day / month, as/admin/v1/keysshows it);today,last_7_days(today and the 6 UTC days before it) andmonth(UTC month to date), each{from, to, requests, tokens_in, tokens_out, cost_usd, cached_tokens, cache_hit_ratio, by_model: [{model, local, requests, tokens_in, tokens_out, cost_usd, cached_tokens}]}, figures from the spend ledger by key id (local upstreams are $0, their tokens counted);top_sessions: today's costliest sessions of this key (sessions, default 5, at most 50), each{session_id, first_ts, last_ts, requests, errors, tokens_in, cached_tokens, tokens_out, cost_usd, subagents, top_model}with its subagent sessions (x-venom-parent-session) rolled in;session:?session=<id>in the same shape plusmodels, ornullwhen this key has no logged request in it;log,notes. Cached tokens,localand sessions come from the request log, which records key names: they count only rows logged under the caller's name, and when another key (live, or deleted but still in the ledger) has that name they are left out (log: false,cached_tokens: null, a note) rather than mixed. A session id another key also used shows only the caller's rows.400for a badsessionsor a session id over 256 bytes;401like the other/v1routes (the admin token is not a client key).GET /v1/routing[?window=24h&alias=a,b&recent=10]— venom's own: where the calling key's requests to each route alias went, and why, for the CLI's/routesandvenom routes. Like/v1/usageit takes no key parameter and reads the request log by the caller's key name (refused,log: falsewith a note, when another key has that name; open mode: requests logged without a key).window:1h,24h(default),7d, any Go duration or<n>d, 1m to 31d;alias: also list these aliases with no traffic (only ones the key'sallowed_modelspermit);recent: how many latest fallbacks (default 10, at most 100). Body (object: "venom.routing"):now,window,since,scope: "key",key {id, name, team},log,notes,requests,cost_usd,aliases(most requests first) andrecent_fallbacks. Each alias:{alias, strategy, configured, requests, unserved, unserved_share, fallbacks, fallback_share, fallback_reasons: {reason: n}, tokens_in, tokens_out, cost_usd, targets, recent_fallbacks}; each target, in configured order (then targets the log names that a reload removed,position: 0,state: "removed"):{position, upstream, model, configured, state, detail, until, capabilities: {tools, responses, decisions, context_window}, requests, share, errors, tokens_in, tokens_out, token_share, cost_usd, cost_share, latency_ms, ttft_ms, live_ttft, fallbacks_in, fallbacks_in_reasons, passed_over, passed_over_reasons}.stateis the target's live state now:ok,half_open(its breaker admits one probe),cooling(a quota 429 cooldown;until),circuit_open(until= when it goes half-open),unhealthy(last health check failed),missing(the upstream lacks the model),disabled(detailnames the missing env var),removed.share= requests the target served (any status) / the alias's requests; every target's share plusunserved_share(requests no target served) is 1.latency_ms/ttft_msare{p50, p90, n}(nearest rank) over the target's successful requests / streams,nullwithout any;live_ttftis the router's own decayed estimate now ({p50_ms, p90_ms, samples, age_s}, as on/admin/v1/routes;nullbefore a sample).fallbacks_in: requests it served after an earlier target did not, by the reason of the first target passed over;passed_over: times it did not serve, by reason; an alias'sfallbackscounts requests with at least one target passed over. Requests routed under a route profile (x-venom-profile, or the key's default) are their own block:profile(absent = none) is on the alias block and its fallbacks, and under a profile that re-targets the alias the block's targets are the profile's, in its order. A fallback is{ts, request_id, alias, profile?, from, to, reason, status, chain}:fromthe first target passed over,tothe one that served ("": none did),chainthe request'sattempt_log. Nothing names an upstream's address or credential.400for a badwindoworrecent.GET /v1/gears— venom's own: the gears the orchestrator's config defines, for the CLI's gear list (see "Gears" below). Same auth as/v1/models. Always200; withoutgears:the list is empty.POST /v1/responses— streaming and non-streaming. To a target whosecapabilities.responsesis true (OpenAI by default) it is passed through. To any other target (Ollama, llama.cpp, vLLM, SGLang and generic servers by default) it is translated to/v1/chat/completionsand the reply translated back into a Responses object or Responses SSE events (response.created,response.in_progress,response.output_item.added|done,response.content_part.added|done,response.output_text.delta|done,response.reasoning_text.delta|done,response.function_call_arguments.delta|done, thenresponse.completed/response.incomplete, orerror+response.failed; every event carriessequence_number). Translated:input(string or message /function_call/function_call_output/reasoningitems, whose text goes on the assistant turn they preceded; text andinput_imageURL parts),instructions, functiontools,tool_choice(incl.allowed_tools),max_output_tokens,temperature,top_p,parallel_tool_calls,text.format,reasoning.effort,user,safety_identifier,prompt_cache_key,service_tier.metadatais echoed on the Response but not sent upstream, andparallel_tool_calls/tool_choiceare dropped when no tools remain: chat completions refuses all three in those cases where Responses accepts them. Usage maps toinput_tokens/output_tokens(+ cached, cache-write and reasoning details, every field present as the API reference requires). The translation is stateless: a request withprevious_response_idorconversationskips translated targets and gets400 {"error":{"code":"stateful_request_unsupported"}}when no native target remains. Hosted tools (web_search,file_search, …),item_referenceinputs,input_fileparts and storedprompts likewise can't be translated (400 unsupported_request, or the next native target). Translated responses are never stored (store: false), except background ones (below).Background responses (
background: true,proxy/background.go). The route's first target decides how one runs. Native (capabilities.responses): the request is passed through, and the queued reply's id is mapped (SQLitebackground_responses) to the upstream, its model and the calling key. The queued reply is logged at cost 0 and its budget and intent pre-charges stay reserved; once the response is terminal (completed,incomplete,failed,cancelled), seen on a client'sGETor by the poller (every 2 s doubling to 1 min, for up to 6 h; then, or if the upstream forgets it, the pre-charge estimate is booked), its real usage is priced and booked exactly once (budget, spend ledger, intent) and written onto the POST's request-log row. A streamed background request that runs to its end is booked then, like any stream. In mesh mode the row also keeps the cloud ticket or relay job the POST came under (mesh_ref), and the settled usage goes to the cloud as a usage report withsettle: true(docs/cloud-mesh.md§6.6). Emulated (any translated target): the orchestrator answers the queued Response itself (its ownresp_…id,background: true,store: true) and runs the request detached as a translated stream; it is billed like any request when it ends (a cancel bills what was generated, like a client hanging up) and its final Response is kept for 24 h. Withstream: truethe POST streams the run's events; the run goes on if the client leaves. A background request's request-log row hasbackground: true(the console'sbgbadge). Mappings of native responses are kept 30 days.GET /v1/responses/{id}— a background response this key created (another key's, or an unknown id:404 not_found). Native: proxied to its upstream with the upstream's credentials, query string included. Emulated: the stored object;?stream=truereplays its events from the start, or afterstarting_after=<sequence_number>, and follows a running one to its end (after 10 min in memory only the terminal event remains).POST /v1/responses/{id}/cancel— native: proxied (the cancelled response settles at the usage the upstream reports for it). Emulated: aborts the run and answers thecancelledobject; a completed one answers400 invalid_state.DELETE /v1/responses/{id}— native: an unsettled response is read (and cancelled if it still runs) and settled first, then the delete is proxied. Emulated: cancels a running one, then{"id","object":"response","deleted":true}.
A model value resolves as: route alias → its target list (an experiment
alias → the target of the variant the request is assigned, see "Model A/B
experiments"); upstream/model → that upstream directly; bare model id →
first upstream advertising it. An id
that contains / but that its prefix upstream can't serve (disabled, or not in
its static models list) is looked up as a bare id, so vLLM's
openai/gpt-oss-20b still reaches vLLM.
Gateway errors use the OpenAI {"error":{message,type,code}} shape. They
name the upstream but never its address or the transport error (dial tcp 10.0.0.5:11434: …); that detail goes to the log, last_error and the admin
request log. An upstream answering 401/403 means the orchestrator's credential
for it is wrong: the upstream is marked failed, the next target is tried, and
if none serves the client gets 502 upstream_auth_failed (the upstream's own
body, which can echo part of the key, is not relayed). A request body is read
within 60 s (408 request_timeout otherwise) and bounded by max_body_bytes
(413 request_too_large). A client that stops reading a response is dropped
once one write to it has waited defaults.write_stall_timeout (60 s; time
waiting on the upstream doesn't count) and its upstream request cancelled. A
stream ends at its terminal event (data: [DONE], or a Responses
response.completed, response.incomplete or response.failed): the
orchestrator closes the client's stream there, reading at most 1 s more for the
upstream's own end, so an upstream that keeps the connection open can't hold
the client. An upstream that sends nothing for defaults.stream_idle_timeout
(5 min) in the middle of a response is cut, and a stream ends with its error
event. A client that closes a stream after the upstream
finished it ([DONE], or a finish_reason plus usage) is logged with the
upstream's status, not 499.
Response headers added: x-venom-request-id, x-venom-upstream,
x-venom-model (the concrete model that served the request),
x-venom-base-model (the model that one is priced as, when its config sets a base_model),
x-venom-translated: chat_completions on a translated /v1/responses reply,
and x-venom-variant: <variant> on every answer to an experiment alias
(errors included; absent for any other model), and x-venom-profile: <name> on every answer
routed under a route profile (errors after the profile is chosen included), x-venom-response-cache: hit | shared on a side request answered from the response cache (the original's headers otherwise,
with this request's own id), x-venom-dedup: joined | replayed on a request answered from an
identical one's upstream call or a stored Idempotency-Key response (the latter also with
Idempotent-Replayed: true; → Request deduplication). x-venom-fallback: <reason>[,<reason>…]
is on an answer the first target it tried did not serve: the reason code of each target
passed over or failed, in order (429_quota, quota_cooldown,timeout); absent when the
first target served. Reason codes (stable): 429 (rate limited), 429_quota (quota or
balance exhausted; the target then cools down), quota_cooldown (skipped while cooling),
circuit_open, unhealthy (skipped: last health check failed), busy (at
max_inflight), unsupported (the target can't serve this request: tools, endpoint),
context_window (passed over: the request's estimated tokens, prompt bytes/4 + max output, exceed its
context_window, so a larger one serves; the response also carries x-venom-context-cost-usd, the
estimated input-cost difference in USD, served minus passed over, when both are priced; nothing is ever
truncated), missing (the upstream lacks the model, or answered 404 for it before),
timeout (no first byte within ttft_timeout / the upstream timeout), 5xx, 404
(404/501), auth (the upstream refused our credential), conn_error, context_length (the upstream
answered 400 context_length_exceeded; only a target with a larger or unknown window is tried next, and if
none serves, that 400 goes to the client), rejected_param (the upstream answered 400 for a field or value it
doesn't take, e.g. reasoning_effort must be one of low/high/max, unsupported_parameter, Model Studio's
InvalidParameter, Z.ai 1210/1212/1214/1215: the next target is tried and the 400 is held for the client if none
serves; a 400 whose body says the request doesn't parse is relayed at once), rate_shaped (passed over: at its
provider's learned rate limit while a later target could serve now, or its rate-shaping queue refused the wait). When no target's context_window holds the request (prompt bytes/4
- its max output tokens), the answer is an OpenAI-shaped 400 before any attempt:
{"error":{"message":…,"type":"invalid_request_error","param":"messages","code":"context_length_exceeded"}}(paramisinputon/v1/responses,prompton/v1/completions). When some target fits but none serves (503 no_healthy_upstream,502 upstream_unavailable), the message names every target of the route, not only those tried or skipped as down: after the attempts' reasons come the targets that can't take this request with why (… but ollama/qwen3:8b has a 4096-token context window, plus theOLLAMA_CONTEXT_LENGTHfix for an Ollama target) and the targets on a disabled upstream (openai: OPENAI_API_KEY is not set), all;-separated.x-venom-upstreamandx-venom-modelare set on every committed answer (chat completions, responses, translated responses,/v1/systemone) before the first byte, since the fallback is decided by then. The CLI's call meter reads both on every request it makes, streamed or not, to show which model answered each call (cli/AGENTS.md→ "Call meter"); per-request cost is priced client-side fromconfig/prices.yaml, since a header can't carry a cost known only at the end of a stream. Every/v1answer, errors included, also carriesx-request-id, where the OpenAI SDKs read a request id from: the same value asx-venom-request-id, unless the upstream sent its ownx-request-id, which is relayed instead. Upstream rate-limit headers (retry-after,retry-after-ms,x-ratelimit-*) are relayed with an upstream 429, so the SDKs' retry timing follows them. Upstream response headers pass by allowlist and nothing else does:content-type,content-length,content-encoding,cache-control,x-request-id,retry-after,retry-after-ms,x-should-retry,x-ratelimit-*,openai-processing-ms,openai-version(what the OpenAI SDKs read, plus what describes the body). Cookies, hop-by-hop headers, account ids (openai-organization,openai-project), server banners and an upstream's ownx-venom-*never reach the client. Retries happen at one layer at a time (see "Routing semantics"): a request carryingx-stainless-retry-count> 0 (an OpenAI SDK's own retry) gets no same-upstream retry here, and an upstream 429/503 asking for a longer wait thandefaults.backoff_maxis returned at once instead of slept on, so an SDK call reaches an upstream at mostmax_attempts + max_retriestimes (unless rate shaping queues that wait, below). The OpenAI SDKs are checked against this surface bymake conformance(services/go-orchestrator/AGENTS.md, "Conformance").
Rate shaping (rate_shaping: in orchestrator.yaml; services/go-orchestrator/AGENTS.md → Rate shaping):
each upstream (one provider credential) learns its provider's limits from the
x-ratelimit-{limit,remaining,reset}-{requests,tokens} and
anthropic-ratelimit-{requests,tokens}-{limit,remaining,reset} headers of its answers and from its 429s
(retry-after-ms / Retry-After; 500 ms when a 429 says nothing). A request that would go over a learned limit
is sent to a later target that has room now, else waits for the limit to refill in that upstream's queue,
ordered by start-time fair queuing across client keys (and mesh teams), at most rate_shaping.max_wait
(default 30s) with at most rate_shaping.max_queue (default 256) waiting. A request that waited carries
x-venom-queued-ms: <ms>. A provider 429 the shaper learned from is retried after waiting in that queue
rather than a backoff, also when its Retry-After is longer than backoff_max but within max_wait (and
the retry budget's time). Only when the queue is full or the wait would pass the bound does the client get
429 {"error":{"type":"rate_limit_error","code":"upstream_queue_full"|"upstream_wait_exceeded",…}} with
Retry-After / Retry-After-Ms (when it would fit, estimated), before any upstream call.
rate_shaping.weights: {<key name>: <weight>} (default 1, at most 1000) gives a key a larger share of a
contended upstream. max_queue: -1 or max_wait: -1s turns waiting off: the learned limits then only steer
requests to targets with room, and the last target is sent the request as before. An upstream that never
sends a rate-limit header (local engines) is not shaped.
Rate limits per client key (rpm_limit, tpm_limit): each request takes one
RPM token. Before it is forwarded, its estimated size (prompt bytes/4 + the
requested max_tokens / max_completion_tokens / max_output_tokens) is
charged to the key's TPM bucket, and afterwards the charge is reconciled to the
usage the upstream reported (unused tokens refunded; a request that produced
no response is refunded in full). A request that doesn't fit yet gets
429 rate_limit_exceeded with Retry-After; one larger than the whole
per-minute limit gets 429 tpm_limit_exceeded (no Retry-After).
Spend budgets per client key (budget, set via the admin API; a key without
one is unlimited). Periods are the UTC calendar day and month and reset at
00:00 UTC. After routing, each request is pre-charged its worst-case cost:
prompt bytes/4 plus its max output (4,096 tokens when it sets none; 0 for
embeddings) at the list price of the costliest candidate it could be served
by. Then:
- spent + in-flight pre-charges + this pre-charge > a hard limit →
429with{"error": {"type": "insufficient_quota", "code": "budget_exceeded", "message": …}},x-should-retry: false(the OpenAI SDKs raiseRateLimitErrorand don't retry) andx-venom-budget-reset: <RFC 3339 UTC>. NoRetry-After. A request whose pre-charge is 0 (every candidate local) is never refused, so local models keep working past a budget. - a key with a hard limit, and an enabled cloud candidate the price table has
no price for →
403{"error": {"type": "permission_error", "code": "unpriced_model"}}: its cost would be booked as $0, so the budget couldn't limit it. Add the model toconfig/prices.yaml. A key with only soft limits or no budget is served (its spend there reads $0). - spent ≥ a soft limit → the request is served and the response carries
x-venom-budget-warning(the period, its spend and limits, when it resets).
When the request finishes, its pre-charge is released and its actual cost
(see Cost) is booked. A response whose usage never arrived (the client
hung up on a stream before its usage chunk, the upstream broke mid-response,
or sends none) is booked at an estimate: prompt bytes/4 plus the output the
events' text adds up to (usage.TextTokens: content, reasoning and tool-call
argument strings at about 4 bytes a token, at least one per event that carried
text; one per event when none did; body bytes/4 when not streamed), capped at
the request's max output, so cancelling a stream just before its end isn't
free, and a provider that streams many tokens per chunk isn't billed as one.
The request log carries those estimated tokens with the cost. Venom Cloud does
the same for what it serves itself (lib/providers/partial.ts) and books a
relayed job's partial end at the daemon's figures (docs/cloud-mesh.md §6.5). Spend is kept per key in memory (O(1), no allocation
on the request path), in a spend ledger in SQLite that is not pruned with the
request log, and is reloaded from the ledger at start.
Request deduplication
Identical concurrent requests from one client key (same endpoint, route profile, Accept and
OpenAI-Beta, and the same body with whitespace removed; a mesh team's requests never meet another
team's) share one upstream call: a client's retry or double submit while the first is still
generating. Eligible: temperature 0 set explicitly, /v1/embeddings and /v1/rerank (unset
temperature is the provider's default, 1 for OpenAI, so never), or any request with an
Idempotency-Key (≤ 255 bytes, else 400 invalid_idempotency_key). Never a stateful or background
/v1/responses request or an audio upload. X-Venom-Dedup: off opts a request out.
- The response is fanned out as it streams, never buffered whole: each client reads it at its own
pace, and one more than
dedup.max_lag_bytes(256 KiB) behind is cut loose (its connection reset, as a broken upstream's would be; before anything was written to it, it goes upstream on its own), so a slow client never stalls the others. A duplicate can join while the response is at mostdedup.max_bytes(1 MiB) and no further along thanmax_lag_bytes; later, it is sent upstream itself. The first client hanging up doesn't stop a call a duplicate is reading; when every client is gone the upstream request is cancelled. - Billed once: the first request's row carries the tokens and cost; each duplicate's row has
attempt_log: dedup:joined:<first request's id>, no tokens andcost_usd0, and its answerx-venom-dedup: joinedwith its own request id. Idempotency-Key: the same key and body is collapsed at any temperature, and after the call completes it is replayed whole fordedup.idempotency_ttl(5 min) withx-venom-dedup: replayedandIdempotent-Replayed: true(attempt_log: dedup:replayed:<id>). A 5xx, a 429, a broken stream or one overmax_bytesis not kept, so a retry goes upstream again. The same key with another body is422 idempotency_key_reused. Kept responses take at mostdedup.idempotency_max_bytes(32 MiB, least recently used dropped).- Counted in
venom_dedup_total{how}, logged (request deduplicated). Venom Cloud does the same for relayed jobs (docs/cloud-mesh.md §4.1.1).
Intent spend accounts
A 24/7 agent needs a budget per task, not per period. A request that carries
X-Venom-Intent: <id> (at most 128 bytes; longer is 400 invalid_intent) is
metered against that intent and against its key's team (internal/intents).
An intent's cap is a running total, not a UTC day or month, and the same
pre-charge as the key budget applies (worst case over the candidates, so
concurrent requests can't jointly overshoot). A request over a cap is refused
before any attempt:
402{"error": {"type": "insufficient_quota", "code": "intent_budget_exceeded", "message": …}}withx-should-retry: false. The message names the scope (intentorteam), the spend and the cap. A request whose pre-charge is 0 (every candidate local) is never refused.- an intent or team with a cap, and a candidate the price table has no price
for →
403 unpriced_model, as for a key budget. - more than 100,000 distinct intents →
503 intent_ledger_full.
The first refusal of an intent stamps it exceeded (exceeded_at) and logs
intent over its cap. GET /admin/v1/intents?state=exceeded is that feed:
it is what an inbox item for the owner is built from. Raising the cap above the
spend clears the mark, so the next overrun is a new item. An intent is created
by its first request (uncapped, owned by the first key's team that names it)
or by PUT; accounts persist in SQLite (intents, team_caps).
GET /admin/v1/intents[?state=ok|uncapped|exceeded]→{intents: [{id, team, cap_usd, spent_usd, reserved_usd, requests, state, first_at, last_at, exceeded_at}], teams: [{team, cap_usd, spent_usd}]}, exceeded first.GET /admin/v1/intents/{id},DELETE /admin/v1/intents/{id}(the team keeps the spend counted so far).PUT /admin/v1/intents/{id}{"cap_usd": 5, "team": "core"}sets the cap (0 removes it) and creates the account.PUT /admin/v1/teams/{team}/intent-cap{"cap_usd": 50}caps the total spend of all the team's intents.
Spend is booked when a request finishes, from the same cost as the request log. Not covered: a team's spend counts only requests that name an intent, and several orchestrators on one database keep separate running totals.
Guardrails (guardrails: in the config, guardrails.md): every
/v1 inference request is checked against the org's rules, its key team's and its
key's own. A block is 403 {"error": {"type": "guardrail_blocked", "code": "GR-ORG-CRED-101", "message", "layer", "action", "url", "request_id"}}; a
redaction, warning or keep-local match passes with one x-venom-guardrail: <code>; action=…; layer=…; url=… entry per rule, and a stream ends with one
more event listing them (venom_guardrails in a chunk, venom.guardrails on
the Responses API). GET /g/{code} (no key) is the code's page. Venom Cloud
answers the same way.
Admin surface (Authorization: Bearer $VENOM_ADMIN_TOKEN)
All JSON, snake_case.
GET /admin/v1/overview→{uptime_s, requests_total, errors_total, tokens_in, tokens_out, cached_tokens, cache_write_tokens, cache_breaks, cache_hit_ratio, cost_usd, latency_p50_ms, latency_p95_ms, ttft_p50_ms, ttft_p95_ms, active_streams, by_model:[{model, requests, errors, tokens_in, tokens_out, cost_usd}], wasted_tokens, wasted_usd, wasted:[{model, reason, attempts, tokens, usd}]}(since start, in memory;wasted*is the wasted spend of/admin/v1/requestswaste, summed by requested model and reason)GET /admin/v1/anomalies→{anomalies:[{scope:"key"|"session", id, name, metric:"spend"|"cache_breaks", value, baseline, ratio, requests?, breaks?, since, last}]}: open cost anomalies, newest first (in memory).valueandbaselineare $/hour forspendand a break ratio (0–1) forcache_breaks;ratiois 0 while the entity has no baseline yet. Rules:services/go-orchestrator/AGENTS.md→ Cost anomalies.GET /admin/v1/upstreams→[{name, kind, base_url, disabled, healthy, circuit:"closed"|"open"|"half_open", inflight, latency_ewma_ms, models:[string], last_error, local, retries:{budget_available, first_attempts, retries, skipped:{budget, retry_after, client_retrying}}, quota_exhausted:[{model, until, reason}], load:{waiting, running, kv_cache_usage, at, saturated}, shaping:{…as GET /admin/v1/shaping}}].shaping(omitted until the upstream sent a rate-limit header or a 429) is its rate shaper (see Rate shaping above).load(omitted when not scraped or stale) is the engine's own queue and KV-cache use (0–1, -1 = not exported) read from its/metrics(vllm, sglang, llamacpp;metrics_url), and whether routing counts it saturated (services/go-orchestrator/AGENTS.md→ KV-cache pressure).quota_exhausted(omitted when empty) lists the models sitting out a cooldown after a "quota exhausted" 429, with when they are tried again (RFC 3339).disabledis true for an upstream that is structurally off (itsapi_key_envis unset): it is never health-checked andhealthyis false, but it is not down.localis true when its tokens cost 0 (see Cost below).retriesis the upstream's retry budget since start, the same figures as thevenom_upstream_attempts_total,venom_upstream_retries_skipped_totalandvenom_upstream_retry_budget_availablemetrics: retries the budget allows now, first attempts and retries sent, and retries not made because the budget was spent, the upstream'sRetry-Afterwas longer thanbackoff_max, or the client was retrying itselfGET /admin/v1/shaping→{max_queue, max_wait_ms, upstreams:[{name, active, learned_at, requests:{limit, remaining, reset_at}, tokens:{limit, remaining, reset_at}, blocked_until, queue_depth, max_depth, admitted, queued, cancelled, rejected:{queue_full, wait_exceeded}, wait_p50_ms, wait_p95_ms, wait_max_ms}]}: every upstream's rate shaper.requests/tokens(omitted until learned) are the provider's limit and what remains now (refilled linearly to the limit byreset_at, less what was sent since);blocked_until(omitted when past) is a 429's Retry-After;admittedwent through at once,queuedafter waiting,cancelledleft the queue (client gone),rejectedwere answered 429 here; the waits are of the last 256 queued requests.GET /admin/v1/routes→[{alias, strategy, targets:[{upstream, model, weight, available, reason, fix, servable, context_window, ttft, capabilities:{tools, tool_choice, responses, reasoning_field, replay_reasoning}}]}].ttftis the target's live time-to-first-token estimate,{p50_ms, p90_ms, samples, age_s}(omitted before its first streamed answer): whatstrategy: fastestandbalancedrank by (docs/configuration.md → Latency-aware routing).availableis false when routing skips the target because its upstream doesn't have the model;reasonthen says why and names the fix (model qwen3:8b is not pulled on ollama: run `ollama pull qwen3:8b` / `make pull-models`) andfixis the fix alone (ollama pull qwen3:8b (or make pull-models)). Both are""when available.servableis true when routing would send a request to the target now (available, its upstream enabled and healthy or not checked yet, breaker not open); the harness resolves an alias to the first servable target to pick the edit tool.context_window(tokens, configured else discovered; omitted when unknown) andcapabilitiesare the target's spec as the capability gate sees it: kind defaults, then config, then discovery (responses: falsemeans a/v1/responsesrequest is translated;reasoning_fieldis""when the upstream emits none;replay_reasoningis true when earlier assistant turns' reasoning is sent back to it inreasoning_field, false when it is removed from requests). The console's model page shows them.GET /admin/v1/routing[?window=24h&key=NAME&alias=a,b&recent=10]→ theGET /v1/routingbody over every key's requests (scope: "all", nokey;?key=NAMEnarrows it to one key name), listing every configured alias (with or without traffic), and each fallback with itskey. The console's Routing page. It also carrieslearned(admin only, read-only): for each alias withstrategy: learned, the cached per-target scores the router orders by (params, and per targetsessions,successes,success_rate,wilson_lower,mean_cost_usd,measured,qualified,rank), plusupdated_atandwindow. Learned routing scores session outcomes against cost; merge and revert rates are a follow-up pending a commit-to-session join and a task-type key (services/go-orchestrator/AGENTS.md→ Learned routing).GET /admin/v1/requests?limit=100(max 1000) →[{id, ts, key_name, model, route, upstream, upstream_model, status, stream, latency_ms, ttft_ms, tokens_in, tokens_out, cached_tokens, attempts, error, session_id, cost_usd, local, parent_session_id, variant, attempt_log, profile, cache_write_tokens, cache_break, wasted_tokens, wasted_usd, waste?:[{upstream, model, reason, tokens_in, tokens_out, usd}], base_model, price_source, role, gear, tools, cache_saved_usd, audio_chars?, audio_ms?}](newest first).audio_charsis a/v1/audio/speechrequest's input characters andaudio_msthe audio a transcription or translation was billed for (migration 20; omitted when 0).roleis the spend lane the request ran as (x-venom-role: the gear rolesworkers,smart,vision,browser,classifier,summary,reflex, or a side request's lane outside a session turn:review,triage,judge,decide,hook,ask; any other value and a main turn are""),gearits gear (x-venom-gear, trimmed and lower-cased; kept only when it is a letter then letters, digits,.,_,-, at most 64, as Venom Cloud reads it; else"") andtoolswhether it offered tools (a non-emptytoolsarray). All three come from migration 15; older rows read"","",false.modelandrouteare what the client asked for,upstream/upstream_modelwhat served it (after any fallback),base_modelthe model that one is priced as (""= itself) andprice_sourcewherecost_usd's price came from:custom,base_model,builtin,localorunpriced(both""on rows before migration 13 and for requests nothing served).wastelists the attempts of the request that were billed and discarded (absent when none), withwasted_tokens/wasted_usdtheir sums:reasonisabandoned(the upstream had started answering and the attempt was dropped, or it broke a committed response),ttft_timeout(cancelled at the TTFT deadline or the upstream timeout after the request was sent),client_disconnect(the client hung up mid-response: 499) orretried(an abandoned or timed-out attempt after which the request went to the same upstream again). Tokens are the upstream's usage when it reported some, else estimates (prompt bytes/4, the relayed events' text at about 4 bytes a token);usdis list price, 0 on local upstreams. A 499 or broken response's own attempt is incost_usdas well; the others are spendcost_usddoesn't show (0 / absent on rows before migration 8).cache_write_tokensis the share oftokens_inthe provider reported writing to its prompt cache (billed at the price'scache_write, else input);cache_breakisidle,model_change,prompt_shrank,concurrentorunknownwhen the request, in a session, read under half of the prefix it shares with the session's previous request on the same served model,""otherwise (both 0 /""on rows logged before migration 7).cache_saved_usdis what the prompt cache saved on the request at list price, priced when listed from the current table (as its base model when it has one): cached × (input − cached_input) − written × (cache_write − input), negative when a write cost more than it saved, 0 for local and unpriced models (also on the rows ofGET /admin/v1/sessions/{id}).profileis the route profile the request was routed under,""for none.variantis the experiment variant of a request to an experiment alias (routeis then the alias),""otherwise.attempt_logis why the request fell back: each target that did not serve asupstream/model:<reason>(thex-venom-fallbackcodes), then the one that did asupstream/model:ok, joined by→(openai/gpt-5.4-nano:429_quota→qwen/qwen3.8-flash:ok);""when the first target tried served (and on rows logged before migration 5). Every admin read flushes the log writer first.&flush=1also waits (up to 5 s) for requests that started but have not been logged yet: a client can read its last byte before the handler logs the request.&key=NAMEscopes that wait to one client key (the rows are not filtered). The reply then carriesx-venom-log-flushed: true, orfalsewhen the wait timed outGET /admin/v1/sessions?limit=50(max 500) → per-session aggregates over the request log, most recently active first:[{session_id, first_ts, last_ts, requests, errors, tokens_in, tokens_out, cached_tokens, cache_hit_ratio, local_tokens, cloud_tokens, cost_usd, models:[{upstream, model, local, requests, tokens_in, tokens_out, cached_tokens, cost_usd}], parent_session_id, subagents:{sessions, requests, errors, tokens_in, tokens_out, cost_usd}, variant, cache_write_tokens, cache_breaks}].cache_write_tokenssums the session's requests' cache writes;cache_breakscounts its requests logged with acache_break.variantis the session's experiment variant (""when none of its requests went to an experiment alias). A session is every logged request that carried the samex-venom-session; requests without the header belong to none. A request may also carryx-venom-parent-session(venom's subagents send their parent's session id): it is logged asparent_session_id(ignored withoutx-venom-session, or when it names the session itself) and changes nothing in routing, except that on an experiment alias the subagent gets its parent's variant. Both headers are at most 256 bytes; a longer one is refused with400invalid_request_error, codeinvalid_session.x-venom-experiment: <id>; arm=<a|b>names a Venom Cloud gear experiment's arm (HarnessOptions.experiment,Harness.setExperiment; venom sets it when a session joins one,docs/cloud-mesh.md§10.14), sent whereverx-venom-geargoes. The orchestrator ignores it and never passes it upstream.x-venom-gear: <name>names the gear a request is made under: the harness sends the engaged gear's name (HarnessOptions.gear,Harness.setGear; venom's/gearsets it) on turns, summaries and side requests, and subagents take their parent's. The orchestrator ignores it; Venom Cloud counts the spend toward the org's gear budgets (docs/cloud-mesh.md§10.2) and answersx-venom-spend-warning(a budget past its warning share) andx-venom-spend-downgrade(a budget at its limit switched the model), which the harness turns into one notice line per budget and window, and429 spend_limit_reachedat a limit, which it explains (switch gears with/gear) without retrying. None of it enters the model context.x-venom-cache-ttlis the client's cache-lifetime intent on/v1chat and Responses requests:long(venom's interactive TUI sends it; a person pauses between turns) asks a model withcapabilities.cache_retentionfor that longer lifetime from the first request;shortnever does; absent, the session asks once one of its requests came more than 5 minutes after the previous one, and keeps asking for the rest of the session. The field is set per upstream kind (OpenAIprompt_cache_retention, Kimi on kindgenericprompt_cache_options) and never overwrites a client's own. An answer to a request it was asked for carriesx-venom-cache-retention: <value>(24h,30m), which venom's keep-warm pings take as the cache's TTL. Unknown values count as absent. Not in the mesh dispatch allowlist: a request relayed by Venom Cloud falls back to the gap rule.parent_session_idis the session that started this one (""for most).subagentssums every request of the sessions that name this one as parent; the other figures cover the session's own requests only, so its spend with its subagents iscost_usd + subagents.cost_usd.modelslists what served the session (costliest first); a request no upstream served counts in the totals only.local_tokens/cloud_tokensare prompt + completion tokens on local / cloud upstreams. Figures cover the rows still in the log (request_log_retentionprunes the oldest).GET /admin/v1/sessions/{id}?limit=1000[&subagent_requests=1](max 5000) →{session: {...as above}, requests: [...request rows, oldest first], subagents: [...sessions as above], outcome: {session_id, success, score, reported_at} | null, subagent_requests?: [...request rows]}(the newestlimitrequests when there are more;subagentsare the sessions this one started, most recently active first, at most 500;outcomeis the task outcome reported for it, below);404when no logged request carries the id.subagent_requests=1adds every request of those subagent sessions, oldest first across them, the newestlimitwhen there are more (absent without the parameter, or when there are none): the console's session tracePOST /admin/v1/sessions/{id}/outcome{success: bool, score?: number}→{session_id, success, score, reported_at, experiment, variant}. Records the session's task outcome (a later report replaces it;scoreis any finite number,nullwhen left out), for the experiment's success rate. The id is 1–256 printable characters; the session need not be logged yet, andexperiment/variantsay what its requests were assigned (""until the log has one of its experiment requests).400for a body withoutsuccess, with unknown fields or over 4 KiB. venom's eval runner reports each trial's pass/fail here; a CI job can report its own verdict the same wayGET /admin/v1/shadow?days=7&route=<alias>&limit=50→{days, routes: [R], samples: [S]}, shadow routing (docs/configuration.md→ Shadow routing).R = {route, configured, config?, capped_today, counts: {sampled, capped, dropped_queue, dropped_capture, skipped, completed, failed}, candidates: [{route, candidate, samples, compared, agreed, errors, agreement, by_check, primary_p50_ms, shadow_p50_ms, primary_ttft_p50_ms, shadow_ttft_p50_ms, primary_cost_usd, shadow_cost_usd, shadow_cost_today_usd}]}: every configured route plus any with samples in the lastdays(1–30); counts since the process started.S = {ts, route, role, primary_model, candidate, candidate_model, request_id, shadow_request_id, check_kind, agree, score, detail, error, primary_ttft_ms, primary_ms, shadow_ttft_ms, shadow_ms, primary_cost_usd, shadow_cost_usd}, newest first (limit0–500);agreeandscorearenullfor a failed replay. Replays' request-log rows carrysource: "shadow"GET /admin/v1/experiments→[{alias, variants: [{name, target, weight}]}], the configured experiment aliasesGET /admin/v1/experiments/{alias}?since=<RFC 3339>→{alias, configured, since, variants: [V]}withV = {name, target, weight, configured, sessions, subagent_sessions, requests, errors, error_rate, tokens_in, tokens_out, cached_tokens, cost_usd, latency_ms: {p50, p95, n}, ttft_ms: {p50, p95, n}, outcomes: {sessions, successes, success_rate, ci95_low, ci95_high, scored, mean_score}}. Over the retained request log's rows for the alias (at or aftersincewhen given;""= all), per variant:sessionsare top-level sessions,subagent_sessionssubagent sessions with requests to the alias;errorsare requests with status 0 or ≥ 400; latency percentiles (nearest rank) are over successful requests, TTFT over successful streams,nthe sample size.outcomesjoins the reported outcomes to the sessions of the variant (a session counts for the variant of its first request to the alias):success_rate= successes / sessions with its 95% Wilson score interval[ci95_low, ci95_high][R190],mean_scoreover outcomes with a score;nullwithout outcomes. Configured variants are listed (in config order) even before they have traffic; a variant a reload removed is listed after them,configured: false, while the log has its rows.404when the alias is neither a configured experiment nor in the log as one;400for a badsinceGET /admin/v1/keys→[{id, name, prefix, created_at, rpm_limit, tpm_limit, allowed_models, team, profile, budget: {daily_usd, daily_soft_usd, monthly_usd, monthly_soft_usd}, spend: {daily: P, monthly: P}}]withP = {spent_usd, reserved_usd, soft_usd, hard_usd, state: "ok"|"soft"|"exceeded"|"none", resets_at}(the current UTC day and month;reserved_usdis pre-charged by requests in flight;none= no limit for the period).prefixis the first 8 characters of a minted key (vk-+ 5 of its 32 random characters), enough to tell keys apart. The bootstrap key (id: "env", fromVENOM_API_KEY) hasprefix: "": it is the operator's own string, possibly short or guessable, so no character of it is listed. It has no stored record, so it takes no team or budget (its spend is still tracked)POST /admin/v1/keys{name, rpm_limit?, tpm_limit?, allowed_models?, team?, budget?, profile?}→{...key, key}(plaintext shown once).budgetis{daily_usd?, daily_soft_usd?, monthly_usd?, monthly_soft_usd?}in USD, each 0–1e9, 0 = none; a soft limit left out is 80% of that period's hard limit, and may not exceed it.teamis a label of at most 64 bytes grouping keys in the spend views.profileis the key's default route profile (used when a request sends nox-venom-profile); it must name a configured profile (400otherwise),""= nonePATCH /admin/v1/keys/{id}{team?, budget?: {...} | null, profile?, rpm_limit?, tpm_limit?}→ the key as listed.profile: ""clears the default profile; a limit of0removes it (negative:400). A changed limit applies to the next request and keeps the bucket's balance (no fresh minute). A field left out is unchanged;budgetreplaces the whole budget,nullclears it. The key's spend so far is kept, so a lower limit applies to the next request.400for the bootstrap key,404for an unknown idDELETE /admin/v1/keys/{id}GET /admin/v1/spend?from=&to=&key=&team=&sessions=20→ the spend ledger for UTC daysfrom..to(YYYY-MM-DD, inclusive; default the last 30 days; at most 366):{from, to, total: T, by_key: [{key_id, key_name, team, deleted, ...T}], by_team: [{team, keys, ...T}], by_model: [{model, base_model, ...T}], by_base_model: [{model, ...T}], by_day: [{day, ...T}], top_sessions: [{...session as in /sessions, total_cost_usd}]}withT = {requests, tokens_in, tokens_out, cost_usd}.by_modelis the servedupstream/model(the requested model for a request nothing served), with thebase_modelthe current config prices it as;by_base_modelsums it per priced-as model (the base model, else the served id without its upstream);by_daylists every day of the range. Teams are the keys' current teams (""= none; a revoked key's spend stays,deleted: true, under"");key_id: ""is spend without a key (open mode).key=<id>andteam=<label>(team=alone = keys without a team) narrow every list.top_sessionsare the costliest sessions active in the range, ranked bytotal_cost_usd= their own cost plus their subagents'; a subagent whose parent is also in the range is counted under it, not listed. They come from the request log, so they are limited to whatrequest_log_retentionkeeps; the other lists come from the ledger, which is never pruned (one row per day, key and model)GET /admin/v1/usage/by-role?since=7d&until=→ spend per lane (VEN-13), grouped from the request log (so withinrequest_log_retention):{object: "venom.usage.by_role", since, until?, requests, cost_usd, roles: [{role, requests, errors, tokens_in, tokens_out, cached_tokens, cost_usd}]}, costliest first.roleis thex-venom-rolethe request named; a request that named none, or an unknown one, ismain, so the lanes add up to the window and the total is the same as without the grouping.sinceis an RFC3339 time, a UTC day or a duration back (7d,36h,90m; default7d; at most 366 days);untilan RFC3339 time or a UTC day (default: no bound;400unless aftersince).errorscounts status ≥ 400.venom cost --by rolereads the same lanes from Venom Cloud'sGET /v1/usage/summary.GET /admin/v1/pricing→ every configured model's effective price:{sources: ["custom","builtin"], table_models, models: [{id: "upstream/model", upstream, model, base_model, source, local, disabled, input, cached_input, output, blended, price_ref, routes: [alias]}], routes: [{alias, strategy, targets: [{id, source, blended}]}], unpriced: ["upstream/model"]}. Models are the route targets, the upstreams' staticmodelslists and the upstream/model idspricing:names; prices are USD per 1M tokens (nullwhen unpriced),blendedthe 3:1 input:output pricestrategy: cheapestranks by,sourceone ofcustom,base_model,builtin,local,unpriced. A route's targets are in its configured order (price order forcheapest).unpricedlists the enabled cloud models booked at $0 (docs/configuration.md → Pricing and cost-based routing).GET /admin/v1/cache?days=7&sessions=10→ the prompt-cache report over the lastdaysUTC days (1–366, today included; from the request log, so withinrequest_log_retention):{since, totals: B, by_route: [B], by_model: [B], by_upstream: [B], by_day: [B], breaks: {reason: n}, top_sessions: [{...B, parent_session_id}]}withB = {key, requests, tokens_in, cached_tokens, cache_write_tokens, hit_ratio, cached_usd, uncached_usd, write_usd, saved_usd, breaks}. Rows are successful requests with a prompt and a serving upstream;cached_tokensis clamped to the prompt andcache_write_tokensto what is left of it;cached_usd+write_usd+uncached_usdis the prompt's list price (cached ×cached_input, written ×cache_writeor input, the rest × input; 0 for local upstreams and unpriced models) andsaved_usd= cached × (input − cached_input).breakscounts rows logged with acache_break.by_modelkeys are the servedupstream/model,by_daylists only days with rows, lists are by prompt tokens (days by date), andtop_sessions(at most 100) are the sessions with logged breaks, most first, then by uncached + write $. The same figures asscripts/cache-report.py --json(make cache-report); a test runs both over one fixture logPOST /admin/v1/reload— re-readorchestrator.yamland the price table.${VAR}references expand from the process's environment as it started, so a changed.envvalue (a new provider key, say) needs a restart, not a reload- Config history (
server/confighist.go,internal/confighist): everyorchestrator.yamlthe process applies (at start, on any reload: this endpoint, SIGHUP, a local-model selection; on a rollback) is stored in SQLite (config_versions, migration 10) as the file's bytes before${VAR}expansion, with its SHA-256; a config equal to the newest row is not stored again, and the newest 50 are kept.GET /admin/v1/config/versions→{versions: [{id, applied_at, sha256, source, bytes, current}], applied_sha256, file: {path, sha256, changed}, keep}, newest first.sourceisstartup,reloadorrollback:<id>;currentmarks the version the running config came from;file.changedsays the file on disk differs from it (an edit not reloaded, or a reload that failed)GET /admin/v1/config/diff?from=<id|file>&to=<id|file>(defaults: the newest version and the one before) →{from, to, diff: {upstreams, routes, profiles, other, changes}};fileis the file as it is now. Each section lists{name, op: added|removed|changed, value?, changes?: [{path, from?, to?}]}: upstreams byname, routes byalias, profiles by name,otherone entry per remaining top-level key. Mappings diff per field (dottedpath), lists whole. Secrets are redacted:${VAR}references stay as written (the env value is never read), a literal header value or a literal under a key likekey/token/secret/passwordbecomes[redacted <6 hex of its SHA-256>](a change still shows), a URL's password and query values becomeredacted. 404 unknown id, 400 a bad ref, 422 a side that isn't YAMLPOST /admin/v1/config/versions/{id}/rollback{overwrite_sha256?}→ writes the version to the config file (temp file in its directory + rename, the mode kept, a symlink's target written) and reloads it the ordinary way:{status: "rolled_back", rolled_back_to, version, upstreams, routes}(versionis the new history row, sourcerollback:<id>), or{status: "unchanged"}when it is what runs already.409 config_changed_on_disk {disk_sha256, applied_sha256}when the file isn't what was last applied, unlessoverwrite_sha256equalsdisk_sha256(the caller saw that content and replaces it);422 invalid_versionwhen the version no longer loads (checked before writing);500 config_not_writable(e.g. compose's:roconfig mount);422 reload_failedputs the previous file back
- Providers (
server/providers.go,internal/providers, ADR 0045): set up provider keys from the console instead of editing.env. Every value is an environment variable the config already reads (api_key_env,${X_BASE_URL:-…},AWS_*); the config never holds a key. Values set here live in$VENOM_DATA_DIR/provider-keys.json({"env": {VAR: value}}, 0600 in a 0700 directory, outside the repo) and are put in the process environment at start, before the first config load, unless the variable is already set (.env, the shell, compose): that value wins and is read-only here. A key is never returned or logged: responses say whether it is set, where from and its last four characters (none for a value under 12 characters).GET /admin/v1/providers→{providers: [P], store: {path, persisted}, config: {path, writable}}.P={id, name, section: cloud|local|voice, kind, upstream, blurb, key_help?, docs_url?, models_help?, verified?, shares?, fields: [F], configured, conflict?, status: {state, reason?, models, last_error?}}. A provider is listed only when this build has its kind (config.KnownKind); a speech card on a chat kind (MiniMax T2A, Qwen-TTS) only when the kind has a native speech adapter; a kind registered by an adapter with no template (a local TTS server added later) gets a generic card.sharesnames the provider whose upstream and key serve the card (no fields).configured: the config file has an upstream namedupstream;conflict: that upstream has another kind.stateisnot_configured,disabled(reason, e.g.X is not set),checking,healthyorunhealthy.F={id, label, type: secret|url|select|text|models, env?, placeholder?, hint?, pattern?, options?: [{value, label}], default?, optional?, set, source?: env|stored|config|default, value? (non-secret only), last4? (secret only), read_only, note?}.envis the variable the configured upstream reads (its ownapi_key_envwins over the template's); abase_urlwritten out literally in the file hassource: config, read-only. Field ids:api_key,base_url(a URL or a regionselectwhose values are base URLs),aws_access_key_id,aws_secret_access_key,aws_session_token,aws_region,modelsPUT /admin/v1/providers/{id}{values?: {field: value}, models?: [string]}→ saves the values (a secret sent empty is unchanged; a key that doesn't matchpatternis saved with a warning) and, when the config has no upstream for the provider, appends its block (the item's indentation matched, every other byte kept,api_key_env/${X_BASE_URL:-default}only) and reloads, recorded in the config history asproviders:<id>.modelsonly when adding (default the template's). →{status: saved|unchanged, provider: P, config: {status: added|unchanged|manual, path, block?}, warnings: [string]}. A read-only config file (compose's:romount) keeps the values and answersstatus: manualwith theblockto add by hand. Errors{error, code}: 400unknown_field,invalid_value,missing_value,shared_provider; 404unknown_provider; 409env_read_only(the variable comes from.env),field_fixed(written out in the config, or models of a configured upstream),field_unused(the upstream has noapi_key_env),kind_conflict,config_not_editable(an inlineupstreams: [...]),key_required(a typed-inurlfield changes while a key is set: the key must be sent again, so a stored key can't be pointed at another host; also on the test endpoint); 422invalid_config(the new block doesn't load: nothing written),reload_failed(the previous file and values are put back)POST /admin/v1/providers/{id}/test{values?, models?}→ runs that upstream's own health check once (GET {base}/modelswith the kind's auth, or the kind's registered check) withvaluesover the current ones, nothing saved:{ok, checked, latency_ms?, model_count?, models? (first 50), error?, message?, warnings}. AWS kinds and voice kinds with no check answerok: true, checked: false. Errors never contain a key. The only outbound call, made only on demandDELETE /admin/v1/providers/{id}/key→ removes the provider's stored secrets and reloads; the upstream stays in the config, disabled. 409env_read_onlywhen the key comes from.env, 404no_stored_key- 404
no_storeon all four when the server runs without a key store (an embedder that didn't setServer.Providers)
GET /admin/v1/debug/pprof/— the Go profiler (net/http/pprof):profile?seconds=N(CPU),heap,allocs,goroutine,block?seconds=N,mutex?seconds=N,trace?seconds=N. Block and mutex sampling are switched on only for the duration of a?seconds=profile. Example:curl -H "Authorization: Bearer $VENOM_ADMIN_TOKEN" -o cpu.pb.gz 'http://127.0.0.1:8787/admin/v1/debug/pprof/profile?seconds=10'thengo tool pprof -top cpu.pb.gz(go tool pprofcan't send the header itself)
Local models (/admin/v1/local/*, internal/localmodels): the daemon's own models, so a user
never has to drive Ollama or vLLM by hand. Off the /v1 path; outbound calls go only to the
configured local upstreams and to llmfit. Errors are {error, code} with a stable code.
GET /admin/v1/local→{hardware, llmfit, backends, selection, strain, pulls, mesh}:hardware:{source:"llmfit", gpu_name, gpu_count, gpu_vram_gb, unified_memory, total_ram_gb, available_ram_gb, cpu_name, cpu_cores, backend}, ornullwithout llmfit;llmfit:{available, via:"exec"|"url"|"", error}.backends: one per enabled upstream of kindollama,vllm,llamacpp,sglang:{upstream, kind, base_url, reachable, version, managed, error, installed:[{model, size_bytes, parameter_size, quantization, family, modified_at}], loaded:[{model, size_bytes, vram_bytes, context_length, expires_at}], served:[string], metrics:{running, waiting, kv_cache_pct}|null}. Every field is always present:installed/loadedare Ollama's (/api/tags,/api/ps),servedvLLM's/v1/models(llama.cpp / SGLang: the health check's list),metricsvLLM's/metrics.managedmarks the Ollama the actions below use (the first enabled one).selection:{default, fast, source:"env"|"console"}: the modelsvenom-defaultandvenom-fastroute to on the managed Ollama;consolewhen a choice made here is in force.strain: per backend{upstream, model, level, inflight, max_inflight, utilization, latency_ewma_ms, healthy, reasons:[string]},levelthe first that applies:down(no answer, or the last health check failed),strained(vLLM requests waiting or KV cache ≥ 90%; an Ollama model partly in system RAM,vram_bytes < size_bytes; everymax_inflightslot taken),busy(half the slots or more),ok(something in flight),idle.pulls:[{model, status, completed, total, percent, error, started_at, finished_at, select}], running pulls and those finished in the last 10 minutes;statusis Ollama's latest line while running, thensuccessorerror.mesh:{enabled, cloud_url, name, preflight?, live?}(cloud_url/namepresent whenVENOM_CLOUD_URLis set;enabledwhen mesh mode runs).live(only while mesh mode runs, in-memory, no network call) is{daemon_id, state:"connecting"|"connected"|"reconnecting"|"draining"|"stopped", error?, last_heartbeat_at?, lease_expires_at?, rtt_ms|null, channel_open, inflight, slots, jobs_done}:connectedwhile the last heartbeat's lease runs,reconnectingonce it ran out,stopped(witherror) after a revoke or a protocol refusal;jobs_donecounts dispatched jobs finished since start. The console's overview reads it.
GET /admin/v1/local/catalog[?refresh=1]→{hardware, llmfit, context, recommended:{default, fast}, models:[{id, backend:"ollama"|"vllm", name, hf_name, agent_ready, size_gb, memory_gb, fit:"perfect"|"good"|"marginal"|"too_tight"|"unknown", run_mode:"gpu"|"cpu_offload"|"moe_offload"|"cpu"|"", est_tps, tools, vision, note, installed, selected:"default"|"fast"|"", pullable, hint, chat}]}.contextis the managed Ollama'scontext_window(OLLAMA_CONTEXT_LENGTH, default 32768). Rows: the curated agent-ready Ollama tags (agent_ready: true;memory_gb= the tag's weights plus its KV cache atcontext, fitted against llmfit's VRAM/RAM);scripts/model.sh's vLLM presets (agent_ready: true, sized for VRAM alone); llmfit's other tool-using Ollama models that fit wholly on the GPU (≤ 20; a model with no Ollama tag but an Unsloth GGUF in llmfit'sgguf_sourcesis listed ashf.co/unsloth/<repo>:<quant>, llmfit'sbest_quantwhen it is a GGUF quant, else Q4_K_M;memory_gbat least the download + 10%, since llmfit counts only an offloaded MoE's GPU part); up to five llmfit vLLM picks from the makers' own repos (Qwen/, openai/, mistralai/, google/, meta-llama/, deepseek-ai/, zai-org/); every vLLM row haspullable: false,hint: "make model BACKEND=vllm MODEL=<repo>"andlaunch(true whenPOST /admin/v1/local/servecan start it: a known tool-call parser, nottoo_tight, and the catalog'svllm_launch.available); then installed tags in neither.chat: falsemarks an installed embedding model (a*bertfamily or "embed" in the name): not selectable. Order: selected, agent-ready models that fit (biggest first), other models that fit, the rest, embedding models last.recommended(Ollama rows only): the largest agent-ready model that fits perfect or good, and the smallest that fits perfect;qwen3:8b/qwen3:4bwithout llmfit. llmfit is read once per 30 minutes (refresh=1re-reads).GET /admin/v1/local/suggest[?days=7&gpu_usd_per_hour=X]→{object: "venom.local.suggest", days, window_days, gpu_usd_per_hour|null, local: {installed, fast|null, default|null}, candidates: [{gear, role, model, requests, requests_per_week, avg_tokens_in, avg_tokens_out, tool_share, avg_latency_ms, cost_usd, cost_per_week, local_model|null, gpu_per_week_usd?, save_per_week_usd, fit: "good"|"maybe"|"poor", note?}], total_save_per_week_usd, note}: the local-model autopilot's read (venom models local suggest). The successful (status < 400) cloud (local = 0) requests of the lastdays(1 to 30, default 7) whose role isclassifier,summary,reflexorworkers, grouped by gear, role and served model (at most 200 groups, costliest first; main turns,smart,visionandbrowserare left out: a small local model is not a stand-in for them).window_daysis what the per-week figures scale from:days, or less when the window's first such row is younger (roles are logged from migration 15 on), at least 1.local_modelis what would serve the role here: venom-fast's model for classifier, summary and reflex, venom-default's for workers, counted only when the managed Ollama has it installed (without a reachable Ollama, the alias's first healthy local target), elsenull.fit(localmodels.SuggestFit):goodfor classifier, summary or reflex averaging ≤ 8k tokens in and ≤ 1k out;maybefor workers, or up to 32k in;poorfor larger, and for a classifier or summary with more than half its requests offering tools.save_per_week_usdiscost_per_week(local tokens are $0) minus, withgpu_usd_per_hour(0 to 1000),gpu_per_week_usd= requests a week × the rows' average latency / 3.6e6 × $/h (the cloud's latency as a rough stand-in for the local model's); without it every candidate'snotesays GPU time and electricity are not counted. Candidates are sorted bysave_per_week_usd, highest first;total_save_per_week_usdsums thegoodandmaybeones that save. Read only: nothing is applied.400 invalid_days,400 invalid_gpu_usd_per_hour.POST /admin/v1/local/pull{model, select?: "default"|"fast"}→202 {pull}: starts (or joins) a backgroundPOST /api/pullon the managed Ollama; it outlives the request and ends with the server. Withselect, the model is selected for that tier when the pull succeeds.400 invalid_model(Ollama's name charset,hf.co/<user>/<repo>:<quant>allowed) /invalid_tier,409 no_ollama(no Ollama upstream),503 ollama_unreachable.POST /admin/v1/local/select{model, tier:"default"|"fast"}→200 {selection, warning?}: setsVENOM_OLLAMA_MODEL/VENOM_OLLAMA_FAST_MODELin the process and reloads the config exactly asPOST /admin/v1/reloaddoes (a failed reload restores the variable,500 reload_failed), then persists$VENOM_DATA_DIR/local-models.json(0600,{"default":{"model","env"},"fast":{…}},env= the variable when the process started). At start, before the first config load, each entry is applied unless the variable now differs from itsenv(make modelor.envchanged it since: that choice wins and the entry is dropped).409 not_installedwhen Ollama doesn't have the tag (pull withselectinstead),409 not_chatfor an embedding model;warningwhen the config doesn't route the tier's alias through its variable. After a select (and a pull), the managed Ollama's health check runs at once, so routing sees the model without waiting forhealth_interval.POST /admin/v1/local/setup(no body or{}) →202 {recommended, pulls, selection}when it started pulls,200withpulls: []when both recommended models are installed and selected: the one-click path, pulling what is missing (selected on success) and selecting what is installed.POST /admin/v1/local/serve{model}→202 {serve}: one-click vLLM. The orchestrator starts vLLM itself as one Docker container,venom-vllm(compose's pinned image,VENOM_VLLM_IMAGEoverrides; flags and parsers asmake model BACKEND=vllm;-p 127.0.0.1:<vllm upstream port>:8000; weights in compose'svenom_hf-cachevolume;--restart unless-stopped), after unloading the managed Ollama's loaded models and removing its own previous container. It follows the start in the background (GET /admin/v1/local→serve: {model, state: starting|ready|error, message, started_at, finished_at},messagethe container's last log line), and once vLLM lists the model setsVLLM_MODELand reloads, persisted asvllminlocal-models.json(same precedence rule). Available only when the orchestrator runs on Linux withdockeron its PATH (VENOM_DOCKER_BIN), Docker has the NVIDIA runtime, and thevllmupstream'sbase_urlis plain http on loopback (GET /admin/v1/local/catalog→vllm_launch: {available, reason}): so on the desktop daemon andmake dev-orchestrator, not in the compose container.400 invalid_model(a Hugging Faceowner/name) /no_parser,409 vllm_launch_unavailable,409 vllm_foreign(a vLLM it didn't start answers on the port: left alone). Selecting an Ollama model (select,setup, a pull withselect) stops this container: one GPU, one engine.DELETE /admin/v1/local/serve→204: removesvenom-vllm;404 not_running.DELETE /admin/v1/local/models/{model...}→204;409 selectedfor a selected model,409 pulling,404 not_installed.POST /admin/v1/local/unload{model}→204(Ollamakeep_alive: 0: frees its VRAM now).
Daemon (/admin/v1/mesh*, /admin/v1/daemon*; internal/meshctl, internal/daemon): what the
desktop daemon's dashboard drives (docs/daemon.md). Errors are {"error": {message, type, code}}.
Nothing here dials out until the user connects (mesh) or turns on or asks for an update (auto-update).
GET /admin/v1/mesh→{source: "env"|"dashboard"|"none", state: "off"|"connecting"|"connected"| "revoked"|"error", cloud_url, name, daemon_id, team_id, region, enrolled_at, error, default_cloud_url: "https://cloud.usevenom.dev", can_change, paused, jobs, tokens_per_s}.paused(below),jobs(dispatched jobs running now) andtokens_per_s(output tokens per second of the requests finished in the last minute,nullwhen none) read memory only; the macOS menu bar app polls them.source: "env"(mesh mode set byVENOM_CLOUD_URL) is read-only here:can_changeis false and connect/disconnect answer409 mesh_managed_by_env.POST /admin/v1/mesh/connect{join_token: "vjt_…", cloud_url?, name?, region?}→200status: enrolls (docs/cloud-mesh.md§6.1), saves$VENOM_DATA_DIR/mesh-settings.json(cloud URL, name, region; never the token) next to the credential inmesh/, and starts mesh mode without a restart; a restart reads the settings file likeVENOM_CLOUD_URL/VENOM_MESH_NAME/VENOM_MESH_GEO.cloud_urldefaults tohttps://cloud.usevenom.dev,nameto the hostname.400 invalid_join_token|invalid_cloud_url|invalid_name|invalid_region,409 mesh_already_connected,502 mesh_join_failed(the cloud's reason, e.g. an expired token).POST /admin/v1/mesh/disconnect→200status: drains running jobs, deregisters, removes the credential and the settings file (and a pause).POST /admin/v1/mesh/pause/POST /admin/v1/mesh/resume→200status: the daemon stays connected and sends itsdrainingheartbeat (the cloud stops dispatching; running jobs finish) until resumed. Kept across restarts as$VENOM_DATA_DIR/mesh-paused; the auto-update's drain and its resume do not end a pause.409 mesh_offwith no connection configured (venom-daemon pause|resume).GET /admin/v1/heal?limit=50(1 to 200) →{state: "ok"|"healing"|"degraded", failing: [{name, kind, state: "healing"|"degraded"|"attention", attempts, since, next_try?, error?}], recovered_total, repairs_total, events: [{at, kind, target, action?, outcome, attempt?, detail?}]}(newest first): the self-heal supervisor (internal/heal, docs/daemon.md → Self-heal).404 not_availablewithout one. Read-only.GET /admin/v1/computers→{enabled, state: "off"|"enabling"|"on"|"error"|"unsupported", source?: "env"|"setting", can_enable, message?, checks: [{id, label, ok, detail?, fix?}], steps: [{id, label, state, detail?, progress?}], image: {ref, present, way?}, capacity?}(VEN-403, Venom Computers one-click setup,docs/public/computers/enable.md).POST /admin/v1/computers/enableruns the checks, gets the image, savescomputers/setting.jsonin the data dir and starts the manager live:202while it works,200once on, idempotent;409 unsupportedon a host that cannot run Computers (macOS, a container); any other failed check ends in stateerrorwith its fix sentence.POST /admin/v1/computers/disablehibernates seats, stops and saves the setting off (409 env_overridewhileVENOM_COMPUTERS=on).404 not_availablewithout the feature. The same switch is the meshcomputerjob opsenableanddisable(seat_idthe nil uuid).GET /admin/v1/daemon→{managed, version, os, autostart: {supported, enabled}, ollama: {installed, path}, ollama_download_url}.managedis true whenvenom-daemonstarted the orchestrator (VENOM_DAEMON_LAUNCHER); Ollama'sinstalledis a file check (PATH and its usual install paths); whether it runs is theollamabackend'sreachableinGET /admin/v1/local.PUT /admin/v1/daemon/autostart{enabled}→ the daemon status; writes or removes the per-user login item (a LaunchAgent, an XDG autostart entry, a Startup-folder script).404 not_managedoutside the launcher.GET /admin/v1/daemon/projects→{folders: [{path, exists}], max, serve: {running, url?, error?}, managed, workspaces?: {root, items: [{id, name, kind, path, origin, created_by?, git_url?, created_at}], scratch_ttl_hours}}: the folders the daemon'svenom serveserves next to its workspaces folder (ADR 0081, docs/daemon.md → Projects;projects.jsonin the state dir) and the workspaces in it (docs/daemon.md → Workspaces).DELETE /admin/v1/daemon/projects?workspace=<id>deletes one and its files (404 workspace_not_found).POST /admin/v1/daemon/projects{path}adds one (absolute, or under the home folder for~/…and relative paths; must be an existing folder, not a disk root; at most 50;400 invalid_project) andDELETE /admin/v1/daemon/projects?path=removes one (404 project_not_found); both run the launcher'sserve-reload(restart serve) and answer the list, withserve.errorwhen serve did not come up.404 not_managedoutside the launcher.- Auto-update (
internal/daemon/update.go, ADR 0031; opt-in, docs/daemon.md → Auto-update). The daemon status carries it asupdate(managed daemons only), and:GET /admin/v1/daemon/update→{supported, reason?, auto_update, state, current, latest?, available, last_check_at?, next_check_at?, error?, detail?, progress?: {done, total}, skipped_versions?, last_result?: {from, to, state, reason?, at}, source}.stateisidle|checking|available|downloading|verifying|waiting_for_idle|draining|restarting|done|failed|rolled_back;supported: false(withreason) for a dev build, Windows, or an install not made byinstall-daemon.sh. No network.PUT /admin/v1/daemon/update{auto_update}→ the update status. Off by default; on, checks run every 6 h ±20% (the first within 10 min), backing off with full jitter on failure.POST /admin/v1/daemon/update/check→ the update status after one GET of<releases>/latest/download/SHA256SUMS(VENOM_DAEMON_UPDATE_URL, defaulthttps://github.com/elishahbiz/venom-releases/releases);502 check_failed.POST /admin/v1/daemon/update/apply→202the update status; the install runs in the background (download, SHA256 against the version'sSHA256SUMS, unpack beside the running release, an idle window of 30 s with no/v1request or mesh job in flight, at most 30 min, adrainingmesh heartbeat, then the new launcher'svenom-daemon upgradeswaps and restarts, or rolls back).409 up_to_date|update_busy|not_supported.- All four answer
404 not_managedoutside the launcher. Files:data/daemon-update.json(switch, last check, versions skipped after a rollback) anddata/daemon-upgrade.json(the swap's result), both 0600.
Cost. cost_usd is list price from the bundled, offline price table
(config/prices.yaml: USD per 1M input, cached_input and output tokens,
with the date checked and the source URL per model; never fetched at
runtime). Per request: (prompt − cached) × input + cached × cached_input + completion × output, matched on the model the upstream served (exact, else
minus a dated snapshot suffix). A local upstream (kind ollama, vllm,
sglang, llamacpp, or a base URL whose host is loopback, private,
link-local, *.local, *.internal or a single label such as a compose
service) costs 0 and is marked local: true, unless the upstream's config sets
local: false (a paid gateway on a private address, such as a LiteLLM proxy:
billed, and counted against budgets) or local: true (a self-hosted server on a
public name); a cloud model missing from the table also costs 0 (logged once at warn, listed by
GET /admin/v1/pricing). pricing: in orchestrator.yaml sets custom prices and base models
(a deployment named my-gpt priced as gpt-5.4) and the lookup order over custom and the table;
a served model with a base model is answered with x-venom-base-model
(docs/configuration.md → Pricing and cost-based routing). It is computed once per request after the response is
relayed, never on the streaming path. The same table ships with the harness
for the CLI's local estimate (packages/harness/src/cost/prices.ts, kept in
sync by a test).
Model catalog: GET /api/models/providers/{id} (no key, not under /v1) serves
<VENOM_MODEL_CATALOG>/<id>.json, a {"models":[...]} shard in the format pi's remote catalog
client reads, with a strong ETag and Last-Modified (If-None-Match → 304), types= ignored;
an unknown or unsafe id, or no directory, is 404. Static files, no per-request allocation beyond
the file handle. The CLI does not read it yet: pi has no setting for the catalog URL (ADR 0042).
Ops: GET /healthz (liveness, 200 {status:"ok"}), GET /readyz (200 with
≥1 healthy upstream, else 503 {status:"no healthy upstream", healthy_upstreams:0, checking_upstreams:0}, or status:"checking upstreams" with checking_upstreams > 0 while enabled upstreams still await their first probe, right after a start or reload: venom doctor waits up to 6 s for those);
an upstream counts as unhealthy until its first probe passes; a probe that only timed out keeps a
healthy upstream healthy when live traffic got a 2xx from it since the previous probe, and otherwise
takes two in a row, while an error status or a refused connection still counts at once),
GET /metrics (Prometheus text format 0.0.4, passes promtool check metrics):
| metric | type | labels |
|---|---|---|
venom_requests_total |
counter | model, upstream, code (0 = no upstream reached, 499 = client closed) |
venom_request_duration_seconds |
histogram | model (successful requests) |
venom_ttft_seconds |
histogram | model (successful streams) |
venom_tokens_total |
counter | model, type = prompt | completion | cached | cache_write |
venom_cache_breaks_total |
counter | model (served upstream/model), reason = idle | model_change | prompt_shrank | concurrent | unknown (see /admin/v1/requests cache_break) |
venom_cache_hit_ratio |
gauge | — (cached / prompt tokens since start) |
venom_cost_anomaly |
gauge | scope = key | session, metric = spend | cache_breaks (open anomalies, /admin/v1/anomalies) |
venom_cost_anomalies_total |
counter | scope, metric (anomalies opened) |
venom_response_cache_hits_total |
counter | kind (side kind), how = hit (stored) | shared (an identical request in flight); billed 0 |
venom_response_cache_stores_total |
counter | kind (side-request responses stored) |
venom_dedup_total |
counter | how = joined (shared an identical request's upstream call) | replayed (a stored Idempotency-Key response) | lagged (cut loose: fell max_lag_bytes behind) | alone (too late to join, went upstream itself) | mismatch (an Idempotency-Key reused with another body, 422); billed 0 |
venom_tool_call_repairs_total |
counter | upstream (served), kind = wrapper | single_quote | python_literal | control_char | trailing_comma | unterminated | unrepaired (responses whose tool-call arguments were mended; see services/go-orchestrator/AGENTS.md → Tool-call repair) |
venom_upstream_shaping_queue_depth |
gauge | upstream (requests waiting for the provider's learned rate limit; only upstreams whose shaper learned a limit) |
venom_upstream_shaping_requests_total |
counter | upstream, outcome = admitted (at once) | queued (after waiting) | cancelled (left the queue) |
venom_upstream_shaping_rejected_total |
counter | upstream, reason = queue_full | wait_exceeded (answered 429 here) |
venom_upstream_shaping_wait_seconds |
histogram | upstream (how long queued requests waited) |
venom_upstream_requests_waiting, venom_upstream_kv_cache_usage |
gauge | upstream (the engine's own queue and KV-cache use 0–1 from its /metrics; fresh scrapes only) |
venom_cost_usd_total |
counter | model (list-price USD from the price table; local upstreams add 0) |
venom_wasted_tokens_total, venom_wasted_usd_total |
counter | model, reason = abandoned | ttft_timeout | client_disconnect | retried (billed attempts whose output was discarded; see /admin/v1/requests waste) |
venom_active_streams, venom_uptime_seconds |
gauge | — |
venom_upstream_healthy, venom_upstream_inflight |
gauge | upstream |
venom_upstream_circuit_state |
gauge | upstream (0 closed, 1 open, 2 half_open) |
venom_upstream_latency_ewma_seconds |
gauge | upstream |
venom_upstream_attempts_total |
counter | upstream, attempt = first | retry (a retry is a later attempt of the same request on the same upstream) |
venom_upstream_retries_skipped_total |
counter | upstream, reason = budget (retry budget spent) | retry_after (the upstream asked for more than backoff_max; its answer went to the client) | client_retrying (x-stainless-retry-count > 0) |
venom_upstream_retry_budget_available |
gauge | upstream (retries the budget allows now) |
venom_upstream_quota_exhausted |
gauge | upstream, model (1 while a target sits out its quota cooldown; no series otherwise) |
venom_route_target_available |
gauge | route, upstream, model (1 = the upstream has the model or can't tell, 0 = routing skips the target) |
venom_key_spend_usd |
gauge | key (key name), key_id, period = daily | monthly (spend in the current UTC day / month) |
venom_key_budget_usd |
gauge | key, key_id, period, limit = soft | hard (only limits that are set) |
venom_budget_soft_limit_crossed_total |
counter | key, key_id, period (at most once per key and period window) |
venom_budget_exceeded_total |
counter | key, key_id, period (requests refused with 429 budget_exceeded) |
venom_request_log_dropped_total |
counter | — |
venom_request_log_rows |
gauge | — (rows in the request log: request_log_retention, plus up to a margin of 10%, at least 2,048, before pruning starts) |
go_goroutines, go_heap_objects_bytes, go_memory_total_bytes, process_open_fds |
gauge | — (process_open_fds on Unix only) |
go_gc_cycles_total, process_cpu_seconds_total |
counter | — (process_cpu_seconds_total on Unix only) |
The four per-key series list a key with a budget (every series, zeros
included), else a key with spend this month or a non-zero counter; an idle
key without a budget adds none. key_id keeps two keys with the same name
apart.
model is the requested alias or id (unknown names are recorded as
(unknown), and a direct upstream/<model> the upstream neither lists nor
has configured as a route target as upstream/(unknown)), so label
cardinality is bounded by the config and the upstreams' model lists. The
observability compose profile (Prometheus + Grafana, config in
deploy/observability/) graphs these.