Upstreams and kinds
Add an inference server or cloud provider to the orchestrator, pick its kind, and set its keys, models, capabilities and limits.
An upstream is one server the orchestrator sends requests to: a local engine, a provider's API or a gateway. Each entry under upstreams: in orchestrator.yaml has a name (what routes and upstream/model ids refer to), a kind (which wire format and parameter rules apply) and a base_url. Routes then list {upstream, model} targets; see Routes and strategies.
Minimal example
upstreams:
- name: ollama
kind: ollama
base_url: ${OLLAMA_BASE_URL:-http://localhost:11434/v1}
context_window: ${OLLAMA_CONTEXT_LENGTH:-32768}
- name: openai
kind: openai
base_url: https://api.openai.com/v1
api_key_env: OPENAI_API_KEY # unset = the upstream is disabled, nothing calls OpenAI
max_inflight: 32
routes:
- alias: venom-default
targets:
- {upstream: ollama, model: "qwen3:8b"}
- {upstream: openai, model: gpt-5.4-mini}An upstream whose api_key_env names an empty variable, or whose base_url is empty, is kept but disabled: it is never health-checked and its targets are skipped. That is why one file works online and off the grid. GET /admin/v1/upstreams shows each upstream's state and why it is disabled.
Kinds
The kind decides the request rewriting ("quirks"), the capability defaults, how the context window is discovered and how the key is sent. Unset, kind is generic.
| Kind | For | Notes |
|---|---|---|
openai |
OpenAI's API | /v1/responses passed through natively; reasoning_effort mapped to the nearest level each model takes; service_tier flex and priority; a shared prompt_cache_key pool per model; native_batch and cache_retention (24h, 30m) apply. |
ollama |
Ollama's OpenAI-compatible API | max_completion_tokens sent as max_tokens; n and logit_bias dropped; tool_choice off by default; reasoning read from reasoning; context window from /api/ps (loaded model) or /api/show; a model without the tools capability is routed around. Local. |
vllm |
vLLM | stream_options only on streams, an empty tools dropped; reasoning in reasoning; window from /v1/models max_model_len; load scraped from /metrics. Local. |
sglang |
SGLang | reasoning_content; window from /v1/models max_model_len; /metrics scraped. Local. |
llamacpp |
llama.cpp's llama-server |
cache_prompt: true; a slot per x-venom-session (id_slot) on builds without the host prompt cache; window from /v1/models meta.n_ctx or /props. slots applies. Local. |
generic |
any other OpenAI-compatible server (LiteLLM, LocalAI, Kimi's platform, Azure OpenAI's v1 API, Together, Speaches, Kokoro-FastAPI) | No rewriting beyond the common rules; cache_retention: 1h is Kimi's. |
zai |
Z.ai's GLM API (https://api.z.ai/api/paas/v4, or …/coding/paas/v4 for the GLM Coding Plan) |
temperature clamped to [0, 1], top_p to [0.01, 1], stop as an array, metadata dropped, GLM-5.3 efforts mapped to the ones it takes; tool_choice off. Context windows come from config. |
dashscope |
Alibaba Cloud Model Studio's OpenAI-compatible API (Qwen, DeepSeek, Kimi and GLM hosted there) | Thinking controls (enable_thinking, thinking_budget) set from reasoning_effort per model family; a forced tool choice turns thinking off; explicit_cache and native_batch apply; rerank defaults to Model Studio's native API. |
mistral |
Mistral AI's La Plateforme (https://api.mistral.ai/v1) |
Only the documented fields are sent (the API rejects others with 422); seed → random_seed; tool call ids shortened to 9 characters; /v1/completions with a suffix goes to Codestral's /fim/completions; embeddings dimensions → output_dimension. Window from /v1/models max_context_length. |
minimax |
MiniMax's OpenAI-compatible API (https://api.minimax.io/v1, or https://api.minimaxi.com/v1 in mainland China) |
temperature and top_p kept in (0, 1]; reasoning_split on; effort and thinking knobs per model; tool_choice off. No /models: list the models. Chat only. |
anthropic |
Anthropic's Messages API (https://api.anthropic.com/v1) |
Chat completions and /v1/responses translated to POST /messages and back; x-api-key auth; prompt-cache breakpoints placed for you; cache_retention: 1h and native_batch apply; embeddings and completions are unsupported_endpoint. |
bedrock |
Amazon Bedrock's Converse API | Chat translated to /model/{id}/converse(-stream); SigV4-signed with IAM keys (aws:); embeddings and rerank through InvokeModel; priced by the model's base name (us.anthropic.claude-sonnet-5-5-v1:0 as claude-sonnet-5-5). models is required. |
sagemaker |
an AWS SageMaker real-time endpoint | Models are endpoint names (endpoint:component sets the inference component); payload: openai or tgi; SigV4. models is required, no health probe. |
typesafe |
TypeSafe's System One decision API | Decision models: they serve POST /v1/systemone only. See Rerank, images and decisions. |
sdapi |
AUTOMATIC1111's web UI API (also Forge, SD.Next, stable-diffusion.cpp's server); base_url is the server root without /v1 |
Image models: they serve POST /v1/images/generations only. |
"Local" kinds (ollama, vllm, sglang, llamacpp), and any upstream whose base URL is loopback, private, link-local, *.local, *.internal or a single-label host, cost $0 and are never counted against budgets. Set local: false for a paid gateway on a private address, local: true for your own server on a public name.
Speech providers with their own wire formats register as kinds of their own (ADR 0044): cloud ones (elevenlabs, fishaudio, byteplus-speech) and local servers (cosyvoice, qwen3-tts, indextts, sparktts, fishspeech, which cost $0 wherever they run). Speech also goes to any kind above that speaks OpenAI's audio API. See the Voice section and Local speech models.
Upstream keys
| Key | Type | Default | What it does |
|---|---|---|---|
name |
string | required | Unique, no /. |
kind |
string | generic |
One of the kinds above. |
base_url |
URL | none (disabled) | The API root including /v1 (except sdapi). AWS kinds default to the regional endpoint. |
api_key_env |
env var name | none (no auth) | The variable holding the key; unset or empty disables the upstream. |
api_key_envs |
list of env var names | none | A credential chain after api_key_env: a request refused for credit (402, insufficient_quota), a revoked key (401, 403) or a rate limit (429) is retried on the next key before the next route target. |
models |
list | discovered | The models it serves. Required for bedrock, sagemaker, minimax; when set it is also an allowlist for upstream/<id> and bare ids. |
timeout |
duration | defaults.timeout (120s) |
Per-attempt time to first byte. A started response runs to its end. |
max_inflight |
int | defaults.max_inflight (0 = unlimited) |
Concurrent requests to this upstream; /v1/models reports it so clients queue on their side. |
headers |
map | none | Extra request headers (values may use ${VAR}). |
health_interval |
duration | defaults.health_interval (15s) |
How often it is probed. |
stream_usage |
bool | by kind | Whether stream_options.include_usage is injected so streams report usage. |
context_window |
int (tokens) | discovered, else unknown | Applies to every model unless model_info overrides it. A request too long for every target is a 400 before any attempt. |
capabilities |
map | by kind | See below; applies to every model. |
model_info |
map of model id → settings | none | Per-model context_window, capabilities, and pricing (base_model, input, cached_input, output, or a nested price with input, cached_input, output, search_unit). |
slots |
int | /props total_slots |
llamacpp only: the slot count sessions are pinned onto. |
metrics_url |
URL | origin + /metrics for vllm, sglang, llamacpp |
Where the engine's load (queue, KV-cache use) is scraped. |
local |
bool | by kind and address | Whether its tokens cost $0. |
native_batch |
bool | false |
openai, dashscope, anthropic only: hand batches and batch requests to the provider's Batch API at its batch price. See Batches. |
aws |
map | from the environment | bedrock and sagemaker: region (default AWS_REGION, then AWS_DEFAULT_REGION), access_key_id_env (default AWS_ACCESS_KEY_ID), secret_access_key_env (default AWS_SECRET_ACCESS_KEY), session_token_env (default AWS_SESSION_TOKEN). Disabled while the region or a key is missing. |
payload |
openai or tgi |
openai |
sagemaker only: the container's body format. |
prompt_template |
plain, chatml, llama3 |
plain |
sagemaker with payload: tgi: how messages become one prompt. |
rerank |
jina, cohere, voyage, tei, dashscope |
dashscope for kind dashscope, InvokeModel for bedrock, else jina |
The dialect POST /v1/rerank is sent in. |
reference_audio_dir |
path | none | Local speech kinds: the directory a speech request's venom_options.reference_audio file path is read from for voice cloning; unset, file paths are refused. See Local speech models. |
Capabilities
capabilities: on an upstream, or under model_info.<model>.capabilities, overrides the kind's defaults. Discovery (after each passing health check) only fills what the config leaves unset.
| Key | Type | Default | What it does |
|---|---|---|---|
tools |
bool | true (false for typesafe) |
The model takes tools; a request with tools skips targets that don't. |
tool_choice |
bool | true; false for ollama, zai, minimax |
Whether tool_choice is sent or removed. |
responses |
bool | true for openai |
Native /v1/responses; otherwise requests are translated to chat completions. |
reasoning_field |
reasoning or reasoning_content |
reasoning_content (reasoning on ollama, vllm; none on openai) |
Where the server puts reasoning text; answers are normalised to reasoning_content. |
replay_reasoning |
bool | true (false for openai, anthropic, bedrock) | Whether earlier turns' reasoning is sent back. |
vision |
bool | discovered, else a known-model list | Image input; /v1/models reports it. |
decisions |
bool | true for typesafe |
A decision model: serves /v1/systemone and nothing else. |
tts |
bool | a known-id list (tts-1, gpt-4o-mini-tts, Kokoro, ...) |
A text-to-speech model for /v1/audio/speech. |
stt |
bool | a known-id list (whisper, *-transcribe, voxtral, ...) |
A speech-to-text model; reported by /v1/models. |
constrained_decoding |
bool | true for local kinds | The server decodes against a JSON schema; false removes json_schema constraints instead of sending them. |
cache_retention |
24h, 30m (openai); 1h (generic, anthropic) |
none | The longer prompt-cache lifetime asked for when a session pauses (x-venom-cache-ttl: long, or gaps over 5 minutes). |
explicit_cache |
bool | off | dashscope only: Model Studio's explicit cache markers on prompts of 1,024+ tokens. |
Examples
A paid gateway on a private address, priced and budgeted:
- name: litellm
kind: generic
base_url: http://10.0.0.5:4000/v1
api_key_env: LITELLM_KEY
local: falseTwo keys for one provider, the second used when the first runs out of credit:
- name: openai
kind: openai
base_url: https://api.openai.com/v1
api_key_env: OPENAI_API_KEY
api_key_envs: [OPENAI_API_KEY_BACKUP]Bedrock and a SageMaker endpoint priced per token:
- name: bedrock
kind: bedrock
aws: {region: us-east-1}
models: ["us.anthropic.claude-sonnet-5-5-v1:0"]
- name: sagemaker
kind: sagemaker
aws: {region: us-east-1}
payload: openai
models: [my-llama-endpoint]
model_info:
my-llama-endpoint: {context_window: 131072, price: {input: 0.30, output: 0.60}}Mistral and MiniMax (both commented out in the shipped config; set MISTRAL_API_KEY or MINIMAX_API_KEY and uncomment):
- name: mistral
kind: mistral
base_url: ${MISTRAL_BASE_URL:-https://api.mistral.ai/v1}
api_key_env: MISTRAL_API_KEY
models: [mistral-large-latest, mistral-small-latest, codestral-latest, mistral-embed]
- name: minimax
kind: minimax
base_url: ${MINIMAX_BASE_URL:-https://api.minimax.io/v1}
api_key_env: MINIMAX_API_KEY
models: [MiniMax-M3, MiniMax-M3.1-Flash]Troubleshooting
- The upstream shows
disabled: itsapi_key_envvariable is empty in the orchestrator's environment, orbase_urlis empty. Set it andPOST /admin/v1/reload(or sendSIGHUP). 503 model_unavailable: the upstream's own model list (Ollama, vLLM) lacks the model. The error names the fix, such asollama pull qwen3:8b.- A target is skipped as
unhealthy: its last health check failed. Kinds without a model list (minimax, AWS) are judged by their circuit breaker instead. 502 upstream_auth_failed: the provider refused the orchestrator's key (not the client's). The provider's body is not relayed, since it can echo part of the key.- Config errors at start or reload: unknown fields and kinds, a duplicate name, a
/in a name, or a kind-specific key on the wrong kind (slotsoffllamacpp,native_batchoff the three batch kinds,payloadoffsagemaker) stop the load with the field named. A failed reload keeps the running config.
Related
- Routes and strategies and Configuration reference.
- Environment variables for the keys and base URLs the shipped config reads.