Routes and strategies
Define model aliases, choose how their targets are ordered, and understand fallback, retries, stickiness, experiments and shadow traffic.
A route gives a model alias (the model a client sends) an ordered list of targets, each a model on a configured upstream. The alias is what clients depend on; the targets are yours to change.
Minimal example
routes:
- alias: venom-fast
strategy: fallback # the default
ttft_timeout: 12s # give up on a target that hasn't sent a byte by then
targets:
- {upstream: ollama, model: "qwen3:4b"}
- {upstream: openai, model: gpt-5.4-nano}
- {upstream: qwen, model: qwen3.8-flash} # skipped while QWEN_API_KEY is unsetcurl -s http://127.0.0.1:8787/v1/chat/completions -H "Authorization: Bearer $VENOM_API_KEY" \
-H "Content-Type: application/json" -D - -o /dev/null \
-d '{"model":"venom-fast","messages":[{"role":"user","content":"ping"}]}' | grep -i x-venomThe x-venom-upstream and x-venom-model headers name the target that served; x-venom-fallback lists why earlier ones did not.
Route keys
| Key | Type | Default | What it does |
|---|---|---|---|
alias |
string | required | The name clients use. Unique. |
strategy |
see below | fallback |
How targets are ordered for each request. |
targets |
list | required (not with experiment) |
{upstream, model, weight} entries; weight (default 1) is used by weighted. A target may also carry base_model, input, cached_input, output to price it. |
ttft_timeout |
duration | the upstream's timeout |
Per-attempt deadline for the first byte. It only applies while another target remains, and only tightens the upstream's own timeout. Keep it well under defaults.retry_budget (the load warns otherwise). |
images |
alias | none | Another ordinary route that serves this alias's requests carrying an image, for an alias whose own models are text-only (the shipped glm sends images to glm-vision). |
rate_limit |
{rpm, tpm} |
none | A token bucket for all keys together on this alias: requests and tokens per minute (0 = no limit on that axis). |
base_model |
model id | none | The model every target is priced as unless it declares its own price. |
learned |
{min_samples, margin} |
20, 0.10 |
Tunes strategy: learned (an error on other strategies). |
balance |
{price, latency} |
half and half | Weights for strategy: balanced (an error on other strategies; both 0 is an error). |
explore |
0–0.5 | 0.05 |
Share of a fastest or balanced route's requests sent first to the target measured longest ago, so a recovered target is seen again. |
experiment |
{variants} |
none | Makes the alias an A/B test (below). Excludes targets, strategy, ttft_timeout and images. |
shadow |
map | none | Mirrors a sample of the alias's chat completions to a candidate model (below). |
Strategies
| Strategy | Order |
|---|---|
fallback |
As configured. The first servable target serves. |
round_robin |
Rotates the starting target per request. |
least_latency |
By each upstream's moving-average latency. |
weighted |
Random, in proportion to each target's weight. |
cheapest |
By effective price (a 3:1 input:output blend from pricing:, base_model and the price table), computed when the table is built: local first, unpriced last, ties in configured order. |
fastest |
By live time to first token (p50 and p90 per upstream/model, from streamed traffic; unmeasured targets first), with explore traffic to the longest-unmeasured target. |
balanced |
By a weighted sum of price and live time to first token, each scaled across the route's targets (balance). |
learned |
By reported session outcomes against cost over the last 30 days (POST /admin/v1/sessions/{id}/outcome); the configured order while a target has fewer than min_samples sessions. A target qualifies within margin of the best success lower bound, and the cheapest qualified one goes first. |
GET /admin/v1/routes shows every alias, its targets in order, each target's live time-to-first-token estimate and whether it can serve now; GET /admin/v1/pricing shows the prices cheapest ranks by.
Fallback and retries
- Before the first byte only. A connection error,
429,5xxor a missed TTFT deadline moves to the next target with full-jitter backoff. Once a byte reaches the client, a failure ends the stream with an error event and the client decides. - Known-down targets cost nothing. A target whose upstream failed its last health check, whose circuit breaker is open, whose upstream lacks the model, or which sits out a quota cooldown is skipped without an attempt while another target can serve. The last remaining target is still tried, so a lone local model off the grid answers.
- Slow is not down. A missed TTFT deadline does not count against the breaker while the health check passes. A target that served this session in the last 5 minutes gets four times the deadline (never past
timeout). - Retries don't multiply. Every target gets one attempt; same-upstream retries come after, within
defaults.max_attempts,defaults.retry_budgetand the per-upstream retry budget (retry_ratio,retry_min_per_second). A request from an OpenAI SDK that is itself retrying (x-stainless-retry-count> 0) gets no same-upstream retry, and aRetry-Afterlonger thandefaults.backoff_maxis returned to the client rather than slept on (unless rate shaping can queue it). - Context windows are checked, never truncated. A target whose window can't hold the request (prompt bytes/4 plus max output) is passed over for a larger one. If none can, the answer is
400 context_length_exceededbefore any attempt.
x-venom-fallback reason codes (stable): 429, 429_quota (quota or balance exhausted; the target cools down), quota_cooldown, circuit_open, unhealthy, busy (at max_inflight), unsupported (the target can't serve this request), context_window, missing (the upstream lacks the model), timeout, 5xx, 404, auth, conn_error, context_length, rejected_param (the upstream refused a field or value; the 400 is held for the client if no later target serves), rate_shaped. When a target was passed over for its context window and both are priced, x-venom-context-cost-usd gives the input-cost difference.
Session stickiness
Send x-venom-session: <id> (the venom CLI does on every request) and the orchestrator keeps the session on one target by rendezvous hashing over the healthy targets, so engine prefix caches and provider prompt caches are reused. Subagents send x-venom-parent-session too, so spend rolls up under the parent. Without the header, the session is a hash of the body's prompt_cache_key, else safety_identifier, else user. On llama.cpp builds without a host prompt cache a session gets a slot of its own; on OpenAI a prompt_cache_key is chosen from a pool sized to the model's traffic (a key the client set is kept). Side requests (x-venom-session: <session>:<kind>) get the body defaults in side_requests: (Profiles and gears).
A/B experiments
- alias: venom-ab
experiment:
variants:
- {name: default, target: venom-default, weight: 50}
- {name: smart, target: venom-smart, weight: 50}| Variant key | Default | What it does |
|---|---|---|
target |
required | Another ordinary route's alias (routed with its own strategy and deadlines) or upstream/model. Not another experiment. |
name |
the target | 1–128 of A-Z a-z 0-9 . _ : / @ + -; sent in x-venom-variant. |
weight |
1 | Share of new sessions; 0 stops new sessions (existing ones keep their variant). |
Assignment is sticky per session (subagents inherit their parent's), by weighted rendezvous hashing, so every orchestrator agrees and adding a variant only moves the share it takes. Every answer carries x-venom-variant. Assignment is not a security boundary: a client can pick a variant by trying session ids, so only put variants in one experiment that every key using the alias may reach. Results: GET /admin/v1/experiments and GET /admin/v1/experiments/{alias}?since=<RFC 3339> (Admin API).
Shadow traffic
- alias: venom-default
targets: [...]
shadow:
candidate: qwen/qwen3.8-max # an alias, upstream/model or bare id
sample: 0.05 # share of the alias's chat completions, (0, 1]
daily_usd: 2 # required: no replay starts once the UTC day's shadow cost reaches itAfter the client has its answer, a sampled chat completion is replayed on the candidate and the two answers are compared; the client only ever sees the primary. max_concurrency (default 1, at most 8) replays run at once, queue (default 16, at most 256) samples may wait, and a primary answer longer than max_capture_bytes (default 1 MiB) drops the sample. Replays carry no client key, are logged with source shadow, and are summarised by GET /admin/v1/shadow?days=7&route=<alias>.
Errors
| Status | Code | Cause |
|---|---|---|
| 400 | image_input_unavailable |
the request carries an image and no target (nor the images route) takes images |
| 400 | context_length_exceeded |
no target's window holds the request |
| 404 | model_not_found |
unknown alias, upstream or model |
| 429 | rate_limit_exceeded |
the route's rate_limit (or the key's) |
| 503 | model_unavailable |
every remaining target lacks its model; each is named with its fix |
| 503 | no_upstream |
no target is enabled; names the missing credential |
Related
- Upstreams and kinds, Profiles and gears, Configuration reference for the retry and breaker settings.