Skip to content
Docs menu

Routes and strategies

Define model aliases, choose how their targets are ordered, and understand fallback, retries, stickiness, experiments and shadow traffic.

View .md

A route gives a model alias (the model a client sends) an ordered list of targets, each a model on a configured upstream. The alias is what clients depend on; the targets are yours to change.

Minimal example

routes:
  - alias: venom-fast
    strategy: fallback          # the default
    ttft_timeout: 12s           # give up on a target that hasn't sent a byte by then
    targets:
      - {upstream: ollama, model: "qwen3:4b"}
      - {upstream: openai, model: gpt-5.4-nano}
      - {upstream: qwen,   model: qwen3.8-flash}   # skipped while QWEN_API_KEY is unset
curl -s http://127.0.0.1:8787/v1/chat/completions -H "Authorization: Bearer $VENOM_API_KEY" \
  -H "Content-Type: application/json" -D - -o /dev/null \
  -d '{"model":"venom-fast","messages":[{"role":"user","content":"ping"}]}' | grep -i x-venom

The x-venom-upstream and x-venom-model headers name the target that served; x-venom-fallback lists why earlier ones did not.

Route keys

Key Type Default What it does
alias string required The name clients use. Unique.
strategy see below fallback How targets are ordered for each request.
targets list required (not with experiment) {upstream, model, weight} entries; weight (default 1) is used by weighted. A target may also carry base_model, input, cached_input, output to price it.
ttft_timeout duration the upstream's timeout Per-attempt deadline for the first byte. It only applies while another target remains, and only tightens the upstream's own timeout. Keep it well under defaults.retry_budget (the load warns otherwise).
images alias none Another ordinary route that serves this alias's requests carrying an image, for an alias whose own models are text-only (the shipped glm sends images to glm-vision).
rate_limit {rpm, tpm} none A token bucket for all keys together on this alias: requests and tokens per minute (0 = no limit on that axis).
base_model model id none The model every target is priced as unless it declares its own price.
learned {min_samples, margin} 20, 0.10 Tunes strategy: learned (an error on other strategies).
balance {price, latency} half and half Weights for strategy: balanced (an error on other strategies; both 0 is an error).
explore 0–0.5 0.05 Share of a fastest or balanced route's requests sent first to the target measured longest ago, so a recovered target is seen again.
experiment {variants} none Makes the alias an A/B test (below). Excludes targets, strategy, ttft_timeout and images.
shadow map none Mirrors a sample of the alias's chat completions to a candidate model (below).

Strategies

Strategy Order
fallback As configured. The first servable target serves.
round_robin Rotates the starting target per request.
least_latency By each upstream's moving-average latency.
weighted Random, in proportion to each target's weight.
cheapest By effective price (a 3:1 input:output blend from pricing:, base_model and the price table), computed when the table is built: local first, unpriced last, ties in configured order.
fastest By live time to first token (p50 and p90 per upstream/model, from streamed traffic; unmeasured targets first), with explore traffic to the longest-unmeasured target.
balanced By a weighted sum of price and live time to first token, each scaled across the route's targets (balance).
learned By reported session outcomes against cost over the last 30 days (POST /admin/v1/sessions/{id}/outcome); the configured order while a target has fewer than min_samples sessions. A target qualifies within margin of the best success lower bound, and the cheapest qualified one goes first.

GET /admin/v1/routes shows every alias, its targets in order, each target's live time-to-first-token estimate and whether it can serve now; GET /admin/v1/pricing shows the prices cheapest ranks by.

Fallback and retries

  • Before the first byte only. A connection error, 429, 5xx or a missed TTFT deadline moves to the next target with full-jitter backoff. Once a byte reaches the client, a failure ends the stream with an error event and the client decides.
  • Known-down targets cost nothing. A target whose upstream failed its last health check, whose circuit breaker is open, whose upstream lacks the model, or which sits out a quota cooldown is skipped without an attempt while another target can serve. The last remaining target is still tried, so a lone local model off the grid answers.
  • Slow is not down. A missed TTFT deadline does not count against the breaker while the health check passes. A target that served this session in the last 5 minutes gets four times the deadline (never past timeout).
  • Retries don't multiply. Every target gets one attempt; same-upstream retries come after, within defaults.max_attempts, defaults.retry_budget and the per-upstream retry budget (retry_ratio, retry_min_per_second). A request from an OpenAI SDK that is itself retrying (x-stainless-retry-count > 0) gets no same-upstream retry, and a Retry-After longer than defaults.backoff_max is returned to the client rather than slept on (unless rate shaping can queue it).
  • Context windows are checked, never truncated. A target whose window can't hold the request (prompt bytes/4 plus max output) is passed over for a larger one. If none can, the answer is 400 context_length_exceeded before any attempt.

x-venom-fallback reason codes (stable): 429, 429_quota (quota or balance exhausted; the target cools down), quota_cooldown, circuit_open, unhealthy, busy (at max_inflight), unsupported (the target can't serve this request), context_window, missing (the upstream lacks the model), timeout, 5xx, 404, auth, conn_error, context_length, rejected_param (the upstream refused a field or value; the 400 is held for the client if no later target serves), rate_shaped. When a target was passed over for its context window and both are priced, x-venom-context-cost-usd gives the input-cost difference.

Session stickiness

Send x-venom-session: <id> (the venom CLI does on every request) and the orchestrator keeps the session on one target by rendezvous hashing over the healthy targets, so engine prefix caches and provider prompt caches are reused. Subagents send x-venom-parent-session too, so spend rolls up under the parent. Without the header, the session is a hash of the body's prompt_cache_key, else safety_identifier, else user. On llama.cpp builds without a host prompt cache a session gets a slot of its own; on OpenAI a prompt_cache_key is chosen from a pool sized to the model's traffic (a key the client set is kept). Side requests (x-venom-session: <session>:<kind>) get the body defaults in side_requests: (Profiles and gears).

A/B experiments

- alias: venom-ab
  experiment:
    variants:
      - {name: default, target: venom-default, weight: 50}
      - {name: smart,   target: venom-smart,   weight: 50}
Variant key Default What it does
target required Another ordinary route's alias (routed with its own strategy and deadlines) or upstream/model. Not another experiment.
name the target 1–128 of A-Z a-z 0-9 . _ : / @ + -; sent in x-venom-variant.
weight 1 Share of new sessions; 0 stops new sessions (existing ones keep their variant).

Assignment is sticky per session (subagents inherit their parent's), by weighted rendezvous hashing, so every orchestrator agrees and adding a variant only moves the share it takes. Every answer carries x-venom-variant. Assignment is not a security boundary: a client can pick a variant by trying session ids, so only put variants in one experiment that every key using the alias may reach. Results: GET /admin/v1/experiments and GET /admin/v1/experiments/{alias}?since=<RFC 3339> (Admin API).

Shadow traffic

- alias: venom-default
  targets: [...]
  shadow:
    candidate: qwen/qwen3.8-max   # an alias, upstream/model or bare id
    sample: 0.05                  # share of the alias's chat completions, (0, 1]
    daily_usd: 2                  # required: no replay starts once the UTC day's shadow cost reaches it

After the client has its answer, a sampled chat completion is replayed on the candidate and the two answers are compared; the client only ever sees the primary. max_concurrency (default 1, at most 8) replays run at once, queue (default 16, at most 256) samples may wait, and a primary answer longer than max_capture_bytes (default 1 MiB) drops the sample. Replays carry no client key, are logged with source shadow, and are summarised by GET /admin/v1/shadow?days=7&route=<alias>.

Errors

Status Code Cause
400 image_input_unavailable the request carries an image and no target (nor the images route) takes images
400 context_length_exceeded no target's window holds the request
404 model_not_found unknown alias, upstream or model
429 rate_limit_exceeded the route's rate_limit (or the key's)
503 model_unavailable every remaining target lacks its model; each is named with its fix
503 no_upstream no target is enabled; names the missing credential