Configuration reference
Every top-level section of orchestrator.yaml with its keys and defaults, plus how the file is loaded, reloaded, versioned and rolled back.
The orchestrator reads one YAML file, $VENOM_CONFIG (default config/orchestrator.yaml). The shipped file is commented line by line and is the best starting point; this page lists every key.
Loading and reloading
${VAR}and${VAR:-default}are expanded from the environment before parsing. An unset or empty variable with no default becomes""; a bare$is left alone.- Parsing is strict: an unknown key, kind or strategy, a duplicate name, or a route target on an unknown upstream stops the load with the field named. Availability is lenient: an upstream whose key variable is empty is kept but disabled.
- Legal but likely-wrong settings (a
ttft_timeoutnot shorter thandefaults.retry_budget, a profile target outside its ownallowlist) are logged as warnings at every load. - Reload without a restart:
POST /admin/v1/reloadorSIGHUP. Both also re-read the price table. A reload that fails keeps the running config.
curl -s -X POST http://127.0.0.1:8787/admin/v1/reload -H "Authorization: Bearer $VENOM_ADMIN_TOKEN"Top-level sections
| Key | Page |
|---|---|
listen |
below |
upstreams |
Upstreams and kinds |
routes |
Routes and strategies |
profiles, side_requests |
Profiles, side requests and gears |
gears, gears_locked, gears_source |
Profiles, side requests and gears |
defaults, response_cache, dedup, rate_shaping, batch_routing, pricing, models, guardrails |
below |
search |
Web search and MCP |
listen
listen: ${VENOM_LISTEN:-127.0.0.1:8787} in the shipped file; 127.0.0.1:8787 when absent. Loopback by default: with no client key the gateway serves anyone who can reach it, so it refuses to start on another address without a key unless VENOM_ALLOW_OPEN=1. The Docker image sets VENOM_LISTEN=0.0.0.0:8787 inside the container and compose publishes the port on the host's loopback only.
defaults
| Key | Default | What it does |
|---|---|---|
defaults.timeout |
120s |
Per-attempt time to first byte. Once a byte arrives the request runs until it ends or the client leaves. |
defaults.max_inflight |
0 (unlimited) |
Concurrent requests per upstream, unless the upstream sets its own. |
defaults.health_interval |
15s |
Health-check period per upstream. |
defaults.max_attempts |
3 |
Total attempts per request. Every target still gets one try when there are more targets; extra attempts retry the same targets with backoff. |
defaults.retry_budget |
20s |
No new attempt starts after this much time. |
defaults.backoff_base, defaults.backoff_max |
100ms, 2s |
Full-jitter exponential backoff. backoff_max is also the longest upstream Retry-After waited on; a longer one goes to the client. |
defaults.retry_ratio, defaults.retry_min_per_second |
0.2, 1 |
Per-upstream retry budget: retries are allowed while retries ≤ ratio × first attempts + min_per_second × 10 s over the last 10 s. Both 0 = no same-upstream retries. |
defaults.breaker_threshold |
3 |
Consecutive failures that open an upstream's circuit breaker. |
defaults.breaker_cooldowns |
[1m, 5m, 30m] |
Growing cooldown per trip (the last repeats) before a half-open probe. |
defaults.breaker_cooldown |
none | One fixed cooldown instead of the ladder. |
defaults.max_body_bytes |
33554432 (32 MiB) |
Request body cap (413 request_too_large). Also caps audio uploads. |
defaults.write_stall_timeout |
60s |
A client that takes no bytes for this long during one write is dropped and its upstream request cancelled. |
defaults.stream_idle_timeout |
5m |
An upstream that sends nothing for this long mid-response is cut off; the stream ends with its error event. Raise it for models that think for minutes without a byte. |
defaults.request_log_retention |
100000 |
Request-log rows kept in SQLite (the spend ledger is never pruned). |
defaults.load_interval |
2s |
How often vllm, sglang and llamacpp engines' /metrics are read for queue and KV-cache use. |
defaults.saturation_kv, defaults.saturation_waiting |
0.95, 4 |
When an engine counts as saturated (KV cache that full, or that many requests queued); a saturated engine goes behind the others. |
response_cache
Exact-match reuse of deterministic side requests (a side kind at temperature 0, or any kind listed in kinds) from the same client key; never a main-session turn. A hit is billed $0 and answered with x-venom-response-cache: hit, or shared when an identical request was in flight.
| Key | Default | What it does |
|---|---|---|
response_cache.ttl |
5m |
How long a response is reused. |
response_cache.max_bytes |
33554432 (32 MiB) |
Memory bound, least recently used evicted; negative turns the cache off. |
response_cache.max_entry_bytes |
1048576 (1 MiB) |
Largest response stored. |
response_cache.kinds |
[] (shipped: [auto-classifier, decide]) |
Side kinds cached at any temperature. |
dedup
Identical concurrent requests from one key (temperature 0 set explicitly, /v1/embeddings, /v1/rerank, or any request with the same Idempotency-Key) share one upstream call, fanned out as it streams. A request opts out with x-venom-dedup: off. Duplicates are billed once and answered with x-venom-dedup: joined; a completed Idempotency-Key response is replayed with x-venom-dedup: replayed and Idempotent-Replayed: true. The same key with another body is 422 idempotency_key_reused; a key over 255 bytes is 400 invalid_idempotency_key.
| Key | Default | What it does |
|---|---|---|
dedup.max_bytes |
1048576 (1 MiB) |
A duplicate may join while the response is shorter; a longer Idempotency-Key response is not kept. |
dedup.max_lag_bytes |
262144 (256 KiB) |
A reader that falls this far behind is cut loose, so a slow client never holds the others. |
dedup.idempotency_ttl |
5m |
How long a completed Idempotency-Key response is replayed. |
dedup.idempotency_max_bytes |
33554432 (32 MiB) |
Bound on kept responses; -1 keeps none (collapsed only while in flight). |
rate_shaping
Each upstream learns its provider's rate limits from x-ratelimit-* and anthropic-ratelimit-* headers and from 429s. A request that would exceed a learned limit goes to a later target with room, else waits in that upstream's queue in fair order across client keys. A request that waited carries x-venom-queued-ms. Only a full queue or a wait past the bound answers 429 upstream_queue_full or upstream_wait_exceeded, with Retry-After. Upstreams that never send rate-limit headers (local engines) are not shaped.
| Key | Default | What it does |
|---|---|---|
rate_shaping.max_queue |
256 |
Requests waiting per upstream; negative = none wait. |
rate_shaping.max_wait |
30s |
Longest wait; negative = none wait (limits then only steer requests). |
rate_shaping.weights |
each key 1 | Fair-queuing weight by client key name (at most 1000). |
batch_routing
Shapes batch requests (x-venom-urgency: batch); see Batches, files and service tiers.
| Key | Default | What it does |
|---|---|---|
batch_routing.window |
30s (at most 24h) |
How long requests for one model collect before their provider batch is submitted. |
batch_routing.max_lines |
1000 (at most 50,000) |
Submit at once at this many requests. |
batch_routing.max_wait |
10m (at most 1h) |
Cap on x-venom-batch-wait and GET /v1/batch_requests/{id}?wait=. |
batch_routing.callback_hosts |
[] (no callbacks) |
Hosts (host or host:port) x-venom-callback-url may name. The orchestrator calls no client-named URL unless you list its host. |
pricing
Costs are list prices from the offline price table (prices.yaml next to the config, or $VENOM_PRICES; never fetched at runtime): USD per 1M input, cached_input and output tokens, with optional cache_write, flex and priority rates. Local upstreams cost $0. A cloud model nothing prices costs $0 and is logged once; GET /admin/v1/pricing lists every model's effective price and source.
pricing:
source: [custom, builtin]
models:
azure/my-gpt: {base_model: gpt-5.4} # priced as gpt-5.4
my-tuned: {input: 3, cached_input: 0.3, output: 12}
tts-hosted/narrator: {per_1m_chars: 15} # text to speech, per 1M characters| Key | Default | What it does |
|---|---|---|
pricing.source |
[custom, builtin] |
Lookup order over your prices (custom) and the table (builtin); a source left out is not consulted. |
pricing.models |
none | Rules keyed by upstream/model (that upstream only) or a bare model id (any upstream). |
pricing.models.*.base_model |
none | The model whose price applies. A served model with a base model is answered with x-venom-base-model. |
pricing.models.*.input, cached_input, output |
none | Your price in USD per 1M tokens; input and output together, cached_input defaults to input. Wins over base_model. |
pricing.models.*.search_unit |
none | USD per search unit, for a reranker billed per search. |
pricing.models.*.per_1m_chars |
none | USD per 1M input characters, for text-to-speech. |
pricing.models.*.per_second |
none | USD per second of audio, for speech-to-text. |
The same pricing keys are accepted in an upstream's model_info and on a route target; one model priced two different ways is a config error.
models
Per-model settings a mesh daemon advertises; today only provenance for the public mesh.
| Key | What it does |
|---|---|
models[].id |
A model id the daemon serves (a route alias or upstream/model). Unique. |
models[].upstream |
The vllm or sglang upstream that runs it; public jobs go to it alone. Required with provenance. |
models[].provenance.registry_id |
The public mesh registry entry it runs. |
models[].provenance.sampler |
Must be venom-sampler/1. |
guardrails
guardrails: holds the layered guardrails checked on every /v1 inference request: the org's rules and catalog picks, teams: keyed by a client key's team label and members: keyed by a key's name. A block is 403 guardrail_blocked with a code such as GR-ORG-CRED-101; a redaction, warning or keep-local match passes with an x-venom-guardrail header per rule. GET /v1/guardrails shows a key what applies to it, and GET /g/{code} (no key) is a code's page. Absent, nothing is checked and nothing costs. The rule format, codes and layers are in the Security section.
Config history and rollback
Every config the process applies (start, reload, SIGHUP, a local-model selection, a rollback) is stored in SQLite as the file's bytes before ${VAR} expansion, with its SHA-256; the newest 50 are kept.
curl -s http://127.0.0.1:8787/admin/v1/config/versions -H "Authorization: Bearer $VENOM_ADMIN_TOKEN"
curl -s "http://127.0.0.1:8787/admin/v1/config/diff?from=12&to=file" -H "Authorization: Bearer $VENOM_ADMIN_TOKEN"
curl -s -X POST http://127.0.0.1:8787/admin/v1/config/versions/12/rollback \
-H "Authorization: Bearer $VENOM_ADMIN_TOKEN" -H "Content-Type: application/json" -d '{}'GET /admin/v1/config/versionslists versions newest first, marks thecurrentone and says whether the file on disk differs (file.changed).GET /admin/v1/config/diff?from=<id|file>&to=<id|file>diffs two versions (default: the newest and the one before) by upstream, route, profile and other key. Secrets are redacted:${VAR}stays as written, literal header values and secret-named fields become a short hash.POST /admin/v1/config/versions/{id}/rollbackwrites that version to the file and reloads it.409 config_changed_on_diskwhen the file isn't what was last applied (sendoverwrite_sha256equal to the reporteddisk_sha256to replace it anyway),422 invalid_versionwhen it no longer loads,500 config_not_writablefor a read-only mount,422 reload_failedrestores the previous file.