---
title: Upstreams and kinds
description: Add an inference server or cloud provider to the orchestrator, pick its kind, and set its keys, models, capabilities and limits.
section: orchestrator
order: 10
audience: [operator, builder]
sources: [services/go-orchestrator/internal/config/config.go, services/go-orchestrator/internal/quirks/quirks.go, services/go-orchestrator/AGENTS.md, config/orchestrator.yaml]
features: [orchestrator.config-upstreams, orchestrator.upstream-openai, orchestrator.upstream-ollama, orchestrator.upstream-vllm, orchestrator.upstream-llamacpp, orchestrator.upstream-sglang, orchestrator.upstream-generic, orchestrator.upstream-zai, orchestrator.upstream-dashscope, orchestrator.upstream-mistral, orchestrator.upstream-minimax, orchestrator.upstream-typesafe, orchestrator.upstream-anthropic, orchestrator.upstream-sdapi, orchestrator.upstream-sagemaker, orchestrator.upstream-bedrock]
---

An upstream is one server the orchestrator sends requests to: a local engine, a provider's API or a gateway. Each entry under `upstreams:` in `orchestrator.yaml` has a `name` (what routes and `upstream/model` ids refer to), a `kind` (which wire format and parameter rules apply) and a `base_url`. Routes then list `{upstream, model}` targets; see [Routes and strategies](/docs/orchestrator/routing.md).

## Minimal example {#example}

```yaml
upstreams:
  - name: ollama
    kind: ollama
    base_url: ${OLLAMA_BASE_URL:-http://localhost:11434/v1}
    context_window: ${OLLAMA_CONTEXT_LENGTH:-32768}
  - name: openai
    kind: openai
    base_url: https://api.openai.com/v1
    api_key_env: OPENAI_API_KEY      # unset = the upstream is disabled, nothing calls OpenAI
    max_inflight: 32

routes:
  - alias: venom-default
    targets:
      - {upstream: ollama, model: "qwen3:8b"}
      - {upstream: openai, model: gpt-5.4-mini}
```

An upstream whose `api_key_env` names an empty variable, or whose `base_url` is empty, is kept but disabled: it is never health-checked and its targets are skipped. That is why one file works online and off the grid. `GET /admin/v1/upstreams` shows each upstream's state and why it is disabled.

## Kinds {#kinds}

The kind decides the request rewriting ("quirks"), the capability defaults, how the context window is discovered and how the key is sent. Unset, `kind` is `generic`.

| Kind | For | Notes |
|---|---|---|
| `openai` | OpenAI's API | `/v1/responses` passed through natively; `reasoning_effort` mapped to the nearest level each model takes; `service_tier` flex and priority; a shared `prompt_cache_key` pool per model; `native_batch` and `cache_retention` (`24h`, `30m`) apply. |
| `ollama` | Ollama's OpenAI-compatible API | `max_completion_tokens` sent as `max_tokens`; `n` and `logit_bias` dropped; `tool_choice` off by default; reasoning read from `reasoning`; context window from `/api/ps` (loaded model) or `/api/show`; a model without the `tools` capability is routed around. Local. |
| `vllm` | vLLM | `stream_options` only on streams, an empty `tools` dropped; reasoning in `reasoning`; window from `/v1/models` `max_model_len`; load scraped from `/metrics`. Local. |
| `sglang` | SGLang | `reasoning_content`; window from `/v1/models` `max_model_len`; `/metrics` scraped. Local. |
| `llamacpp` | llama.cpp's `llama-server` | `cache_prompt: true`; a slot per `x-venom-session` (`id_slot`) on builds without the host prompt cache; window from `/v1/models` `meta.n_ctx` or `/props`. `slots` applies. Local. |
| `generic` | any other OpenAI-compatible server (LiteLLM, LocalAI, Kimi's platform, Azure OpenAI's v1 API, Together, Speaches, Kokoro-FastAPI) | No rewriting beyond the common rules; `cache_retention: 1h` is Kimi's. |
| `zai` | Z.ai's GLM API (`https://api.z.ai/api/paas/v4`, or `…/coding/paas/v4` for the GLM Coding Plan) | `temperature` clamped to [0, 1], `top_p` to [0.01, 1], `stop` as an array, `metadata` dropped, GLM-5.3 efforts mapped to the ones it takes; `tool_choice` off. Context windows come from config. |
| `dashscope` | Alibaba Cloud Model Studio's OpenAI-compatible API (Qwen, DeepSeek, Kimi and GLM hosted there) | Thinking controls (`enable_thinking`, `thinking_budget`) set from `reasoning_effort` per model family; a forced tool choice turns thinking off; `explicit_cache` and `native_batch` apply; rerank defaults to Model Studio's native API. |
| `mistral` | Mistral AI's La Plateforme (`https://api.mistral.ai/v1`) | Only the documented fields are sent (the API rejects others with `422`); `seed` → `random_seed`; tool call ids shortened to 9 characters; `/v1/completions` with a `suffix` goes to Codestral's `/fim/completions`; embeddings `dimensions` → `output_dimension`. Window from `/v1/models` `max_context_length`. |
| `minimax` | MiniMax's OpenAI-compatible API (`https://api.minimax.io/v1`, or `https://api.minimaxi.com/v1` in mainland China) | `temperature` and `top_p` kept in (0, 1]; `reasoning_split` on; effort and thinking knobs per model; `tool_choice` off. No `/models`: list the models. Chat only. |
| `anthropic` | Anthropic's Messages API (`https://api.anthropic.com/v1`) | Chat completions and `/v1/responses` translated to `POST /messages` and back; `x-api-key` auth; prompt-cache breakpoints placed for you; `cache_retention: 1h` and `native_batch` apply; embeddings and completions are `unsupported_endpoint`. |
| `bedrock` | Amazon Bedrock's Converse API | Chat translated to `/model/{id}/converse(-stream)`; SigV4-signed with IAM keys (`aws:`); embeddings and rerank through InvokeModel; priced by the model's base name (`us.anthropic.claude-sonnet-5-5-v1:0` as `claude-sonnet-5-5`). `models` is required. |
| `sagemaker` | an AWS SageMaker real-time endpoint | Models are endpoint names (`endpoint:component` sets the inference component); `payload: openai` or `tgi`; SigV4. `models` is required, no health probe. |
| `typesafe` | TypeSafe's System One decision API | Decision models: they serve `POST /v1/systemone` only. See [Rerank, images and decisions](/docs/orchestrator/rerank-images-decisions.md#decisions). |
| `sdapi` | AUTOMATIC1111's web UI API (also Forge, SD.Next, stable-diffusion.cpp's server); `base_url` is the server root without `/v1` | Image models: they serve `POST /v1/images/generations` only. |

"Local" kinds (`ollama`, `vllm`, `sglang`, `llamacpp`), and any upstream whose base URL is loopback, private, link-local, `*.local`, `*.internal` or a single-label host, cost $0 and are never counted against budgets. Set `local: false` for a paid gateway on a private address, `local: true` for your own server on a public name.

Speech providers with their own wire formats register as kinds of their own (ADR 0044): cloud ones (`elevenlabs`, `fishaudio`, `byteplus-speech`) and local servers (`cosyvoice`, `qwen3-tts`, `indextts`, `sparktts`, `fishspeech`, which cost $0 wherever they run). Speech also goes to any kind above that speaks OpenAI's audio API. See the [Voice section](/docs/voice.md) and [Local speech models](/docs/voice/local-speech.md).

## Upstream keys {#keys}

| Key | Type | Default | What it does |
|---|---|---|---|
| `name` | string | required | Unique, no `/`. |
| `kind` | string | `generic` | One of the kinds above. |
| `base_url` | URL | none (disabled) | The API root including `/v1` (except `sdapi`). AWS kinds default to the regional endpoint. |
| `api_key_env` | env var name | none (no auth) | The variable holding the key; unset or empty disables the upstream. |
| `api_key_envs` | list of env var names | none | A credential chain after `api_key_env`: a request refused for credit (`402`, `insufficient_quota`), a revoked key (`401`, `403`) or a rate limit (`429`) is retried on the next key before the next route target. |
| `models` | list | discovered | The models it serves. Required for `bedrock`, `sagemaker`, `minimax`; when set it is also an allowlist for `upstream/<id>` and bare ids. |
| `timeout` | duration | `defaults.timeout` (120s) | Per-attempt time to first byte. A started response runs to its end. |
| `max_inflight` | int | `defaults.max_inflight` (0 = unlimited) | Concurrent requests to this upstream; `/v1/models` reports it so clients queue on their side. |
| `headers` | map | none | Extra request headers (values may use `${VAR}`). |
| `health_interval` | duration | `defaults.health_interval` (15s) | How often it is probed. |
| `stream_usage` | bool | by kind | Whether `stream_options.include_usage` is injected so streams report usage. |
| `context_window` | int (tokens) | discovered, else unknown | Applies to every model unless `model_info` overrides it. A request too long for every target is a `400` before any attempt. |
| `capabilities` | map | by kind | See below; applies to every model. |
| `model_info` | map of model id → settings | none | Per-model `context_window`, `capabilities`, and pricing (`base_model`, `input`, `cached_input`, `output`, or a nested `price` with `input`, `cached_input`, `output`, `search_unit`). |
| `slots` | int | `/props` `total_slots` | `llamacpp` only: the slot count sessions are pinned onto. |
| `metrics_url` | URL | origin + `/metrics` for `vllm`, `sglang`, `llamacpp` | Where the engine's load (queue, KV-cache use) is scraped. |
| `local` | bool | by kind and address | Whether its tokens cost $0. |
| `native_batch` | bool | `false` | `openai`, `dashscope`, `anthropic` only: hand batches and batch requests to the provider's Batch API at its batch price. See [Batches](/docs/orchestrator/batches-and-files.md). |
| `aws` | map | from the environment | `bedrock` and `sagemaker`: `region` (default `AWS_REGION`, then `AWS_DEFAULT_REGION`), `access_key_id_env` (default `AWS_ACCESS_KEY_ID`), `secret_access_key_env` (default `AWS_SECRET_ACCESS_KEY`), `session_token_env` (default `AWS_SESSION_TOKEN`). Disabled while the region or a key is missing. |
| `payload` | `openai` or `tgi` | `openai` | `sagemaker` only: the container's body format. |
| `prompt_template` | `plain`, `chatml`, `llama3` | `plain` | `sagemaker` with `payload: tgi`: how messages become one prompt. |
| `rerank` | `jina`, `cohere`, `voyage`, `tei`, `dashscope` | `dashscope` for kind dashscope, InvokeModel for bedrock, else `jina` | The dialect `POST /v1/rerank` is sent in. |
| `reference_audio_dir` | path | none | Local speech kinds: the directory a speech request's `venom_options.reference_audio` file path is read from for voice cloning; unset, file paths are refused. See [Local speech models](/docs/voice/local-speech.md). |

### Capabilities {#capabilities}

`capabilities:` on an upstream, or under `model_info.<model>.capabilities`, overrides the kind's defaults. Discovery (after each passing health check) only fills what the config leaves unset.

| Key | Type | Default | What it does |
|---|---|---|---|
| `tools` | bool | true (false for `typesafe`) | The model takes tools; a request with tools skips targets that don't. |
| `tool_choice` | bool | true; false for `ollama`, `zai`, `minimax` | Whether `tool_choice` is sent or removed. |
| `responses` | bool | true for `openai` | Native `/v1/responses`; otherwise requests are translated to chat completions. |
| `reasoning_field` | `reasoning` or `reasoning_content` | `reasoning_content` (`reasoning` on ollama, vllm; none on openai) | Where the server puts reasoning text; answers are normalised to `reasoning_content`. |
| `replay_reasoning` | bool | true (false for openai, anthropic, bedrock) | Whether earlier turns' reasoning is sent back. |
| `vision` | bool | discovered, else a known-model list | Image input; `/v1/models` reports it. |
| `decisions` | bool | true for `typesafe` | A decision model: serves `/v1/systemone` and nothing else. |
| `tts` | bool | a known-id list (`tts-1`, `gpt-4o-mini-tts`, Kokoro, ...) | A text-to-speech model for `/v1/audio/speech`. |
| `stt` | bool | a known-id list (whisper, `*-transcribe`, voxtral, ...) | A speech-to-text model; reported by `/v1/models`. |
| `constrained_decoding` | bool | true for local kinds | The server decodes against a JSON schema; false removes `json_schema` constraints instead of sending them. |
| `cache_retention` | `24h`, `30m` (openai); `1h` (generic, anthropic) | none | The longer prompt-cache lifetime asked for when a session pauses (`x-venom-cache-ttl: long`, or gaps over 5 minutes). |
| `explicit_cache` | bool | off | `dashscope` only: Model Studio's explicit cache markers on prompts of 1,024+ tokens. |

## Examples {#more-examples}

A paid gateway on a private address, priced and budgeted:

```yaml
- name: litellm
  kind: generic
  base_url: http://10.0.0.5:4000/v1
  api_key_env: LITELLM_KEY
  local: false
```

Two keys for one provider, the second used when the first runs out of credit:

```yaml
- name: openai
  kind: openai
  base_url: https://api.openai.com/v1
  api_key_env: OPENAI_API_KEY
  api_key_envs: [OPENAI_API_KEY_BACKUP]
```

Bedrock and a SageMaker endpoint priced per token:

```yaml
- name: bedrock
  kind: bedrock
  aws: {region: us-east-1}
  models: ["us.anthropic.claude-sonnet-5-5-v1:0"]
- name: sagemaker
  kind: sagemaker
  aws: {region: us-east-1}
  payload: openai
  models: [my-llama-endpoint]
  model_info:
    my-llama-endpoint: {context_window: 131072, price: {input: 0.30, output: 0.60}}
```

Mistral and MiniMax (both commented out in the shipped config; set `MISTRAL_API_KEY` or `MINIMAX_API_KEY` and uncomment):

```yaml
- name: mistral
  kind: mistral
  base_url: ${MISTRAL_BASE_URL:-https://api.mistral.ai/v1}
  api_key_env: MISTRAL_API_KEY
  models: [mistral-large-latest, mistral-small-latest, codestral-latest, mistral-embed]
- name: minimax
  kind: minimax
  base_url: ${MINIMAX_BASE_URL:-https://api.minimax.io/v1}
  api_key_env: MINIMAX_API_KEY
  models: [MiniMax-M3, MiniMax-M3.1-Flash]
```

## Troubleshooting {#troubleshooting}

- **The upstream shows `disabled`**: its `api_key_env` variable is empty in the orchestrator's environment, or `base_url` is empty. Set it and `POST /admin/v1/reload` (or send `SIGHUP`).
- **`503 model_unavailable`**: the upstream's own model list (Ollama, vLLM) lacks the model. The error names the fix, such as `ollama pull qwen3:8b`.
- **A target is skipped as `unhealthy`**: its last health check failed. Kinds without a model list (`minimax`, AWS) are judged by their circuit breaker instead.
- **`502 upstream_auth_failed`**: the provider refused the orchestrator's key (not the client's). The provider's body is not relayed, since it can echo part of the key.
- **Config errors at start or reload**: unknown fields and kinds, a duplicate name, a `/` in a name, or a kind-specific key on the wrong kind (`slots` off `llamacpp`, `native_batch` off the three batch kinds, `payload` off `sagemaker`) stop the load with the field named. A failed reload keeps the running config.

## Related

- [Routes and strategies](/docs/orchestrator/routing.md) and [Configuration reference](/docs/orchestrator/configuration.md).
- [Environment variables](/docs/orchestrator/environment.md) for the keys and base URLs the shipped config reads.
