---
title: Local speech models
description: Run CosyVoice, Qwen3-TTS, IndexTTS, Spark-TTS, Fish Speech and other open speech models on your own hardware and use them through the orchestrator's OpenAI-shaped audio API, voice cloning included.
section: voice
order: 25
audience: [user, operator, builder]
sources: [services/go-orchestrator/internal/voice/local.go, services/go-orchestrator/internal/voice/cosyvoice.go, services/go-orchestrator/internal/voice/qwen3tts.go, services/go-orchestrator/internal/voice/omni.go, services/go-orchestrator/internal/voice/indextts.go, services/go-orchestrator/internal/voice/sparktts.go, services/go-orchestrator/internal/voice/fishspeech.go, docs/configuration.md, config/orchestrator.yaml]
features: []
---

Open speech models run on your own GPU keep your text and voice on your machine. Some of their servers already
speak OpenAI's audio API (Kokoro-FastAPI, Speaches, LocalAI, Ollama) and work as kind `generic` or `ollama`.
The others have upstream kinds of their own, and the orchestrator translates each request to the server's API.
Clients keep sending `POST /v1/audio/speech` and `GET /v1/audio/voices`. Aliases, fallback, breakers and the
request log work as they do for cloud models, and every local kind costs $0, wherever it runs.

## Configure the upstreams

These servers don't list models, so name the one you serve. None needs a key. If you put one behind a proxy
that wants one, set `api_key_env` and it goes out as a bearer token.

```yaml
upstreams:
  - name: cosyvoice
    kind: cosyvoice
    base_url: http://gpu-box:50000
    models: [Fun-CosyVoice3-0.5B]
    reference_audio_dir: /srv/voices
  - name: qwen3-tts
    kind: qwen3-tts
    base_url: http://gpu-box:8091/v1
    models: [Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice]
  - name: indextts
    kind: indextts
    base_url: http://gpu-box:8092/v1
    models: [IndexTeam/IndexTTS-2]
    reference_audio_dir: /srv/voices
  - name: sparktts
    kind: sparktts
    base_url: http://gpu-box:8000
    models: [spark_tts]
    reference_audio_dir: /srv/voices
  - name: fishspeech
    kind: fishspeech
    base_url: http://gpu-box:8080
    models: [openaudio-s1-mini]
routes:
  - alias: venom-speak
    targets:
      - {upstream: qwen3-tts, model: Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice}
      - {upstream: cosyvoice, model: Fun-CosyVoice3-0.5B}
```

| Kind | Server | Start it with | Formats | Voices |
|---|---|---|---|---|
| `cosyvoice` | CosyVoice's FastAPI server (CosyVoice 1, 2, Fun-CosyVoice3) | `python runtime/python/fastapi/server.py --port 50000 --model_dir <model>` | wav, pcm | none listed: the server has no speaker list |
| `qwen3-tts` | vLLM-Omni serving Qwen3-TTS (the Qwen3-TTS repo has no HTTP server) | `vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091` | wav, pcm, flac, mp3, opus | the model's speakers and your uploaded voices |
| `indextts` | vLLM-Omni serving IndexTTS2 or 2.5 (the route the index-tts README recommends) | `vllm serve IndexTeam/IndexTTS-2 --omni` (see vLLM's IndexTTS recipe) | wav, pcm, flac, mp3, opus | the voices uploaded to the server |
| `sparktts` | Spark-TTS on Triton Inference Server (`runtime/triton_trtllm`; the repo has no REST server) | the repo's `run.sh`, Triton HTTP on port 8000 | wav, pcm | none: every voice is cloned |
| `fishspeech` | the self-hosted Fish Speech / OpenAudio API server | `python tools/api_server.py --listen 0.0.0.0:8080` | wav, pcm, mp3, opus | the references saved on the server |

A format the server can't make is a `400` with the code `unsupported_by_provider`, and the server is never called.
A pcm or wav answer names its sample rate in `x-venom-sample-rate`: 24000 for CosyVoice 2 and 3 and for
Qwen3-TTS, 22050 for CosyVoice 1 (a model id containing `300M`) and IndexTTS, 16000 for Spark-TTS, 44100 for Fish
Speech (48000 for its opus).

## How the OpenAI fields map

**CosyVoice** picks its mode from what the request carries:

| The request has | Endpoint | Sent as |
|---|---|---|
| `voice` | `/inference_sft` | `spk_id`: a speaker of an SFT model |
| `voice` and `instructions` | `/inference_instruct` (CosyVoice 1) | `spk_id`, `instruct_text` |
| a reference clip and `reference_text` | `/inference_zero_shot` | `prompt_wav`, `prompt_text` |
| a reference clip alone | `/inference_cross_lingual` | `prompt_wav` |
| a reference clip and `instructions` | `/inference_instruct2` (CosyVoice 2 and 3) | `prompt_wav`, `instruct_text` |

For a model id containing `cosyvoice3`, the orchestrator adds the `You are a helpful assistant.<|endofprompt|>`
prefix CosyVoice 3 expects, and it appends `<|endofprompt|>` to any instruction that lacks one. The server has
no speed control, so `speed` is ignored. Its PCM is streamed one sentence at a time.

**Qwen3-TTS** takes OpenAI's body as it is (`voice`, `instructions` as the model's instruct, `speed`,
`response_format`) plus:

- `venom_options.language` → `language` (`en` becomes `English`, `zh` becomes `Chinese`, and so on for its ten
  languages);
- `venom_options.sample_rate` → `sample_rate`;
- `stream: true` → the server's raw audio stream (pcm or wav only), and `stream_format: "sse"` → its OpenAI
  speech events;
- a reference clip → `task_type: "Base"` with `ref_audio`, plus `ref_text` from `reference_text` (without one,
  `x_vector_only_mode`);
- a model id containing `VoiceDesign` → `task_type: "VoiceDesign"`, and `instructions` describing the voice is then
  required.

**IndexTTS** uses the same vLLM-Omni API. It clones every voice, so it needs a reference clip, or a `voice` that
names a voice uploaded to the server. `instructions` becomes its emotion text (`extra_params.emo_text` with
`use_emo_text`), and `venom_options.language` becomes `extra_params.lang` for IndexTTS 2.5's languages (zh, en, ja,
es, ar). The other emotion controls go in `venom_options.provider`:

```json
{"model": "indextts/IndexTeam/IndexTTS-2", "input": "I can't believe it.", "response_format": "wav",
 "venom_options": {"reference_audio": "me.wav",
                   "provider": {"extra_params": {"emo_vector": [0, 0, 0.8, 0, 0, 0, 0, 0], "emo_alpha": 0.7}}}}
```

`emo_vector` holds eight weights from 0 to 1.2, in this order: happy, angry, sad, afraid, disgusted, melancholic,
surprised, calm.

**Spark-TTS** on Triton only clones. The reference clip must be a 16 kHz WAV, 16-bit or 32-bit float, at most 60
seconds. Only the first channel is used, and the server does no resampling. `reference_text` is the clip's
transcript. Spark-TTS's gender, pitch and speed controls exist in its Python API, but the shipped Triton model
doesn't take them. `gender`, `pitch` and `speed` in `venom_options.provider` are sent as extra inputs, for a model
repository you extend with them. `speed` and `instructions` in the OpenAI body are not sent. Triton answers in one
JSON object, so the audio arrives all at once.

**Fish Speech**: `voice` is a reference saved on the server (`reference_id`), and a reference clip goes inline in
`references` with `reference_text` as its text. The server streams only wav, so `stream: true` works for wav and
pcm (the orchestrator takes the wav header off). It has no speed or language field. This is the self-hosted
server's API. Fish Audio's cloud is kind `fishaudio`, a different API.

For every kind, `venom_options.provider` is merged into the request last, so any other server option is
reachable. For CosyVoice each key becomes a form field. For the JSON servers (Qwen3-TTS, IndexTTS, Spark-TTS's
Triton body, Fish Speech) it is merged one level deep, for example `{"extra_params": {...}, "seed": 7}`.

## Voice cloning

Give the clip in `venom_options.reference_audio`, and its transcript in `venom_options.reference_text` when the
model wants one:

```json
{"model": "venom-speak", "input": "Hello from my own voice.", "response_format": "wav",
 "venom_options": {"reference_audio": "me.wav", "reference_text": "This is how I sound."}}
```

`reference_audio` can be:

- **a file path**, read from the upstream's `reference_audio_dir`. Without that setting file paths are refused,
  and a path that leaves the directory (`..`, an absolute path elsewhere, a symlink pointing out) is a `400`;
- **a `data:` URL**, `data:audio/wav;base64,…`, which is bound by the 1 MiB request body;
- **an `http(s)` URL**, handed to a server that fetches clips itself (Qwen3-TTS and IndexTTS through vLLM-Omni).
  The orchestrator never fetches it, and a server that needs the clip's bytes (CosyVoice, Spark-TTS, Fish Speech)
  refuses a URL.

A clip must be WAV, MP3, FLAC or Ogg audio, checked by its first bytes, and at most 10 MiB. A few seconds to
half a minute of clean speech is what these models want.

## Ollama and llama.cpp

- **Ollama** serves OpenAI's `POST /v1/audio/speech` itself. Text to speech was announced in v0.40.11 and reached
  Linux and Windows in v0.40.13. Kind `ollama` passes it through with no adapter. Mark the model
  `capabilities: {tts: true}` and ask for `wav` or `pcm` (16-bit, 24 kHz): Ollama refuses other formats, any
  `speed` other than 1, and `instructions`. Its `POST /v1/audio/transcriptions` takes WAV files.
- **llama.cpp**'s `llama-server` has `POST /v1/audio/transcriptions`, which passes through like any OpenAI-shaped
  transcription. It speaks on its own `POST /tts` route (float32 pcm) rather than `/v1/audio/speech`, so it isn't
  a text-to-speech target yet.

## What isn't verified

Each kind's wire format was read from the project's source on 2026-11-03, but none of it was run against a live
server. A few details come from docs alone:

- CosyVoice's answer has no declared `Content-Type`, so the orchestrator ignores it. The clip formats its server
  decodes beyond WAV aren't confirmed.
- Qwen3-TTS's 24 kHz rate and the `x-audio-sample-rate` header come from vLLM-Omni's documentation.
- IndexTTS's 22050 Hz output through vLLM-Omni, and whether vLLM-Omni streams it, come from vLLM's recipe page.
- Whether Spark-TTS's Triton model accepts an empty `reference_text`.
- Fish Speech's rate with decoders other than the shipped 44.1 kHz codec.
