Local speech models
Run CosyVoice, Qwen3-TTS, IndexTTS, Spark-TTS, Fish Speech and other open speech models on your own hardware and use them through the orchestrator's OpenAI-shaped audio API, voice cloning included.
Open speech models run on your own GPU keep your text and voice on your machine. Some of their servers already
speak OpenAI's audio API (Kokoro-FastAPI, Speaches, LocalAI, Ollama) and work as kind generic or ollama.
The others have upstream kinds of their own, and the orchestrator translates each request to the server's API.
Clients keep sending POST /v1/audio/speech and GET /v1/audio/voices. Aliases, fallback, breakers and the
request log work as they do for cloud models, and every local kind costs $0, wherever it runs.
Configure the upstreams
These servers don't list models, so name the one you serve. None needs a key. If you put one behind a proxy
that wants one, set api_key_env and it goes out as a bearer token.
upstreams:
- name: cosyvoice
kind: cosyvoice
base_url: http://gpu-box:50000
models: [Fun-CosyVoice3-0.5B]
reference_audio_dir: /srv/voices
- name: qwen3-tts
kind: qwen3-tts
base_url: http://gpu-box:8091/v1
models: [Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice]
- name: indextts
kind: indextts
base_url: http://gpu-box:8092/v1
models: [IndexTeam/IndexTTS-2]
reference_audio_dir: /srv/voices
- name: sparktts
kind: sparktts
base_url: http://gpu-box:8000
models: [spark_tts]
reference_audio_dir: /srv/voices
- name: fishspeech
kind: fishspeech
base_url: http://gpu-box:8080
models: [openaudio-s1-mini]
routes:
- alias: venom-speak
targets:
- {upstream: qwen3-tts, model: Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice}
- {upstream: cosyvoice, model: Fun-CosyVoice3-0.5B}| Kind | Server | Start it with | Formats | Voices |
|---|---|---|---|---|
cosyvoice |
CosyVoice's FastAPI server (CosyVoice 1, 2, Fun-CosyVoice3) | python runtime/python/fastapi/server.py --port 50000 --model_dir <model> |
wav, pcm | none listed: the server has no speaker list |
qwen3-tts |
vLLM-Omni serving Qwen3-TTS (the Qwen3-TTS repo has no HTTP server) | vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091 |
wav, pcm, flac, mp3, opus | the model's speakers and your uploaded voices |
indextts |
vLLM-Omni serving IndexTTS2 or 2.5 (the route the index-tts README recommends) | vllm serve IndexTeam/IndexTTS-2 --omni (see vLLM's IndexTTS recipe) |
wav, pcm, flac, mp3, opus | the voices uploaded to the server |
sparktts |
Spark-TTS on Triton Inference Server (runtime/triton_trtllm; the repo has no REST server) |
the repo's run.sh, Triton HTTP on port 8000 |
wav, pcm | none: every voice is cloned |
fishspeech |
the self-hosted Fish Speech / OpenAudio API server | python tools/api_server.py --listen 0.0.0.0:8080 |
wav, pcm, mp3, opus | the references saved on the server |
A format the server can't make is a 400 with the code unsupported_by_provider, and the server is never called.
A pcm or wav answer names its sample rate in x-venom-sample-rate: 24000 for CosyVoice 2 and 3 and for
Qwen3-TTS, 22050 for CosyVoice 1 (a model id containing 300M) and IndexTTS, 16000 for Spark-TTS, 44100 for Fish
Speech (48000 for its opus).
How the OpenAI fields map
CosyVoice picks its mode from what the request carries:
| The request has | Endpoint | Sent as |
|---|---|---|
voice |
/inference_sft |
spk_id: a speaker of an SFT model |
voice and instructions |
/inference_instruct (CosyVoice 1) |
spk_id, instruct_text |
a reference clip and reference_text |
/inference_zero_shot |
prompt_wav, prompt_text |
| a reference clip alone | /inference_cross_lingual |
prompt_wav |
a reference clip and instructions |
/inference_instruct2 (CosyVoice 2 and 3) |
prompt_wav, instruct_text |
For a model id containing cosyvoice3, the orchestrator adds the You are a helpful assistant.<|endofprompt|>
prefix CosyVoice 3 expects, and it appends <|endofprompt|> to any instruction that lacks one. The server has
no speed control, so speed is ignored. Its PCM is streamed one sentence at a time.
Qwen3-TTS takes OpenAI's body as it is (voice, instructions as the model's instruct, speed,
response_format) plus:
venom_options.language→language(enbecomesEnglish,zhbecomesChinese, and so on for its ten languages);venom_options.sample_rate→sample_rate;stream: true→ the server's raw audio stream (pcm or wav only), andstream_format: "sse"→ its OpenAI speech events;- a reference clip →
task_type: "Base"withref_audio, plusref_textfromreference_text(without one,x_vector_only_mode); - a model id containing
VoiceDesign→task_type: "VoiceDesign", andinstructionsdescribing the voice is then required.
IndexTTS uses the same vLLM-Omni API. It clones every voice, so it needs a reference clip, or a voice that
names a voice uploaded to the server. instructions becomes its emotion text (extra_params.emo_text with
use_emo_text), and venom_options.language becomes extra_params.lang for IndexTTS 2.5's languages (zh, en, ja,
es, ar). The other emotion controls go in venom_options.provider:
{"model": "indextts/IndexTeam/IndexTTS-2", "input": "I can't believe it.", "response_format": "wav",
"venom_options": {"reference_audio": "me.wav",
"provider": {"extra_params": {"emo_vector": [0, 0, 0.8, 0, 0, 0, 0, 0], "emo_alpha": 0.7}}}}emo_vector holds eight weights from 0 to 1.2, in this order: happy, angry, sad, afraid, disgusted, melancholic,
surprised, calm.
Spark-TTS on Triton only clones. The reference clip must be a 16 kHz WAV, 16-bit or 32-bit float, at most 60
seconds. Only the first channel is used, and the server does no resampling. reference_text is the clip's
transcript. Spark-TTS's gender, pitch and speed controls exist in its Python API, but the shipped Triton model
doesn't take them. gender, pitch and speed in venom_options.provider are sent as extra inputs, for a model
repository you extend with them. speed and instructions in the OpenAI body are not sent. Triton answers in one
JSON object, so the audio arrives all at once.
Fish Speech: voice is a reference saved on the server (reference_id), and a reference clip goes inline in
references with reference_text as its text. The server streams only wav, so stream: true works for wav and
pcm (the orchestrator takes the wav header off). It has no speed or language field. This is the self-hosted
server's API. Fish Audio's cloud is kind fishaudio, a different API.
For every kind, venom_options.provider is merged into the request last, so any other server option is
reachable. For CosyVoice each key becomes a form field. For the JSON servers (Qwen3-TTS, IndexTTS, Spark-TTS's
Triton body, Fish Speech) it is merged one level deep, for example {"extra_params": {...}, "seed": 7}.
Voice cloning
Give the clip in venom_options.reference_audio, and its transcript in venom_options.reference_text when the
model wants one:
{"model": "venom-speak", "input": "Hello from my own voice.", "response_format": "wav",
"venom_options": {"reference_audio": "me.wav", "reference_text": "This is how I sound."}}reference_audio can be:
- a file path, read from the upstream's
reference_audio_dir. Without that setting file paths are refused, and a path that leaves the directory (.., an absolute path elsewhere, a symlink pointing out) is a400; - a
data:URL,data:audio/wav;base64,…, which is bound by the 1 MiB request body; - an
http(s)URL, handed to a server that fetches clips itself (Qwen3-TTS and IndexTTS through vLLM-Omni). The orchestrator never fetches it, and a server that needs the clip's bytes (CosyVoice, Spark-TTS, Fish Speech) refuses a URL.
A clip must be WAV, MP3, FLAC or Ogg audio, checked by its first bytes, and at most 10 MiB. A few seconds to half a minute of clean speech is what these models want.
Ollama and llama.cpp
- Ollama serves OpenAI's
POST /v1/audio/speechitself. Text to speech was announced in v0.40.11 and reached Linux and Windows in v0.40.13. Kindollamapasses it through with no adapter. Mark the modelcapabilities: {tts: true}and ask forwavorpcm(16-bit, 24 kHz): Ollama refuses other formats, anyspeedother than 1, andinstructions. ItsPOST /v1/audio/transcriptionstakes WAV files. - llama.cpp's
llama-serverhasPOST /v1/audio/transcriptions, which passes through like any OpenAI-shaped transcription. It speaks on its ownPOST /ttsroute (float32 pcm) rather than/v1/audio/speech, so it isn't a text-to-speech target yet.
What isn't verified
Each kind's wire format was read from the project's source on 2026-11-03, but none of it was run against a live server. A few details come from docs alone:
- CosyVoice's answer has no declared
Content-Type, so the orchestrator ignores it. The clip formats its server decodes beyond WAV aren't confirmed. - Qwen3-TTS's 24 kHz rate and the
x-audio-sample-rateheader come from vLLM-Omni's documentation. - IndexTTS's 22050 Hz output through vLLM-Omni, and whether vLLM-Omni streams it, come from vLLM's recipe page.
- Whether Spark-TTS's Triton model accepts an empty
reference_text. - Fish Speech's rate with decoders other than the shipped 44.1 kHz codec.