Skip to content
Docs menu

Local speech models

Run CosyVoice, Qwen3-TTS, IndexTTS, Spark-TTS, Fish Speech and other open speech models on your own hardware and use them through the orchestrator's OpenAI-shaped audio API, voice cloning included.

View .md

Open speech models run on your own GPU keep your text and voice on your machine. Some of their servers already speak OpenAI's audio API (Kokoro-FastAPI, Speaches, LocalAI, Ollama) and work as kind generic or ollama. The others have upstream kinds of their own, and the orchestrator translates each request to the server's API. Clients keep sending POST /v1/audio/speech and GET /v1/audio/voices. Aliases, fallback, breakers and the request log work as they do for cloud models, and every local kind costs $0, wherever it runs.

Configure the upstreams

These servers don't list models, so name the one you serve. None needs a key. If you put one behind a proxy that wants one, set api_key_env and it goes out as a bearer token.

upstreams:
  - name: cosyvoice
    kind: cosyvoice
    base_url: http://gpu-box:50000
    models: [Fun-CosyVoice3-0.5B]
    reference_audio_dir: /srv/voices
  - name: qwen3-tts
    kind: qwen3-tts
    base_url: http://gpu-box:8091/v1
    models: [Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice]
  - name: indextts
    kind: indextts
    base_url: http://gpu-box:8092/v1
    models: [IndexTeam/IndexTTS-2]
    reference_audio_dir: /srv/voices
  - name: sparktts
    kind: sparktts
    base_url: http://gpu-box:8000
    models: [spark_tts]
    reference_audio_dir: /srv/voices
  - name: fishspeech
    kind: fishspeech
    base_url: http://gpu-box:8080
    models: [openaudio-s1-mini]
routes:
  - alias: venom-speak
    targets:
      - {upstream: qwen3-tts, model: Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice}
      - {upstream: cosyvoice, model: Fun-CosyVoice3-0.5B}
Kind Server Start it with Formats Voices
cosyvoice CosyVoice's FastAPI server (CosyVoice 1, 2, Fun-CosyVoice3) python runtime/python/fastapi/server.py --port 50000 --model_dir <model> wav, pcm none listed: the server has no speaker list
qwen3-tts vLLM-Omni serving Qwen3-TTS (the Qwen3-TTS repo has no HTTP server) vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091 wav, pcm, flac, mp3, opus the model's speakers and your uploaded voices
indextts vLLM-Omni serving IndexTTS2 or 2.5 (the route the index-tts README recommends) vllm serve IndexTeam/IndexTTS-2 --omni (see vLLM's IndexTTS recipe) wav, pcm, flac, mp3, opus the voices uploaded to the server
sparktts Spark-TTS on Triton Inference Server (runtime/triton_trtllm; the repo has no REST server) the repo's run.sh, Triton HTTP on port 8000 wav, pcm none: every voice is cloned
fishspeech the self-hosted Fish Speech / OpenAudio API server python tools/api_server.py --listen 0.0.0.0:8080 wav, pcm, mp3, opus the references saved on the server

A format the server can't make is a 400 with the code unsupported_by_provider, and the server is never called. A pcm or wav answer names its sample rate in x-venom-sample-rate: 24000 for CosyVoice 2 and 3 and for Qwen3-TTS, 22050 for CosyVoice 1 (a model id containing 300M) and IndexTTS, 16000 for Spark-TTS, 44100 for Fish Speech (48000 for its opus).

How the OpenAI fields map

CosyVoice picks its mode from what the request carries:

The request has Endpoint Sent as
voice /inference_sft spk_id: a speaker of an SFT model
voice and instructions /inference_instruct (CosyVoice 1) spk_id, instruct_text
a reference clip and reference_text /inference_zero_shot prompt_wav, prompt_text
a reference clip alone /inference_cross_lingual prompt_wav
a reference clip and instructions /inference_instruct2 (CosyVoice 2 and 3) prompt_wav, instruct_text

For a model id containing cosyvoice3, the orchestrator adds the You are a helpful assistant.<|endofprompt|> prefix CosyVoice 3 expects, and it appends <|endofprompt|> to any instruction that lacks one. The server has no speed control, so speed is ignored. Its PCM is streamed one sentence at a time.

Qwen3-TTS takes OpenAI's body as it is (voice, instructions as the model's instruct, speed, response_format) plus:

  • venom_options.language → language (en becomes English, zh becomes Chinese, and so on for its ten languages);
  • venom_options.sample_rate → sample_rate;
  • stream: true → the server's raw audio stream (pcm or wav only), and stream_format: "sse" → its OpenAI speech events;
  • a reference clip → task_type: "Base" with ref_audio, plus ref_text from reference_text (without one, x_vector_only_mode);
  • a model id containing VoiceDesign → task_type: "VoiceDesign", and instructions describing the voice is then required.

IndexTTS uses the same vLLM-Omni API. It clones every voice, so it needs a reference clip, or a voice that names a voice uploaded to the server. instructions becomes its emotion text (extra_params.emo_text with use_emo_text), and venom_options.language becomes extra_params.lang for IndexTTS 2.5's languages (zh, en, ja, es, ar). The other emotion controls go in venom_options.provider:

{"model": "indextts/IndexTeam/IndexTTS-2", "input": "I can't believe it.", "response_format": "wav",
 "venom_options": {"reference_audio": "me.wav",
                   "provider": {"extra_params": {"emo_vector": [0, 0, 0.8, 0, 0, 0, 0, 0], "emo_alpha": 0.7}}}}

emo_vector holds eight weights from 0 to 1.2, in this order: happy, angry, sad, afraid, disgusted, melancholic, surprised, calm.

Spark-TTS on Triton only clones. The reference clip must be a 16 kHz WAV, 16-bit or 32-bit float, at most 60 seconds. Only the first channel is used, and the server does no resampling. reference_text is the clip's transcript. Spark-TTS's gender, pitch and speed controls exist in its Python API, but the shipped Triton model doesn't take them. gender, pitch and speed in venom_options.provider are sent as extra inputs, for a model repository you extend with them. speed and instructions in the OpenAI body are not sent. Triton answers in one JSON object, so the audio arrives all at once.

Fish Speech: voice is a reference saved on the server (reference_id), and a reference clip goes inline in references with reference_text as its text. The server streams only wav, so stream: true works for wav and pcm (the orchestrator takes the wav header off). It has no speed or language field. This is the self-hosted server's API. Fish Audio's cloud is kind fishaudio, a different API.

For every kind, venom_options.provider is merged into the request last, so any other server option is reachable. For CosyVoice each key becomes a form field. For the JSON servers (Qwen3-TTS, IndexTTS, Spark-TTS's Triton body, Fish Speech) it is merged one level deep, for example {"extra_params": {...}, "seed": 7}.

Voice cloning

Give the clip in venom_options.reference_audio, and its transcript in venom_options.reference_text when the model wants one:

{"model": "venom-speak", "input": "Hello from my own voice.", "response_format": "wav",
 "venom_options": {"reference_audio": "me.wav", "reference_text": "This is how I sound."}}

reference_audio can be:

  • a file path, read from the upstream's reference_audio_dir. Without that setting file paths are refused, and a path that leaves the directory (.., an absolute path elsewhere, a symlink pointing out) is a 400;
  • a data: URL, data:audio/wav;base64,…, which is bound by the 1 MiB request body;
  • an http(s) URL, handed to a server that fetches clips itself (Qwen3-TTS and IndexTTS through vLLM-Omni). The orchestrator never fetches it, and a server that needs the clip's bytes (CosyVoice, Spark-TTS, Fish Speech) refuses a URL.

A clip must be WAV, MP3, FLAC or Ogg audio, checked by its first bytes, and at most 10 MiB. A few seconds to half a minute of clean speech is what these models want.

Ollama and llama.cpp

  • Ollama serves OpenAI's POST /v1/audio/speech itself. Text to speech was announced in v0.40.11 and reached Linux and Windows in v0.40.13. Kind ollama passes it through with no adapter. Mark the model capabilities: {tts: true} and ask for wav or pcm (16-bit, 24 kHz): Ollama refuses other formats, any speed other than 1, and instructions. Its POST /v1/audio/transcriptions takes WAV files.
  • llama.cpp's llama-server has POST /v1/audio/transcriptions, which passes through like any OpenAI-shaped transcription. It speaks on its own POST /tts route (float32 pcm) rather than /v1/audio/speech, so it isn't a text-to-speech target yet.

What isn't verified

Each kind's wire format was read from the project's source on 2026-11-03, but none of it was run against a live server. A few details come from docs alone:

  • CosyVoice's answer has no declared Content-Type, so the orchestrator ignores it. The clip formats its server decodes beyond WAV aren't confirmed.
  • Qwen3-TTS's 24 kHz rate and the x-audio-sample-rate header come from vLLM-Omni's documentation.
  • IndexTTS's 22050 Hz output through vLLM-Omni, and whether vLLM-Omni streams it, come from vLLM's recipe page.
  • Whether Spark-TTS's Triton model accepts an empty reference_text.
  • Fish Speech's rate with decoders other than the shipped 44.1 kHz codec.