Voice agents: the core library
The @venom/voice package, the call pipeline under venom's AI phone agents, and how to build and test it.
@venom/voice (packages/voice/) is the library under venom's realtime voice agents. A call runs through a cascaded,
full-duplex pipeline: voice activity detection and turn detection, streaming speech-to-text through the orchestrator, a
harness agent with the agent's business tools, then streaming text-to-speech back to the caller. It has no UI and
reports everything as events. The design, with every interface, is docs/voice-agents.md (ADR 0043).
This is the design's slice 2, the core. The calls API in venom serve, the telephony transports, compliance packs and
the console's Calls page build on it in later slices.
What a call does
startCall(options) runs one call on a transport (Asterisk, Twilio or the browser test line, all behind the
CallTransport interface):
- Caller audio goes to the VAD and, from speech start, to the orchestrator's streaming transcription endpoint
(
GET /v1/realtime?intent=transcription, OpenAI's realtime transcription events) throughopenStt. - When the caller stops, the turn detector decides whether the turn is over: a short silence when the transcript ends like a sentence, up to 1.2 s otherwise. The model starts on the partial transcript at once (preemptive generation) and its reply is used if the final transcript matches.
- The reply is cut into speakable segments (a clause first, never inside a number, time or abbreviation), synthesized
through
POST /v1/audio/speech(createTtsClient, raw PCM) and paced to the transport in real time, at most 60 ms ahead (100 ms on Twilio). - If the caller talks over the agent, playback stops at once. Real speech cancels the answer, and the model's reply is cut to the words the caller actually heard, so the next request shows what was said. A backchannel such as "uh-huh" resumes the answer from the word that was cut.
- Tools that change something (a booking, a payment plan, a transfer) finish when the caller cuts in, and their result stays in the conversation; lookups are stopped. A preemptive run never books before the turn is confirmed.
createCallBrain gives each call a harness session with only the agent's business tools and the call tools
(transfer_to_human, end_call, record_cease_request), the permission gate, and a system prompt that stays the same
for every call of one agent, so the prompt prefix is cached.
Every step is reported as a CallEvent (call.state, caller.final, agent.spoken, interruption, tool,
turn.metrics with the per-turn latency, ...). Call.events() replays them from the start.
Testing calls with CallSim
@venom/voice/testing runs whole calls on a virtual clock: a scripted caller (utterances with their transcripts, at
given times), fake STT, TTS, VAD and transport, and a scripted model. A test asserts the latency budget (under 800 ms
from the end of speech to the first audio), the barge-in cut time, the heard text and the transcript, with no network
and no waiting:
import { CallSim } from "@venom/voice/testing";
const sim = new CallSim({ agent: { greeting: "Hi." }, reply: () => ({ text: "We open at nine." }) }).start();
sim.say("When do you open?", 1000);
await sim.runUntil(5000);
sim.of("turn.metrics")[0].metrics.eosToFirstAudioMs; // within budget for the fake latenciesBuild and test
| Command | What it does |
|---|---|
make install-voice |
installs the package's dependencies |
make build-voice |
builds packages/voice/dist/ |
make test-voice |
typecheck, lint and the unit tests; part of make test and make test-airplane |
The tests never touch the network.