Human speech is often unstructured and messy. To keep up with real conversation, Voice AI bots must parse partial sentences, filler words, and interruptions, all without the visual cues a human listener relies on. On top of that linguistic challenge, real-world latency budgets and network conditions test every Voice AI pipeline, and users don’t forgive one that stutters.
In this episode of WebRTC Live, host Arin Sime talks with Dr. Varun Singh, Chief Product Technology Officer at Daily.co / Pipecat.ai, about the engineering decisions that separate voice agents that succeed from ones that get abandoned. Topics include where Voice AI is working today and where it isn’t, from well-defined tasks like collections to the open-ended conversations everyone is still chasing, and how Pipecat approaches the problem: composing voice agents from STT, LLM, TTS, and everything in between, fitting in speech-to-speech models, and getting from prototype to production. Also discussed: why WebRTC remains the right transport for real-world networks, the production realities of observability, cost, and failure modes, how to weigh build-vs-buy, and where voice-native UIs are headed.
WebRTC Live #116: Building Robust Voice AI Pipelines for Every Use Case with Varun Singh
Episode highlights below.
Episode Highlights
Turn detection, not the STT/LLM/TTS chain, is the hard problem in a voice pipeline
The speech-to-text, LLM, and text-to-speech steps are comparatively straightforward. Knowing when the user has finished talking is the hard problem, because there’s no equivalent of pressing enter. A fixed pause threshold, Pipecat defaults to roughly 1000 milliseconds, breaks down across children, elderly speakers, and non-native speakers, who all pause at different rates. Pipecat’s answer is a dedicated turn-detection model, Smart Turn, that reads intonation and speech patterns rather than relying on silence alone. Varun said the model is on its 3.2 version, developed over roughly 20 months starting around October 2024. He also drew a line between turn detection and the separate question of whether a bot should interrupt before the user finishes speaking at all. A coffee order was his example: if someone says “I want a double shot espresso,” the bot doesn’t need to wait for the rest of the sentence to ask about cream and sugar. Whether that’s appropriate depends heavily on the use case.
Cascade’s advantage over speech-to-speech comes from observability, not latency
“Cascade” is the term for a voice pipeline that handles speech-to-text, the LLM, and text-to-speech as three separate steps in sequence, each one producing a text output the next step consumes. Speech-to-speech turn latency runs uneven once lookups are involved: some turns fast, some slow. Cascade’s real advantage is the text checkpoint it produces at every step. Once you have a text output, you can check whether the utterance should go to the LLM at all, and after the LLM responds, a second model can review that output against the last several turns of conversation and flag a mismatch before it reaches the caller. Speech-to-speech offers far less of that intermediate visibility, you can infer what it heard and generated, but you don’t get the same checkpoints in between. Varun’s restaurant example: a bot handling “do you have availability” can answer instantly from preloaded data without hallucinating, because that data was loaded ahead of the call. Booking a table for a specific party size is a step up in complexity, and that’s where the intervention points matter more. For healthcare and banking, that checkpoint carries real weight.
Hybrid is the recommendation, not a single architecture
Cascade should own the parts of a call where guardrails and evals matter most. Speech-to-speech should own the well-understood, low-risk parts, greetings, or a straightforward coffee order. Varun expects that split to shift further toward speech-to-speech as teams build confidence in their evals, and he framed the payoff in concrete terms: better accessibility scores and higher end-of-call ratings, because the right model handled the moment that needed empathy. Host Arin Sime added a related point from his own company’s work: a WebRTC.ventures client built a training tool for mental health therapists that uses push-to-talk instead of open conversation, which sidesteps the need for turn detection entirely. Not every voice AI product needs to solve for interruption and barge-in.
Pipecat’s core bet is neutrality across vendors
No lock-in to a single speech-to-text, text-to-speech, or LLM provider, and no requirement to use Daily’s own transport. Pipecat also supports websocket ingress from several telephony providers Varun named in the conversation. Some of those names came through the transcript in forms Claude couldn’t confidently match to real companies (see verification list below), so confirm before repeating them anywhere public. This neutrality traces back to Pipecat’s origin: Daily was already doing WebRTC-based meeting transcription before LLMs were part of the product, and the team started assembling ASR, LLM, and TTS orchestration around the time GPT-3.5 made small-prompt outputs reliable enough to build on, late 2022 into early 2023 by Varun’s account. He described text-to-speech quality specifically as weak throughout 2023, which is part of why a wave of TTS-focused startups emerged in 2024. The bet from early on was that no single model or vendor would win across every domain, healthcare-specific jargon behaves differently in an ASR model than financial jargon does, so the value Pipecat adds is in the integration layer, not in one vendor relationship.
Fast/slow model pairing in voice AI mirrors a pattern already common in coding agents
A fast, cheap model handles quick acknowledgments and simple lookups. A slower, more capable model handles harder reasoning. The system decides which to use per turn, the same pattern increasingly used in coding agents and in microservice design generally, not something specific to voice. Varun’s example: a restaurant booking bot can have a fast model confirm “yes, we have availability today” immediately, while a slower model works out party size, seating preferences, or allergy accommodations in parallel. He named specific model tiers as analogues, Claude’s smaller and larger models, Gemini Flash versus its bigger counterparts, used the same way developers already split “cheap and fast” from “slow and thorough” when using coding agents.
Outbound calls are the easier Voice AI build
The bot already knows who it’s calling and why, so most outcomes collapse to a yes/no branch, and teams can skip building expensive fallback logic for the rarer “no” case. Varun’s example: confirming a restaurant reservation. If the answer is yes, the call is essentially done. If it’s no, the system can hand off to a second, “warm” bot that already has the booking context loaded and a different prompt ready to go, rather than trying to solve the rebooking conversation live. Inbound calls carry more uncertainty going in, so more design effort goes into figuring out what the caller actually needs before the bot commits to an action.
WebRTC’s advantage today is network resilience, not raw speed
The advantage shows up on bad networks: hotel wifi, an old DSL line, a switch to cellular tethering. That resilience is built into the browser implementation and standardized jitter buffering, which teams get for free rather than having to build themselves. Varun contrasted this with a newer transport built on WebTransport and MOQ that Pipecat has been merging in, attributed to contributor Luke Curley. He described it as close to landing in Pipecat at the time of the interview. His honest assessment: in the worst-case network conditions, this newer approach performs about the same as WebRTC; in the best case, it may be slightly better, but it doesn’t yet have WebRTC’s browser support or years of production track record. Arin raised this as a build-versus-buy question, similar to the tradeoffs teams already navigate with WebRTC itself, and Varun’s answer didn’t push back on that framing.
The bottleneck in voice UI has been the request-response loop, not the underlying speech tech
Automatic speech recognition has worked well for about 15 years, in Varun’s telling, comparable to how long the computer mouse existed before windowed, multi-app operating systems caught up to it. Early assistants like Alexa underperformed because they were built as request-response loops rather than open-ended conversation; without an LLM behind them, they could only match keywords to a fixed set of “skills.” LLMs make the open-ended version viable. Varun’s team built an open-source game, Gradient Bang, specifically to test turn-taking and barge-in without being constrained by a real business use case, the player can interrupt the model mid-response, issue a new command, and have it queue alongside the original task. Varun’s separate example for where speech-to-speech already works well today is language learning: he described an ESP32 device on his desk that checks his calendar and starts a Finnish conversation with him when he’s free, using no text-to-speech cascade at all, since a good speech-to-speech model already carries the language fluency needed for that use case. His framing for the future: voice as an input layer across apps and operating systems, not a single self-contained assistant.
Pre-Interview Segment: self-hosted voice AI telephony
Before the main interview, WebRTC.ventures’ Alberto Gonzalez and Hector Zelaya were live from the SignalWire-hosted conference, ClueCon. Alberto discussed his conference talk on a self-hosted voice AI telephony architecture using FreeSWITCH with self-hosted speech models, aimed at avoiding dependence on expensive GPU infrastructure. Hector’s talk covered using eBPF to let a turn server written in Python keep up with relay-logic performance demands by running that logic at the kernel level, bypassing user space. Hector also made a broader observation: the AI systems that hold up in production tend to depend less on which model is used and more on the deterministic engineering around it, guardrails and similar scaffolding.
Post-Interview Segment: OpenAI’s GPT-Live Voice AI Architecture and the WARP Protocol
In this month’s WebRTC Industry Chat with Arin Sime and Tsahi Levent-Levi, Tsahi discusses his interview with Justin Uberti about how OpenAI built their newest generation responsive voice AI. Two main takeaways emerged:
- GPT-Live uses a dual-path architecture: a fast, low-latency voice model handles immediate responses and filler phrases, while a separate frontier model, GPT-5.5 in OpenAI’s example, handles deeper reasoning and passes answers back. This setup lets OpenAI swap the frontier model independently without retraining the voice layer, and it supports full-duplex streaming conversation instead of turn-taking. Long sessions become possible through background context compaction across multiple running model instances.
- The conversation covers OpenAI’s WARP (WebRTC Abridged Roundtrip Protocol), which reduces WebRTC call setup from six roundtrips to one by carrying context between protocol layers instead of renegotiating each layer from scratch. Tsahi calls this a useful optimization but not architecturally significant on its own. He frames the bigger insight as Uberti’s ability, drawn from his WebRTC and Google background, to reshape the protocol layer itself rather than work strictly within existing constraints.
Read Justin Uberti’s full post: How we built a realtime system for responsive voice AI in six months
Watch just the post-interview segment the video on YouTube
Varun’s References:
Up Next! WebRTC Live #117
Live from RTC.ON
WebRTC.ventures CTO Alberto Gonzalez will be live from the RTC.ON conference in Poland, interviewing several speakers.
Friday, September 18, 8:00 am Eastern
