September 9, 2026
Comments Off on How Voice AI Agents Handle Interruption: State Machines vs Streaming Approaches

The most frustrating thing about speaking with a Voice AI agent is when it interrupts you, or it doesn’t understand the normal flow of human conversation. There are multiple ways to solve this depending on your use case, and coming up with the best architecture for your needs is crucial to creating positive customer experiences.

For over a decade, our team has built voice and video applications for our clients using WebRTC. This has involved highly specialized knowledge and best practices for architecting scalable and secure applications with low latency, high quality video and audio, and fallback scenarios for users on different devices and in different network conditions.

All of this expertise is still foundational to building a highly performant Voice AI application, but now we have to also worry about the conversation itself. Before it was enough to get two humans talking via WebRTC, and just ensure they can hear and see each other well. When a call participant is a Voice AI however, there are additional challenges that must be solved.

Generally speaking, there are two architectural approaches to building a conversational AI: State machines and streaming approaches. For this blog post, I’ve interviewed three Voice AI engineers in our team who have a variety of experience with these approaches and use cases.

First, let’s cover a quick definition of both approaches, and how they balance control vs natural human speech patterns.

The difference between state machine and streaming approaches

A state machine gives you explicit, named states (idle, listening, processing, speaking, handling interruption) with defined transitions between them. The system waits for one stage to finish before moving to the next: silence detection triggers a transition to “processing,” the LLM produces a full response, then it transitions to “speaking” and plays it back. This makes the conversation flow predictable and easy to debug or audit, since you can enumerate every state and every path between them. It’s a natural fit for structured interactions like intake forms, appointment scheduling, or anything with compliance requirements, since you can enforce exactly what happens at each step. The tradeoff is latency and naturalness: because stages tend to be sequential, users often feel a beat of dead air between turns, and handling interruptions (barge-in) requires you to explicitly model it as another state transition rather than it happening more naturally.

Diagram of the Voice AI state machine approach, showing four explicit states (idle, listening, processing, speaking) and the transitions between them, including a barge-in path back to listening.
In a state machine approach, the conversation moves through explicit, named states, which makes the flow easy to audit but adds a beat of latency between turns.

A streaming approach treats the conversation as continuous, overlapping flows of audio and data rather than discrete steps. ASR emits partial transcripts as the user talks, the LLM can start generating responses before the user has fully finished, and TTS streams audio back token by token so playback starts almost immediately. Interruptions become more natural because everything is already flowing and can be cut off at the audio level rather than requiring a formal state change. Newer speech-to-speech models take this further and skip the discrete ASR-LLM-TTS pipeline entirely, processing audio directly. The cost is control: streaming systems are harder to reason about, more prone to race conditions, and rely on heuristics (VAD tuning, semantic endpointing) to guess when someone’s actually done talking, which can misfire. In short, it’s more complex, but if you implement it correctly, the conversation can be more natural.

Diagram of the Voice AI streaming approach, showing ASR, LLM, and TTS stages overlapping in time as audio flows from microphone input to speaker output.
In a streaming approach, ASR, LLM, and TTS overlap instead of running in sequence, which is what makes low-latency, natural interruptions possible.

In practice a lot of production systems (Pipecat, LiveKit Agents, and similar frameworks) do both: a streaming pipeline underneath for low latency, with a lightweight state machine layered on top to track conversation phase, manage tool calls, and enforce business logic. If you’re weighing this for a specific project, the deciding factor is usually how much you need deterministic control over the flow versus how much you’re optimizing for a natural, low-latency back-and-forth.

Introducing our Panelists

Now that you understand the high-level difference in approaches, let’s hear what our panel of WebRTC.ventures Voice AI engineers has to say. Here’s a quick introduction to each person and the project they are working on:

  1. Andrés Rincon has been working with CETA Global on a real-time clinical training simulator, using voice and video AI roleplay to help healthcare professionals practice patient scenarios. Interruption handling was one of the hardest problems his team had to solve. (Read the case study and see more in WebRTC Live #115.)
  2. Suman Paudel has been working on a professional training platform (details under NDA), building Voice AI systems on both LiveKit and Pipecat. He’s spent significant time on interruption handling in production.
  3. Fahad Mahmood has been working on an AI sales agent platform (details under NDA), building LiveKit-based voice pipelines with custom STT, LLM, and TTS components. Much of his work has focused on turn-taking implementations.

I asked each engineer three questions: what actually happens in their applications, how state machines compare to streaming in production, and the hardest interruption bug they’ve had to chase down. Their answers are lightly edited for clarity and brevity. I think you’ll find a wealth of experience in their answers! 

Question 1: Walk me through what actually happens, step by step, when a user starts talking while the agent is mid-sentence. What’s the real sequence of events in a system you’ve built or worked on?

Andrés: In our system turn-taking is not automatic. The user takes the floor by pressing a push-to-talk button. [Note: this was a conscious architectural decision the team took because it fits their unique use case, and while most Voice AI use cases cannot utilize this simplified approach, it’s a good example of how you can use creative architectures that reduce technical complexity and cost for your use case]

We tried the fully automatic streaming approach. Technically it worked. The real problem is that psychologists pause for a long time when they think, and the automatic system treated that silence as the end of the turn. Then the AI client started talking and the session felt wrong.

So this is the sequence:

  1. The AI client is speaking. TTS is still streaming and the avatar is talking.
  2. The user presses the talk button. That event is the interrupt. We are not waiting for Voice Activity Detection (VAD).
  3. We stop the current LLM generation, we stop TTS, and we stop the avatar.
  4. The browser starts sending microphone audio over a WebSocket connection.
  5. Chirp 3 (the Speech To Text model) transcribes while they speak. We kept speech to text even though Gemini can take audio directly, because we needed to see the transcript to debug the session.
  6. When the user releases the button, that is the end of the turn. A pause does not conclude the turn.
  7. The transcript goes to Google ADK. The client agent writes the next reply. The other agents can run at the same time for parallel tasks like coaching advice.
  8. The new agent audio goes to HeyGen and the avatar talks again.

Suman: In the LiveKit-based pipelines I’ve built, a few things happen in quick succession.

First, the incoming audio goes through noise suppression. On self-hosted deployments I’ve used DeepFilterNet (Krisp is cloud-only). This step matters more than you might expect. On phone calls especially, background noise is one of the biggest sources of false interruptions.

Then Silero VAD detects speech activity. But VAD firing doesn’t automatically mean “the user wants to interrupt.” A cough, an “mm-hmm,” or a TV in the background can all trigger it.

So we put a gating layer on top: a minimum speech-duration threshold combined with a semantic turn detector. I’ve used Pipecat’s SmartTurn and the Namo turn detector for this. The goal is to figure out whether someone is actually trying to take the floor or just making a bit of background noise.

Once we’re confident it’s a real interruption, a few things need to happen almost at the same time:

  • Cancel the LLM generation that’s currently in flight.
  • Stop TTS synthesis.
  • Flush all the audio that’s already queued up i.e. the playout queue, transport buffer, and, on telephony, anything sitting in the SIP bridge’s jitter buffer.

That last one is easy to miss. If you only stop TTS but leave audio sitting downstream, the agent can keep talking for another second after the user has started speaking. From the user’s perspective, that feels like the system simply isn’t listening.

There’s another subtle part: conversation context.

The LLM may have generated an entire sentence, but the user might only have heard half of it. If we leave the full generated sentence in the conversation history, the agent can later refer to something the user never actually heard. I usually sync the conversation context with the TTS playback position so the transcript reflects what was really played.

After that, the agent switches back to listening. The nice thing is that STT never really had to stop, so the user’s new utterance is already being processed.

The whole detection-to-silence path ideally needs to happen in roughly 100–300 ms. People are surprisingly sensitive to an agent talking over them, even when the delay is pretty small.

Fahad: As soon as a customer speaks, the raw audio first passes through a noise cancellation service to strip away background hums. Next, Voice Activity Detection (VAD) monitors the cleaned signal to filter out silence and protect expensive downstream resources. 

Finally, the audio is analyzed by an acoustic turn detection model. Rather than just counting seconds of silence, this model analyzes acoustic traits like pitch, intonation, and rhythm to intelligently determine if the user has truly finished their thought or is just pausing to think. Throughout this entire initial phase, the pipeline remains audio-only, with no text conversion taking place yet.

Based on the turn model config, and if you’ve set the agent as interruptible, the turn model will instantly interrupt the agent’s speech (to ensure both don’t speak simultaneously) and mark it as the user’s turn. This model solves the end-of-user-turn problem because using semantic understanding and acoustic cues, it detects whether the user’s turn is finished or they have paused for a moment to finish their thoughts.

Then, the audio is converted to text using the STT model, passed to an LLM, and finally sent to a TTS model before the agent starts speaking. (the latency here can be improved using pre-emptive generation of responses).

Question 2: Have you worked with both a state-machine style approach to turn-taking (explicit states like listening/speaking/interrupted) and a more continuous streaming approach? Or only one or the other? Have you noticed any practical differences in latency, complexity, or reliability?

Andrés: I worked with both. We prototyped the continuous streaming path and we shipped a small state machine with the following states:

  • idle
  • user speaking
  • processing
  • agent speaking

The Push-to-talk function is how you enter the user speaking state.

There are a few practical differences between state-machine and streaming approaches:

  • Latency: streaming can feel faster because there is nothing analogous to a button. For a support bot that is usually better. For training it was more important that the turn was correct.
  • Complexity: streaming looks simple until you add VAD, echo, and cancelling the LLM, TTS, and avatar at the same time. For our use case, Push to talk is less code and more predictable.
  • Reliability: this is the main reason we chose a strict state machine approach. The hard part was not the models. The hard part was deciding who has the floor.

If I build an always on voice agent later, I would still keep a state machine. Streaming moves the audio. Something still has to decide who is talking, what to cancel, and what the model should remember as already said.

Suman: Yes, I have used both approaches in production.

LiveKit Agents lean more toward explicit agent states like listening, thinking, speaking with events around those transitions. Pipecat is more of a continuous frame-based pipeline, where audio, text, and control signals all move through processors. An interruption is essentially a control signal that propagates through the pipeline and flushes things along the way.

The biggest differences I’ve noticed are:

  • Debuggability: State machines win here. When something breaks, you can look at the state-transition history and usually see where turn ownership went wrong. With a streaming pipeline, you tend to get stranger symptoms like ghost audio, duplicate responses, partially interrupted speech and then have to trace individual frames to find the race condition.
  • Latency: Streaming tends to win by a little. Explicit state transitions can introduce boundaries where the system waits for a clean state change before doing something. With frames, cancellation can propagate while everything is still streaming, which can shave some real milliseconds off barge-in.
  • Reliability: This is where state machines have a big advantage. They tend to fail in ways you can predict. Streaming pipelines sometimes fail more creatively. I’ve seen them end up partially flushed or half-interrupted in states that nobody had explicitly designed for.

Where I’ve landed is a hybrid: streaming for the data plane, with a small explicit state machine on top that answers one question: Who currently owns the floor?

Turn ownership is discrete. Audio is continuous. Trying to force one model onto the other is where a lot of the pain starts.

Fahad: I’ve mostly worked on turn taking based implementations, so I can’t share practical differences about latency or performance based on my personal experience, but in general the turn-taking approach gives you more control.

Andrés: The worst one was not a leftover audio buffer, it was that automatic turn detection kept interrupting the therapist trainee.

  • Therapists pause on purpose while thinking.
  • Sometimes they look at notes.
  • Other times they sit with a question before asking the patient.

The automatic stack treated that silence as an indicator that you are done speaking, and the AI client answered before it was actually their turn. Then the problem got worse because the AI coaching scored a conversation that the therapist did not finish.

The cause was not a bad prompt, it was that we were using silence to decide that the human was done, in a use case where silence is actually part of the work and must be respected. So that’s why we made turn taking explicit.

Suman: My hardest bug was probably a self-interrupting agent on telephony.

On one phone deployment, the agent would randomly stop halfway through a sentence, apologize, and start over, basically behaving as if someone had interrupted it. Except nobody had.

What made it particularly frustrating was that it only happened with certain telephony carriers, and so reproducing it consistently was difficult.

Eventually we tracked it down to acoustic echo. The agent’s own TTS audio was leaking back into the inbound audio path. Some carrier and handset combinations had weak or nonexistent echo cancellation, so the VAD was detecting the agent’s own voice as if it were the user speaking.

The agent was literally interrupting itself!

The fix ended up being layered. We added better echo suppression on the inbound path, raised the interruption threshold from simply VAD firing to VAD + minimum duration + semantic turn detection, and also started comparing detected speech against our own playback timing.

If user speech starts at almost exactly the same time as the agent is playing audio, we treat it as a suspicious echo rather than immediately assuming the user wants to interrupt.

The bigger lesson for me was that you can’t assume the audio path is clean, especially on telephony. Interruption handling that looks perfect in a browser demo can fall apart pretty quickly in the real world.

Barge-in detection really needs to be defense-in-depth: energy, duration, and semantics, rather than relying on a single VAD threshold.

Fahad: I think the most difficult interruption-related bug is the false or backchannel interruptions of an agent. For example, if you cough, your mom says something in the background, or you simply affirm, “That’s correct,” that should not be treated like turn detection. It’s just what happens in real human conversation. But agents may detect that as an interruption and will stop speaking, which derails the conversation.

There’s an easier way to solve this if your agent is deployed on LiveKit Cloud, where you can use their adaptive interruption model (audio based) and add it to the beginning of your pipeline. It will ensure the agent doesn’t stop speaking if the user doesn’t intend to interrupt.

But since it’s not available on self-hosted environments, there’s a multi-step approach you have to take yourself.

First, you need to include VAD and configure the min-word duration and confidence threshold values to make sure small interruptions or noises are ignored. For further safety, you can also introduce noise cancellation to ensure background noises don’t affect the agent.

However, these steps won’t stop accidental interruptions when a user is just reacting without meaning to cut the agent off. I haven’t built a solution yet, but I’m thinking about trying the following options if needed:

  • Word Filter: Simply filter out common backchannel words (“hmm,” “wow”, ‘that’s nice’), and don’t interrupt the agent when one of these is spoken. We can also benefit from the capabilities of a STT model that is used  in the pipeline, For instance, Deepgram’s Nova model consistently transcribe five specific vocalized listener signals: mhmm, mm-mm, uh-uh, uh-huh, and nuh-uh. By actively catching these specific tokens in real time, the pipeline can recognize them as passive listener signals and signal the agent to continue speaking uninterrupted.
  • Small Audio Model: Use a lightweight model to check the user’s tone and semantics before deciding to interrupt. (More accurate, but adds latency).

There’s a lot to consider with Voice AI pipelines

As you can see, a simple question like “How do I stop my Voice AI from interrupting people?” cannot be answered in a few lines of code. It has architectural implications that need to be carefully considered. In addition to our team’s perspectives, you can also check out some recent interviews I’ve had with industry experts on WebRTC Live. 

This is an area where vibe coding will not get you very far. You need a team of experts who have deployed custom and integrated Voice AI solutions across a variety of use cases. Contact us today to tap into that expertise, and we’ll help make sure your Voice AI delivers the positive customer experience you’re aiming for!

Recent Blog Posts