SIP and RTP feed a call and its media into a voice AI pipeline. What happens next runs through the control plane: what the voice agent is told, and what it decides to do about it.
In my talk at ClueCon this year, From SIP to Tokens: Deterministic Telephony Meets Real-Time Voice AI, I walked through this idea. The voice AI control plane, not the model, decides whether an agent hears something correctly and acts on it safely. I showed an open-source project our WebRTC.ventures team built to prove it out: Agentic Contact Center, a reference stack for testing voice-agent reliability.
A quick definition: control plane is a term borrowed from networking. A router’s data plane moves packets; its control plane decides how those packets should be routed. Applied to voice AI, the data plane is the conversation itself: audio, the model’s response, the words spoken back. The control plane is everything that decides whether that response should happen at all: what the agent is allowed to do, whether a proposed action is authorized, when a human needs to step in.
SIP moves the call. The control plane moves the decision.
SIP is still the front door for most real voice systems. It handles signaling and establishes the call. RTP carries the timed media into the application layer. Those telephony boundaries are deterministic and observable. The AI path is probabilistic and variable.
Once the call reaches the AI layer, a system has to decide several things in real time:
- what the agent is allowed to do
- how conversation state gets tracked
- when a tool call is safe to run
- when the system should pause or retry
- when a human should take over
A voice agent can sound polished in a demo and still fail once a real conversation gets messy. Interruptions, partial utterances, background noise, and unexpected branches are normal in production calls. The control layer has to keep working when the transcript stops making sense.
The model question isn’t the one that matters
Most teams start with the model question: which ASR, LLM, and TTS to use. That question matters, but it rarely decides whether a voice AI system survives production.
The outcome depends more on the control plane wrapped around those models: a clear way to decide when to listen, when to speak, when to act, and when to stop. That requires real separation between:
- media handling
- conversation state
- business logic
- permissions
- latency and observability
When these layers blur together, a system becomes hard to debug and harder to trust. A failure mid-call should point clearly to telephony, the model, a tool, or a policy decision. Getting there requires tracing the full turn: STT commit, first-token latency, TTS, tool results, fallback behavior, and final state.
Pipecat’s FlowManager is where we put this in practice. In Agentic Contact Center, incoming speech runs through ASR into FlowManager before anything reaches the response pipeline. FlowManager holds a conversation state and checks each proposed action against policy before it executes. A model might propose a retention offer mid-call. FlowManager confirms the offer exists, is authorized for that customer, and can be executed safely before anything happens. The model proposes. The application layer decides.
A concrete test case: cancellation-rescue
The project’s reference scenario is a customer trying to cancel a subscription. The system has to:
- detect cancellation intent, including indirect phrasing
- avoid making unauthorized billing promises to keep the customer
- enter a policy hold when a proposed action needs review
- preserve state cleanly if a human operator steers or takes over
- produce evidence of what happened, reviewable after the call
This scenario tests the control plane directly. A model with good judgment can still cause a costly failure if nothing checks its proposals before they reach the customer.
Separation of concerns, forced by design
Separation of concerns is easy to argue for and hard to hold onto once a project grows. The instinct to let one team, one service, or one class handle “a bit of everything” is constant. Agentic Contact Center forced the discipline by splitting into four projects, each with one job and a hard boundary around it:
| Project | Owns | Does not own |
| ACC | Reference target, local demo modes, operator control, evidence production | Generic eval UI, model scoring |
| ConversationAgentEvals | Test orchestration, evidence normalization, reports | Call or session runtime |
| rtc-asr | Speech recognition, ASR benchmarks | Policy or call state |
| ASSERT | Scoring, failure taxonomy, canonical evaluation artifacts | Runtime or persistence |
That boundary is what makes the system debuggable. When something breaks, the question is never “where in this codebase,” it’s “which of these four owns this.” A speech recognition failure and a policy failure produce different symptoms in different places, not the same tangle of logs.
Agentic Contact Center is a reference implementation for testing reliability patterns, not production-ready contact-center software. State runs in memory. Security hardening and persistence are not built yet. What it proves is the pattern, and the pattern is portable: policy checks that run before an action reaches a caller, and evidence that survives the call for review afterward. Any team building a production voice agent can apply this same separation, whether or not they use this specific stack.
Voice AI works when a team treats it as an engineering system rather than a feature demo. The teams that get this right tend to keep the telephony layer boring and reliable, put policy and tool authorization in explicit places, choose models with clear fallback paths, and measure latency and design for observability from the start.
The real test
The strongest voice AI systems are not the ones that sound impressive in a short clip. They hold up when the call gets real: when the customer interrupts, when the network drops a packet, when the model proposes something it shouldn’t approve on its own.
That’s the work we do at WebRTC.ventures: helping teams build the control plane around their voice AI, so the system can be trusted with what it hears. Talk to us about your voice AI control plane.
