“Real-time voice AI only works when infrastructure makes latency feel invisible. For us, that meant changing the shape of our WebRTC deployment without changing what clients expect from WebRTC itself.”
A firsthand look at how OpenAI rearchitected their entire WebRTC stack for global scale, straight from Yi Zhang, co-author of their deep-dive post, How OpenAI delivers low-latency voice AI at scale. He was part of the team behind this overhaul, addressing pauses, interruptions, and delayed barge-in for ChatGPT voice users and agents operating in interactive voice workflows.
He breaks down what it took to make WebRTC work cleanly inside Kubernetes, how separating stateless packet forwarding from stateful session ownership reshaped their infrastructure, the ICE ufrag routing technique that enables deterministic first-packet routing, and what this architecture means for developers building with the OpenAI Realtime API today.
A veteran of Meta and Microsoft with a PhD in Engineering from the University of Wisconsin–Madison, Yi knows this territory from the inside out.
WebRTC Live #114: An Insider’s Look at OpenAI’s Rearchitected WebRTC Stack
Episode highlights and key insights below.
Key Insights
⚡ Latency is a key challenge, especially with Voice AI. Yi explained that infrastructure is just as important like connection setup, routing, packet forwarding, interruption handling etc. If those aren’t optimized, even the fastest model feels slow. He says, “You need to have very low latency so that you can feel … more like you are having a conversation with a real human.”
⚡ Sometimes the simplest architecture wins. Traditional SFUs are great for multi-party meetings. But Voice AI is almost always one human, one AI. Instead of forcing a conferencing architecture onto a conversational problem, OpenAI designed a lightweight transceiver model that better matches the actual use case. Yi says, “100% of our use cases are one-on-one, from AI to user. So those scenarios fit nicely into the transceiver model.”
⚡ Scale by separating responsibilities. One of the biggest architectural decisions was separating Stateless packet forwarding and Stateful session ownership. This makes scaling much easier because networking and conversation logic can grow independently instead of competing for resources. Yi says, “The relay will just do one simple thing, using some information in the STUN to find out where this packet should go. So in our case, we’re just using some embedded information in the ufrag as the indication. So I can leverage this one because ufrag is a user fragment, is a username fragment that is designed for this session.”
Episode Highlights
Build for the experience, not the technology
OpenAI didn’t choose WebRTC because it’s the “best protocol.” They chose it because it currently delivers the best real-time user experience across browsers and devices. The goal isn’t WebRTC. The goal is making conversations feel human.
Yi explains: “Yeah, of course, the team members mostly are coming from the WebRTC background. We can kind of have the initial bias towards, okay, we should do the WebRTC. But I wouldn’t say we have a strong opinion about whether the WebRTC or different protocol standards … we don’t think WebRTC is the end goal, right? The end goal is you deliver the real-time experience to your client, to your customers. So WebRTC is just the means to solving this problem.”
WebRTC vs WebSockets
WebSockets are great for getting a voice AI prototype off the ground. But if your vision is natural, real-time conversations at global scale, you’ll eventually need infrastructure built for that future. OpenAI chose WebRTC because it better supports low-latency, conversational experiences across browsers, devices, and unreliable networks. Yi says, “I think WebSockets is still pretty relevant in most of the voice AI domains. It’s easier to pick it up and also easier to build a first demo or prototype you have to find the product market fit. But WebRTC has their own advantage, especially when you have the vision. In the future, your AI voice is going to be more conversational, more natural, and more real-time, then the WebRTC will be the de facto.”
The future isn’t faster responses; it’s simultaneous conversations.
The goal isn’t simply answering faster. It’s making AI behave more like another person in the conversation. Yi explains, “You need very low latency so that you can feel the interruption is more natural. It’s more like you have a conversation with a real human versus like today a lot of the AI voice model is more like a request a question and answer sort of the model but you’re going to have more. Imagine in the near future you’re going to experience the same like when you’re talking and the model is talking you can talk over each other with just like the humans.”
Alberto adds: “Or even if two people are talking over each other, because it has happened, if there is background noise, how you handle that, all these things.”
Up Next! WebRTC Live #115
Using Voice AI and Avatars to Scale Live Training
Wednesday, July 22, tentatively 12:30 pm Eastern
