How can AI scale clinical roleplay training beyond what live sessions with human trainers can deliver? For CETA Global, a nonprofit that trains frontline mental health providers around the world, the answer was an AI “flight simulator”: a real-time platform where psychologists practice difficult client sessions with an AI-generated patient, complete with a lip-synced streaming avatar, live coaching, and replayable feedback. Behind the scenes, a team of six AI agents works in concert to play the patient, coach the trainee, and evaluate the session against evidence-based criteria in real time.
Host Arin Sime sits down with Andrés Rincon, the WebRTC.ventures lead developer on the project, for a look at EBT-Sim, which went from concept to working platform in just eight weeks for a Google.org Accelerator: Generative AI demo.
They dig into what it takes to make concurrent AI feel like a single conversation: a WebSocket-based real-time communication layer, purpose-built Gemini agents orchestrated through Google ADK, low-latency speech-to-text and text-to-speech, and HeyGen streaming avatars. Andrés shares the design decisions that shaped the experience, including why the team chose push-to-talk over automatic turn detection, and how they made the AI client’s resistance adaptive rather than scripted.
If you’re building voice AI applications, multi-agent experiences, or training simulators, this episode is full of practical lessons.
WebRTC Live #115: Using Voice AI and Avatars to Scale Live Training
Episode highlights follow below.
Here are the main episode takeaways for someone focused on voice AI and multi-agent orchestration:
Multi-agent architecture with Google ADK
The system runs six coordinated agents: a patient agent, a coach agent, a step-tracker agent, a whiteboard-control agent, and an evaluation agent. Andreas built this on Google’s Agent Development Kit (ADK), which manages sequencing, parallel execution, retries, and fallbacks. Several agents run in parallel rather than sequentially, since the patient’s audio response doesn’t need to wait on the coach’s feedback. ADK also provides built-in observability (a local web dashboard showing decisions, token usage, input/output, and caching) and deployment scripts for Cloud Run, Vertex AI, and Kubernetes.
Splitting one large prompt into specialized agents
Their first approach used one large prompt to generate the patient’s response, the coach’s feedback, and the whiteboard actions together. This became hard to debug and maintain. Splitting responsibilities across separate agents, each with a narrower prompt and its own tool set, made the system easier to manage. This is a concrete argument for decomposing a monolithic LLM prompt into task-specific agents.
Websockets over WebRTC, and over SSE
The team chose websockets instead of WebRTC because the use case only requires one-directional-feeling audio streaming to a backend, not peer-to-peer, low-latency bidirectional video calling between two humans. They also tested Server-Sent Events (SSE) against websockets and found websockets more stable for this audio streaming pattern, partly because SSE only supports text and would require encoding audio, adding latency. This is a useful case study in matching transport protocol to actual requirements rather than defaulting to WebRTC.
Speech-to-text as an observability layer
Even though their LLM could accept audio directly, they added a speech-to-text step before passing input to the models. The reason was observability: they couldn’t find tooling that displayed audio inputs directly, so converting to text first let them monitor and debug what the models were actually receiving.
Dynamic difficulty via a hidden scoring metric
To simulate a “difficult” patient (for example, one hiding a problem with alcohol use), they didn’t rely on a static instruction like “don’t mention this.” Instead they tracked a running metric that increases as the trainee asks better questions, and the model’s willingness to disclose information scales with that metric. This is a practical pattern for adaptive AI behavior tied to user performance, rather than fixed prompt rules.
Streaming avatar integration (HeyGen, now called Live Avatar)
Key selection criteria were streaming support in both directions: the avatar service could start speaking before a full audio clip was generated, and the app could stream audio to the avatar service incrementally. They also used a “warm up” feature to keep an avatar idling without incurring cost, which is charged only for actual speech audio. This reduces perceived latency at the start of a session.
Whiteboard as a tool-calling surface
Up Next! WebRTC Live #116
Building Robust Voice AI Pipelines for Every Use Case
With Dr. Varun Singh of daily.io and pipecat.ai
Wednesday, August 12, 12:30 pm Eastern
