WebRTC started in the browser, and it now also runs on servers: SFUs, gateways, cloud gaming, recording, and AI media agents. Each of these services needs a server-side WebRTC implementation, the protocol stack that lets a backend program act as a peer. Google’s libwebrtc, the most widely used implementation, was built for browsers and native client apps. Its API centers on capturing, encoding, and rendering media for one user, which makes it awkward in a service that routes or bridges media among many peers. Server-first libraries fill that gap: Pion (Go), aiortc (Python), webrtc-rs (Rust), and GStreamer’s webrtcbin.
Picking among them is expensive to reverse. A library’s limits rarely show up in a demo with two peers. They show up after routing, recording, and scaling depend on it, and working around them at that stage often means a rewrite.
The choice depends mostly on the layer where your server handles media. An SFU forwards encoded RTP packets untouched. A voice AI agent or computer vision pipeline needs decoded audio and video frames. This post explains when the implementation decision falls to you, compares the main server-side WebRTC libraries, and maps each one to the workloads it fits.
When the Implementation Decision Is Yours to Make
An implementation is not the same thing as a media server. Keeping the two separate is the first step.
- A WebRTC implementation is the protocol stack (Interactive Connectivity Establishment (ICE) for connectivity, DTLS-SRTP for encrypted transport, SCTP for data channels, and RTP/RTCP for media) that lets one process be a peer.
- A media server (an SFU or an MCU) is a server application that uses an implementation to route media among many peers.
Think of implementation as engine parts and the media server as the assembled vehicle. This post is about the parts. We covered the full vehicle layer in Open Source WebRTC Media Servers: Choosing the Right One for Your Use Case.
This distinction leads to one unifying rule for scoping your project. Almost every server-side workload already has a higher-level, already-built/managed option, and you drop to the implementation itself when your requirements exceed it. However, that layer wears different clothes depending on the workload:
- Voice AI has managed frameworks and platforms such as Pipecat, LiveKit Agents, or SignalWire Agents. When one fits, it keeps the implementation invisible, which is a feature.
- SFUs and group media have open-source media servers such as LiveKit, mediasoup, or Janus.
- Telephony gateways have SIP plugins.
Some are “already built” rather than “managed,” but the logic is identical: the higher-level option abstracts the complexity until it can’t do what you need.
When there’s a requirement that exceeds that option: cost, data residency, latency control, custom media processing, a media source that isn’t a capture device (a render pipeline, a robot camera), an unusual deployment topology, or compliance control the higher layer won’t give you. At that point, the project becomes a peer embedded in your own Go, Rust, or Python service.
What varies between workloads is how soon you hit that ceiling. Managed Voice AI platforms cover a lot of ground, so most teams stay on them for a long time. Cloud gaming, WHIP/WHEP streaming edge cases, robotics, non-device media sources, and strict self-hosting might hit the ceiling much sooner, which makes “which implementation?” a day-one question for them and a someday question for others.
The Main Server-Side WebRTC Libraries
Three implementations are built for the server from the ground up, each with a distinct strength.
Pion (Go)
A pure-Go stack with no C dependencies, which means static binaries and trivial cross-compilation. It is modular, mature, broadly deployed, and gives you first-class access to raw RTP which is the capability media routing lives or dies on. LiveKit, the Go SFU, is built on it.
aiortc (Python)
An asyncio-based stack that is the shortest path to a WebRTC peer inside Python code. When your server’s real work is a machine-learning model, a computer-vision pipeline, or AI glue, aiortc puts a peer right next to it.
webrtc-rs (Rust)
Pion’s design, rebuilt in Rust: modular protocol crates, a Sans-IO core (protocol logic decoupled from I/O), and memory safety without a garbage collector. It is younger than the other two and moving fast, so it fits teams that want performance and safety and are comfortable on a newer stack.
GStreamer’s webrtcbin
Webrtcbi is a fourth that belongs in the same conversation but arrives from a different direction. implements most of the W3C PeerConnection API (offers, answers, SDP negotiation, media and data channels) so it is a real server-side peer, not just a bridge.
What sets it apart is the integration model. Pion, aiortc, and webrtc-rs are libraries you call: you hold a peer-connection object and drive it from your own control flow. webrtcbin is a single element inside a GStreamer media pipeline. You build a graph of elements around it and react to its negotiation signals, letting the framework own the data flow.
That makes it the natural choice when the server’s job is a media pipeline (capture, transform, encode, especially with hardware-accelerated plugins) rather than routing packets. It is also the heaviest dependency of the four: you are taking on GStreamer, not a single library.
libwebrtc and libdatachannel
Two more WebRTC implementations are worth mentioning.
- libwebrtc is the client and interop baseline. If you are on the browser, you are already using it.
- libdatachannel is a small C/C++ stack for teams that want a minimal footprint rather than all of libwebrtc, which makes it a natural fit on constrained hardware.
Language and maturity matter, but they are rarely the deciding factor. One capability predicts your timeline more reliably than any of them.
The Deciding Question: Media at the Layer You Need
The single most predictive question when choosing an implementation is: At what layer does your workload touch the media, and how cleanly does the library hand you that layer? Media moves through a peer at two layers: the encoded RTP packet on the wire, and the decoded frame (raw PCM audio or a raw video frame) after the codec runs. Different workloads need different layers, and the two cleanest examples pull in opposite directions.
SFUs need encoded RTP
An SFU needs the encoded layer. Its entire job is to read an inbound RTP packet from one peer and write it out to the others, untouched with no decoding, no re-encoding. In Pion that is a documented two-call loop: read the packet, write the packet. The outbound track accepts pre-encoded RTP as-is, so no codec is ever instantiated.
// Outbound track other peers subscribe to. "StaticRTP" accepts
// pre-encoded RTP packets as-is — no codec is instantiated.
localTrack, _ := webrtc.NewTrackLocalStaticRTP(
webrtc.RTPCodecCapability{MimeType: webrtc.MimeTypeVP8}, "video", "pion",
)
// When a publisher's track arrives, forward its packets verbatim.
peerConnection.OnTrack(func(remote *webrtc.TrackRemote, _ *webrtc.RTPReceiver) {
for {
pkt, _, err := remote.ReadRTP() // raw RTP in
if err != nil {
return
}
if err := localTrack.WriteRTP(pkt); err != nil { // raw RTP out
return
}
}
})
AI and vision pipelines need decoded frames
A decode-side workload needs the opposite: pull each frame out as raw data, do something with it, and publish a frame back. A voice AI agent is one instance with inbound audio becoming PCM for speech-to-text, and text-to-speech produces PCM going back out. A computer-vision pipeline is another, and so is any in-process step that transforms media. aiortc expresses this directly: you await a decoded frame off the track, convert it to an array, transform it, and hand frames back out through a track whose job is to produce them. The pattern reads in a handful of lines.
from aiortc import MediaStreamTrack
from av import VideoFrame
class TransformTrack(MediaStreamTrack):
kind = "video"
def __init__(self, track):
super().__init__()
self.track = track # the inbound remote track
async def recv(self):
frame = await self.track.recv() # decoded frame in
img = frame.to_ndarray(format="bgr24") # raw pixels to work on
img = transform(img) # your CV / ML / overlay step
new_frame = VideoFrame.from_ndarray(img, format="bgr24") # frame back out
new_frame.pts = frame.pts # preserve timing
new_frame.time_base = frame.time_base
return new_frameThat is the whole idea behind owning a server-side peer: receive a frame, process it, send a frame. The snippet is adapted from aiortc’s server example, where VideoTransformTrack does almost exactly this. The transform call stands in for your real vision or ML step, and a production build adds signalling, TURN, and the model pipeline on top.
webrtcbin answers the same question from the pipeline side. Instead of a loop that pulls a frame and returns one, you declare the media graph and let it flow: a source and encoder feed the peer, or the peer’s decoded output feeds your processing elements. The media handling is wiring, not control flow. That is a liability for packet routing: forwarding RTP between peers means plumbing pads between webrtcbin instances rather than a two-call loop, and it fights you. But it is exactly right when the media is a pipeline: an embedded camera encoding to a browser, a transcode gateway, anything that leans on GStreamer’s library of hardware-accelerated elements. The layer question still decides, the answer is just expressed as a graph.
Every other workload lands on the same axis. Recording and gateways want raw access and sometimes decode. A cloud-gaming server sits on the encode side, but with frames coming from a render pipeline rather than a camera. A robotics or IoT build may barely touch media at all and live mostly on the data channel. The question is always which layer, how cleanly? Not merely “can it forward a packet.” aiortc is first-class for decoded frames, and Pion and webrtc-rs expose both encoded RTP and decoded samples, so each gives you a layer to work at directly. Once you know your layer, the workload-to-implementation mapping falls out.
Which WebRTC Library Fits Your Workload
For any of these workloads, start with a managed platform or an existing media server. Use the table below once your requirements go beyond that option and you need to choose the library for your own peer.
Load testing also belongs on this list. A load-testing tool creates many simulated peers to stress-test your media infrastructure, which makes it a server-side WebRTC build too.
| Workload | Recommended implementation | Why |
|---|---|---|
| SFU / group media at scale | Pion or webrtc-rs | Raw RTP forwarding; LiveKit runs on Pion |
| Recording / compliance capture | Any server-first lib | Needs raw media access; usually self-hosted |
| Gateways (RTMP/HLS/SIP, WHIP/WHEP) | Pion/GStreamer | Bridging and HTTP-signalled ingest/egress |
| Media transcode / HW-accel pipelines | GStreamer webrtcbin | Capture-transform-encode as a pipeline, HW plugins |
| Cloud gaming / remote rendering | Pion or webrtc-rs | Media from a render pipeline, not a device |
| Single-source streaming / IP camera | GStreamer webrtcbin | One source to a sub-second browser stream |
| Robotics teleoperation | webrtc-rs or libdatachannel | Low-latency video out, control on data channel |
| IoT / edge / data channels | libdatachannel or webrtc-rs | Small footprint; data channel is the point |
| AI voice/video agents | aiortc, or Pion/webrtc-rs | Decode-side media; fit to your AI pipeline |
Choosing Your Server-Side WebRTC Implementation
The one-line rule: decide what your server does with the media, let that fix the layer you need, and let the layer narrow the implementation. The tradeoffs around that rule: language and operational fit with your team, in-house skills, and the maturity gap between a battle-tested stack like Pion and a fast-moving one like webrtc-rs all pull on the answer.
At WebRTC.ventures, we have shipped SFUs implementations, recording and compliance pipelines, telephony and broadcast gateways, streaming ingest and egress, and AI media agents across multiple stacks. We help teams decide whether a media server or managed framework still covers their requirements, or whether they need to build on a WebRTC library directly. If you are weighing that decision, consider an architecture assessment. If you already know what you need and want to build it, let’s build your server-side WebRTC application.
