August 28, 2026
Comments Off on Using eBPF to Debug a WebRTC SFU: Finding Where Your Media Server Eats Packets

When WebRTC quality degrades on your SFU (jitter spikes, frozen frames, choppy audio), your monitoring tells you something is wrong. It doesn’t tell you what is responsible. Is the network dropping packets? Or is your server too loaded to read them from the socket on time? The fix for one is scaling infrastructure. The fix for the other is pushing back on network conditions. eBPF gives you kernel-level observability to find out.

eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that runs small, sandboxed programs directly inside the kernel, without changing kernel source code or loading a custom module. That lets you hook into kernel functions and extract measurements no application-level tool can reach, and catch problems before they become user-visible.

This post shows how to use eBPF to measure where a WebRTC SFU loses time, and tell server delay apart from network jitter. To show how this works, we’ll measure one specific metric: RTP packet dwell time. This is how long each UDP packet waits in the kernel’s socket receive queue before your application reads it. This is one example of the kind of kernel-level observability eBPF enables. It answers the server-vs-network question with sub-microsecond precision, often while getStats() still reports “everything looks fine.

What getStats() Can and Can’t Tell You

WebRTC’s getStats() API gives you receiver-side aggregates: jitter, packetsLost, roundTripTime. Essential for knowing that quality degraded, but not enough to pinpoint what is responsible.

Consider two scenarios:

  1. A packet crosses the internet in 5ms but sits in your SFU’s socket queue for 12ms before your server reads it.
  2. A packet bounces through a congested peering link adding 12ms of network jitter, but your server processes it within microseconds of arrival.

Both produce noise on getStats(), yet the underlying causes (and fixes) are completely different.

In early-stage failures, the signatures can appear distinct. A loaded server might produce jitter with zero packet loss (i.e. the queue is filling but hasn’t overflowed yet), while a lossy network might show packet loss with minimal jitter (i.e. packets either arrive on time or don’t arrive at all). However, these clean signatures don’t hold under sustained pressure. At that point, getStats() alone cannot disambiguate server-side from network-side problems.

What’s missing is a kernel-level measurement that complements application stats: socket queue dwell. This data is key because CPU starvation often manifests first as tail latency spikes, well before aggregate utilization metrics cross any alarm threshold. As a result, your CPU looks fine at 60%, but packets are queuing because the SFU’s event loop can’t call recvmsg() fast enough.

Where Host-Side Latency Actually Hides

Here’s the simplified path a UDP packet takes on your SFU:

A simplified view of a packet entering the server and being processed by the SFU

At 100 participants × 3 simulcast layers × 30fps, you’re dealing with 9,000 packets per second inbound. Your event loop budget is tight. If anything causes 2ms delays at this rate, the receiver’s jitter buffer grows and users hear it.

eBPF: The Right Tool for This Job

eBPF lets you attach small programs to kernel function entry and exit points. Because it operates below the application layer, the same tracer works regardless of whether your SFU is written in Go, C++, Rust, or Node.js, and regardless of whether it’s Pion, mediasoup, Janus, LiveKit, or something else.

The overhead is about 100-200ns per probe invocation. At 9,000 packets/sec with two probes, that’s roughly 3.6ms of CPU per second or 0.36% of one core. Negligible for a diagnostic tool.

The eBPF Program: Measuring Dwell Time

The kernel-side program is compact. We hook _ _udp_enqueue_schedule_skb (where the packet enters the socket receive queue with a valid socket pointer) and udp_recvmsg (where the application reads it). The difference is the dwell time.

The enqueue probe fires each time the kernel places a UDP packet on a socket’s receive queue. It filters for RTP-range ports and records a nanosecond timestamp into a per-socket circular FIFO, marking the start of dwell time for that packet.

SEC("kprobe/__udp_enqueue_schedule_skb")
int BPF_KPROBE(kprobe_udp_enqueue, struct sock *sk, struct sk_buff *skb)
{
    __u64 sock_key = (__u64)sk;

    // Filter: only trace packets destined for RTP port range (10000–60000).
    __u16 dport = BPF_CORE_READ(sk, __sk_common.skc_num);
    if (!is_rtp_port(dport))
        return 0;

    // Lookup or create the per-socket timestamp FIFO.
    struct sock_ts_queue *queue = bpf_map_lookup_elem(&sock_queues, &sock_key);
    if (!queue) { /* ... initialize new queue entry ... */ }

    // Record the enqueue timestamp — this marks the START of dwell.
    __u32 idx = queue->tail & (MAX_QUEUE_DEPTH - 1);
    queue->timestamps[idx] = bpf_ktime_get_ns();
    queue->tail++;

    return 0;
}

The return probe fires when `udp_recvmsg` completes, the moment the application has consumed a packet. It dequeues the oldest timestamp from the FIFO, subtracts it from the current time to compute dwell, and emits the measurement through a ring buffer to userspace.

SEC("kretprobe/udp_recvmsg")
int BPF_KRETPROBE(kretprobe_udp_recvmsg, int ret)
{
    // Correlate this return with the socket pointer saved on entry.
    __u64 pid_tgid = bpf_get_current_pid_tgid();
    __u64 *sock_key_ptr = bpf_map_lookup_elem(&active_recvmsg, &pid_tgid);
    if (!sock_key_ptr)
        return 0;

    __u64 sock_key = *sock_key_ptr;
    bpf_map_delete_elem(&active_recvmsg, &pid_tgid);

    struct sock_ts_queue *queue = bpf_map_lookup_elem(&sock_queues, &sock_key);
    if (!queue || queue->head >= queue->tail)
        return 0;

    // Dequeue the oldest timestamp — this is the END of dwell.
    __u32 head_idx = queue->head & (MAX_QUEUE_DEPTH - 1);
    __u64 enqueue_ts = queue->timestamps[head_idx];
    queue->head++;

    // Compute dwell time: current time minus enqueue time.
    __u64 now = bpf_ktime_get_ns();
    __u64 dwell_ns = now - enqueue_ts;

    // Emit the measurement to userspace via ring buffer.
    struct dwell_event *evt = bpf_ringbuf_reserve(&events, sizeof(*evt), 0);
    if (!evt)
        return 0;
    evt->dwell_ns = dwell_ns;
    evt->sock_cookie = (__u32)sock_key;
    evt->queue_len = queue->tail - queue->head;
    bpf_ringbuf_submit(evt, 0);

    return 0;
}

The Go Userspace: Aggregation and Reporting

On the userspace side, we use cilium/ebpf to load the BPF program and read events from the ring buffer. Events are aggregated into per-socket histograms with 9 buckets spanning from <10µs to >50ms.

The `Record` function receives each dwell event, updates per-socket running totals (count, sum, max), and classifies the dwell time into a histogram bucket. This is the hot path, called once per packet read, so it uses a simple lock and O(1) bucket lookup.

// BucketBounds defines histogram edges in microseconds.
// Produces 9 buckets: [0,10) [10,50) ... [50000,+∞)
var BucketBounds = [8]uint64{10, 50, 100, 500, 1000, 5000, 10000, 50000}

func (sc *StatsCollector) Record(cookie uint32, dwellNs uint64,
    sourceIP string, sourcePort, destPort uint16) {
    sc.mu.Lock()
    defer sc.mu.Unlock()

    // Find or create the per-socket accumulator (keyed by kernel socket cookie).
    acc, ok := sc.sockets[cookie]
    if !ok {
        acc = &socketAccumulator{
            sourceIP: sourceIP, sourcePort: sourcePort,
            destPort: destPort, cookie: cookie,
        }
        sc.sockets[cookie] = acc
    }

    // Update running totals for avg/max computation.
    acc.packets++
    acc.totalDwell += dwellNs
    if dwellNs > acc.maxDwell {
        acc.maxDwell = dwellNs
    }

    // Classify into histogram bucket (ns → µs, then linear scan of 8 bounds).
    dwellUs := dwellNs / 1000
    acc.buckets[bucketIndex(dwellUs)]++

The tracer initialization attaches all three probes (kprobe on enqueue, kprobe + kretprobe on recvmsg) and opens the ring buffer. This is the one-time setup that connects the kernel-side measurements to the Go event loop.

func New(cfg Config) (*Tracer, error) {
    t := &Tracer{
        stats: NewStatsCollector(cfg.MaxSockets),
        cfg:   cfg,
    }

    // Load compiled BPF bytecode into the kernel.
    if err := loadBpfObjects(&t.objs, nil); err != nil {
        return nil, fmt.Errorf("failed to load BPF objects: %w", err)
    }

    // Attach kprobe to __udp_enqueue_schedule_skb — marks dwell START.
    kpUdpEnqueue, err := link.Kprobe(
        "__udp_enqueue_schedule_skb", t.objs.KprobeUdpEnqueue, nil)
    if err != nil { return nil, err }
    t.links = append(t.links, kpUdpEnqueue)

    // Attach kprobe + kretprobe to udp_recvmsg — marks dwell END.
    kpRecvmsg, _ := link.Kprobe("udp_recvmsg", t.objs.KprobeUdpRecvmsg, nil)
    krpRecvmsg, _ := link.Kretprobe("udp_recvmsg", t.objs.KretprobeUdpRecvmsg, nil)
    t.links = append(t.links, kpRecvmsg, krpRecvmsg)

    // Open ring buffer — userspace reads dwell events from here.
    t.reader, _ = ringbuf.NewReader(t.objs.Events)
    return t, nil
}

Video Demo: Three Scenarios on a Live Server

We paired the tracer with a Pion WebRTC echo server: a minimal SFU that reflects incoming video back to the sender, and ran three scenarios on an EC2 instance while a browser client streamed video and displayed both getStats() and eBPF dwell metrics side by side.

  1. Scenario 1: Baseline. The echo server reflects video with no artificial load and no network impairment. Local and echoed video counters stay in lockstep. getStats() reports zero packet loss, stable FPS, ~70ms RTT, single-digit jitter. The eBPF panel shows ~30µs average dwell. Packets enter the queue and get read within microseconds. Everything is healthy.
  2. Scenario 2: Server Load. We enable throttling, every 5th socket read is delayed by 20-50ms, simulating an event loop that can’t keep up. The echoed video stutters and falls behind. getStats() shows jitter climbing, but packet loss stays at zero and FPS holds. From the application metrics alone, you might not flag this as urgent. The eBPF panel tells a different story: average dwell jumps to 76ms, max hits 193ms, status flips to ALERT. The server is holding packets hostage and the tracer caught it before the stream fully degraded.
  3. Scenario 3: Network Impairment. Load is removed, the server returns to baseline, and we add tc netem rules: 10ms±20ms delay, 10% packet loss, 25% reorder. The video degrades again. From the user’s perspective, it looks just as bad as Scenario 2. getStats() lights up: packet loss ticks upward, RTT climbs. But the eBPF panel stays at 33µs dwell, status OK. The server is processing packets the instant they arrive. The problem is in transit, not on our box.

As mentioned before, these application-level signals blur together under sustained production load. The eBPF dwell metric is the tiebreaker: high dwell means the server is guilty; low dwell means it’s innocent.

From Measurement to Action

Socket queue dwell time is one metric. eBPF makes dozens more possible for WebRTC workloads: kernel-side packet drop counting (hooking kfree_skb to distinguish server-caused drops from network drops), run queue latency (measuring how long your SFU thread waits for CPU time before it can read packets), softirq processing delays (detecting when NAPI budget exhaustion defers packet delivery), and per-core IRQ-to-application timing. The same pattern applies: kprobe on entry, kretprobe on exit, compute the delta.

Dwell time happens to answer the most common question SFU operators face, but the observability layer underneath is far more general. When dwell is low, you know the server is doing its job even if users report degradation. When it spikes, you know where to look often before getStats() or user-visible quality has degraded enough to trigger alerts. No more guessing.

Building real-time communication systems is hard. Diagnosing them at scale is harder. Whether you’re choosing between SFU architectures, tuning an existing deployment under production load, or building custom monitoring tools like the one in this post. That’s exactly what WebRTC.ventures does. We help teams design, build, and scale real-time communication systems. If your packets are spending too long in the queue, let’s talk.

Further Reading:

Recent Blog Posts