Debugging video issues in WebRTC applications can be a challenging task. With numerous moving parts under the hood, pinpointing the exact cause of problems can feel like searching for a needle in a haystack. One of the most effective strategies is to reproduce the issue consistently in a controlled environment.

In this post, we explore a powerful technique first featured on the WebrtcH4cKS blog: isolating and reproducing video issues using the video_replay tool. By capturing and replaying raw RTP packets, you can effectively step outside the live environment and analyze video streams in a controlled setting. 

This method, particularly when paired with Google Chrome’s ability to log unencrypted RTP dumps, offers a consistent way to reproduce and debug video problems, ultimately leading to faster and more effective solutions.

Overview of the Video “Replay” Process

WebRTC applications in the browser rely on a complex stack of technologies. When video breaks down, it’s often due to one or more of these components misbehaving. The video_replay tool allows developers to reproduce specific video issues from a previously recorded session, helping to isolate the problem and better understand what’s going wrong.

One way to achieve this is by capturing raw RTP packets of problematic streams, and passing these to libWebRTC’s video_replay tool. This provides a way to isolate such issues by “replaying” the stream using WebRTC framework outside of the browser, providing developers with a consistent way to reproduce and debug them.

However, given WebRTC’s approach for encryption, it is not possible to access such unencrypted packets under normal circumstances. Fortunately, Google Chrome provides a way to log unencrypted RTP dumps that can be used for this purpose.

The overall process is as follows:

1. Start a new Google Chrome session with special flags enabled.
2. Establish the WebRTC connection that you want to debug.
3. Using webrtc-internals, get the relevant information of the stream. This includes the codec, payload type and SSRC.
   • Learn more about webrtc-internals in our previous post: A Quick Tour of webrtc-internals: A Powerful WebRTC Debugging Tool – WebRTC.ventures!
4. Create a PCAP file from the RTP dumps provided by Chrome.
5. Pass such file to video_replay, and/or further analyze using other specialized tools such as Wireshark network protocol analyzer.

Let’s see these steps in an example.

One-Time Set Up: Building video_replay Binaries

The first thing you need is to install the actual video_replay tool. This comes as part of the WebRTC code, so you need to clone the repo, install dependencies and build the binary yourself. The instructions are detailed on the Google Git WebRTC development page, but these can be summarized as follows:

1. Download Chromium depot tools, and add these to your operating system’s PATH variable.
2. Create a new directory (i.e. webrtc_checkout), and use the fetch tool to download the WebRTC code there.
3. Use the gclient tool to finish code synchronization.
4. Inside the src/ folder generate the build files using gn gen.
   1. Optionally you can configure your build according to your needs: i.e. Configure support for H.264.
5. Use the ninja tool to build the video_replay binary.

For instance, below are the commands for Unix-based operating systems. Heads up that this will be a considerably big download.

# 1. Download depot tools
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git ~/depot_tools
export PATH=~/depot_tools:$PATH

# 2. Download WebRTC code
mkdir ~/webrtc_checkout
cd ~/webrtc_checkout
fetch --nohooks webrtc

# 3. Sync code
gclient sync

# 4. Generate build files
cd src
gn gen out/Default

# 4.a (Optional)
#   Configure support for H.264
printf "rtc_use_h264 = true\nffmpeg_branding = \"Chrome\"" >> out/Default/args.gn
#   rebuild
gn gen out/Default

# 5. Build video_replay binary
ninja -C out/Default/ video_replay

After this, you will have the video_replay binary available for troubleshooting video issues.

Capturing Unencrypted RTP Traffic

Now we need to capture the RTP packets of the stream we want to troubleshoot. Start a new Chrome session with the WebRTC-Debugging-Rtp flag enabled as shown in the command below. Make sure to change the chrome binary to the one available in your operating system, which in my case is google-chrome.

Also, if you plan to share the resulting replay (i.e. file a bug in WebRTC issue tracker), you might want to use the --use-fake-device-for-media-stream flag to prevent sending your own camera video.

# Open google chrome with RTP dump logging enabled
google-chrome --enable-logging=stderr \
-v=3 \
--force-fieldtrials=WebRTC-Debugging-RtpDump/Enabled/ \
--use-fake-device-for-media-stream \
> log.txt 2>&1

Now let’s go for some WebRTC traffic! 

For example, let’s capture the receiving stream from the basic Peer connection sample. Before anything else, open chrome://webrtc-internals in a different tab, as we will use it to gather some required information.

Getting back to the WebRTC sample, click on “Start” and then on “Call”. This will initiate two peer connections: one that sends media, and another one that receives it. We’re interested in the second one, so look for it in the webrtc-internals tab, you will recognize it because it has a createAnswer API call in the list of events.

Next, identify the stats for the received video stream. It will likely be named something like inbound-rtp and kind=video. Expand these and take note of the SSRC, codec and payload type values, which in our case are 154136587, VP8 and 96, respectively.

We also need the negotiated payload type for REDundant coding (RED). We get this from the SDP of the receiver peer, located under the setLocalDescription event of such a connection. In the SPD, look for a line similar to a=rtpmap:119 red/9000, and take note of the number assigned to rtpmap (119 in this example).

Depending on your use case you might need additional information, such as the payload type and SSRC for retransmissions of video packets. Such information is also available at the SDP and/or video stream stats.

With this information at hand, you can now end the call by clicking “Hang Up” in the sample tab.

The next step is to get the RTP dumps and generate a PCAP file with them. In Unix-based systems we use the grep tool to extract these from the Chrome log, and then use Wireshark’s text2pcap utility to generate the file, as follows:

# Generate a pcap with RTP dumps
grep RTP_DUMP log.txt > rtp-dump.txt
text2pcap -D -u 1000,2000 -t %H:%M:%S.%f rtp-dump.txt rtp-dump.pcap

Now you have an rtp-dump.pcap capture file that you can further analyze with Wireshark and/or replay it using the video_replay binary you built before.

Replaying the RTP Stream with video_replay

To replay the stream using video_replay, pass the rtp-dump.pcap file as input, along with the information about the stream you got from webrtc-internals. The exact command you’ll need will vary depending on your use case, so be sure to check the list of available params by running it with the tool with the --help flag. In our case, the command we used was:

# Replaying the video stream
~/webrtc-checkout/src/out/Default/video_replay \
    --codec=VP8 \
    --input_file rtp-dump.pcap \
    --media_payload_type=96 \
    --ssrc=154136587 \
    --red_payload_type 119

Behind the scenes, video_replay will use WebRTC’s tooling to depacketize, decode and play the stream offline.

If you’re able to see the particular video issue you’re investigating in the replay, then voila! You have a reproducible test case that you can use to open a ticket in the issue tracker, or investigate deep further in the code.

Troubleshooting WebRTC Video

Troubleshooting video issues in WebRTC applications can be significantly simplified by using the video_replay tool. By capturing and replaying raw RTP packets, you get a controlled environment to analyze and debug video streams outside the complexities of a live browser session. 

By utilizing this method in conjunction with Google Chrome’s unencrypted RTP dump logging capabilities, you obtain a robust and standardized approach to efficiently pinpoint and fix video-related issues. This approach also makes upstream issue reporting easier, which leads to more resilient and dependable WebRTC applications.

Struggling with persistent WebRTC video issues? Don’t let debugging challenges slow you down. At WebRTC.ventures, our team of experts specializes in identifying and resolving complex WebRTC problems. We can help you implement robust debugging strategies, including using tools like video_replay, to ensure your applications run smoothly. 

Contact us today to discuss how we can help you optimize your WebRTC performance and deliver a seamless user experience. Let’s make it live!