August 18, 2021
Comments Off on Adding LiveSwitch into a Telehealth Video Application

In this blog post, we’ll look at how simple it is to add the LiveSwitch CPaaS into your application. Specifically, we’re going to use the LiveSwitch APIs to replace the video chat solution in our SimplyDoc telehealth starter kit.

LiveSwitch is a longstanding and powerful media server option for building live video applications based on the WebRTC standard. Formerly known as Frozen Mountain, LiveSwitch has many features built in and can be used in a cloud configuration or as an on-premise solution.  

The best of both worlds

Open source media server solutions allow lots of flexibility and configuration options, but can be harder to implement and require additional development time. Commercial media server solutions typically come as a CPaaS offering where you have a lot of power and scalability built into a monthly fee, but will usually offer fewer configuration options and less control over the internals. 

LiveSwitch could be described as the best of both worlds, in that you can get lower level access to the media pipeline and media server configurations than many CPaaS’s, but it’s still backed by their Professional Services team as well as their development partners (such as our team at WebRTC.ventures).

WebRTC and Telehealth

Telehealth is a frequent use case for live video applications (and is becoming more popular year-over-year), and we have built many telehealth applications at WebRTC.ventures. For that reason, we created a starter kit called SimplyDoc, which allows us to license existing code to our clients, and then customize that codebase to their unique needs, allowing them to get to market faster with a unique telehealth workflow.  

LiveSwitch has also been used in the telehealth arena many times and it can be used to build HIPAA-compliant applications. Our team wanted to see how easy it would be to replace the commercial media server we currently use in SimplyDoc with the LiveSwitch Cloud offering. It turns out that it was fairly straightforward, as we will describe in this blog post!

How we built it

SimplyDoc is built on a React front end and our current CPaaS provides a javascript importable module that wraps its core functionality for ease of use in the front end. LiveSwitch supports a number of languages and frameworks, so we chose to use the Javascript SDK for the integration with our React front end.

Another reason why we chose LiveSwitch is because it provides a managed instance of the platform. This service provides a dashboard, where you are able to manage various aspects of your application.

Liveswitch supports three media connection modes:

  • Peer-to-Peer (P2P)
  • Selective Forwarding Unit (SFU)
  • Multipoint Control Unit (MCU)

For the SimplyDoc demo, we used peer-to-peer connections (P2P) since the common use case for telehealth is a one to one consultation. In 1-1 calls, you can use a pure Peer-to-Peer model in most cases, whereas in larger group video calls you will need to use an SFU or MCU to provide better quality.  (For more information on scaling your WebRTC video application, check out this episode of WebRTC Live with LiveSwitch CTO Anton Venema as our guest!)

Because LiveSwitch does not provide an importable JS module, we created an ES6 importable wrapper around the core SDK. This provided a similar interface as our current CPassS module which allowed us to minimize the changes necessary to add in LiveSwitch. The LiveSwitch JS SDK is loaded into the frontend using an html <script> tag. 

<script src="./lib/fm.liveswitch.js"></script>

The LiveSwitch.js wrapper is then imported into our React application. During the course of the application, a number of components will call the LiveSwitch module for handling media devices or chat messages. To avoid multiple instances being created, a singleton class was created which calls the wrapper.

import {LiveSwitch} from './LiveSwitch';
 
const liveswitch = new LiveSwitch({
   applicationId,
   sharedSecret,
   deviceId
});
export default liveswitch;

The class was used in a number of application components. The first being the preview, which prepares a call by accessing the local devices.

import liveswitch from '../../liveswitch-instance';
async componentDidMount() {
  this.ls = liveswitch;
 this.ls.setLayoutStage(document.getElementById("video-preview"));
   const levelChangeHandler = (level) => {};
   const mediaStarted = await this.ls.startLocalMedia({levelChangeHandler});
   await this.getDevices();;
   if(!mediaStarted){}
 }

This change was seamlessly integrated into the application and operated alongside the current CPaaS. These calls use the SDK locally and do not require a client to be registered. 

We used LiveSwitch in other parts of the application such as in-call video and audio, message passing (for chat and disconnect notification) as well as screen sharing. For these operations, the client is required to be connected to a channel through a LiveSwitch backend (cloud or server). The process is as follows:

  • Generate a token for the client 
  • Use the token to register the client
  • Join a channel

The wrapper abstracts step one into the register method provided, so the user is not required to generate any tokens.

await this.ls.register();
     await this.ls.startLocalMedia(()=>{});
     const audioDeviceId = this.ls.getAudioInput();
     const videoDeviceId = this.ls.getVideoInput();
     this.setDeviceId('audio', audioDeviceId._id);
     this.setDeviceId('video', videoDeviceId._id);
     try{
       await this.ls.joinChannel({
           messageHandler : ({message})=>{
             switch (message) {
               case "callShouldEnd":
                   this.disconnect();
                 break;
            
               default:
                 break;
             }
           },
           remoteClientHandler : (remotePeers)=>{this.setState({activeConnections :remotePeers});}
       },`${auth.viewer.firstName} ${auth.viewer.lastName}`);

The remoteClientHandler accepts a callback that is triggered when a client joins. As it is peer to peer, the client is responsible for managing connections to and from all connected peers and SimplyDoc uses this to track the number of users in call.

//handler that is raised when a remote client wants to open a peer connection
this.channel.addOnPeerConnectionOffer((remoteClientInfo)=>{});
//handler that is raised when a remote client joins this channel
this.channel.addOnRemoteClientJoin((remoteClientInfo)=>{});

LiveSwitch also provides the ability to pass and receive messages once a user has joined a channel. The SDK offers a message that accepts a call back. We used this functionality for in-call chat and media upload notifications. One challenge with this is that the method is called in a channel after the user has been registered and joined. There are, however, parts of the application that needed to add a listener prior to joining the channel. 

For example, the component handling notification for chat messages is loaded separately from the one where the register() method is called. As such it is possible that the client is not a part of a channel when they are trying to add the handler. To get around the problem, we added a messageHandlers array to the wrapper class and an addMessageHandler method to add new listeners.

export class LiveSwitch{
   constructor(){
	...
    	this.messageHandlers = [];
   }
 
   addMessageHandler(callback){
       this.messageHandlers.push(callback);
   }
}

When a message arrives on the channel, the onMessage handler is triggered. We call each handler that has been added. This requires each handle to check the message it has received.

this.channel.addOnMessage(({_deviceId,_userId : userName
 }, message)=>{
               if(this.deviceId === _deviceId){
                   return;
               }
               if(this.messageHandlers.length !== 0){
                this.messageHandlers.forEach((messageHandler)=>{
                       messageHandler({userName,message});
                   });
               }
           });

Screen sharing functionality was also handled by LiveSwitch. To minimize changes that we had to make to SimplyDoc, an interface for screen sharing in LiveSwitch was created that seamlessly replaced the current CPaaS. 

 //SimplyDoc
 toggleScreenShare = async () => {
   await this.ls.toggleShareSession();
 }

In the LiveSwitch model, the calls to the SDK were abstracted for simplicity. The screen sharing functionality in LiveSwitch is also managed as local media, as such in the constructor for the class, there was a screen media object created.

// local media
this.localMedia = new fm.liveswitch.LocalMedia(withAudio, withVideo);
// media handler for screen sharing
this.screenMedia = new fm.liveswitch.LocalMedia(false, true,true);
// handler for media stopped from browser popups
this.screenMedia.addOnVideoStopped(()=>this.toggleShareSession());

The toggleShareSession method, manages the setup and tear down of screen sessions. It does tasks such as checking if a current session is being shared , if the screen share has been stopped by the browser popup etc. In the peer-to-peer mode, starting a screen share starts a new video-only stream to other peers, while it replaces the local media view with the screen capture rather than the camera input.

async toggleShareSession(){
       const screen_state = this.screenMedia.getState();
       if(!this.channel){
          return false;
        }
       if(this.shareScreenConnections.length){
        }
...
   }

The LiveSwitch SDK allows tags to be set on connections and this is utilized within the class. 

this.screenMedia.setDynamicValue("screenShare" , true);
   ...
this.p2pConnections.forEach(async (client)=>{
   const videoStream = new fm.liveswitch.VideoStream(this.screenMedia);
   const connection = this.channel.createPeerConnection(client, videoStream);
   connection.setTag("screenShare");
   ...
}

The “screenshare” tag is set on these connections so that peers can be aware of screen connections for setting up connections.

this.channel.addOnPeerConnectionOffer(function(peerConnectionOffer) {
// check for screenshare connections
const isScreenShare = peerConnectionOffer.getConnectionTag() === "screenShare";

The end result

With these changes applied, we were able to successfully deploy SimplyDoc with LiveSwitch. The experience was seamless to the end user and the application functioned as usual. It was largely a drop-in replacement that offered a very good developer experience, and the video quality was excellent!

Are you interested in building a live video application? Let our team of experts at WebRTC.ventures help. Contact us today!

To learn more about LiveSwitch, visit LiveSwitch.io to start your free trial!

Recent Blog Posts