Author: Tao Jinliang, senior client development engineer of NetEase Yunxin

In recent years, the field of real-time audio and video has become more and more popular, and many audio and video engines in the industry are implemented based on WebRTC . This article mainly introduces the background and related technology implementation of WebRTC in video auxiliary streaming.

SDP in WebRTC supports two schemes: PlanB scheme and Unified Plan scheme. In the early days, we only supported one video stream in the Plan B scheme of using multiple PeerConnections. This video stream is called “mainstream”. Currently, we use the Unified Plan solution with a single PeerConnection to add a video auxiliary stream. What is a video “auxiliary stream”? Auxiliary video stream refers to the second video stream, which is generally used for screen sharing.

demand background

With the development of business, one video stream cannot meet the needs of more practical business scenarios. For example, in multi-person video chat, NetEase conference and other online education scenarios, two video streams need to be sent at the same time: one is the camera stream, and the other is the video stream. is a screen sharing stream.

However, when using the SDK to share the screen at present, the screen is shared from the camera capture channel. Under this scheme, the sharer has only one upstream video stream. In this scenario, either the camera screen or the screen screen is upstream, and the two are mutually exclusive.

Unless a new SDK is instantiated to collect and send screen images, the solution of instantiating two SDKs is very troublesome to deal with at the business layer and there are many problems, such as how to deal with the relationship between the two streams.

In the WebRTC scenario, there is also a solution that can enable a single upstream video stream for screen sharing, which is called “Substream” . Auxiliary stream sharing means that the sharer publishes two pictures of the camera picture and the screen picture at the same time.

In addition, with this auxiliary stream channel, when the device is a new version…

The post WebRTC series of video streaming first appeared on Lenix Blog .

This article is reprinted from https://blog.p2hp.com/archives/8706
This site is for inclusion only, and the copyright belongs to the original author.