Online video live broadcast means that people can watch the live audio and video happening at the remote end through the Internet, such as events, conferences, teaching, surgeries, etc. The core idea of online video live broadcast is to use existing local area network, metropolitan area network and even world-wide network conditions to realize real-time transmission of audio and video signals, and to achieve smooth viewing at the remote end!
A complete live broadcast process includes but is not limited to the following links: collection, processing, encoding, packaging, streaming, transmission, transcoding, distribution, decoding, playback, etc.
1. Collection
Collection is the first link in the entire video streaming process. It obtains original video data from the system's collection device and outputs it to the next link. Video collection involves the collection of two aspects of data: audio collection and image collection, which respectively correspond to two completely different input sources and data formats.
Audio collection: The audio collection process mainly collects ana-log signals in the environment into PCM-encoded raw data through equipment, and then encodes and compresses the data into MP3 format for distribution. Common audio compression formats are: MP3, AAC, HE-AAC, Opus, FLAC, Vorbis (Ogg), Speex and AMR, etc.
Image acquisition: The image acquisition process is mainly captured by cameras and other equipment into YUV-encoded raw data, which is then encoded and compressed into H.264 format data for distribution. Common video encapsulation formats include: MP4, 3GP, AVI, MKV, WMV, MPG, VOB, FLV, SWF, MOV, RMVB and WebM, etc.
2. Processing
After collecting video or audio, obtain raw data. In order to enhance some live effects or add some additional effects, we usually process the code before compression, such as adding time or company LOGO watermark, beauty freckles and sound obfuscation processing. In the scene where the host and the audience are connected to each other, the anchor needs to have a conversation with one or more viewers and share the conversation results with all other viewers in real time. Part of the processing of the broadcast is also done on the streaming side.
Video processing: beauty, watermark, filter, customization
Audio processing: mixing, noise reduction, sound effects, customization
3. Coding
For streaming media transmission, encoding is very important. Its encoding performance, encoding speed and encoding compression ratio directly affect the user experience and transmission cost of the entire streaming media transmission.
Common video encoders:
1)H.264/AVC
2)HEVC/H.265
3)VP8
4)VP9
5)FFmpeg
Audio encoder: Mp3, AAC, etc.
4. Packaging
Packaging can be understood as what kind of truck is used to transport it, that is, the container of the media. The so-called container mixes and encapsulates the multimedia content (video, audio, subtitles, chapter information, etc.) generated by the encoder.
Several common packaging formats:
1)AVI format (suffix is .avi)
2)DV-AVI format (suffix is .avi)
3) QuickTime File Format format (suffix is .mov)
4) MPEG format (the file suffix can be .mpg .mpeg .mpe .dat .vob .asf .3gp .mp4, etc.)
5) WMV format (suffix is .wmv .asf)
6)Real Video format (suffix is .rm .rmvb)
7) Flash Video format (suffix is .flv)
8)Matroska format (suffix is .mkv)
9)MPEG2-TS format (suffix is .ts)
Currently, we mainly use FLV and MPEG2-TS formats for streaming media transmission, especially live broadcasts, which are used for RTMP/HTTP-FLV and HLS protocols respectively.
5. Push flow
Push streaming refers to the process of transmitting live content to the server using content capture software such as push streaming tools.
There are three main push protocols:
RTSP (Real Time Streaming Protocol): Real-time streaming protocol, a multimedia streaming protocol used to control sound or video, jointly proposed by Real Networks and Netscape;
RTMP (Real Time Messaging Protocol): Real Time Messaging Protocol, an open protocol developed by Adobe for audio, video and data transmission between Flash players and servers;
HLS (HTTP Live Streaming): It is an HTTP-based streaming media transmission protocol implemented by Apple Inc.;
RTMP is the current mainstream streaming media transmission protocol and is widely used in the field of live broadcast. Most of the live broadcast products on the market adopt this protocol.
The RTMP protocol is based on TCP and is a network protocol designed for real-time data communication. It is mainly used for audio, video and data communication between the flash/AIR platform and the streaming media/interactive server that supports the RTMP protocol. Software that supports this protocol includes Adobe Media Server/Ultrant Media Server/red5, etc.
It has three variants:
RTMP is a clear text protocol that works on top of TCP, using port 1935;
RTMPT is encapsulated in HTTP requests and can traverse firewalls;
RTMPS is similar to RTMPT, but uses an HTTPS connection;
The RTMP protocol is like a container used to hold data packets. These data can be data in AMF format or video/audio data in FLV. A single connection can carry multiple network streams through different channels. Packets in these channels are transmitted in fixed-size packets.
6. Transmission
The pushed streaming media needs to be transmitted to the audience, and the entire link is the transmission network.
7. Transcoding
The bit rate of the live video streaming end is determined by the push end, that is, the bit rate of the live streaming end is consistent with the bit rate of the push end. However, the live broadcast effect will be poor when encountering the following scenarios:
The bitrate of the streaming end does not match the bandwidth of the streaming end. When the bit rate of the streaming end is high and the client's bandwidth resources are limited, playback will freeze. When the bit rate of the streaming end is low but the client has high requirements for live broadcast efficiency, the playback effect will be poor.
The player plug-in needs to implement multi-bitrate switching. Front-end player plug-ins can often set bitrate switching, which requires that the same push stream can provide streaming addresses of multiple bitrates at the same time.
Therefore, the live video broadcast provides a real-time transcoding function to provide services to the same push address and multiple stream addresses with different bit rates at the same time.
8. Distribution
The role of the streaming media server is to be responsible for the release and rebroadcast distribution functions of live streams.
9. Decoding
Encoder: a device or program that compresses signals
Decoder: a device or program that decompresses signals
Codec: codec
10. Play
Mainly to realize the display of live programs on the terminal. Because the transmission protocol used here is RTMP, any player that supports the RTMP streaming protocol can use it, for example:
PC: VLC, etc.
Mobile version: Vitamio and ijkplayer, etc.
