surroundings
Introduction
The previous article introduced Streamlit , an open source library that can quickly create web applications, but streamlit itself is not good at processing real-time audio and video. This article introduces streamlit ‘s extension library, streamlit-webrtc , which can help us easily process and transmit real-time audio and video.
Install
Quick install using pip command
pip install -U streamlit-webrtc
Example
The basic use of streamlit-webrtc is very simple, similar to the usage of other components of streamlit , except that there is one more parameter key for unique identification
# app.py from streamlit_webrtc import webrtc_streamer # 这里的key代表唯一标识webrtc_streamer(key="sample")
Then, you can start the service, and the startup method remains the same
streamlit run app.py
After the service starts, the browser automatically opens the link http://localhost:8501/
Click START to start reading camera data
Click STOP to turn off the camera
In many cases, we need to read audio and video and process it. At this time, we can use the callback of audio and video. Let’s see an example of video processing.
from streamlit_webrtc import webrtc_streamer import av def video_frame_callback(frame): img = frame.to_ndarray(format="bgr24") # 图像翻转flipped = img[::-1,:,:] return av.VideoFrame.from_ndarray(flipped, format="bgr24") # 如果是音频的话，使用audio_frame_callback webrtc_streamer(key="example", video_frame_callback=video_frame_callback)
The pyav library is used in the example, which is based on ffmpeg . The frame in the callback function video_frame_callback is an instance of av.VideoFrame (audio is pyav ) in av.AudioFrame , and the image is flipped for each frame of data. The final effect is Such
Next, combine the user input with the processing of real-time video in combination with the checkbox component in streamlit
import streamlit as st from streamlit_webrtc import webrtc_streamer import av # 选择框flip = st.checkbox("Flip") def video_frame_callback(frame): img = frame.to_ndarray(format="bgr24") # 根据用户的选择来判断视频帧是否要进行翻转flipped = img[::-1,:,:] if flip else img return av.VideoFrame.from_ndarray(flipped, format="bgr24") webrtc_streamer(key="example", video_frame_callback=video_frame_callback)
It should be noted here that the callback is executed in the thread from the fork , so the data transfer between the inside and outside of the callback must be thread-safe. In the callback , the components in streamlit , such as streamlit.title , cannot work, and the global keyword cannot meet your data processing expectations, so don’t use it.
References
- https://github.com/streamlit/streamlit
- https://github.com/whitphx/streamlit-webrtc
- https://xugaoxiang.com/2021/08/27/yolov5-streamlit/
- https://github.com/PyAV-Org/PyAV
This article is reprinted from https://xugaoxiang.com/2022/10/30/streamlit-webrtc/
This site is for inclusion only, and the copyright belongs to the original author.