Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one...
Transcript of Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one...
![Page 1: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/1.jpg)
VCA-Bridge
Using GStreamer for Video Analytics
Julian Bouzas Gonzalez
![Page 2: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/2.jpg)
Who I am● Software Developer● From Galicia (Spain)● Working at VCA Technology (London)
● Video Analytics Software forVideo Surveillance
Galicia
![Page 3: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/3.jpg)
Video Analytics Software (VCA5)Focused on Intrusion Detection and Counting
![Page 4: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/4.jpg)
VCA-Bridge
● Home-made device that provides an all-in-one solution for analysing video streams from different sources.
![Page 5: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/5.jpg)
Use Cases
VCA-Bridge
Video in(RTSP)
Video withmetadata out
(HLS)
Alerts withsnapshots out
(SMTP)
Video withmetadata out
(RTSP)
![Page 6: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/6.jpg)
Firmware● Build with:
○ Linux Kernel○ Busybox○ Glibc○ Systemd○ Gstreamer 1.8.3 (with VAAPI plugins)
● Analytics Software:○ VCA-Core (home-made video analytics web server)
![Page 7: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/7.jpg)
Hardware
● Limitations○ 4 channels @ 1080p 30fps○ 8 channels @ 720p 30fps○ 16 channels @ D1 30fps
● Specifications:○ 4GB of RAM○ CPU Intel Core i3-5010U
■ 2 Cores■ 3M Cache■ 2.10 GHz
○ GPU Intel HD Graphics 5500
![Page 8: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/8.jpg)
Third Party Devices● Porting the firmware to many different devices:
○ Ambrella S2○ HiSilicon 3516○ Raspberry Pi○ TI Davinci series○ Qualcomm Snapdragon
● Need to write hardware encoding/decoding gstreamer plugins for those devices
![Page 9: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/9.jpg)
Web UI● Polymer framework● HLS for video streaming● No plugins needed● Supported on multiple browsers:
○ Chrome○ Firefox○ Safary○ Edge○ Internet Explorer
![Page 10: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/10.jpg)
Demo
![Page 11: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/11.jpg)
Gstreamer Pipeline
VCA5 BIA Tee RTSP
Snapshot
HLSAnalytics section
Output section
Input section
Input Decodebin
![Page 12: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/12.jpg)
Input Section
● Decoders (Decodebin):○ RTSP (H264 and Jpeg)○ File (AVI, MP4, MKV...)
● Gstreamer VAAPI pluginsfor hardware acceleration
rtspsrc
filesrc
DecodebinRaw video
buffersOr
![Page 13: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/13.jpg)
Analytics Section● VCA5 element
○ Reads video frame
○ Adds metadata usingthe libvca5 library
● BIA element○ Reads the metadata
from the coming frame
○ Draws metadatain the frame
VCA5 gstreamer
plugin
BIAgstreamer
plugin
Video buffers with metadata
Raw video
buffers from input
To output
tee
Metadata outFrame in
libvca5
![Page 14: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/14.jpg)
Output Section
● Encoders (on-demand):○ H264 (HLS and RTSP server)○ JPEG (Snapshots)
● Gstreamer VAAPI pluginsfor hardware acceleration
OutputTee
H264enc
H264enc
JPEGenc
HLS
RTSPserver
Snapshot
Processed video
frames
![Page 15: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/15.jpg)
Issues we faced● HLS stops generating segments when seeking
○ Something wrong with the keyframes and multifilesink○ Solved by stopping and playing again the pipeline
● RTSP Deadlock (metadata?)○ Seems to only happen with RTSP sources coming from VLM with metadata
● Pipeline might not play again (decodebin?)○ Playing -> Null -> Playing -> Null -> Playing …○ Only with RTSP sources○ Solved by destroying decodebin and creating it again
![Page 16: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/16.jpg)
Future plans● TCP events● Logic rules● Multi-channel rules● Improve latency (Use webRTC instead of HLS)● Improve performance
○ Split VCA5 plugin into different plugins (1 per algorithm)○ Share encoders
● Desktop notifications● Improve stability
![Page 17: Using GStreamer for Video Analytics · VCA-Bridge Home-made device that provides an all-in-one solution for analysing video streams from different sources.](https://reader033.fdocuments.us/reader033/viewer/2022042412/5f2c32bf24df8e2c467c5b47/html5/thumbnails/17.jpg)
http://www.vcatechnology.com
Thanks!