Embedded Streaming Media with GStreamer

and BeagleBoard

ESC-321Presented by Todd Fischer

todd.fischer@ridgerun.com

Agenda

•Introduction to BeagleBoard multimedia architecture features.

•GStreamer concepts

•OMAP GStreamer hands on exercises

•DMAI and GStreamer

•OpenMAX and GStreamer

•Questions

Introduction

•Primary objective for this class is to introduce audience to GStreamer on OMAP3 with hands on exercises

•This doesn’t class won't cover extensively the APIs available and assumes basic knowledge on C programming, Linux and computer graphics.

BeagleBoard and OMAP3 architecture

•Provides architecture with several multimedia features:

•Cortex A8 with Neon

•C64x DSP

•Video accelerators

•How to utilize the hardware features with the software stack?

BeagleBoard and OMAP3 architecture

•Provides architecture with several multimedia features:

•Cortex A8 with Neon

•C64x DSP

•Video accelerators

•How to utilize the hardware features with the software stack?

BeagleBoard and OMAP3 architecture

•Provides architecture with several multimedia features:

•Cortex A8 with Neon

•C64x DSP

•Video accelerators

•How to utilize the hardware features with the software stack?

GStreamer•Streaming media framework – audio and

video

•Close to 200 plug-ins available

•Higher level than just input / filters / output

Networking, audio/video mixed streams, auto data handling

•Various options utilizing hardware accelerators

GStreamer Overview•Elements

•Sources, filters, sinks

•Bins and Pipelines

•Containers, pipeline is the overall bin

•Pads

•Element source / sink connection points

•Caps

•Capabilities organized by stream type with a set of properties

GStreamer Overview•Elements

•Sources, filters, sinks

•Bins and Pipelines

•Containers, pipeline is the overall bin

•Pads

•Element source / sink connection points

•Caps

•Capabilities organized by stream type with a set of properties

•Plugin•Collection of elements

Simple MP3 Player•Create dynamically using gst-launch

•Source element reads from a file

•Filter element converts MP3 to PWM

•Sink element passes to ALSA output

gst-launch filesrc location=a.mp3 ! mad ! alsasink

filesrc mad alsasink

location=music.mp3

src sink sinksrc

pipeline

Hands On Exercise 0

•Start up Windowing environment with two terminals

cd gst

. src

Period character to source the filee

Hands On Exercise 0

•Play audio file

a2•Actual command

gst-launch filesrc

location=bbb.flac !

flacdec ! alsasink

Simple Audio PlayerSource Code

•Create pipeline, source, filter, sink

•Set element properties

•Build into pipeline

•Connect src and sink pads

•Setup pipeline event handler

•End of stream

•Set pipeline state to play

•Run

Simple PA System•Create dynamically using gst-launch

•Source element ALSS audio in

•No filters

•Sink element passes to ALSA output

gst-launch alsasrc num-buffers=100 ! alsasink

alsasrc alsasinksrc sink

pipeline

Keeping Plug-ins Organized

•Each known plug-in is added to registry

•Most aspects of plug-in are tracked in the registry

•Registry support run-in pipeline creation and dynamic filter selection

•Use gst-inspect to list plug-ins

gst-inspect | lessgst-inspect filesrc

Hands On Exercise 1

•Using gst-inspect, list

•All plug-ins

•All video plug-ins

•Element properties for filesrc plug-in

Hands On Exercise 2•GStreamer video pipelines

gst-launch videotestsrc ! ffmpegcolorspace ! fbdevsink

PerformanceData Passing

•Stream held in buffers with data, timestamp, other info

•When possible, buffer memory allocated by sink pad

•Use hardware when data copy is necessary

PerformanceData Transformation•Cortex A8 compiler optimization

•NEON

•Single Instruction Multiple Data

•C64

•Video accelerator

•DMA and other data movers

Davinci MultimediaApplication Interface

AR

M C

orte

x w

ith N

EO

N

GStreamer Libraries

gstplug-in

gstplug-in

DMAIplug-in

Davinci MultimediaApplication Interface

audio video displayframecopy

Codec Engine

VISA OSAL CMEM

DSP Link

Cortex Linux Application

IVA

2 w

ith C

64 D

SP

video hardware accelerator

DSP Link

DSP BIOS

Framework Components

codec codecvideocodec

DMAI and GStreamer

•Davinci Multimedia Application Interface

•Exposes OMAP/Davinci hardware using high level of abstraction

•Stream audio / video

•Graphics display

•Hardware optimized frame/data copy

OMAP3 Codec Engine•Isolates users for audio/video codecs

from those implementing the codecs

•Codec can run in several places without calling application being aware

•Cortex A8, NEON, C64, hardware accelerator

•Uses DSPLink and DSPBios conventions to support DSP based algorithms dynamically

Video Phone

•Vision – video phone application can create audio and video GStreamer pipelines for capture / rendering

•DMAI, Codec engine, DSPLink, C64 algorithms perform heavy lifting

•OMAP3 hardware capabilities utilized

•Video phone application focused on call management, not streaming data manipulation

OpenMAX and GStreamer

•Addresses codecs using hardware accelerators

•GStreamer integration via gst-openmax

•All OpenMax codecs available as GStreamer plug-ins

•Initial focus on high performance codecs

Application LayerDefines high-level playback

and recording API

Hardware

Multimedia Application

Integration LayerDefines media component interfaces

Development LayerDefines media primitives

and concurrency constructs

OpenMAX and OMAP3

•ARM Inc has Cortex-A8 / NEON DL implementation

•Texas Instruments has AAC and MPEG4 IL / DL support

Application LayerDefines high-level playback

and recording API

Hardware

Multimedia Application

Integration LayerDefines media component interfaces

Development LayerDefines media primitives

and concurrency constructs

GStreamer in3 Layers

API for Super Fast Hardware

GStreamer Media Handling

Your Way Cool Application