WebKit and GStreamerBringing lolcatz videos to your desktop

Xabier Rodríguez CalvarIgalia

Who is this dude?

Born in the Basque CountryGrown and living in Galicia

Studied Computer Science at Universidade da CoruñaPartner and developer at Igalia

··

One of the seven Celtic Nations, like Scotland-··

Working in Multimedia and WebKit-

From html5demos.com

Index

Quick review of WebKitWebKit MultimediaPower features

···

WebAudioHardware accelerationAccelerated video compositing

---

5/36

Quick Review

WebKit

Web content engineProcesses and renders web contentStarted as a fork of KTHML and KJSOpen source since 2005Goals: open source, compatibility, compliance, stability,performance, security, portability, usability, hackabilityNon-goals: being a full web browser, being a science project,having reusable components, unlimited scopeForked by Google as BlinkSplit into ports: GTK+, Qt, EFL, Mac, Chromium, Windows

·····

·

··

7/36

Architecture

8/36

WebKitGTK+

Each WebKit port is composed of

WebKitGTK+ platform layer:

API layer is a GtkWidget and a set of GObject APIsWebKitGTK+ is used by Epiphany, Midori, yelp, devhelp

·Platform interfacesAPI layer

--

·libsoup for networkingcairo for rasterizationOpenGL for making the scene graph and WebGLGStreamer for mediaVarious GTK+ APIs for talking with the system

-----

··

9/36

WebKit2

Give the web rendering parts of WebKit their own process

Sandbox web rendering

·Page crashes don't crash the browserCan put vulnerable data into a separate address space

··

·Prevent pages from accessing the disk and operating systeminterface

·

10/36

WebKit2 Architecture

11/36

The Near Future

12/36

WebKit Multimedia

WebKit Multimedia

Mac port: QuickTime, QtKit, AVFoundationsApple Win port: QuickTimeGTK+ and EFL ports: GStreamer 1.0 backendBlackberry port: Bridges over to its own Media playback frameworkChromium: bridges over to WebKit/Chromium backend whichrelays to FFMPeg implementationsQt port: GStreamer on Linux, QtKit on Mac, QuickTime onWindows

·····

·

14/36

WebKit Multimedia Architecture

WebKitWebCore

MediaPlayer

Platform- - - - - - - - - - - -

GStreamer15/36

GStreamer MediaPlayer - Overview

Mainly developed and tested on Linux platformSuccessfully ported to MacOS/Win platfoms (not fully upstreamyet though)Playbin-based: the best GStreamer element for playback use-cases

MediaPlayer registers and uses two WebKit-specific GStreamerelements: webkitwebsrc and a video-sink

··

·

Playbin2 with GStreamer 0.10-·

16/36

GStreamer MediaPlayer - Main Features

On-disk-bufferingFrame-accurate seekingBasic metrics reportingTrick-modesWebAudioHardware decoding support with VA-API (gst 1.2.x)Video accelerated compositingCodec installer support

········

17/36

WebKit and GStreamer Architecture

-mediaPlayer

MediaPlayerClient

+playbackChanged()+volumeChanged()

-mediaPlayerClient

-m_privateHTMLMediaElement+play()+playbackChanged()+setVolume(in volume:float)+volumeChanged()

MediaPlayer+play()+setVolume(in volume:float)

GstPlaybin -playbinHTMLVideoElement HTMLAudioElement

MediaPlayerPrivateInterface

+play()+setVolume(in volume:float)

MediaPlayerPrivateGStreamerBase

+setVolume(in volume:float)

MediaPlayerPrivateGStreamer+play()

18/36

GStreamer MediaPlayer - HTTP source element

a GStreamer bin implementing the URI handler interface forhttp(s):// and blob:// uris.A single source pad provides raw data access to downstreamelements.appsrc is used internally to interface with the WebCoreSubResourceLoader objectElement is registered with high rank in the GStreamer registry soplaybin is more prone to pick it up to read http:// URIs.

·

·

·

·

19/36

GStreamer MediaPlayer - Video rendering

Element inheriting from GstVideoSink

Takes video GstBuffers as inputPass those to the MediaPlayerPrivateGStreamer via a signal

·In practical sense, it is a GstAppSink with video capabilitiesnegotiation

-

··

20/36

GStreamer MediaPlayer - Audio rendering

Use autoaudiosinkCheck for pulseaudio for playback request messagesPitch preservationPreserve audio passthrough when possibleVolume managed by the sink

·····

21/36

GStreamer MediaPlayer - Next steps

Improvements and bug fixes in the player:

MediaStream (getUserMedia) and WebRTC spec implementationMediaSource adaptive streaming API spec implementationIntegration of WebAudio with media elements and mediastreamstackFinish porting to 1.2Remove GStreamer 0.10 code path

·Store on-disk buffered media in WebCore's cache directoryMake the webkitwebsrc element private to WebKit

--

···

··

22/36

WebKit Multimedia Power Features

WebAudio

A JavaScript API allowing to create a new generation of Webgames and applicationsEffects, convolution engine for room simulation, etcRealtime analysis, playback and visualizationW3C spec driven and implemented mainly by Chromium andFirefoxIn WebKit, currently implemented for:

·

···

·MacChromiumGStreamer based Ports: GTK+, EFL, Qt

---

24/36

WebAudio GStreamer - Reading Audio

Spec allows two ways to read audio data:

Pipeline: src ! decodebin2 ! audioconvert ! audioresample !capsfilter ! deinterleave ! queue ! appsinkdeinterleave separates multi-channel audio to n audio monochannels (Pipeline above can contain multiple queue ! appsinkchains)appsink used to collect mono audio buffers, store them in a list andconvert data to floating point valuesData is stored in a WebCore::AudioBus, containing one or multiplechannel Float arrays

·from memory: we can use giostreamsrc GStreamer elementfrom a file path: we can use the filesrc GStremer element

--

·

·

·

·

25/36

WebAudio GStreamer - Playing Audio

The WebAudio stack processed or generated some audio data to besent to the sound cardSimilarly to the "Reading Data" step, a WebCore::AudioBus is usedto store the dataCustom GStreamer element to convert an AudioBus to aninterleaved audio data flow: webkitwebaudiosrcwebkitwebaudiosrc: a bin with n internal pads pushing to aninterleave element. Source pad outputs a single, interleave, wav-encoded data flowPipeline: webkitwebaudiosrc ! wavparse ! audioconvert !autoaudiosink

·

·

·

·

·

26/36

WebAudio GStreamer - Work in Progress

Some layout tests don't pass yetNot yet enabled by default in release buildsSupport for stereo+ streams in the reader side. Playback sidealready handles those.Integration with <video> and <audio> elements: WebAudioMediaElementSourceNode

···

·

27/36

Hardware Accelerated Video

We use GStreamer VA-API. (1.2.x)That's it! Hardcore comes when compositing that.

··

28/36

Accelerated Video Compositing - Acceleration inWebKit

From Joone Hur

29/36

Accelerated Video Compositing

Element inheriting from GstVideoSink

Takes video GstBuffers as input.Video frame is transferred to the MediaPlayerPrivateGStreamer viaa GLib signal.MediaPlayerPrivateGStreamer uploads it to a GL texture andrenders it asynchronously in the WebKit context.

·In practical sense, it is a GstAppSink with video capabilitiesnegotiation

-

··

·

30/36

Accelerated Video Compositing - Architecture

GstPlaybin -playbin

MediaPlayerPrivateInterface

MediaPlayerPrivateGStreamerBase

+paint()

MediaPlayerPrivateGStreamer

GstElement*

WebKitVideoSink <<creates>>

TextureMapperPlatformLayer+paint()

...

31/36

Accelerated Video Compositing - Rendering paths

Software rendering path

Accelerated compositing paths

·It uses Cairo-

·TextureMapperLayer:

TextureMapperGL

-Light-weight implementation of a scene-graphThe TextureMapperLayer tree is equivalent to theGraphicsLayer tree.

--

-Provides a GPU accelerated implementation of thedrawing primitives, using shaders compatible with GL/ES2.0

-

32/36

Accelerated Video Compositing - Next steps

More video decoders with GstVideoGLTextureUploadMeta supportSupport EGLImage in WebKit as video frames handlers

Handle the vertical blank interval in WebKit

··

And video decoders that deliver this kind of capabilities.-·

GTK+ 3.8 seems to fix this issue.-

33/36

From html5demos.com

From html5demos.com

35/36

Thank You!(q&a)

twitter @calvariswww blogs.igalia.com/xrcalvar