Getting the most out of QtMultimediaKit on mobile

Gareth Stockwell

Accenture

• It’s not just a management consultancy ...• I work for a division called Accenture Mobility

– Provides Mobility aspects of end-to-end solutions– Work on many mobile/embedded platforms

• Android, iOS, MeeGo, Symbian, Windows Phone ...

– In a wide range of industry sectors• Healthcare, automotive, retail, consumer electronics ...

– Some of us even get to hack on Qt

About me

• Based in London, UK• 6 years in embedded software

– Mostly on Symbian– Mostly developing multimedia adaptations, lately some

graphics

• 2 years developing Symbian port of Qt– Phonon MMF backend– QAudio backend– QtMultimediaKit, mostly around video rendering

Conventions

• Source code for the demos shown is available at

$DD11/examples/foo

• Logos show which platform(s) a topic or example applies to

https://gitorious.org/garethstockwell-qtdevdays-2011/mainline

• Path within the repository is shown as

Overview

• QtMultimediaKit provides cross-platform abstractions– So why do you need to care whether you are targeting a

mobile device?– Where are the limitations and leaks in those

abstractions?

• How can you tune your code so that it works well in the constrained mobile environment?

Mobile considerations

Latency

Less processing power

Reliance on co-processors

Multiple connectivity options

Limited memory

Cool demos

Low latency processing

Access point control

Memory management and

monitoring

Portable – always in your pocket

Achieving low latency

Sources of latency

Resourceallocation

Buffering Playback

time

Media playback use case

Real-time use cases

Media playback: startup latency

Resourceallocation

Buffering Playback

NoMedia Loading

Loaded Buffering Buffered

audio.source = “foo.mp3” audio.play()

MediaStatus

m_audio->setMedia(...);

• Startup latency can be minimised by pre-loading media sources where possible

• Buffering latency is not controllable via QtMultimediaKit high-level APIs

QML

Native m_audio->play()

Media playback: startup latency

Demo $DD11/audioplayer

Pure QML app

Low level audio APIs

QIODevice

QAudioInput

QIODevice

QAudioOutput

QIODevice *QAudioOutput::start();

void QAudioOutput::start(QIODevice *device);

Push

Pull

Easy to use – no need to implement QIODevice

Requires an extra buffer copy from client data into the audio stack

+

-

Best choice for low-latency processing

Real time audio latency

Buffering Playback

time

Impulsee.g. User input

Real-time data generation

void QAudioOutput::setBufferSize(int value);

• Keep buffer size to a minimum– But beware of underflow

• This is controllable via an API

– But this isn’t supported on Symbian– Here, you can control the buffering instead via

the source QIODevice

qint64 QIODevice::bytesAvailable() const;

Real time audio latency

Demo $DD11/qpiano

Hybrid QML / C++ app

Guess what it does ...

Network streaming

Access point selection

• By default, the access point will be chosen by the platform– May involve popping up a dialog for the user

• Apps may want programmatic control– For application-specific reasons (e.g. video chat protocol

won’t work well over laggy cellular networks)– To simplify the UX

QList<QNetworkConfiguration> QNetworkConfigurationManager::allConfigurations(...) const;

void QMediaPlayer::setNetworkConfigurations( const QList<QNetworkConfiguration> &configurations);

QNetworkConfiguration QMediaPlayer::currentNetworkConfiguration() const;

Access point selection

Demo $DD11/networkplayer

Hybrid QML / C++ app

No access point selection QML

bindings provided in QtMultimediaKit

Network configuration stateUndefined

Defined

Discovered

Active

Memory managementand monitoring

GPU memory constraints

• Current S^3 devices have only 32MB graphics memory– This is shared by the graphics subsystem (GLES and VG

engines) and the multimedia subsystem (video and camera)

– Combination of heavy graphics use cases (e.g. storing large QPixmaps or textures on the GPU) with MM (e.g. playback of high-resolution video clips; starting camera viewfinder) can quickly exhaust the memory

• It can be difficult to understand whether a given bug is due to GOOM– There is an API via which GPU memory usage can be

queried– There are some strategies which can be used to

minimise GPU memory usage

• The good news: the next generation of S^3 devices, starting with the C7-01, have 128MB

Memory usage monitoring

• EGL_NOK_resource_profiling– Provides total memory, total used memory, and per-

process usage– API is not very friendly (particularly when compared to

Qt APIs)– So I wrote a Qt wrapper

class GraphicsMemoryMonitor : public QObject{ Q_PROPERTY(qint64 totalMemory READ totalMemory NOTIFY totalMemoryChanged) Q_PROPERTY(qint64 usedMemory READ usedMemory NOTIFY usedMemoryChanged) Q_PROPERTY(qint64 currentProcessUsage READ currentProcessUsage NOTIFY currentProcessUsageChanged) ...};

$DD11/snippets/graphicsmemorymonitor

Memory usage monitoring

Total usage

Current process usage *

Total memory

• Not all memory allocations are correctly tagged– Here, the 3.84MB is the EGL window surface– Memory consumed by the video decoder is not

represented– So the only reliable measurement is the total usage

Demo $DD11/videoplayer

m_camera->setCaptureMode(QCamera::CaptureModeVideo);

Mimimising camera memory usage

Demo $DD11/viewfinder

• The image capture ISP firmware module consumes several MB– So if you only need a viewfinder, switch

to video capture mode

– You may also need to lower the viewfinder resolution

QMediaRecorder *mediaRecorder = new QMediaRecorder(m_camera);QVideoEncoderSettings videoSettings = mediaRecorder->videoSettings();videoSettings->setResolution(QSize(320, 240));mediaRecorder->setEncodingSettings(audioSettings, videoSettings);

Advanced video rendering

Cool stuff alert

Video post-processing with GLSL

• If QAbstractVideoSurface returns texture handles, we can use QGLShaderProgram to apply effects

• From Qt 4.7.4, this becomes much easier, as we can embed GLSL directly in QML via ShaderEffectItem

ShaderEffectItem { property variant source: ShaderEffectSource { sourceItem: anItem; hideSource: true } property real granularity: 10 property int targetSize: 256

fragmentShader: " uniform highp float granularity; uniform sampler2D source; uniform highp float targetSize; uniform lowp float qt_Opacity; varying highp vec2 qt_TexCoord0; void main() { vec2 uv = qt_TexCoord0.xy; float dx = granularity / targetSize; float dy = granularity / targetSize; vec2 tc = vec2(dx*(floor(uv.x/dx) + 0.5), dy*(floor(uv.y/dy) + 0.5)); gl_FragColor = qt_Opacity * texture2D(source, tc); }"}

Video post-processing with GLSL

• Possible use cases– Video playback transition effects (e.g. pixellated fade-in)– Camera ‘ageing’, applied both to viewfinder and captured

images

Demo $DD11/qmlvideofx

Pure QML app (!)

Video and Qt3D

• Qt3D provides the Material abstraction, which can be a wrapper around a texture– But at present, it doesn’t abstract

texture streams

• With a bit of C++, we can provide the glue which pipes QAbstractVideoSurface output into a Qt3D material– Requires definition of some new

QML elements

import QtQuick 1.0import QtQuick3D 1.0import VideoTexture 1.0import TextureStream 1.0

Rectangle { width: 600 height: 600

VideoTexture { id: video source: "test.mp4” }

Viewport { Cube { id: cube } }

TextureStream { source: video target: cube }}

Video and Qt3D

Demo Source not available yet, sorry...

Hybrid QML / C++ app

C++ provides the glue which sticks

QtMultimediaKit to Qt3D

Summary

• Low latency– Startup– Steady state

• Network access point control

• Graphics memory usage– Monitoring– Reduction

• Advanced video rendering– Shader effects– 3D

Feedback

Remember to send your session feedback via the Qt Developer Days app

Get the app by• Tapping one of the NFC tags on the event

floor• Downloading the “Qt Developer Days” app

from Nokia Store• Downloading from

qt.nokia.com/qtdevdays2011• Visiting m.qtdevdays2011.qt.nokia.com to

use web version

Questions?Thanks for listening