1

Android™ User-Space Tests Multimedia codec tests, status and open discussion

Ashok Bhat

System & Software, ARM

LCU14 – 502

September 2014

2

Use-case - Android Media Codec - AArch64 Port - Validation

Android user-space tests and benchmarks - Discussion

Do we need more?

How to create?

How to share?

How to deploy and maintain?

Agenda

3

Android Core Media Support

Type Format / Codec Encoder Decoder

Audio/ Speech

AAC LC • •

HE-AACv1 (AAC+) • •

HE-AACv2 (enhanced AAC+) •

AAC ELD (enhanced low delay AAC) • •

AMR-NB • •

AMR-WB • •

FLAC • •

MP3 •

Vorbis •

PCM/WAVE •

Video

H.263 • •

H.264 AVC • •

MPEG-4 SP •

VP8 • •

4

Application and Native Media Framework Interaction

Media Codecs

libstagefright.so

Application Framework

android.media.*

Binder IPC Proxies (frameworks/av/libmedia)

Media Player Binder MediaPlayerService.cpp

Media Player Service

(frameworks/av/media/libmediaplayerservice)

OMX Core

MediaSource.cpp

Stagefright Engine

(frameworks/av/libstragefright)

Native Multimedia Framework

(frameworks/av)

OpenMax Integration Layer

OMX Plug-in

Codec Implementation

OpenMAX IL Component

SOURCE : https://source.android.com/devices/media.html

5

Android Media Codecs

6

Codecs external to Android

Active upstream projects

Projects reside in external directory

Audio – AAC, FLAC, Vorbis

Video – VP8, VP9

Typically have validation suite outside Android

Codecs present only in Android

No upstream project

Projects reside in frameworks/av/media/libstagefright/codecs

Speech – AMR-NB, AMR-WB

Audio – MP3

Video – H.263, H.264, MPEG4

Rely on CTS for validation

Android Media Codecs

7

Decoder and Encoder need to conform to a standard

For example, MP3 decoder conforms to ISO/IEC 13818-3 and ISO/IEC 11172-3

For a decoder, standard provides a set of conformance clips and a reference decoder

For an encoder, standard may provide a set of input clips and a test criteria

Decoder test mechanism

Decode conformance test clips

Compare output with the output generated by the reference decoder

Output should be either bit-exact or within a specified range when compared to reference output

Encoder test mechanism

Encode input for different combination of encoding parameters

Check if encoded output conforms to the standard

Typical codec validation

8

Compatibility Test Suite

Part of Android compatibility program

Runs on a desktop machine and executes test cases directly on attached devices or an emulator

Does not claim to be comprehensive

Media codec validation

Part of android.media.cts package

Present in cts/tests/tests/media/src/android/media/cts directory

Tests are Java programs that run on top of the Android software stack

Android CTS

9

Need a method to validate AArch64 port

CTS is a good starting point. But...

Not meant to be comprehensive

Runs on top of Android stack – tricky to use for benchmarking component

Need something in addition to CTS that would

Allow validation/benchmarking/optimization at the component level.

Allow validation of conformance test vectors – preferably file I/O based.

Allow tests to be part of larger testing framework like LAVA.

Media Codec - AArch64 Port - Validation

10

New component level tests developed

By ARM and Linaro

For Video, Audio and Speech components

For encoder and decoder

Mostly file based tests – tests use file input and generate file output

Tests can run independent of Android – Does not require Android Software Stack

Allows porting/optimization/testing of components outside Android

Available in AOSP Gerrit and Linaro Android user-space git repo

Patches pushed to AOSP Gerrit – Not completely merged yet

Part of Linaro Android user-space git repo

Run as part of LAVA for each Linaro Android release

Media Codec - AArch64 Port – Validation (Contd...)

11

Media Codec Test Patches in AOSP Gerrit

12

Tests in Linaro Android user-space git

Repo for Android user-space tests and programs

Link

https://android.git.linaro.org/gitweb/platform/external/linaro-android-userspace-test.git

Contents

13

Android supports extensive list of media codecs

A significant set of codecs exist only in Android and do not have an upstream project

Codec validation requires exhaustive conformance tests as specified by standard

CTS is a good starting point but not the right place for comprehensive component level

validation

Codec component level tests were developed by ARM and Linaro

Test patches are available in AOSP Gerrit

Test code, media files and test script are also available in Linaro Android user-space git

repo

Tests are run as part of each Linaro Android release

Summary

14

Do we need more such component/subsystem level tests?

How to create them?

How do we share them?

Will Android Linaro git repo meet your requirement? Do you need anything different?

How to maintain them?

Is running part of Linaro Android release sufficient?

Anything else that can be done differently?

Discussion

15

Thank you!