1

Yossi CohenLecture atGoogle Technology User Group Tel-Aviv

Cross Platform Computer Vision Optimization

2

The next Instagram is in video processing / Computer Vision

And You Should do it!

3

Computer Vision Application Types

Gestures

Text Recognition

Augmented Reality

Depth Mapping

CV with active IR camera

4

Conflicting RequirementsCross Platform SolutionRun on All Devices

Code Once

Low cost maintenance / update

Platform SpecificUtilize all SoC capabilities for:

Fast processing / fast response

Low Power requirements

5

Conflicting DevelopmentCross Platform DevelopmentHTML5

Java

Platform Specific Development SIMD Optimization (ASM)

Use platform specific GPU, DSP

Use Platform Specific HW accelerators:

CODECs

Rotators

Color Space Convertors…

Doesn’t work for Computer Vision (Yet)

6

Possible Solutions • Too much Power Consumption• Too SloooowwwwDon't Optimize

• Best performance for a single platform (market leader SoC)• Lose (50%+) market share

Optimize for one platform (SoC)

• Good Performance for all ARM platforms• Lose MIPS, X86 Market• Lose GPU, DSP and HW specific acceleration capabilities

Optimize for ARM NEON Only

• Development Costs• Knowledge problem• Fragmented Code, high update & maintenance costs

Optimize for all platforms

7

Optimize for one processor architecture Select a Processor based on Target Market:

For Android its ARM

Optimize for SIMD InstructionNEON Optimization (Alternatively SSE or 3DNow)

Advantages~x1-x8 Acceleration (depending on function)Fit ~95%+ of Android Market

DisadvantagesNot Suited for x86 & MIPSDoes not utilize 100% of SoC capabilities:

Internal DSP GPU HW Accelerators VFU

ARM NEON Optimization Unutilized

8

Optimize for a Single Processor Select a Single Processor based on Target Market:

8960 - the fastest processor 250 Design wins

Optimize NEON

Optimize DSP

Optimize for GPU

AdvantagesYoull have the fastest app on

the best most widely used processor

DisadvantagesDevelopment Timeneed to support inferior/legacy processor as well

Optimized Optimized

CPU

DSP

GPU

VeNum

9

The Optimization Engineer Nightmare:

Selecting between two sub-optimal solutions

Isn’t there someone that will solve this in a better way?

10

The Cross-Platform CV Vision

11

KhronosStandardization organizationGenerates OPEN, Royalty free API (unlike Oracle)

for Cross HW softwareMost Known API – OpenGL In Android:OpenGL ESOpenMAX OpenSL

12

Khronos Vision of Cross Platform Computer Vision

Application LayerSensory Input

OpenCVHigh Level Algorithm

OpenVLIntegration Layer

OpenCL DSP, HW Accelerators, GPU

Camera Input Video Out

13

OpenVL Integration API for Computer Vision

(like OpenGL for graphic )

implements computer vision primitives

14

So there is a cross platform solution

All we have to do is wait 5-7 years for market adaptation…..

If only there was a solution which is both optimized for ARM NEON and for the fastest CPU in the market

15

One Development Toolkit – Two Implementations

CPU

DSP

GPU

Neon

CPU

DSP

GPU

VeNum

FastCV for SnapdragonFastCV for ARM

16

Fast CV Overview Fast CV is an API & library which enables Real-Time Computer Vision (CV)

applications.

FastCV enables mobile devices to run CV applications efficiently.

FastCV allows developers to HW accelerate their CV application.

FastCV is analogous to OpenGL ES in the rendering domain

FastCV is a clean modular library.

17

FastCV Architecture

Video CoreAdreno GPU Hexagon ConnectivitySensors etc

Ha

rdw

are

Ke

rne

l

Display Drivers Camera Drivers

SnapdragonCPU Core (s)

ARCV

A

pp

lic

ati

on

s

Gestures Facial Recognition Other

QC AugmentedReality

QC GestureProcessing

QC FacialRecognition

Op

tim

ize

dF

ram

ew

ork

FastCV Snapdragon FastCV ARM

Computer Vision APIs

Facial Recognition APIsGestures APIsAugmented Reality APIs Defined API

3rd Party CV Frameworks

18

FastCV 1.0 – Feature Grouping Math / Vector Operations

Commonly used vector & math functions

Image processing Image filtering, convolution and scaling operations

Image transformation Warp perspective, affine transformations

Feature detection Fast corner detection, harris corner detection, canny edge detection

Object detection NCC based template matching object detection functions.

3D reconstruction Homography, pose evaluation functions

Color conversion Commonly used formats supported: e.g., YUV, RGB, YCrCb, etc.

Clustering and search K clusters best fitting of a set of input points

19

Industry Computer Vision Solutions FastCV is a processor-core agnostic acceleration API

Khronos is looking to provide a standard CV API Potentially utilizing portions of OpenCV

FastCV will evolve as Khronos standard is defined

Hardware Abstraction Layer

Open source reference implementation

Hardware vendor implementations

High-level CV algorithms library

Media interface

Application

FastCV Hardware Acceleration API

FastCV for ARM (Reference

implementation)

HW Specific Implementations

FastCV for Snapdragon

FastCV for Nvidia

FastCV for Intel

FastCV for Others…

20

FastCV Compared To OpenCV

Function OpenCV FastCVFastCV

Snapdragon

NCC 1.0x 9.0x 23.1x

Dot Product 128x4 1.0x 4.0x 10.0x

Convert YUV420 1.0x 1.4x 1.3x

Sobel 1.0x 1.8x 7.8x

Median3x3 1.0x 3.8x 51.9x

Gaussian3x3 1.0x 2.6x 4.1x

Gaussian5x5 1.0x 1.4x 2.9x

Threshold 1.0x 0.7x 9.7x

Integral Image 1.0x 1.1x 1.3x

Harris Corner 1.0x 2.8x 8.6x

Dilate 1.0x 1.4x 15.0x

Erode 1.0x 1.3x 15.0x

Perspective Fit 1.0x 21.5x 37.8x

LK Optical Flow 1.0x 2.0x 14.3x

21

Gain Is More Than Time Measure CPU frequency along with times

Utilize single CPU in Linux performance mode Legend: CPU Frequency Long algorithm time Short algorithm time

22

ReferencesMore on

OpenMAX http://www.slideshare.net/DSPIP/openmax-overview

OpenCL http://www.slideshare.net/DSPIP/opencl-programming-101OpenSL http://www.slideshare.net/DSPIP/android-audio-opensl

Download FastCV https://developer.qualcomm.com/develop/mobile-technologies/computer-vision-fastcv/g

etting-started-guide

23

Thank you!

Video Expert

Lectures on Video / Android / VoIP

Android Native Developer

More About me:

Yossi Cohenyossicohen19@gmail.comhttp://www.mobilevideotech.com

+972-545-313092