A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very...

18
A First Look at Deep Learning Apps on Smartphones Mengwei Xu, Jiawei Liu, Yuanqiang Liu, Felix Xiaozhu Lin, Yunxin Liu,Xuanzhe Liu Presented: Rahul Bagchi

Transcript of A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very...

Page 1: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

A First Look at Deep Learning Apps on Smartphones

Mengwei Xu, Jiawei Liu, Yuanqiang Liu, Felix Xiaozhu Lin,

Yunxin Liu,Xuanzhe Liu

Presented: Rahul Bagchi

Page 2: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Content

▶ Introduction

▶ Motivation, background, and the problem

▶ Solution

❖ Core idea and the technique used

❖ The workflow of the analyzing tool

❖ Characteristics of early adopter apps using Deep Learning

❖ Deep Learning Frameworks used by the apps

❖ How are the Deep Learning capabilities used in the apps

❖ Deep Learning models used by the apps

❖ Model Resource Footprint

❖ Security and optimization of the Deep Learning models

▶ Findings and Conclusion

2

Page 3: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Introduction

▶ This presentation explains in a simple way the Smart-Device research

article - A First Look at Deep Learning Apps on Smartphones

▶ The article analyses 16,500 Android apps dataset and focuses on apps

that embrace Deep Learning, and the Deep Learning frameworks and

models used in practice

▶ The article aims to bridge the knowledge gap between research and

practice to find out:

❖ Characteristics of early adopter apps using Deep Learning

❖ Deep Learning Frameworks used by the apps

❖ How are the Deep Learning capabilities used in the apps

❖ Deep Learning models used by the apps

❖ Security and optimization of the Deep Learning models3

Page 4: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Introduction

▶ The article findings have direct implications to:

❖ App developers,

❖ Mobile Deep Learning framework developers and hardware

engineers/ vendors

❖ Mobile Deep Learning researchers

▶ To find answer to the article problem the authors uses a software tool

built by them

▶ The tool indirectly sniffs Deep Learning usage of mobile apps by

detecting the use of Deep Learning frameworks

4

Page 5: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Motivation, background, and the problem

▶ Deep Learning has revolutionized many Artificial Intelligence tasks

particularly in computer vision and Natural Language Processing (NLP)

▶ The huge smartphones market provides very promising platform for Deep

Learning based apps

▶ Year 2017 marked noticeable increased use of Deep Learning for

smartphones

▶ Around the same time major vendors like Google, Facebook, Apple and

Baidu launched their mobile Deep Learning frameworks like TFLite,

Caffe2, Core ML and MDL

▶ These frameworks provided affordable on-device rendering of the Deep

Learning inference

5

Page 6: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Motivation, background, and the problem

▶ To bridge the knowledge gap between research and practice, the article

presents the first empirical study on 16,500 popular Android apps to find

out how Deep Learning is used by smartphone apps based on the queries

below:

❖ Characteristics of early adopter apps using Deep Learning

❖ Deep Learning Frameworks used by the apps

❖ How are Deep Learning capabilities used in the apps

❖ What are the popular Deep Learning models used by the apps

❖ Security and optimization of the Deep Learning models

6

Page 7: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Solution

Core idea and the technique used

▶ To automate the analysis of numerous Android apps, a new analyzer tool

was built by the authors

▶ The tool inspects app installation packages, identifies the apps that use

Deep Learning, identify Deep Learning functions and the usage of known

Deep Learning frameworks, and extracts the Deep Learning models from

these apps for inspection

▶ The analysis focuses on apps that embrace Deep Learning, and the Deep

Learning frameworks and models used

▶ The overall workflow of the analyzing tool is depicted in the next slide7

Page 8: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Solution

8

The workflow of the analyzing tool

Page 9: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Characteristics of early adopter apps using Deep Learning

9

Solution

Deep Learning (DL) based apps are significantly more popular than non DL apps

Page 10: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Deep Learning Frameworks used by the apps

10

Solution

Page 11: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

How are the Deep Learning capabilities used in the apps

11

Solution

Page 12: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Deep Learning models used by the apps

Convolution Neural Network containing convolution and pooling layers

12

Solution

▶ Among the DL models extracted

❖ 87.7% models are CNN models

❖ 7.8% models are RNN models

❖ Other models were not confirmed at that time

Page 13: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Deep Learning models used by the apps

Recurrent Neural Network

▶ Recurrent neural network processing sequential inputs

Wikipedia

13

Solution

Page 14: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Model Resource Footprint

14

Solution

▶ Deep Learning models are very lightweight on memory usage and computation

complexity, with median value of 2.47 MB and 10M FLOPs respectively

▶ Running such models on mobile processors is inexpensive

▶ Findings of dominant lightweight Deep Learning models on smartphones:

❖ Deep Learning inference can be as cheap as a few MBs of memory overhead and

❖ Tens of ms execution delay

Page 15: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Optimization of the Deep Learning models

15

Solution

▶ Most DL models lacks optimizations despite availability of well-known

optimizations, e.g. quantization

▶ Quantization can significantly reduce Deep Learning cost with little accuracy loss

▶ The study finds only 6% of models has such optimizations

Page 16: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Security of the Deep Learning models

16

Solution

▶ DL models are poorly protected

▶ The study finds that only 39.2% discovered models are obfuscated and

only about 19.2% models are encrypted

▶ The remaining models were easy to extract and therefore

vulnerable to unauthorized reuse

Page 17: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Findings and Conclusion

▶ The authors carried out the first large-scale study of 16,500 Android

mobile apps to find out how the apps exploit Deep Learning

▶ Findings of the empirical study projects a promising picture of Deep

Learning for smartphones:

❖ showing the prosperity of mobile Deep Learning frameworks available

❖ the prosperity of apps building their cores on top of Deep Learning

❖ on-device Deep Learning inference is fast becoming popular due to stronger

privacy protection, resilient against poor Internet connectivity, and lower

cloud computing cost

❖ massive scope for optimizations on Deep Learning models

❖ huge potential for expanding the footprint of protection for Deep Learning

models

❖ validation of research ideas on these models17

Page 18: A First Look at Deep Learning Apps on Smartphones · The huge smartphones market provides very promising platform for Deep Learning based apps Year 2017 marked noticeable increased

Thank You

18