Mobile Apps quality - a tale about energy, performance, and users’ perception

Ivano Malavolta

VRIJEUNIVERSITEITAMSTERDAM


Ivano MalavoltaAssistant professor

Vrije Universiteit Amsterdam

Mobile Apps quality -a tale about energy,

performance, and users’ perception

Ivano Malavolta


iOS developer

Who’s speaking?

2009

Android developer2010

Cross-platform developer2011

Instructor on mobile apps developmentfor 5 years2012


2014

2016Experiments at theGREEN LAB

Empirical research on hybrid apps

2017

Ivano Malavolta


The Green Lab

Our platform for researching on software– energy efficiency– performance

Students measure real software solutions

OpenSTF+ ADB

Serveweb pages

VMWare manager

SSH

MEASURES

Wifinetwork

Router

Experimentorchestrator

OpenSTF

Web interface

Power meters

A PLATFORM

A COURSE

Ivano Malavolta


Roadmap

Qu

est

ions Deep dive into mobile apps development

Who is speaking?

Energy consumption (of progressive web apps)

User perceptions (of hybrid apps)

Performance (of native apps)

Ivano Malavolta


Deep dive into mobile apps development

Ivano Malavolta (2016). Web-based hybrid mobile apps: state of the practice and research opportunities. In Proceedings of the International Conference on Mobile Software Engineering and Systems, MOBILESoft '16

Ivano Malavolta (2016). Beyond Native Apps: Web Technologies to the Rescue! (Keynote). In Proceedings of the 1st International Workshop on Mobile Development (Mobile!)

Ivano Malavolta


Let’s burst the first myth

htt

p:/

/w

ww

.slid

esh

are

.ne

t/fl

ing

/n

ativ

e-v

-hyb

rid-v

-we

b

Mobile apps development IS NOT CHEAP

Ivano Malavolta


NATIVE APP

010101010101011010101010101011011010010101010101011101010101010101011010

PLATFORM APIs

Native

Development strategies

Ivano Malavolta


• rich user interfaces and heavy graphics• work offline

• visibility via stores

• slow iteration pace for iOS (stores mediation)• no indexing by search engines

• and.....

PROS

CONS

Native apps

NATIVE APP

010101010101011010101010101011011010010101010101011101010101010101011010

PLATFORM APIs

Ivano Malavolta


Fragmentation

Obj-Ccode

Swiftcode

XCode

Javacode

C++code

Eclipse

C#code

C++code

Visual Studio

JScode

Ivano Malavolta


NATIVE APP

010101010101011010101010101011011010010101010101011101010101010101011010

PLATFORM APIs

Native


BROWSER

<html><head><script src=” ...” /></head><body>...

Web

Ivano Malavolta


• fast development• simple maintenance and updates• cross-platform

• 2-steps access– they are launched via a browser

• do not work offline• no background activities

– geofencing

• poor access to system APIs– push notifications, contacts, etc.

Web apps

PROS

CONS

Ivano Malavolta


BROWSER


NATIVE WRAPPER


PLATFORM APIs

NATIVE APP

010101010101011010101010101011011010010101010101011101010101010101011010

PLATFORM APIs

Native WebHybrid


Ivano Malavolta


Web-based hybrid mobile apps

Single code base

Bridge API

Ivano Malavolta


Web-based hybrid apps

• cross-platform portability

• reuse of existing knowledge of web developers

• simpler and less expensive development process

• inherit some cons of native apps• restricted access to system APIs • (slight) decrease in performance

PROS

CONS

Ivano Malavolta


BROWSER


NATIVE WRAPPER


PLATFORM APIs

NATIVE APP

010101010101011010101010101011011010010101010101011101010101010101011010

PLATFORM APIs

Native WebHybrid


Progressive

BROWSER<html><head><script src=” ...” /></head><body>...

Service workers

Ivano Malavolta


The journey of a PWA

1. The user accesses the website asusual

https://goo.gl/KIZydg

Ivano Malavolta




2. After the 3rd-4th visit, the website asks to be installed


Ivano Malavolta





3. The user can decide to add the app to the home screen


Ivano Malavolta



1. The user accesses the website as usual


3. The user can decide to add the app to the home screen

4. From now on, the app is top-level, full-screen, and can receive push notifications


Ivano Malavolta


Components of a PWA

HTTPSWeb

ManifestServiceworkers

{"name": "MyApplication","short_name": "MyApp","scope": "./webApp/","orientation": "portrait","display": "fullscreen","background_color": "#fff","description": "A simple application for testing.","icons": [{"src": "images/touch/homescreen48.png","sizes": "48x48","type": "image/png"

}, ...}],...}

Ivano Malavolta


Service workers

Implemented in JavaScript

Multithreading→ runs in a separate thread w.r.t. the main thread

Used for:• push notifications

• background operations

• content caching– offline functionality

Ivano Malavolta


Offline-first experience

Web AppServiceworker

response

response

request

Backend

Offline cache

this.addEventListener('fetch', function(event) {event.respondWith(

caches.match(event.request).then(function(response) {return response || fetch(event.request);

}););

});

Ivano Malavolta


Energy consumption (of progressive web apps)

Ivano Malavolta, Giuseppe Procaccianti, Paul Noorland, Petar Vukmirovic. Assessing the Impact of Service Workers on the Energy Efficiency of Progressive Web Apps. In Proceedings of the International Conference on Mobile Software Engineering and Systems, MOBILESoft 2017,

Ivano Malavolta


PWAs have been advertised as:

• performance boosters

• network savers

• providers of better UX

However…How does the use of service workers impact the energy efficiency of PWAs under different network conditions?

How does the use of service workers impact the energy efficiency of PWAs?RQ1

RQ2

Ivano Malavolta


PWAs selection • Real PWAs from the pwa.rocks1 repository

• Pseudo-random selection– no toy examples

– data-driven PWAs (aka no videogames)

1 https://pwa.rocks

Name Category Total size

SW size (loc)

Ali Express Shopping 2.1Mb 69

Google I/O 2016 Events 4.2Mb 358

The Washington Post News 4.0Mb 85

Flipkart Shopping 3.8Mb 907

Babe News News 1.2Mb 156

Wiki offline Knowledge 800Kb 1009

The Billings Gazette News 2.1Mb 60

Ivano Malavolta


Experiment design

What we measure

Energy consumption of the device in Joules

Variable name Treatments

SW status <on, off>

Android device <high-end, low-end>

Network condition <2G, WiFi>

What we control

What we measure

Ivano Malavolta


Experiment design

• full 2x2x2 factorial à all possible combinations of treatments

• 8 combinations x 7 PWAs x 8 runs à

• each run executes a typical usage scenario (10-15 gestures)

448 runs

SW status Device Network

on low-end 2G

on low-end WiFi

on high-end 2G

on high-end WiFi

off low-end 2G

off low-end WiFi

off high-end 2G

off high-end WiFi

Ivano Malavolta


7a. HTTP requests as part of the scenario

Mobile device

Server

Measurement infrastructure

Orchestration script

Monkeyrunner

Trepnprofiler

Chrome

Monkeyrunner

1. HTTP requests impersonating phone

2. HTTP responses (recorded)

9. save collecteddata

3. Start experiment run

5. start 6. start

7b. HTTP responses (possibly altered)

8. collect data

Fiddlerproxy

HostedPWA

ADB

OS

4. start scenario

run

Ivano Malavolta


Results - devices

Overall energy consumption across devices

Low-end High-end

High difference across devices

Ivano Malavolta


Results: SW-on VS SW-off

Low-end High-end

Smaller difference in the high-end device

Service workers DO NOT influence the energy consumption of a PWA running on a mobile device

Ivano Malavolta


Results – network conditions

High-end

+ 2G

Low-end

+ 2G

Low-end

+ WiFi

High-end

+ WiFi

PWAs consume less energy on WiFi

Same device + same network condition à low impact of SWs

Ivano Malavolta


PWA-specific trends

Low-end

2G

WiFi

High-endDifferent PWAsà differentimpact of SWs

Same PWA à SWs have a different impact under differentconditions

- - - - + + +

+

-

+

Ivano Malavolta


How does each SW look like?

Ivano Malavolta


Results of manual review

No specific trend here

Name Listened events

Caching Obfuscation/minification

Complexity

Ali Express P, N ✓ ✓ 12

Google I/O 2016 I, A, F, M ✓ 9

The Washington Post I, A, P, N ✓ 131

Flipkart I, A, F ✓ ✓ 5

Babe News I, A, F, P, N ✓ 16

Wiki offline I, A, F, S, M, N ✓ ✓ 7

The Billings Gazette I, A, F ✓ 194

Ivano Malavolta


Conclusions

Ivano Malavolta


PWAs have been advertised as:

• performance boosters

• network savers

• providers of better UX

However…

Ivano Malavolta


7a. HTTP

requests as part

of the scenario

Mobile device

Server



Monkeyrunner

Trepnprofiler

Chrome

Monkeyrunner

1. HTTP requests

impersonating phone


9. save

collected

data


5. start 6. start

7b. HTTP

responses

(possibly altered)

8. collect data

Fiddlerproxy

HostedPWA

ADB

OS

4. start

scenario

run

Ivano Malavolta


Results: SW-on VS SW-off

Low-end High-end

Smaller difference in the high-end device

Service workers DO NOT influence the energy consumption of a PWA running on a mobile device

Ivano Malavolta


Performance (of native apps)

Teerath Das, Massimiliano Di Penta, Ivano Malavolta. A Quantitative and Qualitative Investigation of Performance-Related Commits in Android Apps. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME 2016)

Ivano Malavolta


Goal

Quantitative and qualitative characterization of (documented) performance-related commits

for Android apps

Developer

?App

Ivano Malavolta


Research questions

What are the concerns that developers have when dealing with performance issues of Android apps?

To what extent developers consider performance issues of Android apps?RQ1

RQ2

Ivano Malavolta


Dataset?

Ivano Malavolta


Dataset

Ivano Malavolta


How to identify performance-related commits?

Ivano Malavolta


Results (RQ1)Out of 68,025 commits of 2,443 apps, we discovered a

total of 457 performance-related commits (0.67%) spread across 180 apps

Ivano Malavolta


Open card sorting

– 2 people performed the task, a third one checked the results

1st phasecommits tagged based on keywords e.g., read from file system, swipe lag

2nd phasecommits clustered into meaningful, labeled groups

Ivano Malavolta


Results (RQ2)

Ivano Malavolta


User interface

• Applied solutions:– use of Android’s recycled views instead of plain list views– render images in slices– ...

Problems related to swipe lags, screen layout drawing, lists scrolling responsiveness

FIX layouts for better rendering performance

Ivano Malavolta


Code smells

• Applied solutions:– use more efficient regular expressions– avoid recurrent computations of constant data– avoid usage of deprecated decryption algorithms– ...

Symptoms of poor design and implementation choices

fixed string concatenation performance issue

Ivano Malavolta


Memory

• Applied solutions:– stopping auxiliary services when available memory gets low– avoiding to load potentially unused data– ...

Related to the memory footprint of the app

Fixed major memory leak; should improve responsiveness on older devices

Ivano Malavolta


Local database access

• Applied solutions:– perform queries in an asynchronous task– add indexes to specific fields– ...

Access to local database can be highly inefficient

Added indexes for post.blogID to improve performance of several lookups

Ivano Malavolta


As a developer I can exploit this emerging collective knowledgefor improving the performance of MY mobile app

Ivano Malavolta


User perceptions (of hybrid apps)

Ivano Malavolta, Stefano Ruberto, Tommaso Soru, Valerio Terragni. Hybrid Mobile Apps in the Google Play Store: An Exploratory Investigation. 2nd ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft 2015).

Ivano Malavolta, Stefano Ruberto, Valerio Terragni, Tommaso Soru. End Users’ Perception of Hybrid Mobile Apps in the Google Play Store. IEEE International Conference on Mobile Services (MS 2015).

Ivano Malavolta


Pros

• cross-platform portability

• reuse of existing knowledge of web developers

• simpler and less expensive development processes

Hybrid mobile apps

Cons

• restricted access to hardwarefeatures

• decrease in performance

• variations on user experience

As of today, limited empirical investigations have been performed on hybrid mobile apps

Strong debate about benefits and drawbacks

Ivano Malavolta


Research goal

DeveloperEnd users

createsdownload

& use

App

What is the difference between hybrid and native mobile apps as perceived by end users?

Are hybrid mobile apps published in the Google Play Store? What are their main traits?S1

S2

Ivano Malavolta


Data extraction

Classified apps(hybrid vs native)

Hybrid appsclassifier*

top-500 most popular free apps for each category of the Google Play Store

~11k app binariesApps and

reviewsmining

* analysis tool available here: http://github.com/GabMar/ApkCategoryChecker

Ivano Malavolta


Results

Data-intensive mobile apps[2]

Apps with closer interactionwith the Android platform

Winners, in line withinformal claims[3,4,5]

Ivano Malavolta


Quick test

Ivano Malavolta


Data extraction

Classified apps(hybrid vs native)

Hybrid appsclassifier*

Reviewsanalyzer

top-500 most popular free apps for each category of the Google Play Store

~11k app binaries

50 pages (~255) of reviews for each app

~3M userreviews

apps scores

Apps and reviewsmining

perceived value: 0.5users sentiment: 0.6#reviews: 243performance: 0.6bugginess: 0.1size: 3,456 kb* analysis tool available here:

http://github.com/GabMar/ApkCategoryChecker

Ivano Malavolta


Reviews analysis

Stopwordsremoval

manually performed by 2 domain experts

Single review

Single reviewscore

polaritypos: 0.8 performancepos: 0.6polarityneg: 0.1 performanceneg: 0.05bugginess: 0.2

300 random reviews

Keywordsextraction

Relevant keywords

Lemmatization

Tf-idf basedvectors similarity

computation

Ivano Malavolta


Perceived valueAverage of the ratings as provided by end users

3.35 3.75

Rating = real number in [1, 5] Certain balance, with neglectable differences

Ivano Malavolta


Perceived valuePolarity of sentiment of end users

where posa = #positive reviewsnega = #negative reviews

Balance between hybrid and native apps, with some exceptions

Non data-intensive or requiring multimedia capabilities

Ivano Malavolta


Perceived valueAverage review count

where Ra∈ ℕ

Native apps have been reviewed in average 6.5 times more than hybrid mobile apps

Possible interpretation: hybrid mobile apps are neither perceived as too satisfying nor dissatisfying w.r.t. native ones

Ivano Malavolta


Perceived performance

where posa = #positive reviews w.r.t. performance of the app nega = #negative reviews w.r.t. performance of the app

Balance between hybrid and native apps, with some exceptions

Ivano Malavolta


Perceived bugginess

where buga = #reviews signalling the presence of bugs or failuresreviewsa = #reviews of the app

The highest unbalance between the two development strategies in our study

bugginessa = buga / reviewsa

Possible interpretation: absence of full-fledged testing frameworks for hybrid apps, such as those provided by native apps IDEs like Eclipse and Android Studio

Ivano Malavolta


Results – initial download size

6,586 kb4,625 kb

In line with the average size of Android apps

sizea = file size in kilobytes of the app APK file

Ivano Malavolta


Data extraction Classified apps

(hybrid vs native)

Hybrid apps

classifiers

Reviews analyzer

top-500 most popular free apps for

each category of the Google Play Store

~11k app binaries

50 pages (~255) of

reviews for each app

~3M app

reviews

apps scores

Apps and reviews mining

perceived value: 0.5

users sentiment: 0.6

#reviews: 243

performance: 0.6

bugginess: 0.1

size: 3,456 kb

Apache Cordova (258) and Appcelerator Titanium (116)

Developers reuse JavaScript frameworks for the desktop web

End users value hybrid and native apps similarly

Hybrid may be good for data-intensive apps, whereas it performs poorly when dealing with low-level, platform-specific features

Reviews analysis

Stopwords

removal

manually performed

by 2 domain experts

Single review

Single reviewscore

polaritypos: 0.8

performancepos: 0

.6

polarityneg: 0.1

performanceneg: 0

.05

bugginess: 0.2

300 random

reviews

Keywords

extraction

Relevant keywords

Lemmatization

Tf-idf based

vectors similarity

computation

Ivano Malavolta


Wrap up

Ivano Malavolta


Ivano Malavolta


7a. HTTP

requests as part

of the scenario

Mobile device

Server



Monkeyrunner

Trepnprofiler

Chrome

Monkeyrunner

1. HTTP requests

impersonating phone


9. save

collected

data


5. start 6. start

7b. HTTP

responses

(possibly altered)

8. collect data

Fiddlerproxy

HostedPWA

ADB

OS

4. start

scenario

run

Ivano Malavolta


Reviews analysisStopwords

removal

manually performed by 2 domain experts

Single review

Single reviewscore

polaritypos: 0.8 performancepos: 0.6polarityneg: 0.1 performanceneg: 0.05bugginess: 0.2

300 random reviews

Keywordsextraction

Relevant keywords

Lemmatization

Tf-idf basedvectors similarity

computation

Ivano Malavolta


Open card sorting

– 2 people performed the task, a third one checked the results

1st phasecommits tagged based on keywords e.g., read from file system, swipe lag

2nd phasecommits clustered into meaningful, labeled groups

Ivano Malavolta


BROWSER


NATIVE WRAPPER


PLATFORM APIs

NATIVE APP

010101010101011010101010101011011010010101010101011101010101010101011010

PLATFORM APIs

Native WebHybrid


Progressive

BROWSER<html><head><script src=” ...” /></head><body>...

Service workers

Ivano Malavolta


Contact

Ivano Malavolta | Vrije Universiteit Amsterdam

iivanoo

[email protected]

www.ivanomalavolta.com

A few of the contents of this presentation are from co-authors’ slides

Mobile Apps quality - a tale about energy, performance, and users’ perception

Software

Transcript of Mobile Apps quality - a tale about energy, performance, and users’ perception