Kyungmin Lee*, Jason Flinn*, T.J. Giuli+, Brian Noble*, and Christopher Peplin+

University of Michigan* Ford Motor Company+

AMC: Verifying User Interface Properties for Vehicular Applications

Where do we use mobile apps?

2Kyungmin Lee

Walking Driving Everywhere we go(Not-too-distant future)

Where do we use mobile apps?

3Kyungmin Lee

Walking Driving Everywhere we go(Not-too-distant future)

Mobile apps in a vehicle

4

Pros:- High usability standard- Good form factorCons:- Cost of deployment

- Limited applications

Pros: - Diverse applications- Easy deploymentCons:- Distracting apps- Poor form factor

Kyungmin Lee

Mobile apps in a vehicle

5

Pros:- High usability - Good form factor

Pros: - Diverse apps- Easy deployment

Kyungmin Lee

How can we get the best of both worlds?How can we get the best of both worlds?

Our approach

• Create a tool (AMC) that– Automatically explores the application– No source code & app-specific knowledge– Pinpoints UI guideline violations

6Kyungmin Lee

AMC’s benefits

• Developers– Test own apps– Early design feedback

7

• User interface experts– Fewer properties to check– Focus on hard properties

Kyungmin Lee

AMC’s benefits

• Developers– Test own apps– Early design feedback

8

• User interface experts– Fewer properties to check– Focus on hard properties

Kyungmin Lee

<AMC’s results>20X reduction in testing

False negative: 2%False positive: 0%

<AMC’s results>20X reduction in testing

False negative: 2%False positive: 0%

Outline

• Motivation• How AMC explores the app• How AMC finds UI violations• How closely do apps follow UI guidelines• Conclusion

9Kyungmin Lee

How to use AMC

10

Run AMC Get violation report

Violation reportButton size:1) Button A at state 1 2) …Contrast ratio:1)…2)…

Violation reportButton size:1) Button A at state 1 2) …Contrast ratio:1)…2)…

Kyungmin Lee

- State exploration

- Find violations

- State exploration

- Find violations

ApplicationApplication

MonkeyMonkey

Hierarchy Viewer

Hierarchy Viewer

Modified ApplicationFramewo

rk

Modified ApplicationFramewo

rk

App to test

How does AMC explore?

11

State #0

Kyungmin Lee

How does AMC explore?

12

State #0

Current Location

State #1

Kyungmin Lee

How does AMC explore?

13Kyungmin Lee

How do we define a state

• Different screens -> Different states

14Kyungmin Lee

vs.

Fine-tuning state definition

• Naïve approach: every unique screen -> state

• Solution: Structure hash– Same structure, different content -> same state– Avoids state explosion

15

vs.

Kyungmin Lee

Fine-tuning state definition

• Structure hash is not enough…

• Solution: Consider only 0,1,or many items– If there are 2+ items, ignore the rest– Unless they transition to different states

16

vs.

Kyungmin Lee

Outline

• Motivation• How AMC explores the app• How AMC finds UI violations• How closely do apps follow UI guidelines• Conclusion

17Kyungmin Lee

Finding UI violations

• Checks for 7 UI properties– User actions per task, Text contrast ratio,

Word count, Button size, Button distance,Animation, Scrolling

– Defined by transportation research groups– Report violation, no violation, or defer to experts

18Kyungmin Lee

Finding UI violations

• Properties of individual state

19Kyungmin Lee

Button size

Button closeness

Text contrast ratio

Word count

Animation

Scrolling

Finding UI violations

• Properties of individual state

20Kyungmin Lee

Button size ✔

Button closeness ✔

Text contrast ratio

Word count

Animation

Scrolling

Finding UI violations

• Properties of individual state

21Kyungmin Lee

Button size ✔

Button closeness ✔

Text contrast ratio ✖

Word count

Animation

Scrolling

Finding UI violations

• Properties of individual state

22Kyungmin Lee

Total of 12 words

Button size ✔

Button closeness ✔

Text contrast ratio ✖

Word count ✔

Animation

Scrolling

Finding UI violations

• Properties of individual state

23Kyungmin Lee

Button size ✔

Button closeness ✔

Text contrast ratio ✖

Word count ✔

Animation ✔

Scrolling ✔

Checking user actions per task

24Kyungmin Lee

Few complications

• Buttons in custom view– Empirically discover buttons– If fails, defer to manual verification

• Determining good vs. bad animation

25Kyungmin Lee

vs.

Outline

• Motivation• How AMC explores the app• How AMC finds UI violations• How closely do apps follow UI guidelines• Conclusion

26Kyungmin Lee

Evaluation

• Environmental setup

27Kyungmin Lee

Comparison with expert’s evaluation

28

Application Word count Button size Button distance

Animation Scrolling

Google Finance MyDays

eBay

Twitter

GasBuddy

Google Nav.

TomTom Nav.

Waze

iRadar

Beat the Traffic

Best Parking

MPG

Kyungmin Lee

Tested 12 out of 14 apps &

5 out of 7 properties

Tested 12 out of 14 apps &

5 out of 7 properties

Comparison with expert’s evaluation

29

Application Word count Button size Button distance

Animation Scrolling

Google Finance MyDays

eBay Twitter

GasBuddy

Google Nav.

TomTom Nav.

Waze

iRadar

Beat the Traffic

Best Parking

MPG

Kyungmin Lee

: Violation : No violation

: Maybe

1st column:AMC’s report

2nd column: Expert’s report

Comparison with expert’s evaluation

30

Application Word count Button size Button distance

Animation Scrolling

Google Finance MyDays

eBay Twitter

GasBuddy

Google Nav.

TomTom Nav.

Waze

iRadar

Beat the Traffic

Best Parking

MPG

Kyungmin Lee

: Violation : No violation

: Maybe

1st column:AMC’s report

2nd column: Expert’s report

Comparison with expert’s evaluation

31

Application Word count Button size Button distance

Animation Scrolling

Google Finance MyDays

eBay Twitter

GasBuddy

Google Nav.

TomTom Nav.

Waze

iRadar

Beat the Traffic

Best Parking

MPG

Kyungmin Lee

: Violation : No violation

: Maybe

1st column:AMC’s report

2nd column: Expert’s report

Comparison with expert’s evaluation

32

Application Word count Button size Button distance

Animation Scrolling

Google Finance MyDays

eBay Twitter

GasBuddy

Google Nav.

TomTom Nav.

Waze

iRadar

Beat the Traffic

Best Parking

MPG

Kyungmin Lee

: Violation : No violation

: Maybe

1st column:AMC’s report

2nd column: Expert’s report

Agreed 50 / 60 cases≈ 83%

Agreed 50 / 60 cases≈ 83%

Comparison with expert’s evaluation

33

Application Word count Button size Button distance

Animation Scrolling

Google Finance MyDays

eBay Twitter

GasBuddy

Google Nav.

TomTom Nav.

Waze

iRadar

Beat the Traffic

Best Parking

MPG

Kyungmin Lee

: Violation : No violation

: Maybe

1st column:AMC’s report

2nd column: Expert’s report

One case of false negative<≈ 2%

One case of false negative<≈ 2%

Comparison with expert’s evaluation

34

Application Word count Button size Button distance

Animation Scrolling

Google Finance MyDays

eBay Twitter

GasBuddy

Google Nav.

TomTom Nav.

Waze

iRadar

Beat the Traffic

Best Parking

MPG

Kyungmin Lee

: Violation : No violation

: Maybe

1st column:AMC’s report

2nd column: Expert’s report

Comparison with expert’s evaluation

35

Application Word count Button size Button distance

Animation Scrolling

Google Finance MyDays

eBay Twitter

GasBuddy

Google Nav.

TomTom Nav.

Waze

iRadar

Beat the Traffic

Best Parking

MPG

Kyungmin Lee

: Violation : No violation

: Maybe

1st column:AMC’s report

2nd column: Expert’s report

Animation: 4 out of 381 statesButton size & distance: 13 states

Reduced by 95%!

Animation: 4 out of 381 statesButton size & distance: 13 states

Reduced by 95%!

Assessment of Android apps

36

Application User actions per task

Text contrast

ratio

Word count Button size Button distance

Animation # Item to scroll

Gmail

Google Finance

MyDays

eBay

Twitter

GasBuddy

Google Maps

Google Nav.

TomTom Nav.

Waze

iRadar

Beat the Traffic

Best Parking

MPG

Kyungmin Lee

Assessment of Android apps

37

Application User actions per task

Text contrast

ratio

Word count Button size Button distance

Animation # Item to scroll

Gmail 17 2 2 0 0 0 3

Google Finance 0 1 3 1 4 0 3

MyDays 20 4 2 0 2 0 1

eBay 0 1 1 0 0 0 2

Twitter 24 28 7 0 14 0 9

GasBuddy 12 14 1 0 13 0 4

Google Maps 8 7 3 0 7 0 1

Google Nav. 0 0 0 2 1 0 0

TomTom Nav. 26 0 6 1 0 0 6

Waze 72 15 0 1 5 0 4

iRadar 0 2 1 5 4 0 0

Beat the Traffic 5 7 0 0 1 0 0

Best Parking 0 0 0 0 1 0 1

MPG 0 0 0 0 0 0 0

Kyungmin Lee

Assessment of Android apps

38

Application User actions per task

Text contrast

ratio

Word count Button size Button distance

Animation # Item to scroll

Gmail 17 2 2 0 0 0 3

Google Finance 0 1 3 1 4 0 3

MyDays 20 4 2 0 2 0 1

eBay 0 1 1 0 0 0 2

Twitter 24 28 7 0 14 0 9

GasBuddy 12 14 1 0 13 0 4

Google Maps 8 7 3 0 7 0 1

Google Nav. 0 0 0 2 1 0 0

TomTom Nav. 26 0 6 1 0 0 6

Waze 72 15 0 1 5 0 4

iRadar 0 2 1 5 4 0 0

Beat the Traffic 5 7 0 0 1 0 0

Best Parking 0 0 0 0 1 0 1

MPG 0 0 0 0 0 0 0

Kyungmin Lee

Assessment of Android apps

39

Application User actions per task

Text contrast

ratio

Word count Button size Button distance

Animation # Item to scroll

Gmail 17 2 2 0 0 0 3

Google Finance 0 1 3 1 4 0 3

MyDays 20 4 2 0 2 0 1

eBay 0 1 1 0 0 0 2

Twitter 24 28 7 0 14 0 9

GasBuddy 12 14 1 0 13 0 4

Google Maps 8 7 3 0 7 0 1

Google Nav. 0 0 0 2 1 0 0

TomTom Nav. 26 0 6 1 0 0 6

Waze 72 15 0 1 5 0 4

iRadar 0 2 1 5 4 0 0

Beat the Traffic 5 7 0 0 1 0 0

Best Parking 0 0 0 0 1 0 1

MPG 0 0 0 0 0 0 0

Kyungmin Lee

0.75 violations / state0.75 violations / state

Assessment of Android apps

40

Application User actions per task

Text contrast

ratio

Word count Button size Button distance

Animation # Item to scroll

Gmail 17 2 2 0 0 0 3

Google Finance 0 1 3 1 4 0 3

MyDays 20 4 2 0 2 0 1

eBay 0 1 1 0 0 0 2

Twitter 24 28 7 0 14 0 9

GasBuddy 12 14 1 0 13 0 4

Google Maps 8 7 3 0 7 0 1

Google Nav. 0 0 0 2 1 0 0

TomTom Nav. 26 0 6 1 0 0 6

Waze 72 15 0 1 5 0 4

iRadar 0 2 1 5 4 0 0

Beat the Traffic 5 7 0 0 1 0 0

Best Parking 0 0 0 0 1 0 1

MPG 0 0 0 0 0 0 0

Kyungmin Lee

Assessment of Android apps

41

Application User actions per task

Text contrast

ratio

Word count Button size Button distance

Animation # Item to scroll

Gmail 17 2 2 0 0 0 3

Google Finance 0 1 3 1 4 0 3

MyDays 20 4 2 0 2 0 1

eBay 0 1 1 0 0 0 2

Twitter 24 28 7 0 14 0 9

GasBuddy 12 14 1 0 13 0 4

Google Maps 8 7 3 0 7 0 1

Google Nav. 0 0 0 2 1 0 0

TomTom Nav. 26 0 6 1 0 0 6

Waze 72 15 0 1 5 0 4

iRadar 0 2 1 5 4 0 0

Beat the Traffic 5 7 0 0 1 0 0

Best Parking 0 0 0 0 1 0 1

MPG 0 0 0 0 0 0 0

Kyungmin Lee

0.69 violations / state0.69 violations / state

Assessment of Android apps

42

Application User actions per task

Text contrast

ratio

Word count Button size Button distance

Animation # Item to scroll

Gmail 17 2 2 0 0 0 3

Google Finance 0 1 3 1 4 0 3

MyDays 20 4 2 0 2 0 1

eBay 0 1 1 0 0 0 2

Twitter 24 28 7 0 14 0 9

GasBuddy 12 14 1 0 13 0 4

Google Maps 8 7 3 0 7 0 1

Google Nav. 0 0 0 2 1 0 0

TomTom Nav. 26 0 6 1 0 0 6

Waze 72 15 0 1 5 0 4

iRadar 0 2 1 5 4 0 0

Beat the Traffic 5 7 0 0 1 0 0

Best Parking 0 0 0 0 1 0 1

MPG 0 0 0 0 0 0 0

Kyungmin Lee

Only MPG app had no violationsOnly MPG app had no violations

Conclusion

• AMC is a tool that– Explores the application automatically– Pinpoints UI guideline violations

• Many vehicular apps with UI violations– Shows the need for a tool like AMC

• Can be extended to other UI settings

43Kyungmin Lee

44

Questions?

Kyungmin Lee

45Kyungmin Lee

Backup slides

46Kyungmin Lee

State coverage

47Avg. coverage: 80%Kyungmin Lee

States found & runtime

48

Application # of States found

Runtime(h:mm)

Application # of States found

Runtime(h:mm)

Gmail 27 0:56 Google Nav. 15 0:37

Google Finance

18 0:55 TomTom Nav.

36 4:05

MyDays 26 1:55 Waze 81 6:09

eBay 24 0:50 iRadar 23 1:05

Twitter 65 10:25 Beat the Traffic

18 0:19

GasBuddy 59 4:28 Best Parking 11 0:10

Google Maps 31 1:59 MPG 5 1:22

Avg. runtime: 2hr 31minMedian runtime: ~74 minutes

Kyungmin Lee

Non-deterministic transitions

• Caused by– Configuration option– Previous paths– Mistakes in state classification

• Solution– Avoid using non-deterministic transitions!– Use past exploration information

• Combine previous N transitions into one• Attempt to make it deterministic

49Kyungmin Lee

Non-deterministic transitions

• Caused by unaware hidden property

50

BACK

BACK

Kyungmin Lee

Non-deterministic transitions

• Caused by our heuristics

51≈ ≠

Kyungmin Lee

Related work

• Model checking to find software bugs– KLEE, EXE, eXplode, CMC

• GUI Model checking with pre-specified model– Dwyer et al., Paiva et al., Takala et al.

• GUI checking with no model– GUI Ripping, Amalfitano et al., AutoBlackTest

• Using cognitive model to predict distraction– Salvucci’s work

52Kyungmin Lee

Limitations

• Only two ways to explore the app– GUI buttons & Android actions (e.g., back button)– No voice input, multi-touch, gesture, etc

• No checkpoint & restore– Inability to completely revert back

• Relying on app’s dynamic content– Different time -> different violation result

(e.g., Random pop-ups)

53Kyungmin Lee

Handling text inputs

• Need to input password, address, etc– User pre-specifies the text– User pre-specifies the associated text field’s id

• AMC automatically inputs text

54Kyungmin Lee

How we handle an ad

• No different than a button– Look for button size, button distance violations– Look for animation

• No violation found in our evaluation

55Kyungmin Lee

Industry guideline (TRL, 02)

• Qualitative properties– “Color scheme must not be confusing”– “System should provide precise error message”

• Quantitative properties– “Text contrast ratio should be at least 3:1”– “Buttons should be at least 15mm apart”– Can be verified without human efforts

56Kyungmin Lee