Understanding the Test Automation Culture

of App DevelopersPavneet Singh Kochhar1, Ferdian Thung1,

Nachiappan Nagappan2, Thomas Zimmermann2, David Lo1

1Singapore Management University2Microsoft Research

{kochharps.2012,ferdiant.2013,davidlo}@smu.edu.sg, {nachin,tzimmer}@Microsoft.com

International Conference on Software Testing, Verification and Validation (ICST’15)

2

Smartphones

• Smartphones have become pervasive• Worldwide sales increased by 42.3% from

2012 to 2013

Android has 78.4% of the smartphone

market1

Millions of apps available

[1] Gartner, “Gartner says annual smartphone sales surpassed sales of feature phones for the first time in 2013”

3

Study Goals

• Understand the test automation culture of app developers

• Understand the “pain points” of developers

Are mobile apps adequately tested?

4

Understanding Test Automation Culture

• Analyse over 600 Android apps– Execute test cases – Assess test adequacy

• Survey Android & Microsoft developers– Understand common testing tools used– Challenges faced by them

5

Outline

• Motivation and Goals• Test Adequacy • Data Collection & Basic Statistics• Empirical Results• Recommendations• Conclusion and Future Work

6

Test Adequacy

• Test Adequacy Criterion– Property that must be satisfied for a test suite

to be thorough. – Often measured by code coverage.

• Code Coverage– Percentage of the code executed by test cases

• Line coverage• Block coverage

7

Outline

• Motivation and Goals• Test Adequacy • Data Collection & Basic Statistics• Empirical Results• Recommendations• Conclusion and Future Work

8

Data Collection

• Repository for Free and Open Source Android applications

• We collect apps hosted on• In total, we have 627 apps• Our dataset has large and popular apps

9

First Survey• 3905 distinct email addresses of

developers of 627 apps• Ask questions about testing tools used

and challenges faced• We receive 83 responses (Response rate

2.13%)• Unit of analysis is individual developer

10

First Survey

Questions:• How do you test your app code?• Do you use any test automation tools? • What are the challenges you face during

testing either manually or using automated tools?

11

Second Survey• We improve our survey questions based

on the responses of first survey • Email 678 developers in Microsoft• We receive 127 responses (Response rate

18.73%)• Unit of analysis is individual developer

12

Second Survey

Questions:• How do you test your app code? - Manually, Automated tools, Don’t test• What type of testing do you do? - Unit testing, integration testing, system testing, regression testing etc.• What automated testing tools you use?

13

Second Survey

Questions:• Why do you use testing tools? - Generating test cases, executing test cases etc.• Do you face these challenges? - Time constraints, compatibility issues, lack of exposure to tools, poor documentation etc.• What are the top 2 things you look

for/need/would like to see?

14

Basic Statistics

Tools Number of Apps % of Apps

Without Test Cases 538 85.81%

With Test Cases 89 14.19%

Distribution of Apps w.r.t Test Cases

Total – 627 apps

15

Basic StatisticsDistribution of Apps w.r.t LOC

16

Basic StatisticsDistribution of Apps w.r.t Developers

17

Outline

• Motivation and Goals• Test Adequacy and Code Metrics• Data Collection & Basic Statistics• Empirical Results• Recommendations• Conclusion and Future Work

18

Research Questions

RQ1: Are Android applications well tested? RQ2: Do Android developers use automated testing tools to test their applications? What are the tools commonly used and the challenges faced by them while testing their applications?

RQ3: Do Microsoft developers use automated testing tools to test their applications? What are the tools commonly used and the challenges faced by them while testing their applications?

19

Research Questions

RQ1:Current State of Testing in

Android Applications

20

RQ1: Current State of TestingDistribution of Test Suites (89 apps)

21

RQ1: Current State of TestingLine Coverage (41 apps)

22

RQ1: Current State of TestingBlock Coverage (41 apps)

23

RQ1: Current State of Testing

• Only 14.19% (89 out of 627) of the apps contain test cases

• Only 0.64% (9.75% of 14.19%) have line coverage above 40%

Results

24

Research Questions

RQ2:Survey of Android Developers

25

RQ2: Survey of Android DevelopersAndroid Automated Testing Tools Usage

Tools Number of Respondents

JUnit 18

MonkeyRunner 8

Robotium 7

Robolectric 6

Android unit testing framework

6

Monkey 1

No tool 35

26

RQ2: Survey of Android DevelopersChallenges faced by developers

Tools Number of Respondents

Time constraints 20

Compatibility issues 16

Lack of exposure 11

Cumbersome tool 9

Emphasis on development

6

Lack of organization support

5

Unclear benefits 4

27

RQ2: Survey of Android DevelopersChallenges faced by developers

• Time “Designing and implementing test cases takes some extra time, which makes it difficult to finish the project in time”• Compatibility “I tried robolectric, but ran into several issues, that were probably also related to the fact that I am using Scala on Android”• Usability “I think Monkey runner is kinda cumbersome, and breaks easily when changing layout options”

28

RQ2: Survey of Android DevelopersChallenges faced by developers

• Lack of Support “The advice I was given was ... not bother with trying to use the Android testing tools/frameworks”• Poor Documentation “Testing is documented there, but not very well and there should be far more information”• Steep Learning Curve “I fear it would represent a strong learning curve”

29

RQ2: Survey of Android Developers

• Developers use tools such as JUnit, Robolectric, Robotium etc.

• However, large number of developers prefer manual testing

• Developer face challenges such as time constraints, compatibility issues, lack of exposure, cumbersome tools, unclear benefits, poor documentation etc.

Results

30

Research Questions

RQ3:Survey of Microsoft Developers

31

RQ3: Survey of Microsoft DevelopersTypes of Testing

32

RQ3: Survey of Microsoft DevelopersAutomated Testing Tools Usage

Tools Number of Respondents

Visual Studio 35

Internal tool 8

Selenium 7

Microsoft Test Manager 5

Others (Qunit,Robotium etc.)

27

33

RQ3: Survey of Microsoft DevelopersAutomated Testing Tools Usage

34

RQ3: Survey of Microsoft DevelopersChallenges faced by developers

35

RQ3: Survey of Microsoft Developers

• Top 3 reasons for using automated testing tools are executing test cases, finding potential bugs, and analyzing code coverage.

• Top 3 challenges faced by Microsoft developers are time constraints, compatibility issues, and lack of exposure to tools.

• Developers prefer well documented and easy to use tools

Results

36

Recommendations

• Need to promote new tools that can be easily used by developers

• Need tools which can help test difficult cases• Need tools which have good documentation• Need to spread information about tools

through online channels

Conclusion

37

• Only 14% of the apps contain test cases & 0.64% (9% of 14%) of the apps have coverage above 40%.

• Android developers prefer using JUnit whereas Microsoft developers use Visual Studio, Internal tool, Selenium.

• Developers face challenges such as time constraints, compatibility issues, lack of exposure, cumbersome tools, unclear benefits, poor documentation etc.

38

Future Work

• Expand the study to include more apps– Address the threats to external validity

• Survey more developers to get more responses

• We plan to develop tools to address “pain points”.

Thank you!

Questions? Comments? Advice?{kochharps.2012,ferdiant.2013}@smu.edu.sgdavidlo@smu.edu.sg{nachin, tzimmer} @microsoft.com