Appurify Performance Automation Whitepaper FINAL

The State of Mobile Development

& Performance in 2014

An Appurify Study for Mobile App & Website Developers

© 2014 appurify

Mobile developers understand the importance of performance and quality. In our fast-paced mobile marketplace, consumers have no patience for apps and sites that can’t meet their expectations. Performance testing is essential to deliver the best possible mobile experience—but what kind of testing tools and processes are most effective? Technologies and best practices for app performance are developing rapidly, but to date there has been little data available on the actual state of the industry.

To capture data for our study, Appurify worked with Market Cube, a panel research company, to recruit a total of 400 respondents between Dec. 10 – Dec. 20, 2013. All participants were based in the U.S. and held responsibility for mobile development strategy or mobile testing at organizations with 50 or more employees. Each respondent also had purchasing decision-making power for performance testing and automation tools—so their views are especially

important for understanding the needs, preferences, and plans of companies in this regard. The margin of error for survey results is +/- 4.9%.

The results of the study reveal key concerns in mobile app and site performance today—as well as a clear picture of developers’ plans to create better mobile experiences for users moving forward.

Appurify initiated a study to shed light on several key issues:

2© 2014 appurify

What are mobile developers doing today to ensure quality and performance?

What challenges do we face in meeting the expectations of consumers for their favorite mobile apps and sites?

What kind of testing and automation tools are proving most useful and popular—and what would the ideal tool look like?

3© 2014 appurify

Performance counts. An overwhelming majority of developers—nearly 90%—have seen a direct connection between the performance of their apps and the reviews and ratings they receive in app stores.

Keeping abreast of user reviews. Developers have become too reliant on user reviews to spot quality issues, with 60% of developers scanning user reviews at least daily.

Testing and optimization is getting harder. Top challenges cited include making sure apps work across proliferating devices and platforms, keeping pace with rapid development cycles, and the sheer amount of time testing takes.

Updates are too frequent for manual testing. App updates have increased 62% over the past year, and 42% of apps and mobile websites update weekly, which is too rapid for effective manual testing.

Data is power.Historical and trending data is recognized as an important indicator in a company’s optimization strategy. 98% of companies see testing over time as important.

Automation is the answer.Looking for a smarter way to ensure app performance, 76% of respondents plan to invest in test automation.

Stability — can the app or website function as intended without crashes?

UI responsiveness — does the interface render accurately and respond quickly to user inputs such as panning, scrolling, and taps?

Network performance — Can the app work well with available network connectivity, even in less than ideal signal conditions?

Load time — does the app or website load quickly or does the user have to wait?

CPU usage — can the app make efficient use of available processing power, or does it lag on anything but high-end devices?

Memory usage — can the app make efficient use of available memory, or does is have issues on anything but high-end devices?

Users perceive the performance of an app or website in terms of a variety of factors. These include:

Developers should pay close attention to these factors when testing and optimizing the mobile experiences they create across device types, platforms, and usage conditions.

Key Findings Definition

Part 1:

The Mobile Development

Landscape

4© 2014 appurify

Total

50-499

500-999

1000-4999

5000+

# ofEmployees Percent of Companies with Mobile Apps / Websites by Company Size

MOBILE APPS MOBILE WEBSITE

5© 2014 appurify

48%

28%

24%

28%

Mobile today encompasses a wide range of experiences for business and consumers, a variety of development platforms, and a continued commitment to both apps and websites. Across this diverse space, though, two consistent themes emerge: companies are releasing updates faster than ever to keep users satisfied— and they’re paying close attention to the strong tie between performance and ratings.

Apps vs. Websites—or both

The mobile landscape is shared nearly equally by mobile apps (72%) and mobile websites (76%), with 48% of organizations using both. While mobile sites have a sight edge overall, among larger companies the advantage goes to apps—used by 88% of companies with more than 5,000 employees, compared with 78% for sites.

MOBILE APPS

MOBILE WEBSITES

BOTH

72% 76%68% 73%69% 77%69% 79%88% 78%

Distribution of Mobile Apps / Websites by Company

Type of Applications & Websites Developed

Total Consumer

Apps

90%

PRO

DU

CT

IVIT

Y

RET

AIL

MED

IA

NEW

S

LIFE

STY

LE

GA

MES

BUSI

NES

S A

PPS

TR

AV

EL

EMPL

OY

EE-F

AC

ING

APP

S

BUSI

NES

S SI

TES

OT

HER

OT

HER

CO

NSU

MER

SIT

ES

SOC

IAL

NET

WO

RK

ING

Total Business

Apps

56%

Total Mobile

Websites

72%

Website for Business;Apps for Life

6© 2014 appurify

When it comes to mobile apps, consumers are the name of the game. When asked to select all types of apps and mobile websites developed, 90% of companies that were surveyed focus on consumer apps, 56% focus on business users and 72% focus on mobile websites. The most popular categories of consumer apps include productivity, social networking, and retail.

Mobile web development, on the other hand, tends to be more business-oriented, with 50% of sites targeting business users and only 42% serving consumers.

49% 33%46% 30%42% 27%38% 50%40%34% 42%40%

1%1%

If you could create the ultimate mobile testing solution, what would it include?

I would like a fully automated, fully integrated system that can replicate testing on all major mobile devices and OSs so that we wouldn’t have to spend so much time running redundant tests.

Release Frequency Change vs. 12 Months Ago

INCREASED STAYED THE SAME DECREASED

7© 2014 appurify

To satisfy demanding users, companies are releasing updates faster than ever—at least weekly at 42% of organizations—putting more and more pressure on developers and QA to get things right. Nearly two-thirds of organizations have accelerated their release cycle in the past year, while only 1% has slowed the pace.

Faster and faster—and better?

“”

62%

37%1%

Frequency of Releases

35%

28%

34%

21%

WEEKLY

MONTHLY

QUARTERLY

LESS THAN WEEKLY

YEARLY

3%7%

Frequency of Scanning User Reviews

8© 2014 appurify

As organizations scramble to refresh their apps, they’re also keeping a close eye on the results of their efforts. Of the 73% of developers with apps available in the Apple App Store or on Google Play, 60% scan their user reviews at least daily—and 19% do so every hour. Only 10% check their reviews monthly or less.

When they do, they see a direct correlation between the performance of their app and the ratings given by users. An overwhelming 87% of organizations with apps available through the Apple App Store or Google Play report an impact of performance on ratings—and 47% of these described the effect as a large one.

Extent App Performance Affecting Ratings

LARGE EFFECT

HOURLY DAILY WEEKLY MONTHLY LESS

SOME EFFECT NO EFFECT

47%

41%29%

7%

52%

2%

3% Seen App Performance Affecting Ratings

87%

13%NO

YES

19%

9© 2014 appurify


My ultimate testing solution would involve the ability to test different types of phones with different CPUs, different amounts of memory, operating systems, and platforms. It would also show which ones have more bugs and which have better results.

“

”

Part 2:

Performance & Testing Approaches and

Challenges

10© 2014 appurify

Different companies approach testing differently. This is true in terms of the headcount assigned to this task, the testing methods used, the elements tested, and even, to an extent, the importance of tracking testing over time. But however they test their apps, they often wish they had a better way of doing it. Current tools and process leave something to be desired—and take valuable time that could better go to creation and innovation.

The largest companies tend to have the most people involved with testing mobile apps and sites, as one would expect. Of companies with more than 5,000 employees, 31% have more than 101 people in this role. At smaller businesses of 50 – 499 employees, the norm is to have 10 or fewer. But the correlation isn’t completely consistent—15% of the largest companies surveyed also have 10 or fewer people regularly involved in testing, while 18% of companies with fewer than 500 employees involve more than 101 people. Clearly, different companies can assign varying levels of importance to performance testing.

Companies also use a wide variety of methods for testing. First, we’ll look at functional testing, which verifies whether given input produces the expected output, such as testing the login process for an app. By far, the most popular approach involves manual testing on real devices (78%), followed by the ad hoc discovery of issues by employees or customers—a far from ideal approach (41%). Relatively few outsource (31%) or crowdsource (27%) testing.

11© 2014 appurify

The Who & How of Testing

Number of People Involved with Mobile Testing at Your Organization on a Regular Basis

By Company Size (# of Employees)

1-10

11-50

51-100

101+

Total 50-499 500-999 1000-4999 5000+

39% 54% 36% 30% 15%

24% 27% 19% 27% 22%

24% 14% 34% 27% 32%

12% 4% 4% 16% 31%

A similar range can be seen in approaches to performance testing, which examines metrics such as load time, battery consumption, and network usage. Again, the majority use manual testing on real devices (75%) followed by ad hoc testing by employees or customers (40%).

Significantly, the top two methods for both functional and performance testing involve real devices, showing a clear preference for this approach over simulators and/or emulators. At the same time, two of the top three methods in each area rely on automated testing. From these results, we can infer a strong preference for two factors in testing: real devices and automation.

12© 2014 appurify

Functional Testing of Apps / Mobile Sites

Testing the Performance of Apps / Mobile Sites

Manual Testing on Real Devices

Ad Hoc as Employees or Customers Find Issues

Outsourced Service

Crowdsourced Solution (e.g. uTest)

Automated Testing

Manual Testing on Real Devices

Ad Hoc as Employees or Customers Find Issues

Outsourced Service

Peformance Testing Software (e.g. HP, SOASTA, Appurify)

Crowdsourced Solution (e.g. uTest)

Open Source or Home Grown Tools

78%

75%

41%

40%

31%

29%

26%

27%

29%

10%

28%

Taking a closer look at performance testing, we see the issues of most concern for developers—and, presumably, for users as well. The performance elements actively tested at most organizations include memory usage (71%), load time (71%), CPU usage (64%) and network usage (64%).

Historical and trending data plays an important role in almost every company’s testing strategy. A full 98% of organizations see testing over time as at least somewhat important, and 56% see it as very important. Functional test results are tracked over time at 91% of companies, and performance test results are tracked at 86% of companies.

13© 2014 appurify

Performance Elements Actively Tested

MEMORY USAGE

VERY IMPORTANT SOME-WHAT

NOT VERY

NOT AT ALL

NETWORK USAGE

LOADTIME

BATTERYUSAGE

CPUUSAGE

FRAMES PER SEC.

71%64%

71%

59%64%

48%

Importance of Testing Over Time

35%7%

1% 1%

56%

Mobile performance is a high-level concern at a majority of businesses. Management at 62% of organizations reviews performance reports—more than developers (59%), QA (57%), or operations (46%).

14© 2014 appurify

Tracking Test Results Over Time

When It Comes To Mobile Performance, Who In Your Organization Is Involved In Reviewing The Results

/ Any Performance Reports?

Functional Testing Performance Testing

91% 86%

9% 14%

NO

YES

MANAGEMENT OPERATIONS(OPS)

DEVELOPERS QUALITY ASSURANCE

(QA)

62%

46%

59% 57%

15© 2014 appurify


Accelerate mobile application testing with automated functional testing features. Manage complexity and change while testing on multiple devices and platforms. Prevent service outages and disruptions through load and performance testing. Incorporate location and network speed into functional and performance testing. Reuse scripts to reduce rework and to monitor performance in production.

While businesses clearly understand the importance of performance and functional testing, they’re far from fully satisfied with their current testing processes. 61% rate their testing processes for device coverage as less than ideal, and 58% for operating system coverage. Nearly two-thirds see room for improvement in their ability to pinpoint and address crashes (64%) and in their ability to know how an app will actually perform in the wild in terms of load time, memory usage, and other key factors (65%).

When asked about their top three toughest challenges for mobile testing, companies produced a list that will sound familiar to every mobile developer:

Respondents also provided a rogue’s gallery of the most difficult elements to test using their existing tools, citing the worst as:

“”

Making sure apps work across all device/operating system possibilities

It’s very time-consuming

Keeping pace with quick development cycles

Hard to pinpoint and/or recreate the exact location of bugs and other issues

Accounting for different user and device conditions, such as geographies or low memory on devices

Accounting for different network conditions (networks, signal strengths, etc.)

Crashes

Network usage

CPU usage

Load time

Basic pass/fail functionality

Memory usage

Room For Improvement

16© 2014 appurify

It seems obvious that the most strategic aspect of an app developer’s work is the creation of compelling and differentiated mobile experiences—not routine performance and functional testing. Strikingly, though, a full 78% of companies spend at least as much time testing their mobile apps and sites as they do developing them—and 15% actually spend more time on testing. This underscores the importance of testing automation tools to allow companies to focus their attention and resources on creation rather than troubleshooting.

As discussed above, companies commonly see testing on real devices as the most ideal approach. Nearly half of respondents report doing so very often (47%), while nearly as many say they do so often (42%). But that doesn’t mean it’s easy. The biggest challenges involved in testing on real devices include:

Automating the process

Connecting into continuous integration practices Management of devices, including charging, storage, and keeping them up to date

Getting access to phones running on older operating systems

Buying new devices

Making Time For Testing


Have all the devices need for testing in one place for automatic testing so there would be no need for manual testing.

“”

Part 3:

The Tools of the Trade

17© 2014 appurify

0

1

2

3

4

5

18© 2014 appurify

22%

15%

24%

3%

9%

27%

Where testing tools are concerned, companies prefer not to limit their options. Most use a variety of free and paid tools, and the more people they devote to

testing, the larger their toolbox grows.

Free and paid—there’s room for both

Number of Free and Paid Tools Used

Number of Free Tools Number of Paid Tools

Companies generally use multiple tools for testing, showing no strong preference for either free or paid options. Almost every company uses more than one free tool (88%)—and almost every company uses more than one paid tool as well (90%). Both kinds

have a role to play in the process.

As one would expect, companies with a larger number of people involved in the testing process

tend to use a larger number of free and paid tools.

20%

19%

26%

3%

7%

26%

19© 2014 appurify


Consistent repeatable task or scripts that measure criteria exactly the same way each time, and eliminate the subjectivity of the test.

“”

Number of Free and Paid Tools Used% Using 4 or More

% with 4+ Free Tools % with 4+ Paid Tools

TOTAL TOTAL1-10

TESTERS1-10

TESTERS11-50

TESTERS11-50

TESTERS51+

TESTERS51+

TESTERS

28% 32%38%

43%50%

62%

42% 45%

21© 2014 appurify

When Do You Plan To Invest In Test Automation?

IN THE NEXT 3

MONTHS

IN 3-6MONTHS

IN 6 MONTHS

TO 1 YEAR

IN 1-2YEARS

IN MORE THAN 3 YEARS

29% 34%

15% 1%21%

As testing gets more demanding, automation becomes more critical. Most companies have already adopted test automation, and they’re continuing to expand their capabilities in this area. Those that haven’t yet are only looking for the right way to move forward.

The Time Is Now

Faced with a dynamic marketplace, demanding consumers, and accelerated release cycles, it’s no surprise that a strong majority of companies—74%—have adopted test automation. A similar number

report plans to make further investments in test automation. Time is of the essence; 21% will make these investments within the next three months, and another 29% within the next six months. Only 16% plan

to wait longer than a year.

Irrespective of the popularity of free testing tools, most companies are prepared to devote significant budget resources to these tools.

Of those planning to invest in test automation, 70% plan to spend at least $50,000, and 12% have earmarked at least $500,000.

22© 2014 appurify

How Much Do You Plan To Spend On Implementation?

LESSTHAN $10k

$10kto

$50k

$50kto

$100k

$100kto

$500k

$500k-

$1mm

MORETHAN $1mm

19%

35%

23%8% 4%

11%

Irrespective of the popularity of free testing tools, most companies are prepared to devote significant budget resources to these tools. Of those planning to invest in test automation, 70% plan to spend

at least $50,000, and 12% have earmarked at least $500,000.


Automating your mobile application testing is the best way to achieve the quick, precise results you need to accommodate fast development cycles. For the most realistic results, you must also test on actual devices—without optical recognition, emulation, jailbreaking or tethering.

“”

23© 2014 appurify

If The Process Of Writing Tests Was Significantly Easier, How Likely Would You Be To Adopt Automation

Practices Into Your Organization?

VERY LIKELY

IN 3-6MONTHS

IN 6 MONTHS

TO 1 YEAR

IN 1-2YEARS

IN MORE THAN 3 YEARS

29% 34%

15% 1%21%

If Only We Could...

Of the 26% of organizations not using test automation, reasons cited include uncertainty about how to get started or which framework to use, lack of budget, and lack of resources. Significantly, only 1% has no interest in it—which means that for the vast majority of companies lacking test automation,

this is a matter of circumstances, not of choice.

The question then becomes: how could the implementation of test automation be made more feasible for the companies who have yet to take this step? Test scripts emerge as a key factor. Asked

how an easier process for writing tests would affect their decision to adopt automation practices, 35% said that they would be very likely to do so, and another 41% said that they would be more

likely. Only 1% said that they would remain unlikely to adopt automation even with a significantly easier test writing process.

24© 2014 appurify

For organizations that have adopted test automation, the benefits have been diverse and substantial. Asked about the top benefits they’ve seen, respondents reported:

Saving time

Increasing quality

Reducing cost

Increasing speed of development

Minimizing hours spent on other methods of testing

Greater test coverage

Repeatability of tests

Testing Pays Off

Conclusion

26© 2014 appurify

The results of the Appurify study show that, while companies take a variety of approaches to the mobile experiences they create and the resources they use to do so, they share the need for active testing to meet user requirements in terms of both functionality

and performance.

As release cycles accelerate, organizations can end up spending an inordinate amount of time on testing, increasing the need for

automation. At the same time, they are often less than satisfied with the tools they’re using, even when they’re already using several free and paid tools. What they most want is a way to test on real devices

while simulating the full range of devices, platforms, and usage conditions—simply and efficiently.

A significant number of companies plan to invest in test automation. Time

is of the essence; 21% will make these investments within the next three

months, and another 29% within the next six months. Only 16% plan to wait longer

than a year. Organizations are finding that automation is critical to improve the optimization and performance of

their apps and have received significant benefits including the following:

1 Saving time

2 Increasing quality

3 Reducing cost

4 Increasing speed of development

5 Minimizing hours spent on other methods of testing

6 Greater test coverage

7 Repeatability of tests

About Appurify and AMP

27© 2014 appurify

Designed to meet the needs of mobile app and website developers, AMP (Appurify Mobile Platform) is the mobile app optimization solution that boosts the quality

and performance of mobile apps and sites pre-release in a controlled, measurable environment. With automated analysis across a comprehensive set of real devices

and user conditions, dev, QA, ops and management teams can now get the visibility and actionable intelligence they need to shorten dev cycles, maintain

brand credibility and build better mobile experiences.

To learn more about AMP, please contact Appurify at: [email protected].

Appurify Performance Automation Whitepaper FINAL

Documents

Transcript of Appurify Performance Automation Whitepaper FINAL