Tap into mobile app testing@TDT Iasi Sept2013

40

description

Tap into mobile app testing@TDT Iasi Sept2013

Transcript of Tap into mobile app testing@TDT Iasi Sept2013

Page 1: Tap into mobile app testing@TDT Iasi Sept2013
Page 2: Tap into mobile app testing@TDT Iasi Sept2013

Tap into Mobile App Testing

Andreea Haras

11/09/2013

Page 3: Tap into mobile app testing@TDT Iasi Sept2013

Agenda

• Introduction

• Mobile Testing vs. Traditional Web-Based Testing

• Challenges in Mobile Testing

• Guidelines & Best Practices

• Conclusion

• Q&A

Page 4: Tap into mobile app testing@TDT Iasi Sept2013

About me

• Andreea Haras – [email protected]

• 4+ years experience as software tester

• Mobile testing enthusiastic

• Interested in new technology

Page 5: Tap into mobile app testing@TDT Iasi Sept2013

Why mobile testing?

Page 6: Tap into mobile app testing@TDT Iasi Sept2013

Why mobile testing?

• “The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home” - Forrester

Page 7: Tap into mobile app testing@TDT Iasi Sept2013

Why mobile testing?

• “The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home” - Forrester

• Applications running on mobile devices (e.g., mobile applications running on smart phones or new generation tablets) are becoming so popular that they are representing a revolution in the IT sector.

Page 8: Tap into mobile app testing@TDT Iasi Sept2013

Why mobile testing?

• “The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home” - Forrester

• Applications running on mobile devices (e.g., mobile applications running on smart phones or new generation tablets) are becoming so popular that they are representing a revolution in the IT sector.

• Over 1.6 million mobile applications have been developed, available in Apple App Store and Google Play (www.mobithinking.com)

Page 9: Tap into mobile app testing@TDT Iasi Sept2013

Why mobile testing?

• “The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home” - Forrester

• Applications running on mobile devices (e.g., mobile applications running on smart phones or new generation tablets) are becoming so popular that they are representing a revolution in the IT sector.

• Over 1.6 million mobile applications have been developed, available in Apple App Store and Google Play (www.mobithinking.com)

• Estimates for downloads of apps in 2013 range from 56 to 82 billion (www.idc.com) and a prediction of 200 billion global downloads in 2017 that will be worth US$63.5 billion (www.portioresearch.com)

Page 10: Tap into mobile app testing@TDT Iasi Sept2013

Native, Traditional, Hybrid, Oh My!

• Native Apps: coded with a specific programming language. They are fast, reliable, and powerful, but are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform. Netflix, Pandora, Shazam.

Page 11: Tap into mobile app testing@TDT Iasi Sept2013

Native, Traditional, Hybrid, Oh My!

• Native Apps: coded with a specific programming language. They are fast, reliable, and powerful, but are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform. Netflix, Pandora, Shazam.

• Traditional Mobile Web Apps: mobile web sites designed to match every web-enabled phone. They’re accessed through the mobile device’s web browser and do not need to be downloaded/installed. They re-direct to the site’s M Site. Amazon, eBay, MSN are good examples.

Page 12: Tap into mobile app testing@TDT Iasi Sept2013

Native, Traditional, Hybrid, Oh My!

• Native Apps: coded with a specific programming language. They are fast, reliable, and powerful, but are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform. Netflix, Pandora, Shazam.

• Traditional Mobile Web Apps: mobile web sites designed to match every web-enabled phone. They’re accessed through the mobile device’s web browser and do not need to be downloaded/installed. They re-direct to the site’s M Site. Amazon, eBay, MSN are good examples.

• Hybrid Apps: rely on development frameworks and are written with the same technology used for websites and mobile web implementations. They are hosted or run inside a native container on a mobile device. They ensure cross-platform compatibility and can access the phone’s hardware (camera, GPS, user’s contacts, etc.). Foursquare, Twitter, Wikipedia and Facebook are good examples.

Page 13: Tap into mobile app testing@TDT Iasi Sept2013

Pop Quiz

There are seven (yes, seven!) mobile operating systems that are recognized in the industry, other than the top 2 (iOS and Android), can you name one or more of the other five?

Page 14: Tap into mobile app testing@TDT Iasi Sept2013
Page 15: Tap into mobile app testing@TDT Iasi Sept2013

Can you name the differences between Mobile Testing

and Traditional Web-Based Testing?

Page 16: Tap into mobile app testing@TDT Iasi Sept2013

Mobile Testing vs. Traditional Web-Based Testing

• Integration testing Within areas of the Device Hardware

Battery, light, ringer, sleep, wake, usage, connectivity loss or change in speeds (“elevator test”), camera

Integration with other Apps on the device Facebook, Twitter, etc.

Page 17: Tap into mobile app testing@TDT Iasi Sept2013

Mobile Testing vs. Traditional Web-Based Testing

• Integration testing Within areas of the Device Hardware

Battery, light, ringer, sleep, wake, usage, connectivity loss or change in speeds (“elevator test”), camera

Integration with other Apps on the device Facebook, Twitter, etc.

• Interruption Testing How the App behaves with user-controlled and automatic interruptions

Page 18: Tap into mobile app testing@TDT Iasi Sept2013

Mobile Testing vs. Traditional Web-Based Testing

• Integration testing Within areas of the Device Hardware

Battery, light, ringer, sleep, wake, usage, connectivity loss or change in speeds (“elevator test”), camera

Integration with other Apps on the device Facebook, Twitter, etc.

• Interruption Testing How the App behaves with user-controlled and automatic interruptions

• Device-specific Gesture Testing Pinching, Zooming, Shaking, Orientation, Eye Scrolling, Eye Pausing,

Tilting, Swiping, Press and Hold, Tapping

Page 19: Tap into mobile app testing@TDT Iasi Sept2013
Page 20: Tap into mobile app testing@TDT Iasi Sept2013

Pop quiz

What mobile testing challenges do you think there

are?

Page 21: Tap into mobile app testing@TDT Iasi Sept2013

Challenges in Mobile Testing

• The time frames

• Variety of Devices & OS Versions Display density Different browsers Network connections Installation issues

Page 22: Tap into mobile app testing@TDT Iasi Sept2013

Challenges in Mobile Testing

• The time frames

• Variety of Devices & OS Versions Display density Different browsers Network connections Installation issues

• Simulating real life scenarios Call interrupts Low battery Alarms Network issues

Page 23: Tap into mobile app testing@TDT Iasi Sept2013

• Usability Accessibility Intuitive Mobile experience

More challenges in Mobile Testing

Page 24: Tap into mobile app testing@TDT Iasi Sept2013

• Usability Accessibility Intuitive Mobile experience

• Automation What to automate? Tools Effort vs Value

More challenges in Mobile Testing

Page 25: Tap into mobile app testing@TDT Iasi Sept2013

More challenges in Mobile Testing

• Usability Accessibility Intuitive Mobile experience

• Automation What to automate? Tools Effort vs Value

• Non functional testing Performance Security Load testing

Page 26: Tap into mobile app testing@TDT Iasi Sept2013

Guidelines & Best Practices

• Determining your Mobile Test Matrix

• Setting up your Lab: Phones

• Setting up your Lab: Tablets

• Checklist

• Key types of Mobile QA Testing: Manual or Automated Testing?

• Things to remember

• Test Automation Tool Selection – Few factors to consider

Page 27: Tap into mobile app testing@TDT Iasi Sept2013

The most difficult facet of the testing matrix is device fragmentation. There are nearly 4000 separate Android device models! How do you make sure that all users get a consistent experience?

Device manufacturer market share (Q1 2013)

Determining your Mobile Test Matrix

Page 28: Tap into mobile app testing@TDT Iasi Sept2013

Determining your Mobile Test Matrix

Android device fragmentation

Source: BGR

Page 29: Tap into mobile app testing@TDT Iasi Sept2013

Determining your Mobile Test Matrix

Multiple approaches to build a “Mobile Test Matrix”• Group devices based on the hardware and/or software specification and prioritize

them

• User base – Build a test matrix based on the device and software version the user uses

• Use statistics from vendor market place (especially useful for native apps like games, utilities, etc.)

• Risk based and functionality based matrix

Page 30: Tap into mobile app testing@TDT Iasi Sept2013

Determining your Mobile Test Matrix

Multiple approaches to build a “Mobile Test Matrix”• Group devices based on the hardware and/or software specification and prioritize

them

• User base – Build a test matrix based on the device and software version the user uses

• Use statistics from vendor market place (especially useful for native apps like games, utilities, etc.)

• Risk based and functionality based matrix

Example:• Group 1, priority C: Small devices with a small CPU, RAM and low resolution. Older

software versions and older browsers.

• Group 2, priority B: Mid-range devices with an avg. CPU, RAM (<512 MB), good screen size and resolution. The software is not the latest.

• Group 3, priority A: High-end devices with a dual/quad-core CPU, RAM (>512 MB) and a high screen resolution. Latest software versions.

Page 31: Tap into mobile app testing@TDT Iasi Sept2013

Setting up your Lab: Phones

• Newest, highest resolution, high density, normal sized phone running latest version of OS. (Samsung Galaxy S4, iPhone 5S)

• Oldest, smallest, low density small sized phone running oldest supported version of OS. (iPhone 3, HTC Tattoo)

• Middle of the range phone matches highest usage, normal size, high density, normal resolution, highest usage OS (Samsung Galaxy S2)

• Phone with Highest Market Share (Samsung Galaxy S3, iPhone 4S)

• Budget smartphone - normal size, large screen, mdpi, old OS (Dell Venue)

Page 32: Tap into mobile app testing@TDT Iasi Sept2013

Setting up your Lab: Tablets

• XLarge size high end tablet (10”), xhdpi, latest OS. (Nexus 10)

• Large size Tablet (5-7”), med density, legacy OS (Kindle Fire)

• Phablet (Phone/Tablet Hybrid) (5”) xhdpi, large size, latest OS, strange aspect ratio (LG Intuition, Samsung Note)

• iOS running iOS 5 or lower (iPad 2)

• iOS running iOS 6 (iPad mini, iPad 3)

Page 33: Tap into mobile app testing@TDT Iasi Sept2013

Checklist

Page 34: Tap into mobile app testing@TDT Iasi Sept2013

Key types of Mobile QA Testing: Manual or Automated Testing?

Functional testing 

Performance testing 

Memory Leakage Testing 

Interrupt testing

Usability testing

Installation testing

Upgrade testing

Page 35: Tap into mobile app testing@TDT Iasi Sept2013

Things to remember

• Start testing early. Use emulators / simulators

• Use the app on different carriers and network connections like 3G, 4G, WiFi, or LTE

• Combine testing tools and integrate them into a continuous integration system

• When possible, have a group of BETA testers to help with real-world testing

• Consider the problem space: go mobile to test mobile

• Don’t assume that because it works on one device, it will work on all

Page 36: Tap into mobile app testing@TDT Iasi Sept2013

Test Automation Tool Selection – Few factors to consider

• Supported devices

• Emulator support

• Availability of integrated simulators

• Types of connection supported

• Multi-device support (this is important to run same test on multiple devices)

• Compatibility with existing automation tools

• Security - VPN connections, data sharing, etc.

• Functionality supported - swipe, drag and drop,

zoom, scrolling, pinching, rotating, etc

• Image / text recognition

• Debugging and test reporting

Page 37: Tap into mobile app testing@TDT Iasi Sept2013

Conclusions

• Despite the challenges in mobile application testing, careful selection of target devices, using statistical techniques for test optimization and tools that maximize automation can ensure a cost effective mobile testing process.

Page 38: Tap into mobile app testing@TDT Iasi Sept2013

Conclusions

• Despite the challenges in mobile application testing, careful selection of target devices, using statistical techniques for test optimization and tools that maximize automation can ensure a cost effective mobile testing process.

• Given the stiff competition in the mobile apps market and variety of apps available for users to download, it is vital that you thoroughly test each and every aspect of the app before releasing it into the market, to ensure that the product meets the quality standard demanded by users.

Page 39: Tap into mobile app testing@TDT Iasi Sept2013
Page 40: Tap into mobile app testing@TDT Iasi Sept2013