Mobile Application Testing - SE 3S03 - Tutorial 2khedri/wp-content/uploads/... · testing...

MobileApplication

Testing

Zahra Ali

How to test mobileapps

Emulator, Simulator,Real Device

Traditional Testing

Mobile SpecificTesting

Battery Usage Testing

Stress and interrupttesting

Performance Testing

Mobile AppPerformance Testing

Mobile TestAutomation

Test AutomationTools

What should beautomated

How to select theright tool

Mobile TestingMethods

Summary

Mobile Application TestingSE 3S03 - Tutorial 2

Zahra Ali

Department of Computer ScienceMcMaster [email protected]

Week of Feb 8, 2016

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Outline

How to test mobile appsEmulator, Simulator, Real DeviceTraditional TestingMobile Specific TestingBattery Usage TestingStress and interrupt testingPerformance TestingMobile App Performance Testing

Mobile Test AutomationTest Automation ToolsWhat should be automatedHow to select the right tool

Mobile Testing Methods

Summary

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Emulator, Simulator, Real Device

I EmulatorI Emulates real mobile device software, hardware and OS

I SimulatorI Simulates internal behavior but does not emulate

hardware

I Real Device

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Traditional Testing

I Product FocusedI Used to find bugs

I Process FocusedI Protect software from bugs

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Traditional Testing

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Traditional Testing

I Process FocusedI Methods, languages, tools, guidelines, standards

I Product FocusedI Static testingI Dynamic testing

I White box testingI Black box testing

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Traditional Testing

I White Box TestingI Statement coverageI Branch coverageI Decision coverageI Control Flow testingI Data flow testing

I Black Box TestingI Boundary valuesI Decision TableI Equivalence class partitioning

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Specific Testing

I Mobile Functional Testing

I Mobile Usability Testing

I Accesibility Testing

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Functional Testing

I Execute test cases

I Test if features and requirements have been correctlyimplemented

I Test acceptance criteria

I Test all functions

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Usability Testing

I Less is moreI Avoid useless navigation, text/words

I Self explanatoryI Clear and understandable elements

I Pay attention to patternsI Check compliance with design pattern

I Allow mistakesI User friendly error messages, phrase messages according

to target customer, allow user to correct errors

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Usability Testing

I Check the wordingI Easy wording, free of typos, text size in different

languages

I Check consistencyI Same look and feel in all views, platforms, corporate

identity guidelines

I Write it downI Note down differences, screen capture, categorize bugs

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Accesibility Testing

I Visual ImpairmentI Make elements available for screen readersI Adjustable font sizeI Allow changing color contrastsI Voice recognition

I Auditory ImpairmentI Vibrations or visual notificationsI Subtitles for video contentI Volume controlI Mono audio

I Physical ImpairmentI Use voice recognitionI Processing speed

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Accesibility Testing

I Cognitive ImpairmentI Provide easy interfaceI Screen readers, text highlightingI Auto text feature

I Accesibility guidelinesI Available for Android and iOS

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary


I Fully Charged BatteryI ForegroundI BackgroundI Proxy tools to check requests to back end systems

I Low BatteryI 10% to 15% battery leftI Completely empty battery

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary


I Battery Consumption Tools

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Stress and interrupt testing

I ManualI trigger notifications

I AutomaticI Generates events like touch, click, screen rotation, swipeI Example: Monkey tool for android

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Performance Testing

I Performance AreasI ServerI Mobile data networksI Mobile Device and Applications

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile App Performance Testing

I Measure launch time

I Check for delays during user interactions

I Measure content loading time

I Test on different hardware

I Compare live app version with the new candidateversion

I Profiling tools to measure operation time of the code

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Standby Testing

I Put app into standby mode

I Check UI refreshes

I Check data consistency

I Absence of internet connection

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Installation Testing

I Verify installation on local storage or memory card

I Check installation with different internet connections

I Change connection during installation

I Switch to other apps

I Switch internet off

I Install with less memory remaining

I Uninstall

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Update Testing

I Logged in user should not be logged out after update

I Existing app data should not be modified

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Database Testing

I Validation testing

I Integration testing

I Performance testing

I Procedure and Function testing

I Trigger testing

I CRUD operation testing

I Integration with UI

I Security testing

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Local Storage Testing

I When storage is full

I When local storage is full but extended has some space

I When local has some space but extended is full

I When both are full

I When local is nearly full

I Move app to extended storage

I When local storage is empty

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Security Testing

I Common app security problemsI Cache storageI Unintended local storageI EncryptionI Client side validationI Un encrypted communicationI Unnecessary app permissions

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Security Testing

I Security requirement analysisI Identify user roles and permissionsI Impact of user role on back end systemI Security testing approaches and tools

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Platform Guideline Testing

I Keep platform specific guidelines in mindI Include guidelines in design phase

I Android guidelinesI Apple iOS design guidelinesI Apple iOS approval guidelines

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Conformance Testing

I European Telecommunication Standards Institute

I World Wide Web Consortium

I Institute of Electrical and Electronics Engineering

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Bugs

I Bug IDI Bad: 123I Good: AppXYZ-123

I DescriptionI Bad: App crashed, blank screenI Good: Error code 542 on detail message view, Time out

when sending a search request

I Steps to reproduceI Bad: I tried to execute a searchI Good: Start the app and enter Mobile Testing into

search input field. Press search button and you will seeerror code 783

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Bugs

I Expected resultI Bad: It should work, It shouldnt have crashedI Good: expected to see search results with maximum 20

entries

I Actual resultI Bad: It didnt workI Good: The search result page was empty

I Work-aroundI Bad: I found a work aroundI Good: search results are visible if the device is in

landscape mode

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Bugs

I ReproducibleI Bad: SometimesI Good: occurs two out of ten times

I Operating system, platform, deviceI Bad: AndroidI Good: Android, Version 4.1.2 Google Nexus 4

I Device specific informationI Bad: No informationI Good: Battery state=15%, GPS was activated

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Bugs

I Browser versionI Bad: Google ChromeI Good: Google Chrome version 45.4343

I Software build versionI Bad: No informationI Good: App build version 1.2.3

I Network condition and environmentI Bad: Happened while walkingI Good: Connected to 3G while walking through a place

surrounded with buildings

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Bugs

I LanguageI Bad: No informationI Good: German Language

I Test dataI Bad: No informationI Good: Find attached SQL script

I SeverityI Bad: No informationI Good: Critical

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Bugs

I Bug categoryI Bad: No informationI Good: UI, Performance

I Screenshot or videoI Bad: Screenshot1I Good: 01SearchResultPage

I Log filesI Bad: No informationI Good: Attached, with full stack trace

I Tester informationI Bad: No informationI Good: Name. Email: [email protected]

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Test Automation Pyramid

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Flipped Testing Pyramid

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Test Pyramid

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Test Automation Tools

I Image recognitionI Embed screenshots into scriptI Example: eggPlant, RoutineBot

I Coordinate based recognitionI Rely on predefines coordinates to interact with UI

elementsI Examples: MonkeyTalk, Perfecto Mobile

I OCR/text recognitionI To determine if text is present on screenI Examples: eggPlant, MonkeyTalk

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Test Automation Tools

I Native object recognitionI Accesses UI elements using CSS or object ID of

elementsI Examples: Appium, Calabash

I Capture and replayI Captures activity, replays it to test

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

What should be automated

I Business critical parts

I User workflows and scenarios

I Complex app scenarios

I Sequences that need repetition

I Acceptance criteria

I Automate if economically reasonable!

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

How to select the right tool

I Support for different app types

I Mobile platforms

I Recognition technology

I Tests on real device, simulator, emulator

I Test execution time

I UI and control elements

I Gesture support

I Language tests

I Open source or closed source

I Cross platform tests

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Mobile Testing Methods

I Crowd Testing

I Cloud Testing

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Crowd Testing

I Crowd outsourcing

I Diverse backgrounds: geographically and technically

I Progress of the crowd testers is made visible to theclient

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Crowd Testing Process

I Briefing between crowd provider and client

I Crowd provider selects crowd based on clientrequirements

I Crowd tests the software

I Testers file reports

I Provider ensures quality of report

I Provider writes final test report at the end of the testcycle

I Report is presented to client

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Pros

I Testers with different demographic background and skillsets

I Different mobile devices

I App tested in real world conditions by real users

I Provider filters and categorizes bugs

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Cons

I Testers are generally not testing experts

I Reports may be of low quality

I Access to back end systems may be limited due toprivacy issues

I Time taking process

I Reproducing bugs can be difficult

I Testers may keep using the app after cycle has ended

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Cloud Testing

I Wide range of devices with different hardware andsoftware available on the cloud

I The cloud uses a dynamic, shared and virtual ITinfrastructure

I The cloud provides on demand self service

I Priced according to consumption

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Cloud Testing Types

I Functional Testing

I Performance Testing

I Load Testing

I Mobile Device Testing

I Cross Browser Testing

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Pros

I Easy access to physical devices

I Easy access to emulators and simulators

I Fast and easy setup of mobile devices

I Accessibility from anywhere in the world

I Lower costs

I No maintenance cost

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Cons

I Less control over devices

I Network issues

I Security and privacy issues. Delete app after sessioncomplete.

I Sensors and interfaces cannot be tested

I No physical contact with the device

MobileApplication

Testing

Zahra Ali



Traditional Testing




Performance Testing







Summary

Summary

I Mobile usability, accessibility and functional testing

I Performance Testing

I Bug reporting

I Test automation

I Testing methods

MobileApplication

Testing

Zahra Ali

Appendix

References

References I

Daniel KnottHands-On Mobile App Testing.

Mobile Application Testing - SE 3S03 - Tutorial 2khedri/wp-content/uploads/... · testing...

Documents

Transcript of Mobile Application Testing - SE 3S03 - Tutorial 2khedri/wp-content/uploads/... · testing...