Mobile App Testing: Design Automation Patterns You Should Use
-
Upload
techwellpresentations -
Category
Software
-
view
33 -
download
0
Transcript of Mobile App Testing: Design Automation Patterns You Should Use
6/2/15
1
MOBILE APP TESTING: DESIGN AUTOMATION PATTERNS YOU SHOULD USE (OR CONSIDER USING)
Jon Duncan Hagar, Grand Software Testing
Grand Software Testing (GST) Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
Where is the App World Today?
Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
2
What do you think of when we say mobile test automation?
3 Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
Many Automation Tools
Definitions: Tool - any aid for doing your job (not just software) Automation - Hardware and software that helps get the job done Examples
4 Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
3
• Introduction
• Problem
• Automation you may not have thought about
• Implications, Conclusions and Recommendations
• Summary/Conclusions
• References
Agenda
5 Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
• Software is now in most systems • CPUs, microprocessors, FPGAs, etc. • Provides features, flexibility and “smarter systems”
• V&V/Test of software has been advocated for years, but • Requirements verification-checking is necessary, but not sufficient o Continued existence of fielded “issues” can be COSTLY
• Numerous concepts, ideas, and tools exist o Project context determines how to mix and match - Context includes: budget, schedule, skills, regulations, and domain
o There is NO best practice, tool, or single technique
There is NO MAGIC
Problem: Mobile System-Software Quality Is Important
6 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
4
Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices” 7
Hard situation - We need to find bugs, but where? - We need good user experiences, but how?
Understanding an Issue Taxonomy
Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices” 8
Taxonomy (researched) Super Category
Aero-‐Space Med sys Mobile General Time 3 2 3 Interrupted -‐ SaturaQon (over Qme)
5.5 Time Boundary – failure resulQng from incompaQble system Qme formats or values
0.5 1 Time -‐ Race CondiQons
3 1 Time -‐ Long run usages 4 1 20 Interrupt -‐ Qming or priority inversions
0.7 3 Date(s) wrong/cause problem
0.5 1 Clocks 4 2 ComputaQon -‐ Flow 6 23 19 ComputaQon -‐ on data 4 1 3 1
6/2/15
5
Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices” 9
Taxonomy part 2 Super Category
Aero-‐Space Med sys Mobile General Data (wrong data loaded or used) 4 5.00 2 IniQalizaQon 6 2.00 3 5 Pointers 8 2.00 18 10 Logic and/or control law ordering
8 43 3 30 Loop control –Recursion
1 Decision point (if test structure) 0.5 1 1 Logically Impossible & dead code
0.7 OperaQng system – (Lack of Fault tolerance , interface to OS, other) 1.5 2 6 Software - Hardware interfaces
16 13 SoCware -‐ Software Interface
5 2.00 3 SoCware -‐ Bad command- problem on server 3 5 UI -‐ User/ operator interface
4 5.00 20 10 UI -‐ Bad Alarm 0.5 3 UI -‐ Training – system fault resulQng from improper training
3 Other 10.6 9.00 5 5
Note: one report on C/C++ indicated 70% of errors found involved pointers
LET’S CONSIDER POSSIBLE AUTOMATED TOOL OPTIONS
10 Copyright 2015 Jon D. Hagar – SoCware Test AFacks to Break Mobile and Embedded Devices
6/2/15
6
AUTOMATED MOBILE TESTING TO IMPROVED THE USER EXPERIENCE
Developer testing Exploratory Testing
with capture playback and regression automation Usability Checklist
11 • Copyright 2015 Jon D. Hagar – Software Test Attacks to Break Mobile and Embedded Devices
Unit – Developer Testing Attacks
12 Copyright 2015 Jon D. Hagar – SoCware Test AFacks to Break Mobile and Embedded Devices
Example Automation Concepts Static Code Analysis Tool Review/Inspection Tool Databases Online Modeling – classes and boundary Combinatorial tool Data generation tool Data dictionary database Code Coverage Tool
- Levels Stub-Drivers Metric analysis
6/2/15
7
Most developers love to Automate (it is their job) Agile and TDD expect it Continuous integration
What can testers do?
13 Copyright 2015 Jon D. Hagar – SoCware Test AFacks to Break Mobile and Embedded Devices
Mobile Capture Playback Tooling Supporting Exploratory Testing
What? Impossible?
Inconceivable?
14 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
8
• Verification checking (tests) of requirements is most common
• Additionally, successful teams practice concepts such as risk-based and exploratory attack-based testing
• ISO 29119 is a risk-based testing standard • Whittaker, Hagar, and others advocate • Attacks are design patterns which can include many test techniques • Allows rapid test exploration due to lack of highly scripted tests • Requires “skilled” test teams
• Exploratory testing must be balanced with other V&V
Attack and Risk-based Software Test Planning with Exploration Concepts and Automation
15 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
Conduct exploration while running a capture-playback tool Plan exploration with a risk-based analysis tool Design exploratory test data with a combinatorial test tool
- Boundary value analysis - Equivalence classes
Emulators and simulators to support exploratory testing “Rack” or cloud hardware device testing
Mobile Exploration with Automation (Examples)
16 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
9
The Simple UI Checklist “Tool”
17 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
Database of what has been checked On-line Reviews Pop-up reminders on work flow
MATH -BASED TESTING WITH AUTOMATION Combinatorial and others
18 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
10
Underused Math-based (Tool) Concepts for System and Software Level Testing
19
Math-based Concepts for System and Software Level Testing
General Technique Concept Tool Examples Examples where techniques can be used
Specific sub- technique examples
Combinatorial Testing
ACT [4], Hexawise[5] Medical, Automotive, Aerospace, Information Tech, avionics, controls, User interfaces
Pairwise, orthogonal arrays, 3-way, and up to 6-way pairing are now available
rdExpert [6]
PICT[7]
Design of Experiments DOE ProXL[8] Hardware, systems, and software testing where there are "unknowns" needing to be evaluated
Taguchi [12]
(DOE) DOE++ [9] JMP [10] DOE
Random Testing Random number generator feature used from most systems or languages
Chip makers, manufacturing quality control in hardware selection
Testing with randomly generated numbers includes: fuzzing and use in model-based simulations
Statistical Sampling SAS [10] Most sciences, engineering experiments, hardware testing, and manufacturing
Numerous statistical methods are included with most statistical tools
Software Black box Domain Testing
Mostly used in manual test design, though some tools are now coming available [11]
All environments and types of software tests. These are “classic” test techniques, but still underused
Equivalence Class, Boundary Value Analysis, decision tables
Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
• Not clear if engineers know about these math-based concepts or if they are restricted due to budget/schedule constraints
• Tools and training are improving for many of these concepts
• NIST – Book and ACTS tool
• Kaner et. al. – Advanced book and training on domain testing
• Management, tool and expert support for these concepts needs to continue
Underused Math-based Concepts
20 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
11
Detailed Example from Attack 32: Combinatorial Tests
21
When to apply this attack? • There are numerous related variables and
variable values which interact, while there maybe larger numbers of combinations that other math based approaches can easily handle
What faults make this attack successful?
• An organized and minimized selection approach but with “coverage” of pairings, e.g. 1600 samples reduced to 64
Who conducts this attack? • Tester, analyst
Where is this attack conducted? • Tool running in the lab or field
How to determine if the attack exposes failures?
• A test fails to meet success criteria
How to conduct this attack • Identify combinatorial situation • Identify combinatorial tool • Identify variables • Identify values • Identify constraints on values • Enter variables and values into tool with constraints • Exercise resulting combinations in usage scenario tests or automated tests • Look for failure • Repeat and refine as needed Note: may not be this simple
Copyright 2015 Jon D. Hagar – So7ware Test A=acks to Break Mobile and Embedded Devices
SAE INTERNATIONAL
A Possible Cool Future
Combinatorial testing with high numbers of cases based in equivalent
classes or edge bases
and
no oracles needed to find bugs 22
Copyright 2015 Jon D. Hagar – Software Test Attacks to Break Mobile and Embedded Devices
6/2/15
12
MORE POSSIBILITIES
MODELING
WITH TOOLS TO SUPPORT MOBILE APPS
Copyright 2015 Jon D. Hagar – Software Test Attacks to Break Mobile and Embedded Devices 23
• Interest and use of model-based testing is growing in industry segments • Telecom, finance, automotive, aero, space • European and U.S. interests • UML and UML testing profile (UTP)
• Model-based testing with tool automation can support (examples): • Generation of test cases from models into test automated execution
engines directly using scripts or through the use of keywords • Improved understanding of the system and risks • Use of models to support simulations to drive test environments • Verification via compares between development and test models • Generation of test result oracles or judges • Support of independent testing such as Independent V&V (IV&V) • Model analysis
Model-based Testing in Mobile Apps
24 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
13
25
An Example Test Flow with Modeling
Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
• OMG UTP and ISO Standards currently in place
• Tools to support second “test” analysis model • Produce test automation • Graphic views aid understanding • Serve as an oracle
• Aids in avoidance and/or identification of issues early in lifecycle
• Considerations for growth and continuing usage • N-version problem • Self-checking problem if only one model is created • Skilled modelers and testers needed • Correct development/test environment must be place
Model-based Test Advantages and Considerations
26 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
14
FASTER DEV-OPS/AGILE FEEDBACK FROM TOOL AUTOMATION
27 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
THIS IS THE END MY FRIEND
Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
15
Software Attacks with Automated Exploratory Test
29
Software Test Attack Type Attack Finds Tool-Automation Notes on the Attack
Developer level attacks Code and data structure problems Automate, Automate, Automate
Control system attacks Hardware and software control system errors Modeling Automation
Hardware-software attacks Hardware and software interface issues Automate in cloud or “rack”
Communication attacks Digital communications problems Emulator and simulators
Time attacks Time, performance, sequence, and scenario errors
Load and stress testing using captured scenarios
User interface attacks Problems between man and machine Checklist
Smart/Mobile attacks Issues specific to smart device configurations including cloud issues
Lifecycle and move to left may mean automation
Security test hacking attacks
Software errors that can expose devices to security threats
Fuzzing, Pen-attacks, and identity spoofing
Generic functional verification attacks
Requirements and interoperability errors Modeling, mind mapping, and combinatorial testing
Static code analysis attacks Hard to find errors that classic testing often misses
Testers run the static code analysis tools
Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
• When done correctly and continuously with automation V&V- testing will be valued
• Activities will vary from project to project since there is no one “best” • Combination of automation, V&V, and testing actions needed
• Developers, support, testers, engineers, and customers
• Specifically underused test bases: • Exploratory testing supported by attacks • test automation (more than just execution of tests) • mathematical techniques • model-based
• Testers and project staff should increase their knowledge and skill
Summary and Conclusions
30 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”
6/2/15
16
Notes: Thank You (ideas used from)
• James Whittaker (attacks) • Elisabeth Hendrickson (simulations) • Lee Copeland (techniques) • Brian Merrick (testing) • James Bach (exploratory and tours) • Cem Kaner (test thinking) • Jean Ann Harrison (her thinking and help)
• Many teachers • Generations past and future • Books, references, and so on
Copyright 2015 Jon D. Hagar – SoCware Test AFacks to Break Mobile and Embedded Devices
Book Notes List (my favorites) “Software Test Attacks to Break Mobile and Embedded Devices”
– Jon Hagar “How to Break Software” James Whittaker, 2003
And his other “How To Break…” books “A Practitioner’s Guide to Software Test Design” Copeland, 2004 “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993 “Computer Related Risks”, Neumann, 1995 “Safeware: System Safety and Computers”, Leveson, 1995 Honorable mentions:
“Systems Testing with an Attitude” Petschenik 2005 “Software System Testing and Quality Assurance” Beizer, 1987 “Testing Computer Software” Kaner et. al., 1988 “Systematic Software Testing” Craig & Jaskiel, 2001 “Managing the Testing Process” Black, 2002
Copyright 2015 Jon D. Hagar – SoCware Test AFacks to Break Mobile and Embedded Devices
6/2/15
17
More Resources
• www.stickyminds.com – Collection of test info • www.embedded.com – info on attacks • www.sqaforums.com - Mobile Devices, Mobile Apps -
Embedded Systems Testing forum • Association of Software Testing
– BBST Classes http://www.testingeducation.org/BBST/
• Your favorite search engine
• My web sites and blogs (listed on front page)
Copyright 2015 Jon D. Hagar – SoCware Test AFacks to Break Mobile and Embedded Devices
1. IEEE 1012, Standard for System and Software Verification and Validation- http://standards.ieee.org/findstds/standard/1012-2012.html, IEEE press, 2012 2. ISO 29119, Software Test Standard - http://www.softwaretestingstandard.org/ 3. Hagar, J. Software Test Attacks to Break Mobile and Embedded Devices, CRC press, 2013 4. Kuhn, Kacker, Lei, Introduction to Combinatorial Testing, CRC press, 2013 (includes the tool ACTS) 5. Tool: Hexawise - app.hexawise.com/ 6. Tool: rdExpert – www.phadkeassociates.com/ 7. Tool: PICT – msdn.microsoft.com/en-us/library/cc150619.aspx 8. Reagan, Kiemele, Tool: DOE Pro XL - Design for Six Sigma, Air Academy Associates, self publish, 2000 9. DOE++ - www.reliasoft.com/ 10. SAS - www.sas.com/ 11. Kaner, Hoffman, Padmanabhan, The Domain Testing Workbook, self publish, 2013 12. Bailey, Design of Comparative Experiments. Cambridge University Press, 2008 13. Kacker, Kuhn, Hagar, Wissink, "Introducing Combinatorial Testing to a Large System-Software Organization,” scheduled-2015, IEEE Software 14. Whittaker, James 2003, How to Break Software, Pearson Addison Wesley 15. Whittaker, James and Thompson, Herbert, How to Break Software Security, Pearson Addison Wesley, 2004 16. Andrews, Whittaker, How to Break Web Software, Pearson Addison Wesley, 2006 17. Levy, Tools of Critical Thinking: Metathoughts for Psychology, 1996 18. Bach, Bolton, “Testing vs. Checking,” www.developsense.com/blog/2009/08/testing-vs-checking/ 19. Hagar, “Why didn’t testing find the embedded GM Truck fire system error?”- www.breakingembeddedsoftware.wordpress.com/ 20. OMG UTP 1.2, www.omg.org/spec/UTP/1.2/ 21. Baker, Dai, Grabowski, Schieferdecker, Williams, “Model-Driven Testing:Using the UML Testing Profile,” 2008 22. Green, Hagar, “Testing Critical Software: Practical Experiences,” IFAC Conference 1995 23. Boden, Hagar, “How to Build a 20-Year Successful Independent Verification and Validation (IV&V) Program for the Next Millennium,” Quality Week Conference 1999 24. Port, Nakao, Katahira, Motes, Challenges of COTS IV & V, Springer press, 2005
34
References
Copyright 2015 Jon D. Hagar “SoCware Test AFacks to Break Mobile and Embedded Devices”