Software Testing Attacks for Mobile and Embedded Devices

Software Testing for Mobile and Embedded

Devices

Jon Hagar hosted by XBOSoft

XBOSoft info

• Founded in 2006• Dedicated to software quality

• Software QA consulting• Software testing services

• Offices in San Francisco, Beijing, Oslo and Amsterdam

On the XBO side:

Amy Jan

Housekeeping• Everyone except the speakers is muted

• Questions via the gotowebinar control on the right side of your screen

• Questions can be asked throughout the webinar, we’ll try to fit them in when appropriate

• General Q & A at the end of the webinar

• You will receive info on recording after the webinar

About Jon Hagar

Jon….• has over 30 years in software testing/verification and

validation• is the owner of Grand Software Testing, a company

specializing in software test consulting and training for mobile and embedded systems.

• author of “Software Test Attacks to Break Mobile and Emedded Devices”

Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 6

SOFTWARE TESTING CONCEPTS FOR MOBILE AND EMBEDDED

DEVICES

Jon [email protected]@gmail.combreakingembeddedsoftware.wordpress.com

mailto:[email protected]

mailto:[email protected]

Owner

Need to fix the footer on the slides

Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 7How to Attack Embedded Software 7

AGENDA FOR TODAY

• Introductions• Definitions• Risk based concepts• Exploratory approaches• Attacking the scenario(s)• Mobile and embedded tester skills• Wrap up and references


MOBILE, SMART, AND HANDHELD• As the names imply, these are devices—small, held in the hand, often connected to

communication networks, including

• Cell and smart phones – apps

• Tablets

• Medical devices

• Typically have:

• Many of the problems of classic “embedded” systems

• The power of PCs/IT

• More user interface (UI) than classic embedded systems

• Fast updates

• Are getting more power, memory, and features (software e.g., apps)

• The “hot” area of computers/software

• Testing rules are “evolving”


CLOSE COUSINS:Embedded Software Systems . . .

• Interacts with unique hardware/systems to solve specialized problems in the “real world”

• IT software runs with largely “generic” hardware• Users are barely aware the device uses or has software

• Usually have significant hardware interface issues and concerns• Initialization, noise, power up/down, timers, sensors, etc.

• Often are resource constrained• RAM, ROM, stack, power, speed, time, etc.

• Typically has a restricted or no Human User/Computer Interface (HCI) but is evolving rapidly

• Often no way (or only a risky way) to update and/or change the software• Involves risks, hazards, safety, and/or some specialized domain knowledge

and logic/algorithms usually controlling hardware


WHAT DO THESE LOOK LIKE?

Examples– Avionics systems: planes, cars, rockets, military,…..

– Telecom: switch, routers, phones, cell devices,….

– Transportation: traffic control, railroad, trucking, ….

– Industrial control: lighting, machines, HVAC, nuclear/power,…

– Medical: pacemaker, dispensers, …….

– Home and office systems: control, entertainment (TV box), …

– And the list goes on

• Now starting to include PDA’s and other items that “blur” the lines


TYPES OF MOBILE-EMBEDDED APPS

• Native Applications• Local to device

• Hybrid Applications• Local to device but

interacts w/internet

• Web Applications• Not local to device

All interactions on internet

And JeanAnn Haarrison


THE “WORLD” OF MOBILE-SMART/EMBEDDED SOFTWARE

• Inputs and outputs involve hardware, software, and humans

• Time dependent

• NOTE: most software has “time” (performance) issues but here things are often “hard real time”

• In Mobile/Embedded real-time may be a requirement

Software

Stimulus-Inputs Response-Outputs

Expected

Unexpected

Wanted

Unwanted

BugHardware


MOBILE AND EMBEDDED SOFTWARE TESTING: THE SAME BUT DIFFERENT

• Same patterns in programming errors => same test patterns to be applied

• But there are Differences• The development lifecycle

• Mobile and embedded error patterns• “Embedded” unique sensor (input and output)

• Mobility• Network connections

• Resources


Hardware ChangeHardware New Product

EXAMPLE HIGH LEVEL MOBILE-EMBEDDED LIFECYCLE DIFFERENCE

System Creation

Hardware Build

Software BuildSoftware Build

Software BuildSoftware Build

HwIssue\

Feature

Results: Software is “late”

Software BuildMonths, Days, Hours


THE ERROR (BUG) PATTERN:SIMILAR AND DIFFERENT

• Handheld/Embedded software has similar defects to other software

• Requirements & Design

• Logic & Math

• Control Flow

• Data

• Initialization & Mode changes

• Interfaces

• Security

• Gaming

• etc. . .

Mobile/Embedded additions• Software and hardware

development cycles done in parallel, where aspects of the hardware may be unknown to the software development effort

• Limited resources• Control/use of unique hardware

(sensors)• Hardware problems which are often

fixed with software late in the project

• (a big one) Very tight real-time (often in milli- or micro-second ranges) or load performance issues


MORE SAMPLE MOBILE-EMBEDDED PRODUCT ERRORS TESTERS SHOULD CONSIDER

• Mobility -• Environment and device susceptible to outside influence: noise, weather, heat, cold,

• Network connect: wifi, data, changing signal, etc

• Battery factors: Heat, charge level, charge duration, device conflicts

• Output —outputs to devices and electronics that are susceptible to noise influences, calibrations, changes to hardware, and time

• Inputs – Sensors, speed, timing, and interrupts all can impact devices/software

• Complexity—the size of the system or some aspect of the system makes missed bugs likely

• Device Resources -• Screen size

• Memory

• CPU speed


THUS TESTERS SHOULD THINK: RISKS

• You cannot test everything

• Risk-based testing helps bound the test scope problem

• Testing is about providing information and understanding

• Test exploration gets you started with whatever you have (or don’t have)


IN MOBILE AND EMBEDDED: EXPLORATORY TESTING

• Rapid feedback• Learning• Upfront rapid

learning• Attacking• Address risk

• Independent assessment• Target a defect• Prototyping• Need information• Test beyond the

requirements

Yes, we must test requirements (necessary, but not sufficient)

So include exploratory risk-based testing:


MUCH INFORMATION EXISTS ON EXPLORATORY RISK-BASED TESTING

• Exploratory Testing: Based in the scientific method, where we plan-design, execute, learn, and change the test effort with limited “scripting”

• Check out works by:

• Kaner

• Bach

• Whittaker

• ISO 29119

• I like to recommend and do exploratory attack-based testing


WHY ATTACK?

• Attacking your software is in part, the process of attempting to demonstrate a system (hardware, firmware, software and operations) does not meet requirements, functional and non-functional objectives.

• Mobile/embedded software testing must include "the system" (hardware, software, operations, users)

• Attack common modes of failure, especially where the application is engaged and visible by the user.

Attack testing with approaches to include:

Tools LevelsPatterns Techniques


SO, WHAT IS AN ATTACK?

• Based on a common mode of failure seen over and over• Maybe seen as a negative, when it is really a positive• Goes after the “bugs” that may be in the software• Uses one or more classic test techniques and concepts

• A Pattern (more than a process) which must be modified for the context at hand to do the testing

• Many testers learn these for a domain after years and form a mental “pattern” model (most good testers attack)


KINDS OF ATTACKS

• Whittaker offers a good starting point for software attacks in general that can be applied to embedded:

• User Interface Attacks

• Data and Computation

• File System Interface

• Software/OS Interface

• Whittaker’s “How to Break Software” lists 23 attacks

• “Software Test Attacks to Break Mobile and Embedded Devices” lists 33 attacks and 8 sub attacks


MOBILE AND EMBEDDED ATTACK CLASSIFICATION

• Developer Attacks (unit/code testing)

• Control System Attacks

• Hardware-Software Attacks

• Mobile and Embedded Software Domain Attacks

• Time Attacks (Performance)

• Human User Interface Attacks

• Smart and/or Mobile Phone Functional App Attacks

• Mobile/Embedded Security Attacks

• Generic Attacks• Functional, mind mapping, and combinatorial tests

Jon Hagar Copy right 2013 Software Test Attacks to Break Mobile and Embedded Devices 24

A DETAILED LIST OF ATTACKS • Attack 1: Static Code Analysis

• Attack 2: Finding White–Box Data Computation Bugs

• Attack 3: White–Box Structural Logic Flow Coverage

• Attack 4: Finding Hardware–System Unhandled Uses in Software

• Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs

• Attack 6: Long Duration Control Attack Runs

• Attack 7: Breaking Software Logic and/or Control Laws

• Attack 8: Forcing the Unusual Bug Cases

• Attack 9 Breaking Software with Hardware and System Operations

• 9.1 Sub–Attack: Breaking Battery Power

• Attack 10: Finding Bugs in Hardware–Software Communications

• Attack 11: Breaking Software Error Recovery

• Attack 12: Interface and Integration Testing

• 12.1 Sub–Attack: Configuration Integration Evaluation

• Attack 13: Finding Problems in Software–System Fault Tolerance

• Attack 14: Breaking Digital Software Communications

• Attack 15: Finding Bugs in the Data

• Attack 16: Bugs in System–Software Computation

• Attack 17: Using Simulation and Stimulation to Drive Software Attacks

• Attack 18: Bugs in Timing Interrupts and Priority Inversion

• Attack 19: Finding Time Related Bugs

• Attack 20: Time Related Scenarios, Stories and Tours

• Attack 21: Performance Testing Introduction

• Attack 22: Finding Supporting (User) Documentation Problems

• Sub–Attack 22.1: Confirming Install–ability

• Attack 23: Finding Missing or Wrong Alarms

• Attack 24: Finding Bugs in Help Files

• Attack 25: Finding Bugs in Apps

• Attack 26: Testing Mobile and Embedded Games

• Attack 27: Attacking App–Cloud Dependencies

• Attack 28 Penetration Attack Test

• Attack 28.1 Penetration Sub–Attacks: Authentication — Password Attack

• Attack 28.2 Sub–Attack Fuzz Test

• Attack 29: Information Theft—Stealing Device Data

• Attack 29.1 Sub Attack –Identity Social Engineering

• Attack 30: Spoofing Attacks

• Attack 30.1 Location and/or User Profile Spoof Sub–Attack

• Attack 30.2 GPS Spoof Sub–Attack

• Attack 31: Attacking Viruses on the Run in Factories or PLCs

• Attack 32: Using Combinatorial Tests

• Attack 33: Attacking Functional Bugs


AN EXAMPLE DETAILED TAXONOMY: A2D AND D2A BUG POSSIBILITIES

Type Situation Impact NotesA2D A2D representation information is lost

because measurement is not precise Software computation is based on incorrect data

Number of bits used to store the analog converted data is not large enough or sampling rate to get bits is not correct.

A2D A2D information is contaminated with noise Software computation use noise when it should not

The noise term may not be known, accounted for, or misrepresented.

A2D A2D information is calculated correctly Computation has unknown error

Sources of error can come from: calibrations used on variables, variables lacking initialization, or calculations are not done with enough accuracy (single versus double floating point

D2A D2A conversion losses “least significant bits” (LSB) in conversions, but bits are, in fact, important because computer word sizes are too small

Output to analog device is wrong

Number of bits stored from the digital world to the analog world do not have enough precision, so analog data is incorrect.

D2A D2A information does not account for noise of the real world

Software computation does not include a factor for noise

The analog values are not correct given the noise of the real world (output data may be lost in the noise).

D2A D2A information is calculated correctly because of internal factors

Computation has unknown error

Sources of error can come from: calibrations used on variables, variables lacking initialization, or calculations are not done with enough accuracy (single versus double floating point


FINALLY, IN ADDITION TO TESTER SKILLS: SKILL FOR THE MOBILE/EMBEDDED TESTER

• Tester skills: planning, design, execution, reporting, exploration, techniques, tools…………………….

• Exposure or knowledge about products from the domain in which you are testing: aerospace, medical, automobile manufacturing, smart apps, airplanes, factory systems, robotics, regulated environments, etc.

• Knowledge of hard sciences: math, physics, electronics, hardware, engineering, etc. for logical thought processes

• Knowledge of Soft sciences: psychology, philosophy, sociology, human factors (human machine interface), gaming, arts… for creative & conceptual thought processes

• Passion – follow your bliss


FOR THE SOFTWARE YOU TEST ASK YOURSELF

• Do you know how it fails?

• Do you test for success or failure?

• Both?

• Where can you improve?


SUMMARY: THANK YOU (IDEAS USED FROM)

• James Whittaker (attacks)• Elisabeth Hendrickson (simulations)• Lee Copeland (techniques)• Brian Merrick (testing)• James Bach (exploratory & tours)• Cem Kaner (test thinking)• JeanAnn Harrison

• Many teachers• Generations past and future• Books, references, etc.


BOOK LIST (MY FAVORITES)

• “Software Test Attacks to Break Mobile and Embedded Devices” – Jon Hagar,

• “How to Break Software” James Whittaker, 2003• And his other “How To Break…” books

• “Testing Embedded Software” Broeckman and Notenboom, 2003• “A Practitioner’s Guide to Software Test Design” Copeland, 2004• “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993• “Computer Related Risks”, Neumann, 1995• “Safeware: System Safety and Computers”, Leveson, 1995• Honorable mentions:

• “Embedded System and Software Validation” Roychoudhury, 2009• “Systems Testing with an Attitude” Petschenik 2005• “Software System Testing and Quality Assurance” Beizer, 1987• “Testing Computer Software” Kaner et. al., 1988• “Systematic Software Testing” Craig & Jaskiel, 2001• “Managing the Testing Process” Black, 2002

Questions & Answers

Follow us: @XBOSoftQuestions at: [email protected]

Thanks for joining us and hope to hear from

you soon!

Software Testing Attacks for Mobile and Embedded Devices

Technology

Transcript of Software Testing Attacks for Mobile and Embedded Devices