Finally… Reliable

Software!

Bryan Bakker

Eindhoven 2015

bryan.bakker@sioux.eu

@Bryan_Bakker

Contents

Intro

Reliability

Four step model

Different steps in detail

Conclusion

© Sioux 2015 | Confidential | 2

Thanks

Thanks to:

Rob de Bie

René van den Eertwegh

Peter Wijnhoven

© Sioux 2015 | Confidential | 3

About Bryan Bakker

Test Expert

Certifications: ISTQB, TMap, Prince2

Member of ISTQB Expert Level on Test Automation

Tutor of several test related courses

Domains: medical systems, professional security

systems, semi-industry, electron microscopy

Specialties: test automation, integration testing, design

for testability, reliability testing

© Sioux 2015 | Confidential | 4

About Sioux

HERENTALS NUENEN

EINDHOVEN

UTRECHT MOSCOW

© Sioux 2015 | Confidential | 5

DANANG

Did you ever experience unreliable software?

Reliability

© Sioux 2015 | Confidential | 6

Reliability

What is reliability?

“Software Reliability is the probability of failure-

free software operation in a specified

environment for a specified period of time.” IEEE 729

In short:

Something can be functional correct

But is it reliable? © Sioux 2015 | Confidential | 7

Reliability

What is reliability?

“Software Reliability is the probability of failure-

free software operation in a specified

environment for a specified period of time.” IEEE 729

In short:

Something can be functional correct

But is it reliable? How reliable is it?

© Sioux 2015 | Confidential | 8

Four steps

© Sioux 2015 | Confidential | 9

Example: Security and Surveillance System

Cameras

Recording

Event Handling

Define user domain reliability targets:

Define customer profiles

Identify operation modes and critical

functions

Determine reliability requirements per

operation mode / critical function

First step

© Sioux 2015 | Confidential | 10

Define customer profiles

ATM security

First step

© Sioux 2015 | Confidential | 11

Define customer profiles

ATM security

Parking lot surveillance

First step

© Sioux 2015 | Confidential | 12

Define customer profiles

ATM security

Parking lot surveillance

Airport surveillance

First step

© Sioux 2015 | Confidential | 13

Identify operation modes and critical

functions

Recording mode

Playback mode

Auto-start (critical)

Out-of-the-box

User triggered

Software reset

First step

© Sioux 2015 | Confidential | 14

Determine reliability requirements per

operation mode / critical function

Segment between 0.5s and 2s missed

failure rate ≤1x per day

Playback command does not function as expected

failure rate ≤ 1x per hour of viewing

Not auto-started failure rate ≤ 3 * 10-7

failures/restart

First step

© Sioux 2015 | Confidential | 15

Define operational profile for

user functions

Decompose software reliability targets

Decompose operational profile with

probabilities per component

Second step

© Sioux 2015 | Confidential | 16

How will the product be used by

customers?

Focus development resources based

on expected usage

Operational profile: a quantitative

characterization of how a system will be used

Developed by John Musa

To be used for development and test

activities

Also very useful for reliability

Operational profile

© Sioux 2015 | Confidential | 17

Critical functions can be missed in

operational profiles.

Treat them separately!

What about?

© Sioux 2015 | Confidential | 18

Define operational profile for

user functions

Second step

© Sioux 2015 | Confidential | 19

Decompose software reliability targets

Not auto-started failure rate <= 3 * 10-7

failures/restart

Stream Handler contribution: 20%

“Target” = 0.6 * 10-7 failures / restart

Second step

© Sioux 2015 | Confidential | 20

Focus on Stream

Handler

component

Decompose operational profile with

probabilities per component

Second step

© Sioux 2015 | Confidential | 21

Playback function Occurrence Probability % Stream Handler commands

Play / Pause (toggle) 48 out of 100 48% Start / Stop Playback

Fast forward 20 out of 100 20% Start Playback

Fast reverse 20 out of 100 20% Start Playback

Setup playback windows

10 out of 100 10% Stop / Start Playback

Search and select event for playback

2 out of 100 2% Start Playback with time stamp = time of selected event

Define the engineering processes

Process steps to prevent reliability

faults

Process steps to detect reliability faults

Design choices to minimize effects of faults

Next steps

© Sioux 2015 | Confidential | 22

Measure software reliability growth

Design and execute reliability tests

Based on the operational profiles

Randomly execute test set according to the

operation profile

Extra focus on critical functions

Visualize reliability growths

Next steps

© Sioux 2015 | Confidential | 23

Reliability growth curve

Next steps

© Sioux 2015 | Confidential | 24

Conclusion

Reliability is not binary but a characteristic

that can be measured

Based on theory of John Musa

Practical 4 step approach

Reliability is not reached by accident

For a full description and worked out case

study, see next page…

© Sioux 2015 | Confidential | 25

Questions

© Sioux 2015 | Confidential | 26

Published February 2015:

www.amazon.com

ISBN: 978-1499226669

Rob de Bie

Bryan Bakker

René van den Eertwegh

Peter Wijnhoven

Backup slides

© Sioux 2015 | Confidential | 27

Reliability Curves

Bathtub curve Hardware Reliability

Sawtooth curve Software Reliability

© Sioux 2015 | Confidential | 28

29 © Sioux 2015 | Confidential |

www.sioux.eu

bryan.bakker@sioux.eu

+31 (0)40 26 77 100