Post on 21-Dec-2015
Reliability Modeling for Design Diversity: A Review and Some Empirical Studies
Teresa Cai
Group Meeting
April 11, 2006
2
Outline
Introduction Reliability modeling on design diversity Empirical Studies Possible directions Conclusion
3
Software reliability engineering techniques and their application domains
Fault (Defect) Fault (Failure)
... -> Defects -> Faults -> Errors -> Failures ->...
AvailabilityReliability Safety Security
AvoidanceFault
RemovalFault (Error)Tolerance Prediction
4
Introduction
Software fault tolerance adopts two main techniques on top of fault avoidance and fault removal Single version techniques:
checkpointing and recovery; Exception handling; Data diversity
Multiple version techniques: Recovery blocks N-version programming N-version self-checking programming
5
N-Version Programming (NVP) fault tolerant software architecture
Version 1
Version 2
Version N
A.T.
A.T.
A.T.
Input
DecisionFunction
VersionOutput
Recovery
Result
Fault
Examples: N-Version Programming (NVP)
6
Introduction
The rationale is the expectation that software components built differently will fail differently
The probability of coincident failures in multiple versions remains the key issue in design diversity
Reliability models attempt to the modeling of reliability and fault correlations in diverse systems
Empirical data are highly demanded for evaluation and cross-validation of the usefulness and/or effectiveness of these models
7
Eckhardt and Lee (1985) Variation of difficulty on demand space Positive correlations between version failures
Littlewood and Miller (1989) Forced design diversity Possibility of negative correlations
Dugan and Lyu (1995) Markov reward model
Tomek and Trivedi (1995) Stochastic reward net
Popov, Strigini et al (2003) Subdomains on demand space Upper bounds and “likely” lower bounds for reliability
Conceptual
models
Structural
models
In between
Current Reliability Modeling
8
Eckhardt and Lee Model
Assumption: Failures of an individual program π are deterministic and a
program version either fails or succeeds for each input value x;
There is a randomness due to the development process. P(π) is the probability that a particular version π will be produced from the set of all possible program version Π
There is a randomness due to the demands in operation. P(x): probability of selection of a given input demands x in the set of all possible demands X.
9
Eckhardt and Lee Model
Score function ω(x) = 0: program πsucceeds for input x ω(x) = 1: program πfails for input x
Difficulty function: the average probability of a program version failing on a given demand
10
Eckhardt and Lee Model
The average probability of failure per demand (pfd) of a randomly chosen single version:
The average pfd of randomly chosen pair of program versions:
11
Eckhardt and Lee Model
If for process A and B, the difficulty functions are identical and constant:
Otherwise, it is always the case that:
12
Littlewood and Miller Model
Assumption: the same as EL model The LM model generalizes the EL model to
take account to forced diversity by defining different distributions over the population of all program.
13
Littlewood and Miller Model
Independence: Cov(A<B) = 0 Positive correlated: Cov(A<B) > 0 Negative correlated: Cov(A<B) < 0
Basic intuition: What you find difficult, I may find easy (or at least
easier) Forced diversity: diverse processes and tech
niques are employed to force the diversity of final program versions
14
Popov and Strigini Model
Alternative estimates for probability of failures on demand (pfd) of a 1-out-of-2 system
15
Popov and Strigini Model
Upper bound of system pfd
“likely” lower bound of system pfd- under the assumption of conditional independence
16
Empirical studies
Various projects have been conducted to investigate and evaluate the effectiveness of design diversity evaluations on the effectiveness and cost issues of the fina
l product of diverse systems Avizienis and Chen (1977) Knight and Leveson (1986) NASA-4 University project (1990)
experiments evaluating the design process of diverse systems Avizienis (1995)
adoption of design diversity into different aspects of software engineering practice Popov & Strigini (2003)
17
Recent simple empirical studies on difficulty functions
From Center for Software Reliability, City University First experiment: “Online Judge”
On-line spec / submission / testing 3444 C program versions Source code: about 20 Input domain: 2500 input pairs
Second experiment: ACM program contest Contest Host problem 2666 program versions fall into 34 equivalence classes,
while 5 classes dominates 98% of the population Input domain: 40401 input pairs
18
First experiment
Difficulty functions
Expected pdfs
Dominated faults are spec faults
19
Second experiment
Difficulty functions with / without spec faults for initial release
20
Second experiment
Difficulty functions without spec faults
21
Second experiment
Comparison of expected probability of failure on demand
22
Second experiment
Observation Difficulty functions are relative flat The increase in the pfd of a diverse pair relative to
the independence assumption is small for all population
Limitations Simple programs The input domain may not be representative for
real world application
23
Our on-going work
Two program pools on RSDIMU project C vs. PASCAL 34 versions vs. 20 versions 100 million test cases generated
Fault correlation analysis in and between: 34 C program versions 20 Pascal program versions Between the two
24
Our objective
Is there any difference between the fault correlation in two different program pools?
Does diversity (different programming languages and process) have any influence on the fault correlation of the final programs?
What’s the benefit or loss in diversity with this empirical study?