Software Architecture based Performance and Reliability Evaluation Vibhu S. Sharma Ph.D. Scholar...

Software Architecture based Software Architecture based Performance and Reliability Performance and Reliability EvaluationEvaluation

Vibhu S. SharmaVibhu S. Sharma

Ph.D. Scholar Ph.D. Scholar

CSE, IITkCSE, IITk

Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur

Software architecture is the fundamental Software architecture is the fundamental organizationorganization of a system, embodied in its of a system, embodied in its componentscomponents, their , their relationshipsrelationships to each other and to each other and the the environmentenvironment, and the principles governing its , and the principles governing its design and evolution. - design and evolution. - IEEE 1471-2000.IEEE 1471-2000.

Software architecture styles are like templates Software architecture styles are like templates which are followed by commonly used systems:which are followed by commonly used systems:• Layered style : Web-based systemsLayered style : Web-based systems• Pipe and Filter style : Compilers Pipe and Filter style : Compilers • . . .. . .

Software ArchitectureSoftware Architecture

Client 1

Web-server

Application Server

Client 2

Client nc

A web-based software system

DB


Modeling the workflow of the Software Modeling the workflow of the Software ArchitectureArchitecture

We use finite absorbing Discrete Time Markov We use finite absorbing Discrete Time Markov Chains (DTMC) for representing the software Chains (DTMC) for representing the software architecture workflow.architecture workflow.

Examples:Examples:

These DTMCs can be used to find the average These DTMCs can be used to find the average visit counts to various components for a typical visit counts to various components for a typical request as well as the limiting probabilities.request as well as the limiting probabilities.

Very useful for reliability as well as performance Very useful for reliability as well as performance prediction.prediction.

1 2 3 Cp12 p23 1

1-p12

1-p23

ESA Antenna array configuration tool architecture

S0

S1 S2 Snl-1 Snl

S2nl+1

1

1p12 p23 p(nl-2)(nl-1) p(nl-1)nl

. . . . . . . . . .

Snl+1 Snl+2S2nl-1 S2nl

11111

. . . . . . . . . .

1- p(nl-1)nl1-p231-p12

1A typical layered architecture


Architecture based Reliability predictionArchitecture based Reliability prediction Issues/Questions:Issues/Questions:

• What is the reliability of a software system What is the reliability of a software system composed of unreliable components?composed of unreliable components?

• How does increasing the reliability of a How does increasing the reliability of a particular component affects overall particular component affects overall reliability ?reliability ?

• How does changing the architecture affect How does changing the architecture affect reliability ?reliability ?

Reliability modeling using DTMCsReliability modeling using DTMCs • Model the component failure as a state in Model the component failure as a state in

the DTMC and associate probabilities of the DTMC and associate probabilities of transition (unreliabilities). transition (unreliabilities).

• Overall reliability is the probability of Overall reliability is the probability of ending in the correct state.ending in the correct state.

• Another option is using a reward based Another option is using a reward based method method

1

23

4

5

S

F

pf1

pf2

pf3

pf4

pf5

(1-pf1)p12


Architecture based performance Architecture based performance predictionprediction

Issues/Questions:Issues/Questions:• What is the maximum number of clients the

system can able to handle before it saturates ?• What effect does varying the number of clients

have on the throughput and the average response time ?

• How should software components be allocated to the hardware nodes ?

• What changes should be made in the system to improve performance ?

We take an example of layered software We take an example of layered software architecture.architecture.• A Discrete Time Markov Chain (DTMC) is used

to characterize the control flow in the layered software architecture.

• The DTMC is used to calculate the visit counts to various layers and then a closed queueing network (QN) model is constructed and solved.

Client 1

Layer 1 Layer 2 Layer nl Client 2

Client nc The Software System

S0

S1 S2 Snl-

1

Snl

S2nl+

1

1

1 p12 p23 p(nl-2)(nl-1) p(nl-1)nl

. . . . . .

. . . .

Snl+

1

Snl+

2

S2nl-

1

S2nl

11111

. . . . . .

. . . .

1- p(nl-1)nl1-p231-p12

1


Some Analysis ResultsSome Analysis Results

Average response time and throughput and bottleneck analysis

Effect of change in individual reliabilities & workload on overall reliability


Reliability and Performance TradeoffReliability and Performance Tradeoff

Given a set of components and a set of machines to deploy these onto• What are the performance & reliability characteristics of the setup

under a given load ?• Which architecture should be chosen for optimal performance and

reliability ?• What is the tradeoff ?

Software failures of different types cause different effects on the performance of the system.

Hardware availability also needs to be taken into account.Hardware availability also needs to be taken into account. Some configurations will result in high throughput with Some configurations will result in high throughput with

decreased reliability and vice-versa.decreased reliability and vice-versa.

Combining Static Analysis and Combining Static Analysis and Testing Frameworks Testing Frameworks

to find Software Defectsto find Software Defects

Vipindeep VVipindeep V


Defect Prevention during Software Defect Prevention during Software Development Development

Write Code Compile/Build Customer

Evaluate

ProgramSpecification

Static analysis Test Suite

Testing


Testing + Static Analysis – How?Testing + Static Analysis – How? Static Analysis Vs TestingStatic Analysis Vs Testing

• Different areas altogether, but motive – identify defectsDifferent areas altogether, but motive – identify defects• Combining the approaches… improved program checkingCombining the approaches… improved program checking

Motivations for the Proposed ApproachMotivations for the Proposed Approach• Static analysis concernsStatic analysis concerns• Testing is done.. Bugs are revealedTesting is done.. Bugs are revealed• Ideal test suite => identify all bugsIdeal test suite => identify all bugs• Non-ideal scenario of TestingNon-ideal scenario of Testing

Identify defects, partiallyIdentify defects, partially Some paths/blocks are “not (properly) tested” .. Why??Some paths/blocks are “not (properly) tested” .. Why??

• Static analysis effort on unexplored parts of the codeStatic analysis effort on unexplored parts of the code

Our proposed approach Our proposed approach • Focused Static Analysis with Path Pruning using Coverage Focused Static Analysis with Path Pruning using Coverage

Data Data


Overview of the approachOverview of the approach


Pros and ConsPros and Cons

• Advantages:Advantages:1.1. New real errors can be identified which escaped New real errors can be identified which escaped

in _all_paths analysisin _all_paths analysis

2.2. Rigorous testing effort counted for better static Rigorous testing effort counted for better static analysisanalysis

3.3. Reduce execution timeReduce execution time

4.4. Reduced warnings => Reduced NoiseReduced warnings => Reduced Noise

• Disadvantages:Disadvantages:1.1. Some errors escape in the pruned path analysisSome errors escape in the pruned path analysis

2.2. Depends on testing frameworkDepends on testing framework


Applied the approach with FindBugs, Jlint for JAVAApplied the approach with FindBugs, Jlint for JAVA Results from testing were encouragingResults from testing were encouraging Tested the applicability using Random Path Selection Tested the applicability using Random Path Selection

% of Blocks Pruned VS Percentage of new errors

0

2

4

6

8

10

12

14

0 5 10 15 20 25 30 35 40 45 50

% of Blocks Pruned

% o

f n

ew e

rro

rs

Series1

Preliminary ResultsPreliminary Results


ConclusionsConclusions Works well in practiceWorks well in practice

• Finds enough real defects to be usefulFinds enough real defects to be useful• Noise is low enough that people can use itNoise is low enough that people can use it• Scales well, so works on large very large code Scales well, so works on large very large code

basesbases More issues to explore..More issues to explore..

• Better algorithm to use testing dataBetter algorithm to use testing data• Path profiles, Crash dumps – focused static analysisPath profiles, Crash dumps – focused static analysis• Use static analysis results for focusing the testing Use static analysis results for focusing the testing

efforteffort• Combining static and dynamic analysisCombining static and dynamic analysis• Usability related issuesUsability related issues

State Based Testing State Based Testing AutomationAutomation

Atul GuptaAtul Gupta

Ph.D. ScholarPh.D. Scholar


State Based TestingState Based Testing

Program behavior can be modeled as a Program behavior can be modeled as a state graphstate graph

A formal testing approach at all levels-unit, A formal testing approach at all levels-unit, component or system level. component or system level.

Effective testing strategy to O-O softwareEffective testing strategy to O-O software Test Cases and Test Oracles can be Test Cases and Test Oracles can be

automatically generated and evaluatedautomatically generated and evaluated

Atul Gupta PhD Scholar


Class CoinBox {unsigned totalQtrs;unsigned curQtrs;unsigned allowVend;

public:CoinBox( ) {

totalQtrs = 0;curQtrs = 0;allowVend = 0;

}void retQtrs( ) {

curQtrs = 0;}void addQtr( ) {

curQtrs = curQtrs +1;if (curQtrs > 1)allowVend = 1;

}void vend( ) {

if (allowVend) {totalQtrs = totalQtrs + curQtrs;curQtrs = 0;allowVend = 0;}

}};

An Example : A CoinBox ClassAn Example : A CoinBox Class

WAIT Q1 Q2

allowVend = 0curQtrs = 0


allowVend = 1curQtrs >= 2

addQtr( ) addQtr( )addQtr( )

retQtrs( )

retQtrs( )

vend( )


State Based Testing CriteriaState Based Testing Criteria

All-Transition Coverage (AT)All-Transition Coverage (AT) All-Transition-pair Coverage (ATP)All-Transition-pair Coverage (ATP) Full Predicate coverage (FP)Full Predicate coverage (FP) Transition-Tree Coverage (TT)Transition-Tree Coverage (TT) All-Round-Trip path coverage (ART)All-Round-Trip path coverage (ART) Complete Sequence (CSComplete Sequence (CS))

WAIT Q1 Q2





retQtrs( )

retQtrs( )

vend( )


Generating Test Generating Test SequencesSequences

CriterionCriterion PrefixPrefix(State)(State)

Test SequenceTest Sequence Test OracleTest Oracle(State)(State)

All-Transition (AT):All-Transition (AT): WAITWAITQ1Q1Q2Q2

addQtr( )- retQtrs( )addQtr( )- retQtrs( )addQtr( )-addQtr()- retQtrs( )addQtr( )-addQtr()- retQtrs( )vend( )vend( )

WAITWAITWAITWAITWAITWAIT

All-Transition-Pair All-Transition-Pair (ATP)(ATP)

At Q1At Q1 WAITWAITWAITWAIT

addQtr( )- retQtrs( )addQtr( )- retQtrs( )addQtr( )- addQtr( )addQtr( )- addQtr( )

WAITWAITQ2Q2

At Q2At Q2 Q1Q1Q1Q1Q1Q1Q2Q2Q2Q2Q2Q2

addQtr( )- retQtrs( )addQtr( )- retQtrs( )addQtr( )- addQtr( )addQtr( )- addQtr( )addQtr( )- vend( )addQtr( )- vend( )addQtr( )- retQtrs( )addQtr( )- retQtrs( )addQtr( )- addQtr( )addQtr( )- addQtr( )addQtr( )- vend( )addQtr( )- vend( )

WAITWAITQ2Q2WAITWAITWAITWAITQ2Q2WAITWAIT

At WAITAt WAIT Q2Q2Q1Q1Q2Q2

retQtrs( )- addQtr( )retQtrs( )- addQtr( )retQtrs( )- addQtr( )retQtrs( )- addQtr( )vend ( )- addQtr( )vend ( )- addQtr( )

Q1Q1Q1Q1Q1Q1

Transition-Tree (TT)Transition-Tree (TT)

WAITWAITWAITWAITWAITWAITWAITWAIT

addQtr( )- retQtrs( )addQtr( )- retQtrs( )addQtr( )- addQtr( )-retQtrs( )addQtr( )- addQtr( )-retQtrs( )addQtr( )- addQtr( )-vend( )addQtr( )- addQtr( )-vend( )addQtr( )- addQtr( )- addQtr( )addQtr( )- addQtr( )- addQtr( )

WAITWAITWAITWAITWAITWAITQ2Q2

WAIT Q1 Q2





retQtrs( )

retQtrs( )

vend( )

WAIT

Q1

Q2

addQtr( )

addQtr( )

addQtr( )

retQtrs( )

retQtrs( )

vend( )

WAIT

WAIT Q2WAIT


Research IssuesResearch Issues

How to generate test cases How to generate test cases automatically from a state modelautomatically from a state model

How to execute them in an How to execute them in an automated mannerautomated manner

How to evaluate the results How to evaluate the results automaticallyautomatically


Our ApproachOur Approach

Identify State Model of a Class Identify State Model of a Class Generate test sequences based on Generate test sequences based on

some Coverage Criteriasome Coverage Criteria Convert these sequences into JUnit Convert these sequences into JUnit

format for Automatic Execution and format for Automatic Execution and Result evaluationResult evaluation

AIM : Complete end-to-end AutomationAIM : Complete end-to-end Automation

Software Architecture based Performance and Reliability Evaluation Vibhu S. Sharma Ph.D. Scholar...

Documents

Transcript of Software Architecture based Performance and Reliability Evaluation Vibhu S. Sharma Ph.D. Scholar...