CmpE 550 – Advanced Software Engineering

Selim Özyılmaz & Elif SürerSelim Özyılmaz & Elif Sürer

““Scenario-Based Assessment of Scenario-Based Assessment of Nonfunctional Requirements”Nonfunctional Requirements”

INTRODUCTION

Purpose: Validation of Requirements Specification using Scenarios

“Scenarios are applied to the analysis of non-functional requirements using dependency tables to assess the relationship between goals (functional and non-functional requirements) and the agents and tasks that achieve them in the i* language”

Introduction

Many NFR’s are influenced by human properties, they inherit the diverse nature of human characteristics (System Reliability is influenced by human characteristics such as ability)

This work is a revised version of a previous one that prompted designers with questions about potential problems in a scenario event sequence. (Psychology-based taxonomy of failure)

Intro(Cont’d)

The problem is that too many scenario variations were generated. To prevent, transform the taxonomy of human and system failures into a model to predict the errors in the system design.

Bayesian Belief Nets (BNs) are used to predict the reliability.

Intro(Cont’d)

Model-checking techniques have been used extensively to verify and validate requirements.

Communication problem occurs between user-stakeholders and the model developers.

Software Cost Reduction (SCR)

[ tabular representation ]

Related Work

Related Work

A combination of visualizations, examples, and simulations is necessary to explain complex requirements to end users.

Scenario-based representations and animated simulations help users see the implications of system behavior and, thereby, improve requirements validation.

Related Work

Related Work

Animation simulation tools -> by Dubois et al. in the ALBERT II language

KAOS language and supporting GRAIL tool which enable formal reasoning about dependencies between goal model, system behavior and constraints.

Animator-validator tool, TROLL, uses a formal object-oriented language for modeling information systems

Related Work

Related Work

What is a sufficient set of scenarios to enable validation to be completed with confidence?

While we believe there is no quick answer to this vexing problem, one approach is to automate the process as far as possible so more scenarios can be tested.

Related Work

Related Work

Intent specifications provide a hierarchical model to facilitate reasoning about system goals and requirements in safety critical systems.

Goals are decomposed in a means-ends hierarchy, widely practiced in requirements engineering.

Automated support for reasoning about conflicting system states and behavior is provided by the SpecTRM-RL tool.

Related Work

Related Work

Assessment of nonfunctional system requirements, such as system reliability, has to use probabilistic reasoning since the range of potential system behaviors is either unknown, in the early requirements phase, or too large to specify.

Bayesian Nets (BNs) have been developed to assess software quality from properties of the code and software engineering process.

Related Work

Related Work

BNs have been widely applied as a probabilistic reasoning technique in software engineering and other domains; however, previous work used single nets to evaluate a set of discrete states pertaining to a software product or development process.

More automated tools for scenario analysis of NFR conformance for requirements specifications with multiple BN tests have been developed.

Related Work

Modeling Uncertainity

There are four different techniques to model: Bayesian Probability Dempster-Shafer Fuzzy Sets Possibility Theory

Bayesian Probability offers an easier combination of multiple influences on probability than Dempster-Shafer and a sounder reasoning than Fuzzy sets.

Bayesian Probability provides a decision theory of how to act on the world in an optimal fashion under circumstances of uncertainity.

Modeling Uncertainty

Bayesian Belief Nets

Directed acyclic graphs of causal influences, where the nodes represent the variables and the arcs represent the relationships between variables.

Variables can have any number of states in a BN, so the choice of measurement is left to the analyst.

Network Probability Table (NPT)

BN (Cont’d)

Input evidence are propagated through the network updating the values of other nodes (Computationally complex but effective algorithms exist)

BN (Cont’d)

Three possible ways to compute the probability of each node in the network Input a posterior probability into a BN as a prior

observation each run should be assumed to be independent

Combine output probabilities from a sequential run using a summarizer probabilities of particular run has to be set initially

Output probability of each event is compared with a threshold value, if surpassed success else failure

Pinpoint some steps in the scenario which are weak in reliability

Sensitiviy analysis can be carried out with multiple BN runs by varying environmental variables

BN Model of System Reliability

Influencing factors are divided into two: Slips: Attention based lapses and omissions in

skilled behavior Mistakes: failures in plans and knowledge of

processing. System environmental variables have an indirect

effect on an individuals ability whereas organizational factors (management culture) have a direct effect.

High Cognitive Complexity more prone to mistakes High Physical Complexity more prone to slip errors

System Reliability

Remarks– Input variables are all discrete states– The BN is a run with a range of scenarios

that stress-test system design against operational variables.

– Scenarios can be either taken from domain-specific operational procedures or by interviewing with users

– Two different outputs: slips and mistakes

Operational Performance Time

Same variables with reliability case, the difference is that except two output nodes only one output node

The output is used to increase the best case task completion time to reflect the less than ideal properties of human and machine agents.

ET: estimated time BT: best task completion time To reflect the case of reverting to manual when an

automated technology fails, highly automated tasks worst completion time is set greater than the those of manual tasks

SRA System Architecture

Analysis starts with the selection of the i* model to be evaluated and creating the test scenarios.

Scenarios are narratives taken from real life experience describing operation of similar systems from which event sequences are extracted.

SRA Tool’s Components

The Session Controller implements the user command interface for selecting designs and scenarios and executes the algorithm that assesses a set of scenarios with the BNs. It calls the system reliability or operational performance BN assessors to execute the BN runs with all possible environmental combinations.

SRA Tool’s Components

The i* model editor allows interactive construction of i* models with typical CASE tool-type functions.

The Interactive Scenario Constructor produces test scenarios from the system model based on user directions. Scenarios are stored in a database in an array of tuples.

SRA Tool’s Components

The Model Controller controls the BN models. It selects the appropriate BN model for each task step, then populates the input nodes, runs the model, and receives the belief distributions of the output nodes.

The Model Controller also manages the back propagation of the BN model to identify required technology and agent characteristics.

SRA Tool’s Components

SRA Tool’s Components

The BN assessor modules run the net by calling the HUGIN algorithm for each task step and for each set of environmental variable combinations.

The output from each run is compared with the desired NFR threshold and the survivor runs are passed to the results visualizer.

SRA Tool’s Components

SRA Tool’s Components

The Visualizer provides a visual summary of all qualified BN runs for a set of scenarios for one or more system designs.

This enables different designs to be compared and problem areas in the requirements to be identified, i.e., task/technical component combinations which show low potential NFR assessments. The Visualizer displays results at three levels: System, Scenario, and Phase views.

SRA Tool’s Components

NFR Analysis Method

Remarks Scenarios are composed of a number of

phases and each phase is composed of a number of task-steps each modeled as <Agent,Task,Technology>

Phases are used to structure task sequences that fulfill a higher order goal.

The best design is generally the one who has more surviving BN runs.

The best design also needs to be resilient to environmental conditions.

NFR Analysis Method

NFR Analysis Method

Impact of Environmental Variables (1) Survivor runs with ER x set to best case (2) Total survivor runs for all settings

If an overall design or a particular task fails to meet the threshold back propagation alaysis is used to discover the necessary settingto achieve the NFR value. All input nodes are unconstrained

(calculation for all) One/few inputs are unconstrained

(calculation for unconstrained)

NFR Analysis Method

Case Study

The application of the SRA tool in validating the operational performance and system reliability of a complex socio-technical system.

The requirements question is to assess the impact of new automated technology on the task of loading weapons on to aircraft in an aircraft carrier.

Case Study

Tasks in Design 1 are manual or semiautomated, while, in Design 2, they are semi or fully automated.

The second design saves manpower since it can be operated by one WA and is potentially more rapid to operate, but it is more expensive.

Case Study

When the operational performance times are compared, Design 2 is quicker for nearly all tasks, which is not surprising since it has more automated tasks.

Case Study

Case Study

The critical environmental variables for both designs, shows that incentives, motivation, duty time concurrency, and time constraints were all marked as vulnerable for Design 1.

Design 2 in contrast, fares better with only motivation, concurrency, and maintenance marked as vulnerable.

Case Study

Case Study

After identifying the most appropriate design, the problematic tasks, and the critical environmental variables, the analyst investigates the improvements required for the Autoload palette component, which was the weakest link in Design 2.

Case Study

Validating the BN Models

Data mining techniques are used to test the assumptions in BN models.

All possible permutations are simulated and created a database of reliability and performance time predictions

Data mining Techniques:Relevance Analysis: ranks input parameters of

the model based on their relevance with output

Association Rules: describe how often two or more facts cooccur in a dataset and were employed to check the causal associations in our model.

Classification: partitions large quantities of data into sets with common characteristics and properties

Validating BN Models (Cont’d)

Results– Sea state had only a minor influence on

system error relevance analysis– IF (Duty-time=high) survived=fail– IF (Workload=high) survived=fail

(Association)– These rules indicate that causal

influences of these variables are higher than assumed, alter NPT settings to overcome.

– Crew motivation and agent ability problems classification

Validating the BN Models

Discussion & Conclusions

Automated testing of requirements specifications and designs for conformance to nonfunctional requirements using a set of scenarios and variations in the system environment have been developed.

Discussion & Conclusions

The SRA could be applied to any class of component-based problems where the selection of components needs to be optimized for nonfunctional requirement types of criteria.

The SRA tool was a development from previous BN requirements analyzer and has partially addressed the difficult problem of scenario-based testing

Discussion & Conclusions

Discussion & Conclusions

The SRA tool is aimed at requirements investigation in complex socio-technical systems and, hence, it complements model-checking tools which are more appropriate to later stages in development when specifications of agent behavior are available.

Discussion & Conclusions