1 Experience-Based Approaches to Process Improvement © Otto Vinter / Quality Week 2000...

1Experience-Based Approaches to Process Improvement© Otto Vinter /

Quality Week 2000

Experience-Based Approaches toProcess Improvement

-

Otto VinterProject Manager Software Process Improvements,

DELTA Software Engineering,Tel: +45 4586 7722, Fax: +45 4586 5898

[email protected]

&

Software Engineering Mentor

Tel/Fax: +45 4399 2662, Mobile: +45 4045 0771

[email protected] http://inet.uni2.dk/~vinter


Normative Models for SPI

CMM

BOOTSTRAP

SPICE (ISO15504)

Defined comprehensive process

Levels of maturity (capability)

Key process areas (KPAs) for each level

Assessed by certified assessors


Fundamentals in Normative Models

Software Process Assessment & Improvement(The BOOTSTRAP Approach)ISBN 0-631-19663-3


Problems with Normative Models

Abstract model

Assessment by external body

Costs money

Organisational focus

Points out KPAs to be improved

Little help on precisely what to do

Raises a lot of expectations


Alternative Approaches to SPI

Experience-based improvement actions

• Analyse problems from previous projects to extract knowledge on frequently occurring problems

• Change the development process through the use of an optimum set of methods and tools available to prevent these problems from reappearing

• Measure the impact of the changes in a real-life development project

• Diffuse the results to the rest of the organisation


Characteristics of Alternative Approaches

Experience-based

No specific model

Hot-spot driven

Focus on prevention

One issue at a time (incremental)

Piloting at project level

Evolve rather than define (feed-back)

Fits CMM level 1-2 cultures (where most of us are)


Examples of Alternative Approaches

Analysis of• defects• progress reports, etc.

Structured/Selective interviews• project managers• project members• customers, etc.

Goal-Question-Metric paradigm (GQM)

Some frameworks for alternative approaches:• Experience Factory (V. Basili e.a.) • Product Process Dependency Models

(PROFES: www.ele.vtt.fi/profes)• but primarily: you must find your own way


Brüel & Kjær - Sound & Vibration Measurement

Transducer Signalconditioning

Digitalconversion

Signalprocessing

Postprocessing

Output

Brüel & Kjær

Conditioning Amplifier

ISO 9001CERTIFIED


Defect Analysis from Error Logs

Definitions• Bugs are anything between serious defects and

suggestions for improvements• Problem reporting starts in the integration phase

Error Logs Analysed• Embedded and PC Windows development projects• Project sizes app. 5-7 person years• In total app. 1000 bugs analyzed in detail• Problem reports covered a period until 18 months after

first release


Problem Report Distribution over Time

Example of an Error-prone Product

Weeks

Bug

s R

epor

ted

Trial-releaseProduction-release


Defect Analysis Technique

Interview sessions• 1-2 developers and 1-2 process consultants• app. 5 minutes / bug

Bug Categorisation• based on a bug taxonomy by Boris Beizer:

- Boris Beizer: Software Testing Techniques, Van Nostrand Reinhold, 1990

• comprehensive set of bug categories and statistics

Capture Subjective Information on the Bugs • where and how the bug occurred• quality factor (reliability, usability, functionality ...)• complexity of bug (correction cost)• what could prevent the bug


The Beizer Bug Taxonomy

1. Requirements and Features

2. Functionality as Implemented

3. Structural Bugs

4. Data

5. Implementation (standards violation, and documentation)

6. Integration

7. System and Software Architecture

8. Test Definition or Execution Bugs

9. Other Bugs, Unspecified

Each category detailed to a depth of up to 4 levels


0,0 5,0 10,0 15,0 20,0 25,0

1. Requirements and Features

2. Functionality as Implemented

3. Structural Bugs

4. Data

5. Implementation

6. Integration

7. System and Software Architecture

8. Test Definition or Execution Bugs

9. Other Bugs, Unspecified

%

Our Analysis

Beizer

Problem Report Categorization

Category


1st Action: The Prevention of Errors through Experience-driven Test Efforts (PET)

Results of the analysis of error logs• no special bug class dominates embedded software

development• requirements problems, and requirements related

problems, are the prime bug cause (>36%)• problems due to lack of systematic unit testing is the

second largest bug cause (22%)

Action: Improve Testing Processes• static analysis

- source code complexity, data flow anomalies, coding standards

• dynamic analysis- code coverage by test cases, cross references code to test cases

Funded by CEC. Final Report: http://www.esi.es/ESSI/Reports/All/10438


Static Flowgraph (McCabe = 10)


Static Flowgraph (McCabe = 20)


Test Coverage Strategy

Test coverage must take into consideration• perceived complexity

- based on a visual inspection• criticality

- product and project issues• developer experience

- domain, product, and unit• bug and change request history• degree and type of reuse

Test cases for dynamic analysis are selected by normal testing techniques

• equivalence partitioning, boundary value analysis etc.


Results of the Improved Testing Process

46% Improvement in Testing Efficiency• Removing static bugs

• Increasing unit branch coverage to > 85%

75% Reduction in Production-Release Bugs• Compared to Trial-Release

70% Requirements Bugs in Production-Release

Next Action: Improve Requirements Engineering


2nd Action: A Methodology for Preventing Requirements Issues from Becoming Defects (PRIDE)

Results of the analysis of error logs• Requirements related bugs 51%• Usability issues dominate 64%• External software (3rd party & MS products) 28%

Action: Introduce requirements techniques• Use Situations (Scenarios)

- Relate demands to use situations. Describe tasks for each scenario.

• Usability Test, Daily Tasks, Navigational Prototype- Check that the users are able to use the system for daily tasks,

based on a navigational prototype of the user interface.

Funded by CEC. Final Report: http://www.esi.es/ESSI/Reports/All/21167


Scenarios

“The defining property of a scenario is that it projects a concrete description of activity that the user engages in when performing a specific task, a description sufficiently detailed so that design implications can be inferred and reasoned about”

John M. Carroll,Scenario-Based Design:

Envisioning Work and Technology in System Development,

Wiley 1995.

Many names for the same concept:• scenarios• use situations• work situations• work setting


Road Test Scenario

Road tests are done in the car when it is driving on special test roads. The purpose of the recordings is to identify noise sources, comparing them to earlier measurements, and eventually removing the noise through changes to the car design.

The engineer will record noises from various parts of the car when it is driving at various speeds, when it is turning, when it is breaking, etc. The microphone will have to be mounted at various places not accessible from the drivers seat.

Usually, the engineer has a plan for what to measure, but circumstances may change so that he has to do something different and later find out what he actually did and which sounds relate to what.

Back at the lab the sounds will be analyzed by the engineer himself, or - in many cases - someone else.


Usability Test Environment

PC

V

• •

Experiment Leader

User

Log Keeper

Tape Recorder

Microphone


Results of the 2nd Improvement Action

Product is selling steadily more than twice as many copies

• compared to a similar product developed for a similar domain by the same team

Usability is exceptional in the market• Users’ interaction with the product was totally changed

as a result of the early usability tests

Almost 3 times increase in productivity for the development of the user interface

27% reduction in problem reports


Improvement Approach Based on Project Manager Perceived Problems

Interview project managers• 1 project manager and 3 process consultants• app. 1 hour per interview• process-oriented interview guide

Analyze problems raised• classify the problems according to software process • present the issues at a project manager workshop• establish consensus on a few major issues• let each manager select an issue to improve

Monitor development projects• introduce and train the team on the selected issue • coach, and support regularly and on request• collect experience and results


Improvement Approach Based on Project Manager Perceived Problems

Major Issues • iterative software development model• requirements• project monitoring (estimation, progress evaluation, etc.)• project conclusion (test, configuration management, etc.)

Improvement Actions:• diffuse and adopt the test and requirements techniques • define new iterative development models• improve project monitoring

Results:• improvement actions were performed with enthusiasm• project related actions completed successfully


3rd Action: Iterative Software Development Model

DemoPrototype

FunctionallyComplete

Model

Specification

Spec.

Test

Design

Test

Code Code Code Code

Design Design Design

Test Test

Spec. Spec. Spec.

Customer Req. Spec. Validation

ReleaseVers. 1.0

FinalCustomerValidation

IntermediateCustomerValidation

Pre-Project

StabilizationPartial Implementation

Full Implementation

Specificationand user interface

mock-up

ReleasedProduct

CustomerViews:

ProductViews:

Cycles:

Phases: Specification Development Operation

DevelopmentViews:


Iterative Software Development Model

Types of Iteration• user validation (usability test)• exploration (e.g. performance)• proof-of-concept (e.g. architecture)• well-defined functionality (increment)

Duration • time-boxed

- less than 6 weeks, with 2 weeks spacing

• full increments- app. 3 months, with 2 weeks of stabilization

Essential for Success• user validation • stabilization


Results of the 3rd Improvement Action

• drive and motivation of team increased dramatically• turbulence within and around team decreased • control over project progress increased • requirements creep was kept under control• resources used more efficiently• serious problems were uncovered early• quality improved step-by-step


Comparison of Recommendations

Recommendation Defect 1st Bootstrap Project Mgr. 2nd Bootstrap Analysis Assessment Interviews Assessment

Development Model - iterations x x x - risk management x

Requirements x x x

Project Monitoring - estimation x x x

- time & resource usage x x

- monitor progress x x x

Project Conclusion - configuration mgmt. x x x

- testing x x x ()

- release criteria x x

Reuse x

Process Descriptions x


Problem Diagnosis Results

Defect analysis from error logs• has established a basic process for testing and release

• has improved our requirements process and products

Improvements from interviews• successful diffusion and adoption of previous

improvement actions

• established a new software development model

Impacts on maturity• development model, test, and requirement issues are no

longer on the Bootstrap recommendation list


In Conclusion

Problem diagnosis approach• is a simple and effective way to find problems in the

software development process

• good starting point for process improvement programmes in companies

• step-wise improvements with quick wins

• changes assessment recommendations

However, normative models are needed• comprehensive framework

- KPAs are important, levels are not

• established assessments

- effect of improvements

1 Experience-Based Approaches to Process Improvement © Otto Vinter / Quality Week 2000...

Documents

Transcript of 1 Experience-Based Approaches to Process Improvement © Otto Vinter / Quality Week 2000...