Tutorial joaopascoalfaria-2confcmmiportugal-v1-3-split

Portugal

Quality Management and Process Improvement with

the Team Software Process

João Pascoal FariaAssistant Professor

FEUP

2012-07-06

22

Questions

1. Do you have problems/concerns with software quality?

2. Do you have problems/concerns with cost of quality?

3. Do you know your current quality and cost of quality levels?

4. How do they compare with industry benchmarks?

5. How can you improve quality and reduce the cost of quality?

33

Agenda

Motivation

TSP Performance Results

Factor I – Bottom-Up Performance Improvement

Factor II - Personal Responsibility

Factor III – Feedback Loops for Continual Improvement

Factor IV – Cost-Effective Defect Removal Methods

Factor V - Measurement and Quantitative Methods

Conclusions

44

Motivation

55

Top 10 software engineering trends

1. Increasing emphasis on rapid development and adaptability;

2. Increasing software criticality and need for assurance;

…

Barry Boehm 2011

66

Increasing software criticality and need for assurance

“Although people’s, systems’, and organizations’ dependency on software is becoming increasingly critical, dependability is generally not the top priority for software producers.”

Barry Boehm 2011

77



“(In fact) the IT industry spends the bulk of its resources, both financial and human, on rapidly bringing products to market.”

Barry Boehm 2011

88



“(In fact) the IT industry spends the bulk of its resources, both financial and human, on rapidly bringing products to market.”

“This situation will likely continue until a major software-induced systems catastrophe similar in impact on world consciousness to the 9/11 World Trade Center catastrophe stimulates action toward establishing accountability for software dependability.”

Barry Boehm 2011

99

Why should quality be the top priority?

1010


Increasing dependency on software systems

1111



– 80%+ of functionality of modern aircrafts provided by sw [Humphrey, 2002]

1212




Increasing criticality of functions provided by software

1313





– e-banking, aircraft control software, medical device software, …

1414






Software defects are a primary cause of security vulnerabilities

1515







– Buffer overflow, SQL injection, etc.

1616








Huge economic impact of software errors

1717









– Direct costs of sw errors represent 0,6% of USA’s GNP [NIST, 2002]

1818










Quality work saves time and money

1919










Quality work saves time and money

– Defect prevention and early defect removal reduce rework costs

2020

Quality economics: Cost of Quality (COQ)

2121


For producers, the goal is to reduce the cost of quality (COQ)

2222



– Prevention costs

2323



– Prevention costs • identifying and resolving defect causes • formal spec, prototyping, and design work• process analysis and improvement

2424




– Appraisal (defect detection) costs

2525




– Appraisal (defect detection) costs• reviews and inspections• test development and execution (once)

2626





– Internal failure costs

2727





– Internal failure costs• repair & rework before delivery

– External failure costs• repair & rework plus any scrap after delivery• law suites, loss of customers

http://images.google.com/imgres?imgurl=http://imgsrv.1010wins.com/image/DbGraphic/201001/1475889.jpg?1264110577&imgrefurl=http://www.1010wins.com/Toyota-Recalls-2-3M-US-Vehicles-to-Fix-Gas-Pedals/6173514&usg=__56lu78goz0UatgJs0A-9Eut5kSo=&h=409&w=640&sz=65&hl=pt-PT&start=5&sig2=6Uo3Vrutkh_No6NNeFiCQA&um=1&itbs=1&tbnid=P_S5d5_PauVyuM:&tbnh=88&tbnw=137&prev=/images?q=toyota+recalls&um=1&hl=pt-PT&sa=N&rls=com.microsoft:pt:IE-SearchBox&rlz=1I7TSEA;&tbs=isch:1&ei=887QS8DSNJCh-Aa0xL2bDA

2828

Example: Security vulnerabilities

2929


3030


What happened

?

3131


What happened

?What is the

cause?

3232


What happened

?What is the

cause?

5 min

3333

TSP Performance Results

3434

Better product quality (industrial results)

7,5

6,24

4,73

2,28

1,05

0,06

0

1

2

3

4

5

6

7

8

Def

ecst

/KL

OC

CMM L1 CMM L2 CMM L3 CMM L4 CMM L5 TSP

Average Defect Density of Delivered Software

TSP impact study 2003, 20 projects in 13 organizations, several maturity levels

Capers Jones, 2000

3535

Mea

n N

umbe

r of

Com

pile

and

Tes

t D

efec

ts P

er K

LOC

111098765432100

10

20

30

40

50

60

70

80

90

100

110

120

Program Number

PSP0 PSP1 PSP2

Defects

Better product quality (training results)

PSP Training results, SEI, 298 developers

1/3 with 0 defects

3636

Reduced security vulnerabilities (Microsoft)

TSP + secure coding standards, security design patterns, static analysis tools, secure code inspections and reviews

8-person software development team

Created 30 thousand lines of new and modified code in 7 months

Source: TSP Secure, Noopur Davis et all, TSP Symposium 2009, New Orleans

3737

Better, faster, cheaper (Microsoft IT)

Source: The TSP Story @ Microsoft IT, TSP Symposium, Phoenix, 2008

4 releases with TSP

compared to previous

releases without

TSP

TSP versus non-TSP

projects

3838

Reduced cost of quality (Adobe)

J.Sartain, Senior Director, Quality,

Adobe 2009

3939

Factor I – Bottom-Up Performance Improvement

4040

Build high-performance teams from the bottom-up

TeamMemberSkills

TeamBuilding

TeamManagement

Personal Software Process

Team Software Process

Instance of high-maturity practices for teams

Instance of high-maturity practices for individuals

http://www.sei.cmu.edu/tsp/


4141


TeamMemberSkills

TeamBuilding

TeamManagement

Process disciplinePerformance measuresEstimating & planning skillsQuality management skills







4242


TeamMemberSkills

TeamBuilding

TeamManagement


Goal settingRole assignmentTailored team processDetailed balanced plans







4343


TeamMemberSkills

TeamBuilding

TeamManagement


Goal settingRole assignmentTailored team processDetailed balanced plans

Team communicationTeam coordinationProject trackingRisk analysis







4444

Build personal performance from the bottom-up

Personal practices are introduced stepwise, through a sequence of small projects in a training environment

– The objective is to convince people by seeing their own performance improving as they practice

Team practices are introduced in real projects with the help of a coach

Establish a measured performance baseline

Size and effort estimation

Defect & yield management; design practices

4545

Example: Personal PSP training experience

4646


Goal: achieve 0 defects in unit testing and near 0% time estimation error, with the same productivity as before

4747



PSP Fundamentals: Steady reduction of time estimation error, from 70% to 1% in 4 projects/days

4848




PSP Advanced: Initial decrease of productivity because of the introduction of design documentation templates

4949





8th project: compute the minimum difference between two Java source files, in lines added, delete and modified, ignoring comments and blank lines (considering that deletions have null cost)

5050






– Almost recovered the initial productivity

5151






– Almost recovered the initial productivity– 5% effort estimation error

5252






– Almost recovered the initial productivity– 5% effort estimation error– 2 defects in unit testing, caused by requirements problems!

5353






– Almost recovered the initial productivity– 5% effort estimation error– 2 defects in unit testing, caused by requirements problems!– Next step: improve requirements analysis!

5454

Factor II – Personal Responsibility

5555

Personal responsibility: project management

There is only one way to manage knowledge workers: they must manage themselves [W. Humphrey]

5656



Traditional teamThe leader plans, directs,

and tracks the team’s work.

Team Leader

5757



Self-directed teamAll team members participate in planning, managing, and tracking their own work.

TSP Coach

Team Leader



Team Leader

5858




TSP Coach

Team Leader



Team Leader

Planning Manager

Process Manager

Quality Manager

Support Manager

Test Manager

Implementation Manager

Design Manager

Costumer Interface Manager

5959




TSP Coach

Team Leader



Team Leader

Planning Manager

Process Manager

Quality Manager

Support Manager

Test Manager

Implementation Manager

Design Manager

Costumer Interface Manager

OwnershipCommitmentMotivation

Performance

6060

Personal responsibility: quality management

The only way to build high-quality products in a cost-effective way, is by having developers being personally responsible for the quality of their products.

Since defects can best be managed where they are injected,developers should

– remove their own defects– determine the causes of their defects– learn to prevent those defects

6161

Factor III – Feadback Loops for Continual Improvement

6262

Feedback loops: project lifecycle

Developmphaseor cyclephase

or cycle

Phase or cycleRetrospective

Developmentphase

or cycle

Launch

Re-launch

ProjectRetrospective

Lessons, new goals, new requirements, new risks, etc.

Business and technical goals Estimates, plans,

process, commitment

Work products, status, metrics, results

Weekly team meetings

Release planning

Iteration planning

Status

Updated status and plans

6363

Feedback loops: estimating & planning frmwk

Estimate resources (effort)

Define requirements

Produce conceptual

design (PBS)Estimate

size

Produce task list, distribute effort

Produce schedule

Develop product

Size database

Productivity database

Resources available

Size, resource, schedule data

Process analysis

Product delivery

Tracking reports

PROBE method (PROxy Based Estimating)

Hist. distrib., Process def.

Costumer need

Items

Tasks

Costumer

Management

6464

early defect detection &

removal

Feedback loops: quality framework

Development (and defect injection)

phase

Defect Data

Defect removal phase

Checklists, Test Criteria, …

Process/Data analysis

defect prevention

Subsequent phases late defect

detection & removal

Scripts, Standards, Awareness

Learn from past errors, because we tend to repeat the same errors!

6565

Feedback loops: coaching (& leadership)

A coach helps improving individual and team performance through a continuous coaching/learning cycle

3. ConsciousCompetence

2. Conscious Incompetence

1. UnconsciousIncompetence

External Independent

voice

Guidance and

Feedback

4. UnconsciousCompetence

Next Stag

e

Practice

6666

Example: Historical defect data analysis

6767


6868


Defect Types10 - Documentation60 – Checking80 – Function40 - Assignment70 - Data50 - Interface30 - Build100 - Environment

6969



Most frequent

7070



Most frequent Most expensive*

7171




(*) further analyses showed that they were mainly injected in design and removed in test.

7272




Þ Improve design review checklistsÞ Improve design standards and scriptsÞ Produce test spec after design spec and before design

reviewÞ Minimize documentation and use spell checker

(*) further analyses showed that they were mainly injected in design and removed in test.

7373

Factor IV - Cost-Effective Defect Removal Methods

7474

Defect removal methods (V-Model)

Requirements & ST Spec

REQ TeamInspection

High-LevelDesign

& IT Spec

Detailed Design

& UT Spec

DLD Personal Review

DLD TeamInspection

CodeCode

PersonalReview

Unit Test(UT)

Code TeamInspection

SystemTest (ST)

IntegrationTest (IT)

Compile

HLD TeamInspection

7575

Efficiency of defect removal methods

Even experienced developers introduce ~100 defects/KLOC

Such high-number of defects must be removed using the most cost-effective methods, such as personal reviews and team inspections(Continue to use unit, integration and system testing)

Source: Jim Sartain, Adobe,2009

Minutes to Find and Resolve a Defect by Discovery Activity

7676

Use checklists derived from historical defect data

Peter Pronovost (Dr. Checklist)

“Most influential people of 2008” (Time magazin)

http://www.youtube.com/watch?v=YKm8NUmPg58





7777


Make the review more effective








7878


Make the review more effective– focus the attention on most frequent problems








7979



Make the review more efficient








8080



Make the review more efficient– don’t waste time looking for too rare problems








8181




Reduce the risk of missing critical issues








8282




Reduce the risk of missing critical issues– even experts benefit from checklists








8383





But:








8484





But:

– keep it simple and short








8585





But:


– keep it specific for the technology, language, etc.








8686





But:


– keep it specific for the technology, language, etc.

– combine with other analysis techniques








8787

Take enough review time

The review rate (size reviewed per hour) is one of the best leading indicators of review effectiveness (review yield)

Recommended code review rate: about 200LOC/hour

Revewing too slow or too fast is a waste of time

Also recommended to review in a quiet environmenttake a break between producing and reviewing

8888

Use multiple inspectors to improve yield(yield = % of defects detected)

(source: Introduction to the Team Software Process, Watts Humphrey)

8989

Estimate defects remaining with the capture-recapture method

Based on the degree of overlapping between different reviewers

Found by A = 8

Found by B = 6

Common defects: C = A B = 4

Total defects:T A*B/C = 12

Remaining defects:R A*B/C-(A+B-C) = 2

http://en.wikipedia.org/wiki/Mark_and_recapture

http://en.wikipedia.org/wiki/Mark_and_recapture

9090

Measure the review process and use data to improve the reviews

Effort(review, rework)

Size(of artifact reviewed)

Defects found(number, type, description, severity)

Escaped defects(collected later)

9191





Defect removal rate (indicator of review efficiency)


9292





Defect density (indicator of product quality)



9393






Review rate (leading indicator of review effectiveness)



9494






Review rate (leading indicator of review effectiveness)



Review yield(lagging indicator of review effectiveness)

9595

Factor V - Measurement and Quantitative Methods

9696

The need for measurement

9797


W. Edwards Deming

In God we trust, all others

bring data.

9898


W. Edwards Deming


bring data.

You can't manage

and improve what you

don't measure.

Tom DeMarco

9999


W. Edwards Deming


bring data.

You can't manage

and improve what you

don't measure.

Tom DeMarco Watts Humphrey

When performance is unmeasured or improperly measured, the results are often disappointing and can even be disastrous. Unless your measures cover all important aspects, you will likely motivate counterproductive action.

http://www.youtube.com/watch?v=MpYW6oAE-1U

100100

Base measures

In the TSP, 4 core measures are the basis for quantitative project management, quality management, and process improvement at the personal, team and organization level.

Size

ScheduleQuality

Effort Actual and Planned

101101

Defect tracking

Defects are the measure of quality in the TSP.

Any change to an interim or final work product, made to ensure proper subsequent utilization is considered a defect.

102102

Quality planning: Defect injection and removal plan

• Yields (% of defects removed) based on historical data or benchmarks• Defects injected based on historical defects injected per size unit

Removed

103103

Quality tracking: Defect removal profile

The defect removal profile shows– plan and actual defects removed by phase– early vs. late defect removal plan

Defects Removed by Phase for Assembly SYSTEM

0.0

100.0

200.0

300.0

400.0

500.0

600.0

700.0

800.0

900.0

Phase

Defe

cts

Rem

oved

by P

hase

Plan

Actual

104104

Quality tracking: Defect removal profile

The defect removal profile shows– plan and actual defects removed by phase– early vs. late defect removal plan

Defects Removed by Phase for Assembly SYSTEM

0.0

100.0

200.0

300.0

400.0

500.0

600.0

700.0

800.0

900.0

Phase

Defe

cts

Rem

oved

by P

hase

Plan

Actual

Typical software project

105105

Quality Tracking: Quality profile

It examines criteria that are effective predictors of system test and post-release component quality

Quality Profile for Assembly Common Query Changes (BE)

0

0.2

0.4

0.6

0.8

1Design/Code Time

Code Review Time

Compile Defects/KLOCUnit Test Ddefects/KLOC

Design Review Time

Plan

Actual

Quality Profile for Assembly Common Query Changes (BE)

0

0.2

0.4

0.6

0.8

1Design/Code Time

Code Review Time

Compile Defects/KLOCUnit Test Ddefects/KLOC

Design Review Time

Plan

Actual

1

10

½ design time ½ coding time

4

Design Quality

Design Review Quality

Code Review Quality

Code QualityProgram Quality

106106

Conclusions: TSP Quality principles

107107


To produce quality products, developers must feel personally responsible for the quality of their products. Superior products are not produced by accident; developers must strive to do quality work.

108108



It costs less to find and fix defects earlier in a process than later.

109109




It is more efficient to prevent defects than to find and fix them.

110110





The right way is always the fastest and cheapest way to do a job.

111111





The right way is always the fastest and cheapest way to do a job.

To maximize productivity, focus on quality first.

112112

Conclusions

The TSP approach for producing high-quality software in a cost-effective way is based on learning from your errors and using cost-effective defect removal methods

Data shows that TSP teams produce software with very low defect density and reduced cost of quality

If you have the need to develop top quality software, you should take a look at TSP principles and practices

113113

References

Software Assessments, Benchmarks, and Best Practices, Capers Jones, Addison-Wesley, 2000

Winning with Software, Watts Humphrey, 2002

Introduction to the Team Software Process, Watts S. Humphrey, 2000

TSP: Coaching Development Teams, Watts S. Humphrey, Addison-Wesley, 2006

PSP: A Self-Improvement Process for Software Engineers, Watts S. Humphrey, 2005

“The Economic Impacts of Inadequate Infrastructure for Software Testing”, National Institute of Standards and Technology (NIST), 2002

“Inspiring, enabling and driving the Evolution of Quality at Adobe leveraging the TSP”, Jim Sartain, Senior Director, Quality at Adobe Systems, TSP Symposium 2009

“Some Future Software Engineering Opportunities and Challenges”, Barry Boehm, in The Future of Software Engineering, Springer, 2011

Portugal

Thank you!Contact information:

João Pascoal Faria (*)Departamento de Engenharia InformáticaFaculdade de Engenharia da Universidade do PortoRua Dr. Roberto Frias, s/n 4200-465 Porto, PORTUGALEmail: [email protected]: +351225082134Mobile: +351966494914(*) SEI Qualified PSP Developer, PSP Instructor, TSP Coach

Tutorial joaopascoalfaria-2confcmmiportugal-v1-3-split

Technology

Transcript of Tutorial joaopascoalfaria-2confcmmiportugal-v1-3-split