3 rd Annual CMMI Technology Conference and User Group Kent Schneider, President Defense Enterprise...

3rd Annual CMMI Technology Conference and User Group

Kent Schneider, PresidentDefense Enterprise SolutionsNorthrop Grumman Information TechnologyNovember 18, 2003

Copyright Northrop Grumman Company 2003210/19/03

Organizational Perspective

Northrop Grumman• $26 B in revenue; 120,000 employees; 50 states; 25 countries

Information Technology (IT) Sector• $4.7 B in sales; 23,000 employees; 48 states; 15 countries

Defense Enterprise Solutions (DES) Business Unit• $610 M in sales; 3,045 employees, 23 states, 4 countries

DES provides enterprise-wide technology solutions to the Defense marketplace

Major Application Areas:• Logistics• Mission Support• Science & Technology• Simulation, Analysis, and Training• Communications & Infrastructure

Copyright Northrop Grumman Company 2003310/19/03SW

CM

M

SW C

MM

SE CMSE CM

IPPD-CMM

IPPD-CMM

CoreCore

CoreCore

CoreCore

IPDIPD

SESE

SWSW

OtherOther CMM’sCMM’s

Information TechnologyProducts & Services in

Constant Change

•IT Consulting

•Sys Arch, Engin & Delivery

•Enterprise Integration

•IT Infrastructure Management

•Applications Management

•Sys Eng & Tech Assistance (SETA)

•Functional Process Outsourcing

CMMICMMI

Core Core ProcessesProcesses

IPD DisciplineIPD Discipline

SE Discipline

SE DisciplineSW

SW

Dis

cipl

ine

Dis

cipl

ine

Core Processes Common to Multiple Disciplines


Logicon LIS

Litton TASC

Logicon LTS

L5Litton PRC

(to other units)

Northrop GrummanInformation Technology

Defense Enterprise Solutions

2001 20021999 2000

Logicon LATL3

L5

(to other units)

L3

CMMI

(to other units)

Logicon LISS L3

ENABLER

LIEB

SPII

Organizational Process Maturity Pedigree


DES Business Goals DES Process & Quality Performance Objectives

Achieve Customer Satisfaction rating (CPAR or equivalent) of Blue or Green on 90% of contracts over $5 million annual revenue or deemed to be of significant strategic value. The percentage of contracts with a Blue rating will be at least 33%.

Achieve Revenue, Profit and other financial goals

1. Achieve Cost Perf. Index = 1 0.1 (monthly)

2. Achieve Schedule Perf. Index = 1 0.1 (monthly).

3. Achieve Est-To-Complete-Variance = 1 0.1 (monthly)

4. Achieve 10% improvement in Defect Density for each life cycle phase (from previous baseline)

Objectives are monitored on a monthly basis for the projects; organizational baselines are evaluated on semi-annual basis.

2003 Goals for Performance

Business performance linked to Process and Quality Performance


11

Organizational and Project Quantitative Management Process Overview

DES Business Objectives

Projects improve performance by removing root causes for out-of-bound conditions.

Projects track process performance over time.

Projects select related goals & measurements for each life cycle phase.

Projects check performance against project goals and business objectives.

DES management checks org and project data against DES goals (process capability baseline).

DES management selects quality and process goals & measurements

22

3344

55

66

Organization

Project


Quantitative Management

Measurable contractual commitments:• Delivering products and services on time and under budget• Delivering quality (low-defect) products

Measurable process and quality commitments; DES developed process performance and product quality goals to tract program performance:• Cost• Schedule• Quality

Demonstrable senior management commitment• DES tracks this through its senior management operations

reviews (a sample follows)


-0.055

-0.030

-0.005

0.020

0.045

0.070

0.095

0.120

1/9/

02

1/22

/02

2/5/

02

2/13

/02

3/5/

02

3/28

/02

5/10

/02

6/6/

02

6/17

/02

6/25

/02

3/29

/02

6/3/

02

7/11

/02

7/26

/02

8/5/

02

8/8/

02

8/14

/02

7/2/

02

8/20

/02

9/6/

02

9/16

/02

9/25

/02

9/27

/02

10/4

/02

10/2

2/02

10/2

4/02

10/2

8/02

10/3

1/02

11/8

/02

8/29

/02

10/2

8/02

10/3

1/02

11/1

2/02

11/1

4/02

11/1

5/02

X

DDr DDr Mean UNPLX LNPLX U2S L2S U1.5S L1.5S

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.020

0.025

1/9

/02

1/2

2/0

2

2/5

/02

2/1

3/0

2

3/5

/02

3/2

8/0

2

5/1

0/0

2

6/6

/02

6/1

7/0

2

6/2

5/0

2

3/2

9/0

2

6/3

/02

7/1

1/0

2

7/2

6/0

2

8/5

/02

8/8

/02

8/1

4/0

2

7/2

/02

8/2

0/0

2

9/6

/02

9/1

6/0

2

9/2

5/0

2

9/2

7/0

2

10/4

/02

10/2

2/0

2

10/2

4/0

2

10/2

8/0

2

10/3

1/0

2

11/8

/02

8/2

9/0

2

10/2

8/0

2

10/3

1/0

2

11/1

2/0

2

11/1

4/0

2

11/1

5/0

2

X


S&T/ITS Program and QM Indicators

Goal: Critical/Urgent Defect Total Defects

5/KLOC 10/KLOC Action:

• DP Cycle #1 countermeasures are effective. Reduced targeted “checking” defects from 2.6/KLOC to 1.2/KLOC.

• DP Cycle #2 focused on defects escaping the PSP Cycle. Follow up from countermeasures show improvement in schedule and quality.

• DP Cycle #3 focused on units with highest test defects. Countermeasure approved to peer review code again prior to customer dry run testing.

• SCoVs – 6 complete, 2 in progress, 5 approved for Builds 4 & 5.

• DDt – SPC for Defect Density by test (DDt) is not useful, investigating alternate metrics, began Rayleigh Curve.

Technical Highlights: Construction Complete

• Inspection & Test - 100% of all code was peer reviewed. 100% Unit Test and Internal Integration Test complete.

• Total DDt in Unit Test - 1.39/KLOC (Goal 2)

• Total DDt Internal Integration Test - .44/KLOC (Goal 1)

S Technical Code

Total Defect Density/KLOC (Peer Review)

Critical/Urgent Defect Density/KLOC (Peer Review)

5.3 4.93.2

1.31.4

18.324.1

19.3 21.613.2


0

20

40

60

80

100

120

140

160

180

Unit Test 0.1 - 0.6 0.7 0.75 0.80(Informal)

0.85 0.9

# of

Def

ects

Critical + Urgent Total


Goal: Critical/Urgent Defect Total

Defects DDt Unit .5/KLOC 2/KLOCDDt .25/KLOC 1/KLOC

Action:• None for Defect Density goals.

Technical Highlights:• UNDER Cycle Goals!

E Technical Test ITS Test Defects By Test CycleDefect Density by KLOC

DP Cycle

Build Critical + Urgent

Total

Unit Test .33 1.4

0.1 – 0.6 0.056 0.18

0.7 0.065 0.46

0.75 0.233 0.73

0.8 0.132 0.38

0.85 0.066 0.38

0.9

DP Cycle

Completed Dry Run Test (.85) with enough time to review enhancements. Decrease in showstopper defects.

DP Cycle remeasure led to new ideas for team and enhancements analysis

Remeasure



Goal: •DDs – Defect Discovery

Critical/Urgent Total Defects .25/KLOC 1/KLOC

Action:•None for Defect Discovery goals.

Technical Highlights:•Total Test Defect Density - Sept 2003 is 5.2/KLOC! There is no significant change to defect density this month. This is relative to 19/KLOC in Peer Review!

•DDs– Dry Run Test Latent = 0.044/KLOC– Discovery Predictions next = 0.405/KLOC– Discovery Predictions next = 0.132/KLOC

E Technical Test

Projected Latent Error Discovery Efficiency

Next Phases Estimates

4.725 99.85% 43, 14

ITS Phase Based Estimates after Dry Run Test (8/31/03)



E Technical Documentation

Goal: Total Defects 1/PageActual:

Total STF Defects = .688/PageTotal User Manual Defects = .709/page

Total Misc. Doc Defects = .701/page(Critical & Urgent = .165/page)

Action: No new SCoV Technical Highlights:

Reviewed 100% documentation

100% documentation DDr exceeds goal (Misc Documentation Mean Critical + Urgent continues to decrease)

Total Defect Density – Misc. Documentation

Critical/Urgent Defect Density – Misc. Documentation

-0.75

-0.50

-0.25

0.00

0.25

0.50

0.75

1.00

Feb-0

5-2

002

Mar-

05-2

002

Apr-

25-2

002

Apr-

30-2

002

Jun-0

3-2

002

Aug-0

9-2

002

Oct-

02-2

002

Oct-

02-2

002

Oct-

03-2

002

Oct-

03-2

002

Nov-0

6-2

002

Nov-1

3-2

002

Dec-1

9-2

002

Dec-2

0-2

002

Dec-2

4-2

002

Jan-2

7-2

003

Mar-

07-2

003

Mar-

07-2

003

Apr-

17-2

003

May-1

5-2

003

May-1

5-2

003

May-1

6-2

003

May-1

6-2

003

May-1

6-2

003

Jun-1

2-2

003

Jul-08-2

003

X


-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Feb-0

5-2

002

Mar-

05-2

002

Apr-

25-2

002

Apr-

30-2

002

Jun-0

3-2

002

Aug-0

9-2

002

Oct-

02-2

002

Oct-

02-2

002

Oct-

03-2

002

Oct-

03-2

002

Nov-0

6-2

002

Nov-1

3-2

002

Dec-1

9-2

002

Dec-2

0-2

002

Dec-2

4-2

002

Jan-2

7-2

003

Mar-

07-2

003

Mar-

07-2

003

Apr-

17-2

003

May-1

5-2

003

May-1

5-2

003

May-1

6-2

003

May-1

6-2

003

May-1

6-2

003

Jun-1

2-2

003

Jul-08-2

003

X




Goal: 1 0.1 (0.9 - 1.10) Actual: Current CPIm Mean = .730

Action: Continue ECP plans and attempt cost impact recovery from customer as part of the DMSI schedule alignment ECP.

Technical Highlights: As expected the CPIm remains out of goal. Low number of defects allowed a positive cost variance to increase the mean by 5.7%.

Goal: 1 0.1 (0.9 - 1.10)

Actual: Current SPIm Mean = .753

Action: Continue with ECPs and Test Activities

Technical Highlights: Released CBT. Preparing Software Test Report.

S Financial

S Schedule PerformanceCLIN 1 and CLIN 2 – SPIm (Monthly – Sep 03)

CLIN 1 and CLIN 2 – CPIm (Monthly – Sep 03)

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

12/1

/00

1/1

/01

2/1

/01

3/1

/01

4/1

/01

5/1

/01

6/1

/01

7/1

/01

8/1

/01

9/1

/01

10/1

/01

11/1

/01

12/1

/01

1/1

/02

2/1

/02

3/1

/02

4/1

/02

5/1

/02

6/1

/02

7/1

/02

8/1

/02

9/1

/02

10/1

/02

11/1

/02

12/1

/02

1/1

/03

12/1

/02

1/1

/03

2/1

/03

3/1

/03

4/1

/03

5/3

/03

6/3

/03

7/1

/03

8/3

/03

9/1

/03

X

CPIm CPIm Mean UNPLX LNPLX U2S L2S U1.5S L1.5S

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

12/1

/00

2/1

/01

4/1

/01

6/1

/01

8/1

/01

10/1

/01

12/1

/01

2/1

/02

4/1

/02

6/1

/02

8/1

/02

10/1

/02

12/1

/02

1/1

/03

4/1

/03

6/3

/03

8/3

/03

X

SPIm SPIm Mean UNPLX LNPLX U2S L2S U1.5S L1.5S



As stated in the ITS Quantitative Management PlanCategory 1: Customer Satisfaction Goal

• Deliver no known open Level 1-3 severity defects• Achieve contractual expectations as defined in the ITS SRS

and implied customer expectations as defined in the ITS SQASP

Actual• 100% known peer review and test defects have been repaired.• Customer completed performance survey. ITS was rated

“Exceptional” in all categories. Looking to receive CPARs in CLIN 2. None received to date.

Action None

Technical Highlights• Customer continues to fund contract mods to keep team

momentum and buy conversion data schedule time. Customer continues to fund T&M CLIN. Requested synchronized ITS schedule to DMSI schedule.

E Customer Satisfaction

.


Organizational Innovation

To provide higher quality and better services to our customers, DES:

Set aggressive goals based on past quantitative performance

Research innovations internally and externally to measurably improve our performance.

Report out monthly on performance…

An example follows…


Project Comparisons

Two projects with: • Same customer (different divisions)• Similar applications• Both Client-Server (PowerBuilder / Oracle)• Similar level of team expertise and training• Estimated to be similar size and effort (~$2 Million)


Key Process Insertion Differences

Project A: • Disciplined team design process used to create sound

developer work packets• Personal Software Process used consistently by developers

SEI developed course for software developers which provides process at an individual level for producing software components and documentation (user & technical)

• Peer Reviews conducted on the most critical 20% of the software

Project B:• We decided that this project was “too far along” to benefit from

process insertion


Project Test Time / Cost

Project A Project B

Integration / Acceptance Test Cost

$78.K $612K

Normalized (per KLOC) $0.95K $4.05K

Time to Accept (months) 3.7 14.6

Normalized (months per 100 KLOC)

4.5 9.7


Project Quality

Project A Project B

Duration (months) 31.8 43.0

Size (KLOC) 82 151

Developer Defect Density 9.4 17.3

Peer Review Exit Density 4.78 17.3

Delivered Defect Density 1.55 5.27


Positive Effect of Process Insertion

• Mature Processes & PSP led to–Project A’s Integration and Acceptance Test Costs

were 24% of Project B–Project A’s Test Duration was 46% of Project B–Project A’s delivered defect density was 29% of

Project B

• Process saved cost, reduced schedule, and improved quality


2002 SCAMPI Appraisal Results

Process Maturity• CMMI SE/SW Level 5 for Staged and

Continuous Representations• Software CMM Level 5 also

• Organizational Process Strengths• Integrated SE and SW Eng• Sharing and reusing process assets• Data driven corrective action• Technology Change Management

• Global Best Practices• Requirements Elicitation• Quantitative Data Management

Process Maturity• CMMI SE/SW Level 5 for Staged and

Continuous Representations• Software CMM Level 5 also

• Organizational Process Strengths• Integrated SE and SW Eng• Sharing and reusing process assets• Data driven corrective action• Technology Change Management

• Global Best Practices• Requirements Elicitation• Quantitative Data Management

F1=Fully Implemented Out of Scope

Cost, Schedule and Quality Performance JEDMICS exceeds software quality goals

(.0156 vs .06 defects/KSLOC) PERMS exceeds revenue goals by 11% AIT has 100% on-time delivery Only 2% of total defects found in

delivered SIGS product ITS is on cost and schedule (CPIm = .965

& SPIm = .985, target = 1.0 +/- .1) SIGS averages a customer satisfaction

of 98.5% over last four years

Cost, Schedule and Quality Performance JEDMICS exceeds software quality goals

(.0156 vs .06 defects/KSLOC) PERMS exceeds revenue goals by 11% AIT has 100% on-time delivery Only 2% of total defects found in

delivered SIGS product ITS is on cost and schedule (CPIm = .965

& SPIm = .985, target = 1.0 +/- .1) SIGS averages a customer satisfaction

of 98.5% over last four years


Knowledge Management and Process Management

We manage our Processes as a Community of Practice

We help our integrated teams share and put process into action

Processes underpin each “community of practice”


CMMI Extensibility and Flexibility

Engineering Management Processes

Program Management Processes

Process Infrastructure Processes

Integrated Enterprise Process

Engineering Processes


Conclusions

CMMI Practices have a direct and positive influence on cost, schedule, and quality performance• It helps to tie the quantitative goals and measures to customer

needs and relevant project problems

CMMI high maturity enables organizational and project agility in high priority areas• Projects can improve dramatically in 6 months or less using

defect prevention cycles of 1 week or less

The transition from SW-CMM to CMMI is not inherently difficult, costly, and time-consuming, but…• It pays to proactively improve against emerging models and

standards• It is much easier to attain the same maturity level the second

time• It is much easier to transition between models when you focus

on performance

3 rd Annual CMMI Technology Conference and User Group Kent Schneider, President Defense Enterprise...

Documents

Transcript of 3 rd Annual CMMI Technology Conference and User Group Kent Schneider, President Defense Enterprise...