Harmonizing Systems and Software Estimation23rd International Forum on COCOMO and Systems/Software Cost Modeling and ICM WorkshopUSC Campus, Los Angeles, CA Oct. 27, 2008

Gan Wang, BAE Systems, Reston, VA gan.wang@baesystems.comGarry J. Roedler, Lockheed Martin, Philadelphia, PA garry.j.roedler@lmco.comRicardo Valerdi, MIT, Cambridge, MA rvalerdi@mit.eduAaron Ankrum, BAE Systems, Reston, VA aaron.ankrum@baesystems.comJohn E. Gaffney, Jr., Lockheed Martin, Rockville, MD j.gaffney@lmco.comJared Fortune, USC, Los Angeles, CA fortune@usc.edu Don Reifer, Reifer Consultants, Neptune, CA dreifer@earthlink.net

2

Acknowledgement

• This presentation uses material from the following:– PSM Workshop on Harmonizing COSYSMO and COCOMO Results

Briefing, July 2008

– USC COSYSMO Workshop, March 2008

– BAE Briefing for COSYSMO Workshop, October 2008

– Lockheed Martin Project Briefings

3

Problem & Motivation

• Estimating tools available today (e.g., COSYSMO, COCOMO, PRICE, SEER) are functionally oriented

– We assemble a total engineering bid from functional estimates

• Understand the issues related to integrating systems and software estimation for software-intensive projects

– What are the exact estimate scopes of COSYSMO and COCOMO II?

– How do we deal with potential gaps and overlaps?

4

Summary of Work So Far

• COSYSMO Workshop in conjunction with 2008 USC Annual Research Review

– Scoped the problem

– Prioritized with other COSYSMO projects – assessed as high priority

– Identified that operational guidance may be as important as model constructs

• COSYSMO Workshop at 2008 PSM Users Group Conference (Mystic, CT)

– Assigned WBS elements to functions

– Assessed the coverage by COSYSMO and COCOMO

– Identified potential gaps and overlaps

• Workshops within engineering communities at BAE Systems

• Project in Lockheed Martin to address Integrated Cost Estimation for Systems and Software Engineering

• Joint paper to INCOSE Symposium

5

Areas for Consideration in Analysis of Harmonization

• Overlap/Gaps of tasks– Per WBS, work products, and combined activities

• Analysis of Cost Drivers

• Commonality of Terminology, Constructs, Life Cycle Phases, Units, …

• Compatibility issues from any findings or recommendations

• Consideration of common size drivers

• Base assumptions of the models

6

Approach

• Determine estimate coverage in a common project framework – Engineering Work Breakdown Structure (WBS) vs. Organizational Breakdown Structure (OBS)

– Defined based on MIL-HDBK-881A and EIA/ANSI 632

– Generic contract structure

• Two-step exercise conducted in roundtables/workshops:– Identify functional ownership of the tasks (leaf elements)

– Determine estimate coverage by current COCOMO II and COSYSMO

• Additionally, address the other areas for consideration through analyses in the workshops

Note:

The use of COCOMO is incidental. It addresses similar issues with other models.

7

Contract Engineering WBS Based On Standards

1.0 – System/Project1.1 – Integrated Project Management (IPM)

1.2 – Systems Engineering

1.3 – Prime Mission Product (PMP)1.3.1 – Subsystem / Configuration Item (CI) 1…n (Specify Names)

1.3.2 – PMP Application Software

1.3.3 – PMP System Software

1.3.4 – PMP Integration, Assembly, Test & Checkout (IATC)

1.3.5 – Operations/Production Support

1.4 – Platform Integration

1.5 – System Test & Evaluation (ST&E)

1.6 – Training

1.7 – Data Management

1.8 – Peculiar Support Equipment

1.9 – Common Support Equipment

1.10 – Operational / Site Activation

1.11 – Industrial Facilities

Product-oriented construct, by tailoring MIL-

HDBK 881A and EIA/ANSI 632

Six Functions:

1. Systems Engineering

2. Software Engineering

3. Electrical Engineering

4. Mechanical Engineering

5. Support Engineering

6. Project Engineering Management

Six Functions:

1. Systems Engineering

2. Software Engineering

3. Electrical Engineering

4. Mechanical Engineering

5. Support Engineering

6. Project Engineering Management

8

The Exercise

Y: COSYSMO; S: COCOMO; X: Ownership but no model coverage; U: Uncertainty

9

How Does It Work?

• We compare the systems and software columns.  If any WBS element a mark (Y, S, or X), then it is “owned” by a function.

• Gaps: If an element has no mark or only “X” in either column, then we have a potential gap.  That means neither COSYSMO nor COCOMO, as they stand today, covers that task.  

• Overlaps: If an element has both a “Y” *and* an “S”, then we have a potential overlap.  That means both models estimate that scope and we have to reconcile/deconflict.

• Uncertainty: “U” for an elements means it is not consistently estimated by the model

• A task may be “owned” by other functions, e.g., HW, in which case it should be covered by other functional models

• Important: – Task ownership not (necessarily) execution

You, the IPT lead, can use anyone you like to do the job

– With this analysis, we attempt to cover nominal projects or majority behavior.  There can be exceptions and it could be different from the last program you worked on, which is irrelevant!

10

Summary of Analysis

• WBS vs. OBS Cross-reference Matrix identifies areas of gaps and overlaps

• Analysis identified more potential gaps than overlaps

• Potential gap areas:– Project engineering management

Variations in costs for types of life cycle models Quality Management Technical Process Strategy/Definition/Mgt (at SW level) Accounting for subcontract or supplier mgt (at SW level)

– Prime Mission Product (PMP) systems software SRS Development Development for Reusability (at system level)

– ST&E test equipment, facility, and support Accounting for SW support as needed Test Database Size (at system level)

– Ownership issues for the following major areas (often responsibility of Supportability) Training Data management Site construction/conversion Industry facilities

11

Summary of Analysis (Cont’d)

• Additional gap, not related to WBS – COSYSMO does not address duration or schedule

• Potential overlap areas:– PMP system design

Algorithm Development

– Development test and evaluation (DT&E)

• Areas of Uncertainty (due to lack of explicit COSYSMO guidance)– Discrepancy Report (DR) work-off within PMP or support equipment (CI level)

– ST&E Mock-ups / Prototypes / Simulations & Test Equipment

– Contractor Technical Support – onsite during/after system activation/turnover

12

Results of Other Analysis

• Analysis of Cost Drivers – Most of the drivers have mappings between the models, albeit different

in granularity or handling

– Potential concerns covered in Gaps or Recommendations

• Commonality of Terminology, Constructs, Life Cycle Phases, Units, …

– Additional commonality could improve concurrent usage, but is not essential

• Compatibility issues from any findings or recommendations – No apparent compatibility issues (backward compatibility or with other

models in COCOMO Suite) from recommendations

• Consideration of common size drivers– No essential to harmonization, but may add utility to COCOMO

• Base assumptions of the models– Need to document assumption for COSYSMO

13

Additional Recommendations

• Standard phase alignment for both models – Per definitions used in ISO/IEC 15288 and 12207

• Establish means to adequately account for recursion (at level of hands-off to SW)

– Needed to resolve gaps

• Establish operational guidance to minimize variation in usage• Add Guidance to COSYSMO Drivers TO:

– Account for constraints (e.g. Time & storage) as requirements in the size.

– Describe volatility covered in Requirements/Architecture understanding

• Look into ability to use COSYSMO size drivers in COCOMO for early estimates

• Add documented list of assumptions to COSYSMO

14

Next Steps

• Provide results and recommendations to USC team for these models

• Conduct 2-day workshop at USC in conjunction with COCOMO Forum

– 29-30 OCT 2008

– Workshop to be led by Ricardo Valerdi, Garry Roedler, and Jared Fortune

– All PSM Workshop participants are invited

15

Backup

16

Gaps – Details

17

Gaps – Details (cont.)

18

Overlaps - Details

19

Uncertainties/Inconsistencies

20

Notes & Discussions