October 2003 Costing COTS Integration - macOS...

31
1 OCERA Office Of Cost Estimation & Risk Analysis Costing COTS Integration Costing COTS Integration Linda Brooks October 2003 Los Angeles 18 th International Forum on COCOMO and Software Cost Modeling/ LA SPIN Meeting

Transcript of October 2003 Costing COTS Integration - macOS...

Page 1: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

1OCERA

Office Of Cost Estimation & Risk Analysis

Costing COTS IntegrationCosting COTS Integration

Linda Brooks

October 2003

Los Angeles

18th International Forum on COCOMO and Software Cost Modeling/ LA SPIN Meeting

Page 2: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

2OCERA

Office Of Cost Estimation & Risk Analysis

ObjectiveProvide a roadmap for doing an estimate for a Commercial Off-the-Shelf (COTS) software intensive system development

Defining characteristics of COTS: • Source code not available• Future evolution of product not

controlled by system developer

Page 3: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

3OCERA

Office Of Cost Estimation & Risk Analysis

Topics

Challenges of Estimating COTS IntegrationComponents of an EstimateCOTS Integration Software Estimating MethodsAvoiding Common COTS Estimation PitfallsValidation ExampleSummary

Page 4: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

4OCERA

Office Of Cost Estimation & Risk Analysis

Topics

Challenges of Estimating COTS IntegrationComponents of an EstimateCOTS Integration Software Estimating MethodsAvoiding Common COTS Estimation PitfallsValidation ExampleSummary

Page 5: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

5OCERA

Office Of Cost Estimation & Risk Analysis

Challenges of Estimating COTSSizingProductivity (size/effort)Limited availability of historical data and seasoned parametric modelsNon-separable effort in developmentActivities, life cycle phases and risks substantially different than new developmentCOTS integration typically underestimated

Page 6: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

6OCERA

Office Of Cost Estimation & Risk Analysis

COTS vs. Custom Analyze Design Implement Integrate Test Deliver Sustain

* Cost of COTS versus Custom development needs to be evaluated across life-cycleMost factors are project dependent

Cost Factor * COTS CustomMarketing researchSelection AnalysisDesign/TailoringImplementationInterfaces/IntegrationSecurityLicensesTechnical supportTestingSystem Cut-OverTrainingUpgradesReplacement

$$$$$$$$$$$???$$$$$$$$$$$$$$$

$$$$$$$$$$???$$$$$S$$$$$$$

Source: The MITRE Corporation. Used with permission.

Page 7: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

7OCERA

Office Of Cost Estimation & Risk Analysis

TopicsChallenges of Estimating COTS IntegrationComponents of an EstimateCOTS Integration Software Estimating MethodsAvoiding Common COTS Estimation PitfallsValidation ExampleSummary

Page 8: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

8OCERA

Office Of Cost Estimation & Risk Analysis

Sample Cost Estimation Process Flow

Review with higher level

management/signoff

Plan the estimateIdentify cost

categories/ select estimating methods

Estimate software size

and cost drivers

Generatecost

estimate/risk analysis

Perform sanity checks/ conduct

reviews

Prepare Basis of Estimates (BOEs)

Presentation will cover in more detail

Page 9: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

9OCERA

Office Of Cost Estimation & Risk Analysis

Basic Cost Estimation Methods

Direct comparison to historical dataAdapted ComparisonCost Estimation Relationships (CERs) and standards Bottoms Up (engineering judgment based on detailed plan)Parametric cost model (preferably calibrated to org historical data)Level of effort (LOE)Quote for services or material to be purchased (COTS, Government-Off-the-Shelf [GOTS], NDI, etc.)

Page 10: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

10OCERA

Office Of Cost Estimation & Risk Analysis

Estimate Software Size and Cost DriversIdentify and document software components to the lowest level possibleIdentify components expected to be satisfied by

Developed softwareNew software to be writtenAdapted Software

Commercial off-the-shelf (COTS) softwareIncludes COTS, GOTS, and NDI (Non-Developmental Items) where no source code is available

Size each developed component and identify cost drivers*Identify COTS size and cost drivers*Identify size/cost driver ranges if appropriate (for risk analysis)

* Can use definitions from selected parametric cost model

Page 11: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

11OCERA

Office Of Cost Estimation & Risk Analysis

Risk Analysis

Identify the the cost and risk drivers and quantify their impactUse cost categories, activities, cost driver settings, lessons learned from this briefing or other sources to trigger identification of risksIdeally develop at least three estimates (both cost and schedule)

Best, worst, and expected baseline casesMonte-Carlo techniques can provide more sophisticated risk analysis

Page 12: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

12OCERA

Office Of Cost Estimation & Risk Analysis

Components of a Complete Estimate

Ground rules, assumptionsDefinitions & quantification of termsWhat the estimate includes (WBS Element, WBS Dictionary)Estimate of size, other cost drivers Required resources (labor hours)End products (Software, Specs, Manuals, etc.)Estimated scheduleEstimated costs (labor and non-labor)Basis of Estimate (BOE)Risk Assessment

Page 13: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

13OCERA

Office Of Cost Estimation & Risk Analysis

Topics

Challenges of Estimating COTS IntegrationComponents of an EstimateCOTS Integration Software Estimating MethodsAvoiding Common COTS Estimation PitfallsValidation ExampleSummary

Page 14: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

14OCERA

Office Of Cost Estimation & Risk Analysis

COTS Specific Activities

S/W Development Phase Activities: Custom & COTS

REQUIREMENTS DESIGN IMPLEMENTATIONACCEPTANCE

& DEPLOYMENT

SUSTAINMENT

Custom Development Activities

•Preliminary Design•Detailed Design

•Code & Unit Test•Component Test•System Test

•Planning•System Reqs•Architecture•Concept of Ops•SW Reqs

•Acceptance Test•Site Installation•Site Activation

•Operations•Maintenance•Enhancements

•COTS Approach (Reqs Negotiation)•COTS Component Identification•COTS Assessment/Selection•Prototyping

•Glue Code Development•Tailoring/Configuring•COTS Integration & Test•Component refresh

•Component Refresh•Maintenance

Page 15: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

15OCERA

Office Of Cost Estimation & Risk Analysis

COTS Software Integration Activities (1 of 2)Assessment (COCOTS)

Research and evaluation of COTS products to select products to be used, 2 parts:

Initial filtering of productsDetailed assessment and evaluation

Cognition (SEER-SEM)

Understanding the functionality of the COTS software so as to have insight into, knowledge of, and comprehension of the receiving system/software in order to integrate it with the system or with other COTS components

Page 16: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

16OCERA

Office Of Cost Estimation & Risk Analysis

Tailoring (COCOTS)Customization of the product for its intended use

Provides required functionality missing in the COTS product, orIsolates users and/or developed software fromunrequired/undesired capabilities or overly complex interfaces (e.g, “wrappers”)

“Glue” code (COCOTS)Developed software that

Facilitates data or information exchange between the COTS component and the system or other COTS components into which it is being integratedConnects or “hooks” the COTS component into the system or to other COTS components but does not necessarily enable data exchange

COTS Software Integration Activities (2 of 2)

Page 17: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

17OCERA

Office Of Cost Estimation & Risk Analysis

COCOTS/COCOMO Costing ViewCOCOTS/COCOMO Costing View

TIME

STA

FFIN

G

Used with permission Chris Abts/Dr. Barry Boehm (USC)

Page 18: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

18OCERA

Office Of Cost Estimation & Risk Analysis

SEER-SEM COTS Costing View

PROGRAM…

SYSTEM

UNIT

CUSTOM COMPONENT

COTS COMPONENT

PROGRAMPROGRAM

UNIT…

Summary level, define overall system characteristics

Define overall program/CSCI characteristics, sums both custom and COTS components

Custom software component • New, reused, modified software – SLOC, FP, Proxy sizing• COTS glue code• COTS Tailoring scripts, DDL, etc.

COTS Software Component•COTS functional sizing•COTS specific cost drivers•COTS Cognition•License/Maintenance Fees

Page 19: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

19OCERA

Office Of Cost Estimation & Risk Analysis

SEER-SEM COTS Component Sizing

IFPUG function point (FP) compatible sizing Quick Size - Computes sizes based on selection from COTS application list and % usage parameterFeature Sizing - FPs translated to terminology related to common COTS featuresObject Sizing – FPs translated to object related terminology

Page 20: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

20OCERA

Office Of Cost Estimation & Risk Analysis

Candidate COTS Integration Estimation MethodsOverall

Method #Specific Method N

EW

AD

APT

ED

GLU

EA

SSES

SMN

TTA

ILO

RIN

GC

OG

NIT

ION

VOLA

TILI

TYIN

TEG

RA

TIO

NSY

STEM

I &

T

1 SEER-SEM b b b b b b b b

COCOMO II b b b

COCOTS b b b b

Parametric Model

b b b b b b

Non-Model Estimate

b b b b b

2

3

C C C

Custom Code COTS Custom/COTS C SEER COTS Unit

C

b Normally Included May Be Underestimated

b

bb

b

Page 21: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

21OCERA

Office Of Cost Estimation & Risk Analysis

Topics

Challenges of Estimating COTS IntegrationComponents of an EstimateCOTS Integration Software Estimating MethodsAvoiding Common COTS Estimation PitfallsValidation ExampleSummary

Page 22: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

22OCERA

Office Of Cost Estimation & Risk Analysis

COTS Tasks Typically Overlooked or Underestimated

Systems and software engineeringHands-on prototypingIntegrated system training and documentationAcquisition of COTS software in-depth knowledge

-e.g., mentors, toolsmiths, vendor supportComponent and system performance testsComponents and system regression tests with each upgradeRelated software changes to support COTS software upgrades

Glue code, database changes, configuration filesDeveloper/operator training needed for COTS upgrades

Source: The Aerospace Corporation. Used with permission

Page 23: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

23OCERA

Office Of Cost Estimation & Risk Analysis

COTS Software Development Cost Estimation Method - 3System Physical

Architecture/ReqsWBS

Estimated Labor

Estimate Labor Per COTS Package

Parametric Model Estimate

Components

• New Code• Adapted Code• COTS Tailoring w SLOC/FP (e.g.

Scripts, DDL …)

Total Development Effort Estimate

COTS Tailoring(not in model est.)COTS Assessment

COTS VolatilityCOTS Cognition

Additional Integration(See Worksheet)

Page 24: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

24OCERA

Office Of Cost Estimation & Risk Analysis

Topics

Challenges of Estimating COTS IntegrationComponents of an EstimateCOTS Integration Software Estimating MethodsAvoiding Common COTS Estimation PitfallsValidation ExampleSummary

Page 25: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

25OCERA

Office Of Cost Estimation & Risk Analysis

Standard Process for Performing a Parametric Validation of Historical Data

Detailed code count

Identify COTS Products

Identify Development Environment

Map WBS to Common Model

Identify Project Milestones

Allocate Shared Support Functions

Parametric Model of

Software Size, Effort,

Schedule

Detailed Actual Cost,

Effort by Subsystem,

Phase, Function

Source: K. Bradford, Northrop Grumman Mission Systems, GSAW2001

Page 26: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

26OCERA

Office Of Cost Estimation & Risk Analysis

Example of SEER-SEM Validation

Successful calibration of SEER-SEM to Mission Systems COTS intensive completed projectUsed SEER COTS Components with SEER quick sizing to model COTS elementsUsed developed S/W components with SLOC counts for 4GL tailoring scripts and DDLMatched total actuals to within 7% of SEER estimate

Included both COTS and non-COTS development laborWith adjustments came within 2%

Provides confidence in method for future COTS estimates for similar COTS intensive developments

Page 27: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

27OCERA

Office Of Cost Estimation & Risk Analysis

Actual Vs Model of Delivered Software

A B C D E F G H

Product

Actual Effort

Model ofDelivered Product

A calibrated model provides:• Validation of provided data• Basis of estimate for future

enhancements

• Product F was forced to restart with alternate COTS product 28% into the schedule.• Product C was severely constrained by staffing problems.• Difference <2% with these corrections

Product % DiffA 12%B -9%C 31%D 4%E 20%F -43%G 0%H 6%Total 7%

Source: K. Bradford, Northrop Grumman Mission Systems, GSAW2001

Page 28: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

28OCERA

Office Of Cost Estimation & Risk Analysis

Topics

Challenges of Estimating COTS IntegrationComponents of an EstimateCOTS Integration Software Estimating MethodsAvoiding Common COTS Estimation PitfallsValidation ExampleSummary

Page 29: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

29OCERA

Office Of Cost Estimation & Risk Analysis

Summary

There are lessons learned from using and integrating COTS software applications, use themThe necessary components of a complete estimate are always required, even with estimating for COTS usageUse cost drivers and sizing inputs from parametric models (such as SEER SEM or COCOTS) to derive your COTS integration estimateCross check estimates by using results from two estimating methods Calibrate your estimating models and methodologies with historical dataEvaluate the cost of COTS versus custom development across the life-cycleIdentify the the cost and risk drivers and quantify their impactRevise estimates as you obtain more or better information

Page 30: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

30OCERA

Office Of Cost Estimation & Risk Analysis

References

SEER SEM™ Estimating Model, Galorath, Inc., http://galorath.com/tools_soft.shtmSoftware Cost Estimation With COCOMO II, Barry Boehm et al, Prentice Hall, 2000COCOTS Estimating Model, USC Center for Software Engineering, http://sunset.usc.edu/research/COCOTS/Kathy Bradford, Lessons Learned in Estimating the Software Cost of a Ground Station with COTS Integration, GSAW2001, http://sunset.usc.edu/gsawIFPUG Counting Practices Manual 4.1, http://www.ifpug.org

Page 31: October 2003 Costing COTS Integration - macOS Serversunset.usc.edu/events/2003/Presentations/3_Brooks_COOCMO COTS... · Costing COTS Integration Linda Brooks October 2003 Los Angeles

31OCERA

Office Of Cost Estimation & Risk Analysis

Acronym ListBOM – Bill of MaterialsCER – Cost Estimating RelationshipCM – Configuration ManagementCOCOMO – COnstructive COst MOdelCOCOTS – COntstructive COTSCOTS – Commercial Off-the-ShelfDDL – Data Definition LanguageFP – Function PointsGFE – Government Furnished EquipmentGOTS – Government Off-the-ShelfGSAW – Ground System Architectures WorkshopGUI – Graphical User Interface

H/W – HardwareIFPUG – International Function Point Users GroupLOE – Level of EffortNDI – Non-Developmental ItemNGMS – Northrop Grumman Mission SystemsOCE – Office of Cost EstimationQA – Quality AssuranceSEER-SEM™ – System Evaluation and Estimation of Resources -Software Estimation ModelS/W – SoftwareUSC – University of Southern CaliforniaWBS – Work Breakdown Structure4GL – Fourth Generation Language