Achieving Effective Acquisition of Information

Technology in the Department of Defense

Study Conducted byComputer Science and Telecommunications Board

National Research CouncilPresented by

Dr Steven KimmelSenior Vice President

Alion Science and Technologyto

NDIA 26th Annual T&E ConferenceMarch 2, 2010

22

Background

The National Academies formed a Study Committee of experts from defense and commercial IT sectors and charged them to assess: Legislative requirements and regulatory processes

governing IT systems acquisition in DOD Commercial IT acquisition best practices DOD’s acquisition environment, culture, and barriers Concepts for systems engineering and testing in virtual

environments

3

Committee on Improving Processes and Policies of the Acquisition, Tech, & Evaluation of IT in the DOD

William Campbell, Co-Chair, BAE Systems Dawn Meyerriecks, Co-Chair, Dawn Meyerriecks, LLC* Robert Behler, The MITRE Corporation Philip Coyle, World Security Institute Renato Dipentima, SRA International (ret.) John Gilligan, Gilligan Group, Inc. John Goodenough, Software Engineering Institute Paul Kern (NAE), The Cohen Group Steven Kimmel, Alion Science and Technology Deidre Lee, Professional Services Council Joshua Levine, ESP Technologies Corp. Nachiappan Nagappan, Microsoft Research Frank Perry, Science Applications International Corporation Vaho Rebassoo, The Boeing Company Daniel Sturman, Google Inc.

*Dawn Meyerriecks resigned from the committee in October, 2009, upon her appointment as Deputy Director of National Intelligence for Acquisition and Technology.

4

Agenda

Study Scope Findings and Recommendations Closing Remarks

4

New Acquisition process based on agile acquisition and

Iterative Incremental Development (IID) principles

5

Study Scope

This study focused on IT systems that support the DOD Information Enterprise

To align acquisition approaches with technical characteristics and risks, IT systems were categorized in this report as: Software development and commercial-off-the-shelf integration

(SDCI) programs Commercial-off-the-shelf hardware, software, and services

(CHSS) programs Excluded are the IT-based components embedded

in weapons systems or DOD-specific hardware

5

6

Findings and

Recommendations

6

7

Major Findings

1. DOD systems acquisition policies, expertise, practice, and culture reflect the norms associated with large weapon systems programs

2. Weapon system acquisition processes are often applied to IT systems acquisition, without addressing unique aspects of IT

3. Dollar thresholds for assigning oversight levels on IT programs are much lower than for weapons system oversight -- a disparity that subjects too many IT programs to OSD-level oversight rather than delegation to lower levels that are more agile

4. DOD acquisition, budgeting, and requirements processes are being inappropriately applied to relatively low-dollar IT programs

5. IT program requirements are often written with overly detailed specifications that are inconsistent with the pace of technological change and need for rapid delivery of end-user capabilities

7

8

Major Findings (continued)6. The “waterfall” process used for large IT programs is too document-

intensive, time-consuming, and process-bound to respond effectively to end user needs

7. Although program tailoring is an option, DOD has no established best practice for tailoring and it is seldom used

8. The DOD acquisition training curriculum does not adequately address IT system acquisition or facilitate adoption of applicable commercial methods, processes, products, and services

9. DOD is unable to keep pace with the rate of IT innovation in the commercial marketplace, cannot fully capitalize on IT-based opportunities, and seldom delivers IT-based capabilities rapidly

10. With the exception of IT purchased via vehicles like Enterprise Software Initiative contracts, COTS technologies are insufficiently leveraged, excessively tailored, inefficiently tested, and delayed

8

9

Major Findings (continued)11. Absent discipline and end-user advocacy, large acquisition

oversight bodies can give undue leverage to low-value-added process requirements or “corner case” desires of any participant, which disproportionately impacts on program cost and schedule

12. Testing is integrated too late and serially in current DOD IT systems acquisition practices with testing in realistic operational environments deferred until the mandated operational test

13. Without regular feedback from a user perspective on IT system development, insight necessary to manage and oversee such programs is inadequate

14. The acquisition community has been reluctant to embrace virtualized testing and overtly precluded from re-using or accessing operationally-relevant test data and environments

9

10

Major Findings (continued)15. To more rapidly deliver software capability, the commercial

world has widely embraced the iterative, incremental, development (IID) approach which deals with complexity and features:

The prominence of the end user’s voice A focus on big R requirements—a concise description of the

purpose, mission, and expected outcome of an IT systems acquisition—during early planning

Close integration of developmental and operational test and evaluation into the development cycle

Breaking down a project into incrementally deliverable increments

10

Recommendations1. Adopt a new acquisition process tailored for IT

systems2. Adopt an iterative, incremental approach for

acquiring IT systems3. Perform continuous testing, with early

involvement from end users, in acquiring IT systems

12

Recommendations (continued)

3. Perform continuous testing, with early involvement from end users, in acquiring IT systems3-1 Adopt continuous testing in IT systems development, and insist on the use of metrics, especially emphasizing measures of end-user satisfaction3-2 The Acceptance Team should play a lead role in recommending deployment decisions3-3 Test with users in their actual work or field environment (sometimes referred to as a Beta deployment)3-4 Accept certification and functional IT system component test results across organizational boundaries3-5 Use virtual test environments to support both continuous feedback and certification of operationally suitable and effective solution increments

12

1313

Concluding Remarks The imperative—and opportunity—for change has

never been more obvious. To fully leverage the potential of IT, it is essential that DOD not simply alter a process that has repeatedly failed.

Instead, it should shift from deliberate application of the weapons system acquisition process to IT programs to a new process tailored specifically to IT system acquisition.

Full commitment to this change will touch every aspect of the culture, the processes, and the people.

14

The Final Study Report Report available at www.cstb.org Prepub currently available as download Final edited edition available for download and printed

books available for purchase March. 2010 Contact Jon Eisenberg, CSTB Director jeisenbe@nas.edu 202-664-1235

Back-up Study Details

16

Recommendations

16

1. Adopt a new acquisition process tailored for IT systems1-1 Emphasize timeliness and end user mission success in the IT acquisition culture rather than rigid oversight and process compliance1-2 State IT systems requirements as top-level mission expectations (that is, “big R”) rather than as detailed processes or technical solutions; develop the details (“small r”) by iterative refinement with users1-3 Leverage flexibilities within IT acquisition funding to achieve speed and agility in the new acquisition process1-4 Provide IT systems acquisition professionals with education in modern IT systems and establish minimum competency standards1-5 Use pilot programs to rapidly implement these recommendations1-6 Propose legislative and regulatory changes to

(1) codify a new agile process for acquiring IT systems and (2) revise dollar thresholds for IT system acquisition oversight to foster decentralization

17

Recommendations (continued)

2. Adopt an iterative, incremental approach for acquiring IT systems2-1 Establish iterative, incremental development (IID) based on agile

software development and related approaches as the default2-2 Allocate top-level mission expectations (i.e., big R requirements)

across increments and use each increment to define and satisfy detailed requirements (i.e., little r requirements)

2-3 Establish separate and distinct strategies and processes for acquiring custom versus off-the-shelf IT systems

2-4 Establish, employ, and report measures of success that emphasize the end user experience, including timeliness to field

2-5 Provide a stable budget profile for IID IT programs across multiple increments

17

18

Iterative Incremental Development (IID) & Agile

Acquisition Processes

18

19

Iterative Incremental Development (IID) Long history of success using IID dating to 1950s in hardware and

1970s in software Trident, LAMPS, Space Shuttle avionics, CCPDS-R, Canadian ATC

More than 20 year history of recommendations to abandon waterfall and adopt IID

1987 DSB Task Force on Military Software, Brooks et al. 1994 DSB on Acquiring Defense Software Commercially 2000 DSB Task Force on Defense Software 2009 DSB Acquisition of Information Technology

Yet Defense IT Acquisition remains dominated by waterfall thinking IID is fundamentally a communications & learning process that

addresses two key issues: Difficulty in specifying detailed requirements up front Complexity in software systems

20

Agile Acquisition Characteristics

20

Emergent Agile characteristics Source: P. Crowe and Robert Clouter. “Evolutionary Capabilities Developed and Fielded in Nine Months” in CrossTalk. Available at http://www.stsc.hill.af.mil/crosstalk/2009/05/0905CroweCloutier.html

21

Recommended Approach for SDCI IT System Acquisition

Business Case Development

22

Time-boxed Iterations Within Each Capability Increment

22

System Development & Demonstration (SDD)

4 to 8 Week Iterations

Integrated T&E / Voice of the End User

Requirements Analysis, Re-prioritization &

Planning

Architecture Refinement

Test Cases

Implementation

Integration

Testing

Verification & Validation

Design

Requirements Analysis, Re-prioritization &

Planning

Architecture Refinement

Test Cases

Implementation

Integration

Testing

Verification & Validation

Design

Requirements Analysis, Re-prioritization &

Planning

23

Recommended Approach for CHSS IT System acquisition

23