Toward mission assurance: a framework for systems...

Toward Mission Assurance:A Framework for SystemsEngineering ManagementBrian Sauser*

Systems Engineering and Engineering Management, Stevens Institute of Technology, Castle Point on Hudson, Hoboken,NJ 07030

TOWARD MISSION ASSURANCE

Received 9 March 2005; Revised 12 February 2006; Accepted 13 February 2006, after one or more revisionsPublished online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/sys.20052

ABSTRACT

From its inception, NASA has pushed the boundaries of science and engineering and prideditself on knowing how to manage these engineering innovations. However, pushing themanagement boundaries requires a careful adaptation of risks, resources, and procedures,and projects must clearly assess the complexities and uncertainties of the task. Too oftenattempts to resolve errors in these areas are through technical solutions when the root causeis managerial failure. The purpose of this paper is to illustrate the value of using a systemsengineering management framework based on a contingency approach for associating acorrect management style to a project classification, thus reducing the ultimate failure point(i.e. managerial error). Shenhar and Dvir’s Novelty-Complexity-Technology-Pace (NCTP)framework was used as a representative framework to provide an empirical analysis of fournoted NASA projects. The NCTP framework uses a contingency approach to provide amultidimensional categorization of projects based on their novelty, complexity, technology,and pace with a correlation to an appropriate management style. The paper will conclude witha discussion of how a contingency framework can have implications on NASA and systemsengineering. © 2006 Wiley Periodicals, Inc. Syst Eng 9: 213–227, 2006

Key words: case study; contingency theory; systems engineering management

1. INTRODUCTION

Historically, NASA projects have been difficult, largeventures that inevitably carried high costs. What re-

Regular Paper

*E-mail: [email protected]

Systems Engineering, Vol. 9, No. 3, 2006© 2006 Wiley Periodicals, Inc.

213

sulted were expensive products, long developmentprocesses, complex management, the need for largerrockets, cost overruns that lengthened missions, andsome troublesome failures [Kerr, 1994]. Therefore,there have been concerted efforts within industry andgovernment to try and understand the development ofthese large systems projects. Large systems projectsconsist of customized, interconnected subsystems,carry high cost, are designed for one customer, areproduced in low volume, require broad and deep knowl-edge and skills, engage multiple collaborators, involvethe customer and suppliers throughout the life cycle,and have strong political considerations [Floricel andMiller, 2001; Miller and Lessard, 2000]. The knowl-edge base of large systems projects is limited in the areaof project management and systems engineering. Thereis limited theory on how these projects develop and aremanaged. This need for management theory has be-come even more important as there has been increasingattention on large systems projects in the economicactivities of firms, industries, and nations [Hansen andRush, 1998]. Hobday, Rush, and Tidd [2000] stateconventional innovation wisdom is derived from re-search on high volume consumer products; new evi-dence, models, and concepts are needed to properlyunderstand the innovation process in complex productsand systems.

In NASA’s long history, it has conducted numerousprojects of various kinds, but its most noted has been inthe development of large systems (e.g., Apollo, SpaceShuttle, Mars Pathfinder). However, with the recent lossof the Space Shuttle Columbia, NASA has had to takea concerted look at how it manages these types ofprojects. The Columbia Accident Investigation BoardReport, Volume 1 (CAIB Report) detailed many of theorganizational problems that existed not only within theSpace Shuttle Program but within NASA, and chron-icled some of these problems back to the Challengeraccident [CAIB, 2003]. The CAIB would later state thatone of the root causes in the Space Shuttle Programfailures was a “mischaracterization of the Shuttle asoperational rather than developmental” [CAIB, 2003:9]. The NASA guidelines and requirements on projectmanagement (NPR 7120.5C) do not address charac-terization differences among projects and the ways theyshould be managed. These challenges are not unique toNASA as other government agencies documentationand guidelines have endeavored to address the develop-ments of management frameworks for characterizingand managing projects. For example, the U.S. FederalAviation Administration (FAA) in cooperation with theU.S. Department of Defense has developed an inte-grated model as an “effective and efficient frameworkto guide process improvement efforts” [Ibrahim et al.,

2004]. Key to the FAA model is the construct of appli-cation areas. An application area classifies related ap-plication practices that are deemed necessary foraccomplishing the essential outcomes distinct to theapplication or discipline. These application areas pro-vide a guide for distinguishing which identified processareas and practices in a reference model need to be putinto practice to concentrate on the purpose of the appli-cation area. Within NASA, in spite of several attemptsand suggestions, there is still no agency-wide frame-work for distinction among projects, and differentiatingthe systems engineering management principles inthese projects. Additional investigations have also sug-gested that the two Space Shuttle accidents or otherproject failures were the result of incorrect managementstyle used in these projects [Dimitroff, Schmidt, andBond, 2005; Guthrie and Shayo, 2005; Griner andKeegan, 2000; Shenhar, 1992].

The purpose of this paper is to illustrate how asystems engineering management framework can use acontingency approach to identify the appropriate man-agement style for a project and thus reduce the potentialrisk of managerial error in a project. A contingencyapproach analyzes an appropriate fit between projectcharacteristics and project type to define a managerialapproach or style. To show the potential impact of thiscontingency approach, Shenhar and Dvir’s [2004]NCTP (Novelty-Complexity-Technology-Pace) frame-work was used as a representation of a systems engi-neering management framework that uses acontingency approach. The NCTP framework providesa multidimensional categorization of projects based ontheir novelty, complexity, technology, and pace with acorrelation to an appropriate management style. A casestudy methodology was used with the NCTP frame-work to analyze four NASA projects. This paper willuse this analysis to describe the managerial error of oneof the projects in detail (i.e., Comet Nucleus Tour), andshow how it defined the managerial approach to theother three projects (i.e., Mars Pathfinder, Lunar Pros-pector, and Mars Climate Orbiter). I will conclude withan explanation of how a systems engineering manage-ment framework, using a contingency approach, canhave implications on NASA and the discipline of sys-tems engineering.

2. CASE OVERVIEWS

Comet Nucleus Tour (CONTOUR) was proposed dur-ing the original Announcement of Opportunity (AO)from the Discovery Program, but this proposal was notselected, and did not even make it past the first roundof selections. Between that AO and the next AO, a

214 SAUSER

Systems Engineering DOI 10.1002/sys

complete revamp of how CONTOUR would work as amission and how close it would fly by the nucleus wasperformed. One of the most significant changes, thatlater would became the focus of its technical failure,was the use of Earth swing-bys which resulted in inte-grating the solid rocket motors (SRM) with the space-craft. Proposed by Cornell University as a principalinvestigator (PI)-led mission, CONTOUR would be ajoint project between Cornell University and JohnsHopkins Applied Physics Laboratory (APL), alongwith 14 other university, government, and industry co-investigators. Cornell led the science and APL led thespacecraft development. APL had a string of successeswith the most recently noted being the Near EarthAsteroid Rendezvous (NEAR). With a budget of $159million, CONTOUR was scheduled to fly within 60miles of three comets. The team was feeling confidentwith the complexity of CONTOUR that would make itanother Discovery Program success and erase the nega-tivity NASA was receiving from two Mars failures(Mars Climate Orbiter and Mars Polar Lander).

CONTOUR was scheduled to initiate a SRM burnto accelerate the spacecraft and place it on a heliocentrictrajectory toward Encke (its first comet encounter).While operations continued based on the assumptionthat the firing took place on schedule, shortly afterinitiation of the burn, no signal was received from thespacecraft. A few days later, three objects were identi-fied from the University of Arizona’s Lunar and Plane-tary Laboratory Spacewatch Project near the expectedposition of CONTOUR. This led investigators to be-lieve that the firing took place and that these objectswere parts of the spacecraft and rocket engine. NASA’sconclusions were confirmed by images taken by theDepartment of Defense. Communications attemptscontinued with the spacecraft the next few months, andthe mission was declared officially lost after these at-

tempts showed no signs of success. NASA establisheda Mishap Investigation Board (MIB) to review thecircumstances and potential lessons learned from CON-TOUR. The Board concluded that the probable causewas a failure in the integration of the SRM, but wasunable to determine with certainty the root cause due tothe lack of data during the SRM firing.

The additional three cases are summarized in TableI. Combined, these four case were chartered underNASA’s “faster, better, cheaper” (FBC) mantra andrepresented the maturation of this initiative. These casesrepresented a challenging time in the development of anew management initiative in NASA and some failuresand successes of these types of projects.

3. THEORY FOR A SYSTEMSENGINEERING MANAGEMENTFRAMEWORK

Although there are texts and guidebooks written onproject management and systems engineering that givestandard definitions, theories, processes, and strategiesto apply to projects in general, some have theorized thatprojects carry a complexity all their own and that nosingle management style can fit all projects [Amara,1990; Shenhar, 2001]. What is traditionally not welldefined is the complex activity of a project and how thisactivity is managed. In spite of a growing use of projectmanagement and systems engineering as a practice,most of the classical studies have often advancedknowledge in a single focused area. In the most recog-nized project management professional society, ProjectManagement Institute (PMI), they have fostered anddeveloped a classical project management process forall projects [PMI, 2004]. For systems engineering, theInternational Council for Systems Engineering (IN-

Table I. Case Overviews

TOWARD MISSION ASSURANCE 215


COSE) has likewise produced a fundamental body ofknowledge on systems engineering [INCOSE, 2004].Fundamental bodies of knowledge are essential tobuilding project success, as is the tailoring of thesepractices to fit project diversity, which both of thesedocuments uphold. However, the scope of engineeringsystems has changed dramatically and become a signifi-cant challenge in our ability to achieve project success[Calvano and John, 2004]. Therefore, no one approachcan solve these emerging problems, and thus no onestrategy is best for any one project [Drejer, 1996]. Itbecomes challenging to know what practices to effec-tively apply to these increasingly complex projectswithin varying domains. For instance, the managementstrategies for building the Space Station or designing aMP3 player can be effective and efficient, but both arevery different.

PMI has recently begun to recognize the need forunique management principles for different projecttypes with the development of government, U.S. De-partment of Defense, and construction extensions to theGuide to the Project Management Body of Knowledge(PMBOK®) [PMI, 2003a, 2002, 2003b]. These advanc-ing concepts in management were fostered by severalstudies that have expressed that not only are all projectsnot the same but a categorization, classification, ortypology system is needed with corresponding manage-ment practices and styles. Most noted in these studiesare Lewis et al. [2002], who proposed a framework thatshowed that management styles fluctuate over time andblending of styles enhances performance; Archibaldand Voropaev [2003; Archibald, 2003], who proposedproject categorizes and subcategorizes as essential stepsin the project portfolio management process; Crawford,Hobbs, and Turner [2004], with support from PMI, whostudied project categorizations and their purposes andattributes, as used by companies around the world; andShenhar and Dvir [2001; Shenhar, 2004], who devel-oped a typological theory of project management and afour dimensional framework for project analysis. Thesestudies are referenced because they approach projectmanagement with a systematic interpretation, allow formanagerial decision-making in their application, take auniversal, multiproject approach to project categoriza-tion, and address issues related to project complexity.However, there are few grounded research studies thatuse a typology matched to a project classification andmanagement style.

The linkage of a project classification to a manage-ment style becomes decisive, as organizational errorshave been shown to often be the root of failures inengineering systems [Paté-Cornell, 1990]. In Reason’s[1997] Managing the Risks of Organizational Acci-dents, he contends that the engineering and manage-

ment processes are the primary areas for failure factorsin projects. In a study of space systems, Newman [2001]analyzed 50 space system failures from 1960 to 2000and concluded that failure in these projects was causedby the need for “across the board systems engineeringrigor.” In spite of this recognized gap, projects are stillfailing, and routinely the failure is caused by an inabilityto perform rigorous project management and systemsengineering practices. Furthermore, there is still confu-sion on the effects of these practices to organizationaloutcomes [Gatignon et al., 2002].

4. NCTP FRAMEWORK

Based on classical contingency theory, Shenhar andDvir [Shenhar, 2001; Shenhar and Dvir, 1996, 2004]have proposed a typology based on an elemental foun-dation in contingency theory for managing differenttypes of projects (e.g., systems projects). This typologytheorizes a framework for project managers and sys-tems engineers for the planning and execution phasesof a project with a correlation to a management stylethat includes systems engineering principles for a sys-tems engineering management framework. Assessingthe environment and the task, a project is classified onfour dimensions, and the right management style to fitto the project type. Shenhar and Dvir state that projectscarry contingencies based on the four dimensions ofnovelty, complexity, technology, and pace, the NCTPframework. These four dimensions and the subfactorsfor these dimensions are defined in Table II.

Once a project is classified based on these fourdimensions, it defines certain characteristics of thatproject that make it unique in how it is managed. Figure1 shows how the four dimensions are reflected on agraph, and that connecting the NCTP classification witha straight line to form a diamond gives a qualitativerepresentation of the level of risk associated with aproject.

5. METHODOLOGY

A case study research methodology was chosen becauseit allowed for the characterization of real-life events,such as organizational and managerial processes, andthere was no requirement for control over behavioralevents, thus allowing for the capture of holistic andsignificant experiences [Eisenhardt, 1989; Gillham,2000; Yin, 1994]. Eisenhardt [1989] states that casestudy research provides a conduit to go from theory todata and back to theory.. Eisenhardt describes a funda-mental difference in case study research is that cases are

216 SAUSER


chosen for theoretical reasons, not statistical reasons.This research used the following steps.

5.1. Case Selection and Definition

This step involved the definition of the cases to beevaluated, the framework in which the cases were de-scribed, and the possible alternatives and their conse-quences. The cases were defined as a project thatrepresented a maturation of FBC with the followingattributes:

• Tasked to be completed under a FBC manage-ment style.

• Defined as a large systems project.• Classified as a government or industry aerospace

development project.• U.S.-based prime contractor.• Completed project in the last 12 years.• Project life cycle was completed at least through

launch.

5.2. Data Collection Techniques

A descriptive case study methodology was used wherecases were defined by a descriptive theory (i.e., NTCPframework) [Yin, 1994]. To address any threats to va-lidity as defined by Yin [1994], multiple sources ofevidence supported by data source triangulation [Den-zin, 1984; Stake, 1995; Yin, 1994] and a study protocolwere established for future replication and to reduce anybias in the collection of data [Shenhar, 1999]. Datacollection was performed using the following sourcesof evidence:

• Interviews: Interviews were conducted in a sem-istructured, open-ended conversational format toallow interviewees to speak freely and openlyabout their experiences. Interviews ranged fromFigure 1. NCTP framework.

Table II. Definitions of NCTP Framework Dimensions



30 minutes to 2 hours based on the interviewee’savailability and depth of information. A singleinterview session was performed with each sub-ject, with follow-up interviews on an as-neededbasis. Each interview was conducted by the primeresearcher of this investigation while performingfield research as an aerospace contractor at aNASA center. Six key personnel related to theproject were interviewed. These people repre-sented program management, project manage-ment, systems engineering, team members, andcustomer. Each interview was recorded on audio-tape and transcribed.

• Documentation (related to the project, but not aproduct of the parent organization): formal stud-ies, evaluations, journal articles, survey data,mass media, and physical artifacts (samples ofwork done).

• Archival and Historical Information (directly re-lated to a product of the project or parent organi-zation): letters, memoranda, policy statements,regulations, proposals, guidelines, procedures,summary reports, organizational records, andpersonal records.

• Participant Observation: NASA gave permissionfor participation in its Academy of Program andProject Leadership training programs. This in-cluded project management training classes.

5.3. Data Analysis

Data were collected in an iterative process as documen-tation and archival information was extensively ana-lyzed before interviews were conducted. After eachinterview, data were triangulated against documenta-tion and archival information to determine if additionaldata were needed from any data sources (e.g., follow-upinterviews, additional documentation) before the nextinterview was conducted. Once the data collection andtriangulation was completed for a single case, a 30–40page case summary was written based on a predeter-mined case format [Shenhar]. The case summary wasthen coded and analyzed with the NCTP framework.This resulted in an attribute-versus-alternatives matrix[Scholz and Tietje, 2002] for evaluating the projects(e.g., an attribute in the NCTP framework would beTechnology and the alternatives would be Low-Tech,Medium-Tech, High-Tech, and Super High-Tech). Thecross-cases analysis followed an iterative process wherethe completion of a subsequent case was followed byan analysis to gain familiarity with the data and evaluateor reevaluate the proposition.

5.4. Evaluation and Discussion

Once the final analysis was complete, a final iterationwas performed to develop and refine a final theoreticalstatement about the findings. This was defined as theevaluation and discussion. Evaluation and discussiondescribed the results of the evaluation of the cases,provided a case analysis, and offered recommendationsbased on the investigations objectives and purpose.

6. NCTP FRAMEWORK ANALYSIS

6.1. CONTOUR

CONTOUR was a strategic project that would help APLmaintain and build upon their growing competitiveedge in deep space missions. In addition, Cornell wouldbegin to build a strategic position as an academic leaderin space exploration. They believed that they wereimproving on existing products using technology andconcepts from previous missions (e.g., NEAR and Star-dust). CONTOUR required a significant level of insightand creativity both technically and managerially builtaround a core of talented, experienced people to pro-duce a value added product. The NCTP framework inFigure 2 represents the analysis of CONTOUR in re-spects to its novelty, complexity, technology, and pace.Connecting the NCTP classification with a straight lineto form a diamond gives a qualitative representation ofthe level of risk associated with the project.

Also presented in the model is how CONTOURdiffered in its level of risk. The solid line represents thepreferred managerial approach, while the dashed linerepresents the actual approach. While there is not alinear relationship between the area of the diamonds forpreferred and actual, it does represent a qualitativedifference in the degree of risk. Choosing the preferred

Figure 2. CONTOUR NCTP Classification.

218 SAUSER


approach in managing a project is a strategic decisionthat can have significant impact on a project’s success,and can be just as important as time, budget, and sched-ule. CONTOUR represented an attempt to use the FBCapproach within an organization that was founded ontaking high-level risks, and while sacrificing many ofthe previous practices to address such levels of risk.Indeed, this reinforces what some have suggested thatthis culture cultivated the eventual failure of FBC (e.g.,CONTOUR) and perhaps demonstrates that FBC couldnot be used in cases of extremely high risk. In the nextsections I will discuss CONTOUR’s NCTP classifica-tion in more detail.

6.1.1. NoveltyCONTOUR was a breakthrough project. At the time, noone in the history of space exploration had brought aspacecraft as close to a comet as was planned forCONTOUR. Almost all of the technology they wereusing existed, was previously tested, or was being testedon NEAR and Stardust. Although APL had been suc-cessful with recent missions, not one successful missionhad been repeated (key factor in a breakthrough classi-fication). Each mission, even if using similar, proventechnology from past missions, had some degree ofunproven technology and was a new venture. CON-TOUR was no exception. Traveling to a comet incorpo-rated a lot of calculated risks, intuition, and trial anderror. In some respects, CONTOUR was new becausethis technology was being packaged into a new product.CONTOUR believed that they were building upon the

success and technology of past missions and ap-proached this project more as a platform project. Thisgave the perception that CONTOUR would be a nextgeneration in existing technology. Few projects in spaceexploration can be defined as platform projects (e.g.,Expendable Launch Vehicles—Delta II, Atlas) becauseseldom are they repeated. Novelty is related to a prod-uct’s uniqueness to the market, and even though NASAproducts could make their way to the market, they arenot designed or developed for traditional market appli-cations. Also, novelty is defined based on a comparisonto the market history. NASA projects are rarely repeatedand thus are almost always unique to the market (break-through). From inside the organization, some NASAprojects may not always be viewed as breakthroughprojects, but often the technology or its application isstill novel.

While CONTOUR may be classified as a break-through project, it will be discussed later how an organi-zation such as NASA may need a fourth classificationbetween platform and breakthrough. Table III shows theattributes versus alternatives for the novelty classifica-tion for CONTOUR and a representative statement ofthe distinctive factors that characterize its classification.It also indicates the preferred project management stylebased on project characteristics, compared to the actualstyle used in the project.

6.1.2. ComplexityCONTOUR was a system project that understood theintegration and complexity of that integration to de-

Table III. Novelty vs. Alternatives Matrix



velop a spacecraft. CONTOUR management ap-proached the project with a formal and bureaucraticstyle mixed with some informal relationships with sub-contractors and customers. It required tight and formalcontrol on technical, financial and schedule require-ments supported by many internal and external subcon-tracts. APL led the spacecraft development, CornellUniversity led the science, and other institutions weresubcontracted to develop the scientific instruments.Multiple key customers from industry, government, thepublic and the scientific community were dependent onCONTOUR’s success. As a systems project, CON-TOUR was a complex project that required extensiveplanning, computerized tools and software, tight andformal control, financial and schedule requirements,reviews with customers and management, and exten-sive documentation. While much of the developmentoccurred in-house at APL, a significant portion of thescientific development occurred outside of APL. Thisgeographic separation between major systems and

management (principal investigator and project man-ager) restricted real-time, face-to-face communication.While CONTOUR understood the subsystems of thespacecraft very well, this confidence resulted in per-forming integration testing late in the development,thus not giving them a full understanding of the uncer-tainty of the system. In addition, being a heavily ma-trixed organization, team members were not able to cutties with their parent organizations to fully commit tothe project.

6.1.3. TechnologyCONTOUR was a high-tech project. The technologywas mostly proven but being applied in a new way.Significant improvements were made to the technologyto develop CONTOUR from its original design, withminor modification to bring it together to function as acomplete system. CONTOUR required long periods ofdesign, development, testing, and redesign with multi-ple design cycles that had to start before the project

Table IV. Technology Attributes vs. Alternatives Matrix

220 SAUSER


started. In-depth, technical reviews were mandatoryand had to be supported by frequent and active commu-nication. The complexity and communication demandsrequired an intimately involved management team withgood technical skills. Management also had to recog-nize the unique challenges of CONTOUR and be flex-ible to extensive testing and design changes; therefore,design freezes had to occur as late as possible. Withoutproperly distinguishing CONTOUR’s unique technol-ogy characteristics, CONTOUR assumed a lower levelof uncertainty in the technology that they were usingfor the mission and managed the project more like amedium-tech project. This resulted in approaching thetechnology as a nonrevolutionary improvement to pastmissions when the application of the technology wasvery much one of the first of its kind. Development andtesting of the integration of the Solid Rocket Motor(SRM) was limited and did not receive extensive re-view. Communication was not as frequent with subcon-tractors and key contributors as was needed for ahigh-tech project. Table IV shows the attributes versusalternatives for the technology classification for CON-TOUR and a representative statement of the distinctivefactors that characterize its classification. It also indi-cates the preferred project management style based on

project characteristics, compared to the actual styleused in the project.

6.1.4. PaceCONTOUR was a time-critical project. From the timethe project got the green light, they were under pressureto meet a specified launch opportunity with a strictschedule. Time was critical for project success, anddelays meant project failure. The project team had to bespecifically picked for CONTOUR, and they were con-sidered a special group trying to achieve a rapid solutionto a vital project. Procedures had to be shortened, madesimple, and nonbureaucratic, while top managementhad to remain highly involved and constantly suppor-tive. The huge success of FBC at the time of CON-TOUR built a confidence in Discover Program missionsand gave the appearance that they could be managed asfast-competitive projects. While management under-stood that they were under a constrained time factor,they believed that the project could be accomplishedwith team members maintaining a 40-hour workweek.In addition, inadequate risk analysis led management tobelieve that events such as integrations and testingcould occur late and less frequently in the project. Theconsequence was that there was not the sense of urgencyas seen in successful FBC projects. Table V shows the

Table V. Pace Attributes vs. Alternatives Matrix



attributes versus alternatives for the pace classificationfor CONTOUR and a representative statement of thedistinctive factors that characterize its classification. Italso indicates the preferred project management stylebased on project characteristics, compared to the actualstyle used in the project.

6.1.5. Implications on CONTOURAfter the failure of CONTOUR, two independent re-view boards came to similar technical conclusions onthe failure of CONTOUR. While the technical failurebelieved to be related to the ignition of the SRM and itsintegration with the spacecraft, this will never be ableto be confirmed because of incomplete informationduring the launch phase. Irrelevant of the technicalfailure, the analysis presented in this paper indicatesthat CONTOUR’s management style and approach didnot fit a project of this classification. Therefore, theconception of CONTOUR’s technical failure may nothave occurred during launch and operation but duringthe project planning and initiation phases. While wemay never know exactly why the chosen managementstyle and approach were used for CONTOUR, it is theconclusion of this investigation and has been shown inother studies that as FBC progressed, it created pres-sures of working under constraints that were impossibleto achieve under the combined requirements of FBCand high project risk. FBC was pushing the envelopedof doing high-tech projects cheaper, and NASA hadbegun to produce a line of FBC successes. CONTOUR

management believed that they were building uponthese successes. This may have led to a false sense ofconfidence by CONTOUR management based on thesuccesses of past missions. To compound this, the im-plementation of the NASA Integrated Action Team’s(NIAT) recommendation from their investigation oftwo failed FBC missions, the Mars Climate Orbiter andMars Polar Lander, were coming out almost three quar-ters of the way through CONTOUR’s project life cycle.This resulted in CONTOUR management directing asignificant amount of attention to similar issues onCONTOUR, and may have drawn attention away fromother subsystems that needed attention.

6.2. Additional Cases

Table VI shows how the analysis of the three otherprojects resulted in their NCTP classification and oneshowing an incorrect approach. In all the projects ana-lyzed the team members interviewed said they had tomodify required project management and systems en-gineering principles and practices to be successful.Team members used heuristics to make determinationson what to cut to meet time and budget constraints.These heuristic decisions in some cases resulted in acostly failure (i.e., CONTOUR and MCO). The analysisused by the NCTP framework showed inconsistency inthe actual to preferred classification and thus appropri-ate management style. An analysis such as that of theNCTP framework or any applicable systems engineer-

Table VI. NCTP Classification for Other NASA Projects

222 SAUSER


ing management framework may have revealed that theprojects could not have been successful under theplanned approach. Perrow [1999] states that while adecision may appear perfectly rational, the operatormay be using it in the wrong context. An effectiveframework, while it may not guarantee success, canprovide practitioners with the tools so they can rely lesson heuristics. While it is clear from this investigationthat NASA can value from a unique framework forproject management and systems engineering, the exactprinciples to define that framework are not clear.

7. SUMMARY

7.1. Implications on NASA

These projects represented the maturation and evolu-tion of FBC, and Perrow [1999] states that as onebecomes more comfortable with a technology (in thiscase FBC), one is more willing to take the risks that mayultimately lead to failure. In the case of FBC, NASAcontinued to push projects to faster, better, and cheaperand Dan Goldin, former NASA administrator, laterstated that NASA pushed the “envelope too far.” Paté-Cornell [1990] stated that organizational error gener-ally is caused by factors such as excessive timepressures, failure to monitor hazard signals, and oftencaused by or encouraged by rules and goals set by thecorporation. In The Challenger Launch Decision: RiskyTechnology, Culture, and Deviance at NASA, Vaughn[1996] adds to this argument as she claimed that adisaster’s roots are found in the nature of an institution’slife. NASA is an institution that was founded on takinghigh-level risks. Therefore, it could be concluded thatFBC was destined to fail over some period of time.Could a framework have obverted or delayed the failurein FBC and mitigated the risk of this type of project?

To address the risk associated with a mission, acommon term in NASA for defining and mitigating riskis “mission assurance.” Mission assurance is defined bythe safety and enhancement of success of all NASAactivities through the development, implementation,and oversight on a project. One of NASA’s most press-ing challenges since the loss of Columbia, and theCAIB stated since Challenger, has been in how toeffectively achieve mission assurance. In any organiza-tions these same issues related to mission assurance aredefined by its ability to realize effective and efficientperformance, reliability, maintainability, supportabil-ity, and process. Therefore, can a systems engineeringmanagement framework help an organization such asNASA achieve mission assurance? While it may not beable to guarantee mission assurance it can provide anobjective analysis to support a mission assurance deci-

sion, and no framework should ever be the comprehen-sive answer to mission assurance. It was stated earlierthat an effective framework could help project manag-ers rely less on heuristics to make decisions about aproject. Why this may be true, no framework will everor should ever completely confine a project manager orsystems engineer from using their heuristic knowledge.

The NCTP framework defined the projects well andcould provide a fundamental framework for planningand managing projects. Currently within NASA thereare few frameworks for classifying a project, and noneof these frameworks makes a distinction among differ-ent project types, or how to tailor project or systemsengineering management to project characteristics.Some of these frameworks, like Technology ReadinessLevel (TRL), classify a technology based on its devel-opmental maturity and readiness in relation to beingacceptable for launch. While TRL may give some indi-cation as to a technology’s developmental state, it doesnot allow for the interplay between multiple technolo-gies, does not correlate to any specific managementprinciples or guidelines and only indicates where youare and where you need to be, not how to get there[Mankins, 2002].

Likewise NASA’s risk classification system is pre-dominately used to make executive approval decisionsby management, and not how to manage a project. Thisframework classifies projects into classes A to D (Aequals the highest level of risk) by using criteria suchas project priority and acceptable risk, national signifi-cance, complexity, mission lifetime, cost, launch con-straints, and achievement of mission success criteria. AtJPL the Strategic Systems Technology Program Officehas developed a systematic approach for selection ofNASA technology portfolios called the STrategic As-sessment of Risk and Technology (START) [Weisbin etal., 2004]. START offers a systems analysis for quanti-fying the feature of each development system, assessingits risk, and calculating its probable return-on-invest-ment. NASA has identified the need for a project cate-gorization scheme, but this was not reflected in the newrelease of their procedures and requirements on pro-gram and project management. Some of these sug-gested distinctions on how to differentiate amongprojects are based on a maturity hierarchy for productline management with five levels, or a distinction basedon investment and risk, which is based on a 3 by 3matrix [Buschmann, 2003]. While all of these princi-ples and practices have proven to be successful, they arenot all used agency-wide on all projects, and they donot correlate the classification with an appropriate man-agement approach.



7.2. Implications on Systems Engineeringand Systems Engineering Management

This paper presented a framework that used a contin-gency approach to analyze an appropriate fit betweenproject characteristics and project type to define amanagerial approach or style. While a contingencyapproach or fit to managing projects is not new, it hasnot been used to understand project failure and providethe practitioner with the tools to make an informeddecision on effectively managing a project. This paperaimed to show that an effective framework for systemengineering management using a contingency approachcould take the practitioner past heuristics and guide-lines, toward understanding the results and conse-quences of their actions to reduce managerial error.Using a contingency approach to study project successand failure need define if a project used good or badmanagement, but if it was the right management to thesituation, the task, and the environment. What workswell in one situation may be the wrong approach inanother. Since almost no project is done in isolation andmost organizations are involved in more than one pro-ject, organizations would benefit from developing theirown organization-specific frameworks and teach man-agers to adopt the right approach to the right project.Future investigations could seek additional variables ofsituation and management and explore richer and wideropportunities for analyzing the fit of styles.

One of the underlying contentions of this paper isthat a project is a system; therefore, a framework forsystems engineering management should take a sys-tems approach. This is a fundamental reason why theNCTP Framework was used and viewed as a well-de-veloped representation of a systems engineering man-agement framework. Despite the advanceddevelopments of the NCTP framework, it still does notanswer all of the questions in defining the managementof a system project or systems engineering managementand still leaves issues unanswered such as:

• How do you know when you have correctly clas-sified a project and how can this be verified tosome level of confidence?

• How do you determine the most effective andefficient cost and resources to a project classifi-cation?

• How can a correct or incorrect classificationquantitatively correlate to project risk?

• What is the consistency in using the frameworkamong different practitioners (e.g., project man-agers, program managers, partners, system engi-neers)?

• How do you address discrepancies in a classifica-tion, and who is held accountable?

• What is the significance or impact of an incorrectclassification on any single dimension?

• Do individual classifications have weighting fac-tors?

With these questions in mind, a systems engineeringmanagement framework should be organization-spe-cific (no project or organization is the same), maintainsimplicity for a universal use within an organization,and provide two fundamental functions, definition andarrangement. Definition is the determination of classesof entities that share characteristic attributes; and ar-rangement involves a systematic ordering of classes thatexpresses conceptual relationships within the overallstructure [Jacob, 1991]. The next step is the correlationof the framework to organizational practices and anappropriate managerial style.

7.3. Research Limitations and FutureDirections

Despite the analysis of this research, it was not withoutlimitations. While the sample set used in this investiga-tion was well defined, it was also this well-definedsample set that may limit the ability to correlate theresults to other NASA projects and systems engineeringmanagement. The sample set covered only unmannedspace projects performed under the constraints of FBC.NASA’s projects cover a multitude of technology de-velopments at ten NASA centers. Each of these centersbrings a unique culture and way of doing business thatcould reveal variations in the results of this research.This sample set did not represent all of the 144 possibleproject scenarios of the NCTP framework. This cancause limitations in generalizing the conclusions toother projects, not only in NASA but also across thediscipline of systems engineering. This research statedthat without a framework, the projects were unable toproperly identify a correct approach. These projectswere viewed in retrospect; therefore, the frameworkwas imposed upon the projects. While this allowed fora more extensive data collection pool, further researchwould have to validate the effectiveness of the NCTPframework. A retrospective view also can create somebias in the analysis as some interpretations of the inter-viewees may be based on popular opinion and notobjective analysis.

Finally, all four projects were classified as break-through products. Novelty is related to a product’suniqueness to the market, and is defined based on acomparison to the market history. NASA projects arerarely repeated and thus are almost always unique to the

224 SAUSER


market. It is believed that NASA projects that areunique to the NASA vision of exploring and extendinglife into the universe could fundamentally not be clas-sified as derivative. Therefore, it could be concludedthat most NASA projects are breakthrough. In thisstudy it was challenging for the interviewees to makethe distinction of a NASA product based on the currentdefinitions for product novelty, thus it may also bepossible that NASA projects require a third classifica-tion between platform and breakthrough, or a refineddefinition of Novelty.

8. CONCLUSIONS

The failure of some of these projects to perform somefundamental project and systems engineering manage-ment practices, as identified in this study, can be iden-tified in project failures across the discipline of systemsengineering management and do not enhance our un-derstanding of performing these types of project. Thevalue in this investigation is the significance in beingable to indicate a relationship between a framework orproject classification system to an appropriate manage-rial style with corollary contingencies. The analysisusing the represented systems engineering managementframework supported the premise that despite a projectstechnical failure, project failure is ultimately a manage-rial error. The failure of almost any system can be tracedback to the operator of the system (e.g., project man-ager, systems engineer) and rarely is independentlyrelated to the technology. Eliminating or replacing thetechnology does not solve the managerial problems andsometimes can even complicate them. Therefore, thefundamental value in a systems engineering manage-ment framework or any management framework isbeing able to correctly associate a correct managementstyle to a project classification, thus reducing the ulti-mate failure point (i.e., managerial error) and risk in asystem. In the academic literature and in practice, thiscontingency approach for project and systems engi-neering management is still unexplored.

9. ACRONYMS AND DEFINITIONS

APL: Applied Physics Laboratory—a division ofThe Johns Hopkins University—is a researchand development organization dedicated to solv-ing a wide range of complex problems.

FBC: Fetter, Better, Cheaper—a way of doing busi-ness within NASA in the 1990s, which was in-tended to decrease time and cost for missions andincrease the number of missions and scientificreturn.

Discovery Program—focused on launching smallermissions with fast development times, each for afraction of the cost of NASA’s larger missions.

MIB: Mishap Investigation Board—established byNASA after a mission failure to determine rootand probable causes.

Mission Assurance—defined by the safety and en-hancement of success of all NASA activitiesthrough the development, implementation, andoversight on a project.

NASA: National Aeronautics and Space Admini-stration.

NEAR: Near Earth Asteroid Rendezvous—devel-oped by the Johns Hopkins Applied PhysicsLaboratory—was the first spacecraft to orbit anasteroid.

PMI: Project Management Institute—the leadingproject management professional discipline andthe publisher of the Guide to the Project Manage-ment Body of Knowledge (PMBOK).

SRM: Solid Rocket Motor.Stardust—developed by the Jet Propulsion Labora-

tory, Stardust was dedicated solely to the explo-ration of a comet, and the first robotic missiondesigned to return extraterrestrial material fromoutside the orbit of the Moon.

TRL: Technology Readiness Level—a metric/meas-urement system that supports assessment of thematurity of a particular technology and the con-sistent comparison of maturity between differenttypes of technology.

REFERENCES

R. Amara, New directions for innovation, Futures 22(2)(1990), 142–152.

R.D. Archibald, Managing high-technology programs andprojects, Wiley, New York, 2003.

R.D. Archibald and V.I. Voropaev, Commonalities and differ-ences in project management around the world: A surveyof project categories and life cycle models, Proc 17thIPMA World Cong Project Management, Moscow, Rus-sia, 2003.

S. Buschmann, Improving the management of NASA’s in-vestment, Presentation made to the NASA/USRA Work-shop, Res Topics Prog Project Management, Columbia,MD, 2003.

CAIB, Columbia accident investigation board report, Na-tional Aeronautics and Space Administration, Washing-ton, DC, 2003, Vol. 1.

C.N. Calvano and P. John, Systems engineering in an age ofcomplexity, Syst Eng 7(1) (2004), 25–34.

L. Crawford, J.B. Hobbs, and J. R. Turner, Project categori-zation systems, Project Management Institute, NewtownSquare, PA, 2004.



N. Denzin, The research act, Prentice Hall, Englewood Cliffs,NJ, 1984.

R.D. Dimitroff, L. Schmidt, and T. D. Bond, Organizationalbehavior and disaster: A study of conflict at nasa, ProjectManagement J 36(2) (2005), 28–38.

A. Drejer, Frameworks for the management of technology:Towards a contingent approach, Technol Anal StrategicManagement 8(1) (1996), 9–20.

K.M. Eisenhardt, Building theories from case study research,Acad Management Rev 14(4) (1989), 532–550.

S. Floricel and R. Miller, Strategizing for anticipating risksand turbulence in large-scale engineering projects, Int JProject Management 19 (2001), 445–455.

H. Gatignon, M.L. Tushman, W. Smith, and P. Anderson, Astructural approach to assessing innovation: Constructdevelopment of innovation locus, type, and charac-teristics, Management Sci 48(9) (2002), 1103–1122.

B. Gillham, Case study research methods, Continuum, NewYork, 2000.

C.S. Griner and W.B. Keegan, Enhancing mission success—aframework for the future: A report by the NASA chiefengineer and the NASA integrated action team, NationalAeronautics and Space Administration, Washington, DC,2000.

R. Guthrie and C. Shayo, The Columbia disaster: Culture,communication & change, J Cases Inform Technol 7(3)(2005), 57–76.

K.L. Hansen and H. Rush, Hotspots in complex productsystems: Emerging issues in innovation management,Technovation 18(8/9) (1998), 555–561.

M. Hobday, H. Rush, and J. Tidd, Innovation in complexproducts and system, Res Policy 29(7–8) (2000), 793–804.

L. Ibrahim, J. Jarzombek, M. Ashford, R. Bate, P. Croll, M.Horn, L. LaBruyere, and C. Wells, Safety and securityextension for integrated capability maturity models, F. A.Administration (Editor), FAA, 2004, September, pp. 1–141.

INCOSE, Incose systems engineering handbook, Interna-tional Council on Systems Engineering, Seattle, WA,2004, Vol. 2a.

E.K. Jacob, Classification and categorization: Drawing theline, Proc 2nd ASIS SIG/CR Classification ResearchWorkshop Adv Classification Res, 1991, pp. 67–83.

R.A. Kerr, Scaling down planetary science, Science 264(1994), 1244–1246.

M.W. Lewis, M.A. Welsh, G.E. Dehler, and S.G. Green,Product development tensions: Exploring contrastingstyles in project management, Acad Management J 45(3)(2002), 546–564.

J.C. Mankins, Approaches to strategic research and technol-ogy (R&T) analysis and road mapping, Acta Astronautica51(1–9) (2002), 3–21.

R. Miller and D.R. Lessard, The strategic management oflarge engineering projects, MIT Press, Boston, 2000.

J.S. Newman, Failure-space: A systems engineering look at50 space system failures, Acta Astronautica 48(5–12)(2001), 517–527.

M. E. Paté-Cornell, Organizational aspects of engineeringsystem safety: The case of offshore platforms, Science 250(1990), 1210–1217.

C. Perrow, Normal accidents: Living with high-risk technolo-gies, Princeton University, Princeton, NJ, 1999.

PMI, Government extension to a guide to the project manage-ment body of knowledge (PMBOK® guide), Project Man-agement Institute, Newtown Square, PA, 2002.

PMI, Construction extension to a guide to the project man-agement body of knowledge (PMBOK® guide), ProjectManagement Institute, Newtown Square, PA, 2003a.

PMI, United States Department of Defense extension to: Aguide to the project management body of knowledge(PMBOK® guide), Project Management Institute, New-town Square, PA, 2003b.

PMI, Guide to the project management body of knowledge,Project Management Institute, Newtown Square, PA,2004.

J.T. Reason, Managing the risks of organizational accidents,Ashgate, Burlington, VT, 1997.

R.W. Scholz and O. Tietje, Embedded case study methods:Integrating quantitative and qualitative knowledge, Sage,Thousand Oaks, CA, 2002.

A.J. Shenhar, Project management style and the space shuttleprogram: A retrospective look, Project Management J23(1) (1992), 32–37.

A.J. Shenhar, Real life project analysis—guidelines, StevensInstitute of Technology, Hoboken, NJ, 1999.

A.J. Shenhar, One size does not fit all projects: Exploringclassical contingency domains, Management Sci(3) 47(2001), 394–414.

A.J. Shenhar, Strategic project leadership: Toward a strategicapproach to project management, R&D Management34(5) (2004), 569–578.

A.J. Shenhar and D. Dvir, “How projects differ, and what todo about it,” The Wiley guide to managing projects,P.W.G. Morris and J.K. Pinto (Editors), Wiley, Hoboken,NJ, 2004, pp. 1265–1286.

R. Stake, The art of case research, Sage, Newbury, CA, 1995.D. Vaughn, The challenger launch decision: Risky technol-

ogy, culture, and deviance at NASA, University of Chi-cago, Chicago, 1996.

C.R. Weisbin, G. Rodriguez, A. Elfes, and J. H. Smith, Towarda systematic approach for selection of nasa technologyportfolios, Syst Eng 7(4) (2004), 285–302.

R.K. Yin, Case study research: Design and methods, Sage,Thousand Oaks, CA, 1994.

226 SAUSER


Brian J. Sauser holds a B.S. from Texas A&M University in Agriculture Development with an emphasisin Horticulture Technology, an M.S. from Rutgers University in Bioresource Engineering, and a Ph.D.from Stevens Institute of Technology in Technology Management. He has worked in government, industry,and academia for more than 10 years as both a researcher/engineer and director of programs. He hasmanaged an applied research and development laboratory in life sciences and engineering at NASAJohnson Space Center, was Program Director of the New Jersey–NASA Specialized Center of Researchand Training, where he managed a multi-institutional, multi-disciplinary science and engineering researchcenter working to generate new knowledge and technology for life support systems, and was a ProjectSpecialist with ASRC Aerospace responsible for managing technology utilization and assessment, andcommercial partnership development at NASA Kennedy Space Center. He is currently a ResearchAssistant Professor at Stevens Institute of Technology in the Systems Engineering and EngineeringManagement (SEEM) Department and Director of the SEEM Systems Engineering Management Program.He is a member of the IEEE Engineering Management Society, International Council on SystemsEngineering (INCOSE), and Chairman of SE Development for the INCOSE Liberty Chapter.



Toward mission assurance: a framework for systems...

Documents

Transcript of Toward mission assurance: a framework for systems...