fl˝˙ˆˇ˘ ˙ ˆ ˙ˆ ˆ ˝˙ˆ · CONTENTS 05 Letter from the Editor(s) 07 Message from the...

THE MEASURABLE NEWS

2018.04The Quarterly Magazine of the College of Performance Management

mycpm.org

INSIDETHISISSUE

09 13 21 27On the Psychology of Human Misjudgment: Charlie Munger on Decision-Making

By Paul Bolinger with Steven Phillips

Artificial Neural Networks, Dovetailing Goals, and Rapid Skill Acquisition: Why Sharing Your Experience Helps Us All

By Nathan Eskue

Applying Earned Benefit Management The Value of BenefitsIf you can’t track it, you can’t manage it!

By Crispin “Kik” Piney

Increasing the Probability of Program Success with Continuous Risk Management

By Glen Alleman, Tom Coonce, and Rick Price

CONTENTS05 Letter from the Editor(s)

07 Message from the President

09 On the Psychology of Human Misjudgment: Charlie Munger on Decision-MakingBy Paul Bolinger with Steven Phillips

13 Artificial Neural Networks, Dovetailing Goals, and Rapid Skill Acquisition: Why Sharing Your Experience Helps Us All(Or–Three Things I Realized While Being Held in an Italian Police Station)By Nathan Eskue

21 Applying Earned Benefit Management The Value of BenefitsIf you can’t track it, you can’t manage it!By Crispin “Kik” Piney, PgMP, PfMP

27 Increasing the Probability of Program Success with Continuous Risk ManagementBy Glen Alleman, Tom Coonce, and Rick Price

48 Vendors/Services

2018 ISSUE 04

THE MEASURABLE NEWS

2018.04The Quarterly Magazine of the College of Performance Management

mycpm.org

THE COLLEGE OF PERFORMANCE MANAGEMENT

2018 BOARD & STAFFPRESIDENTWayne Abba 703-658-1815 • [email protected]

EXECUTIVE VICE PRESIDENTKym Henderson +61 414 428 537 • [email protected]

VICE PRESIDENT OF FINANCEBrian Evans 703-201-3278 • [email protected]

VICE PRESIDENT OF ADMINISTRATIONLauren Bone +44 (0) 7766 97 40 63 • [email protected]

VICE PRESIDENT OF CONFERENCE & EVENTSKathy Evans703-201-3278 • [email protected]

VICE PRESIDENT OF EDUCATION & CERTIFICATIONBill Mathis703-825-5588 • [email protected]

VICE PRESIDENT OF RESEARCH & STANDARDSMarty Doucette317-727-1237 • [email protected]

VICE PRESIDENT OF COMMUNICATIONSTimothy Fritz [email protected]

PAST PRESIDENTGary W. Troop 310-365-3876 • [email protected]

EXECUTIVE DIRECTORBeatrice Barry703-234-4147 • [email protected]

THE MEASURABLE NEWS IS AN OFFICIAL PUBLICATION OF THE COLLEGE OF PERFORMANCE MANAGEMENT

EDITORIAL STAFFPublisher: College of Performance ManagementStory Editors: Robin Pulverenti and Nicholas PisanoDesign: id365 Design + CommunicationsCommunications VP: Timothy Fritz

EDITORIAL COPY Editorial contributions, photos, and miscellaneous inquiries should be addressed and sent to the editor at the College of Performance Management (CPM) headquarters. Please follow the author guidelines posted on the CPM web site. Letters submitted to the editor will be considered for publication unless the writer requests otherwise. Letters are subject to editing for style, accuracy, space, and propriety. All letters must be signed, and initials will be used on request only if you include your name. CPM reserves the right to refuse publication of any letter for any reason. We welcome articles relevant to project management. The Measurable News does not pay for submissions. Articles published in The Measurable News remain the property of the authors.

ADVERTISINGAdvertising inquiries, submissions, and payments (check or money order made payable to the College of Performance Management) should be sent to CPM headquarters. Advertising rates are $1000 for inside front or back cover (full-page ad only), $800 for other full-page ads, $500 for half-page ads, and $300 for quarter-page ads. Issue sponsorships are available at $2,500 per issue. Business card ads are available for $100 per issue (or free with full-page ad). Rates are good from January 1, 2018 – December 31, 2018. College of Performance Management reserves the right to refuse publication of any ad for any reason.

SUBSCRIPTIONSAll College of Performance Management publications are produced as a benefit for College of Performance Management members. All change of address or membership inquiries should be directed to:

College of Performance Management 11130 Sunrise Valley Drive, Suite 350Reston, VA 20191 Ph 703-370-7885 • Fx 703-370-1757 www.mycpm.org

All articles and letters represent the view of the authors and not necessarily those of College of Performance Management. Advertising content does not signify endorsement by College of Performance Management. Please notify College of Performance Management for single copy or reproduction requests. Appropriate charges will apply.

© 2018 by the College of Performance Management. All rights reserved.

mailto:[email protected]

SOLUTIONS THAT BRING ALL THE PIECES TOGETHER

�t S�� So�ware we have combined the latest, most powerful business intelligence technology with the exper�se of the best minds in project management to deliver powerful Business Intelligence, Business �naly�cs, and Integrated �rogram Management solu�ons. These solu�ons are transforming the way both government and industry conducts business.

The fusion of COTS domain knowledge augmented by an open, powerful, business intelligence pla�orm will put your organiza-�on ahead of the power curve rela�ve to your compe�tors, maximizing your return on investment and allowing you to build your corporate knowledge and exper�se to meet your current and future needs. This COTS/BI approach enables you to address your immediate needs, realizing immediate ROI, while providing flexibility for new and emergent requirements.

Our �roteus and Envision solu�ons eliminate redundancy, transform, and op�mize data, regardless of size, or source, or for-mat. �e have moved organiza�ons away from needing to manage the confusion of di�ering terminologies and applica�on, to an organiza�on with one source of truth that provides mul�ple pathways of analysis.

S��’s fourth genera�on so�ware is agnos�c to underlying data, and provides an open system that puts your organiza�on and SMEs back in the driver’s seat in determining how and which analysis they view. Our low-code environment allows changes and new func�onality to be released in a ma�er of days or weeks, rather than the structured yearly or bi-yearly releases that require new applica�on installa�ons found in tradi�onal third genera�on so�ware.

B�� I�� P�� P�� S�� E�� S�� P�� I�� A�� S�� A�� C��

� �0�8 S�� SO�T��RE, ��C �� R�� R��. O�� T�� , ��

866-389-6750 sna-so�ware.com

THE MEASURABLE NEWS

The Quarterly Magazine of the College of Performance Management | mycpm.org

2018.04

05

This issue of Measurable News marks a transition in the life of our organization in many ways. This transition is measured not only in the change in CPM leadership, but also in our approach in adapting to the needs of our constituency in the Project and Program Management discipline.

This issue represents the last, fourth issue of the publication for this year. The importance of doing this is twofold. First, CPM has a contractual obligation to its advertisers and its membership to publish its periodical four times in a year. This issue ensures the organization meets its commitments. Second, without a voice in seeking new and important contributions from our membership, and the discipline as a whole that contribute to the literature, the relevancy of Measurable News and, by extension, CPM is called into question.

I mention this because in 2018 finding contributions to the literature turned out to be somewhat challenging. Our membership and the larger readership of this publication should consider this to be more than problematic, and indicative that the organization—and its media—need to change with the times.

This is why in the last issue we announced the transition of Measurable News into a twice-yearly professional journal with an academic and scholarly focus. This transition will begin in 2019 after the publication of this issue. Furthermore, the College will be launching a more frequently published

“Knowledge from the College, the Measurable Newsletter” that will be a more immediate forum for the discussion of the latest news and trends in our community, and which will leverage a number of digital formats in delivering this content.

The need for these changes reflect the challenges that we see among other professional organizations: the effects of social networking that undermine personal participation and face-to-face contact, larger economic trends in industry—driven by technological change—that emphasize integration of larger and more diverse elements of performance data, inherent changes in the composition and perspectives of the people that make up Project and Program Management Teams, years of shrinking travel and personnel development funding, and the multiplication of competing information and venues that confound the average member of our community, especially those professionals just entering this important discipline.

As CPM shifts its emphasis to Integrated Project and Program Management (IPPM) it no doubt, from an intellectual perspective, will entail an evolution in thinking. The Measurable News seeks to be at the forefront of this evolution.

What this means is that any previous emphasis on any particular subdiscipline of IPPM will need to rediscover its appropriate level among the other elements that make up the program and project management information ecosystem through the process of formulating the question of the

problem to solve, observational or quantitative research, hypothesis, measurement, and conclusion. While this sounds challenging, it is understood that we are dealing with complex social systems which defy testing against an analogue. Thus, at the end of the day, the process of challenging old assumptions and the debate that will ensue, will lead our community to find new ways of looking at things—and provide insight and value that would otherwise not been realized. The emphasis will be to foster new ideas and debate through civil discourse and empiricism. Our intent is to make it stimulating and valuable to anyone who works in the acquisition and program management community.

Organizations such as CPM are as vulnerable as they are essential. Once an ecosystem is lost it is very hard to regain it. The roots of this organization began as serving the needs of those engaged in the field of program and project management in the public interest. These needs include sharing expertise, teaching necessary skills, the prosaic critical skills of learning what works and what does not, exploring new and better ways of doing business in our area of expertise, government transparency and outreach, and finding fellowship among those with similar issues and challenges.

With the introduction of the Web, Big Data, and social networking, we are constantly bombarded with all types of information—some of it contradictory and almost all of it unfiltered. It has created the paradoxical condition of expanding one’s universe while atomizing our support systems. Management advice and raw data without a gatekeeper is equivalent to diet and self-help literature. We apply it at great risk given the people, resources, and importance of the work in which we are engaged. CPM is that necessary safe haven that encourages ideas, but also separates the wheat from the chaff. It is that necessary social support system that is sorely needed in our community.

At the very basic level, this is a human enterprise that constantly requires care and feeding. So, I encourage you to contribute new insights into what you have learned to either this publication or the newsletter, to attend the local chapter meetings, to speak at or just attend the workshops, and to reach out to others in your place of work to encourage them to join and participate. Only with a diverse membership can this organization remain both relevant and valuable to the program and project management community at large.

LETTER FROM THE EDITORNicholas Pisano

®

35th Annual International WorkshopMay 22 - 24, 2019 • The Westin Fort Lauderdale, FL954-467-1111 • www.westinftlauderdalebeach.com

• Update your skills with the latest Earned Value Management (EVM) trends, tools, and techniques

• Learn through training, practice symposia, as well as workshops

• Earn PDUs (for PMPs)

• Network with earned value professionals from around the world

For more information visit:

www.mycpm.org or www.evmworld.org

©2018 CPM. EVM World is a registered trademark of CPM. R.E.P. PMI, and the Registered Education Provider logo are registered marks of the Project Management Institute, Inc.

SAVETHE DATE

“Maximizing Program & Performance Management Through Collaboration”

EVM World 2019INTEGRATED PROGRAM PERFORMANCE MANAGEMENT (IPPM)

THE MEASURABLE NEWS


2018.04

07

This message marks the end of my three-year tenure as President of the CPM Board – a time of change in our profession and our association. Beginning January 1, 2019, Gordon Kranz will assume the role, joined by new board members Tim Fritz (VP, Communications) and Mike Bensussen (VP, Administration). Welcome aboard! I will continue on the board as Immediate Past President. We bid farewell with thanks for their dedicated service to outgoing Past President Gary Troop and VP Administration Lauren Bone. After Tim was elected, the board appointed him to finish Lisa Mathew’s term following her resignation due to a job change.

The outgoing board leaves CPM in good shape, poised to build on our successful education programs and find new ways to serve our members. Gordon brings a vision of integrated performance management founded on EVM principles and incorporating other program performance metrics – an initiative consistent with our Integrated Program Performance Management certification.

This year’s Integrated Program Management Workshop was held at George Washington University in Washington, DC. The venue was a departure from the typical hotel/conference center, offering direct contact with faculty and students who participated enthusiastically in our program. The university setting, however, posed logistical challenges and attendance limitations and was inconvenient for attendees whose hotels were not nearby. We’ll return to a commercial venue next year.

While not optimum for our major workshops, the university model is attractive on a smaller scale. Our chapters are located near universities and one – Tennessee Valley in Huntsville, Alabama – enjoys a working relationship with Defense Acquisition University. Our first international chapter, in Colombia, is based at the University of the Andes in Bogota. The Washington, DC chapter manages a track at the annual University of Maryland project management symposium and the emerging Southern California chapter is developing relationships with academic institutions in the San Diego area.

This approach, combining attractive campus meeting venues and student populations, is a promising way to increase our membership and to engage students. Chapters might, for example, provide lecturers for management and engineering disciplines, sponsor/facilitate research and intern programs, mentor students, and help faculty incorporate performance management subjects in their curricula. Chapters are also our “bench” for CPM leadership positions.

I have been privileged to serve twice as CPM president. Thank you for your confidence and for your continued support. Our member volunteers are the heart of CPM!

Wayne Abba

MESSAGE FROM THE PRESIDENTWayne Abba

Since 1978, Humphreys & Associates, Inc. has passionately advocated and promoted the integration of technical, schedule, and cost components to achieve the full benefit of using a performance measurement system to enhance management visibility and control. We have a long tradition of leadership in the industry and creativity seeking a balance of proven and innovative EVMS processes to apply to project management challenges.

H&A is unique in the common sense approach we bring to organizing, planning, and controlling projects. We use proven methods and techniques that have been honed over decades of hands-on experience to achieve the most appropriate, timely, and cost effective results. Our solutions are always compliant with the ANSI-748 Standard for Earned Value Management Systems. We are the “911” of EVMS because government and industry both call us when they absolutely and positively must resolve an issue with an EVMS.

Humphreys & Associates offers a complete range of EVMS consulting services for the entire project life cycle. From proposal preparation and management system gap analysis to mock compliance reviews or contractor self-surveillance and third party validations, H&A is the authority in EVMS.

Because of our unparalleled breadth of knowledge and hands-on experience, H&A is the industry recognized leader in EVMS trainingand hands-on mentoring for project managers and control account managers.

Interested in learning more?

Visit our website at www.humphreys-assoc.com or call us today at (714)685-1730

THE MEASURABLE NEWS


2018.04

09

ABSTRACT Earned value management (EVM) enables informed decision-making. It requires an understanding of the political agendas and psychology of key stakeholders. Charlie Munger, the long-term Vice Chairman of Berkshire Hathaway and partner of Warren Buffet, offers project managers a valuable list of conceptual errors to avoid in his “The Psychology of Human Misjudgment.” In this brief article, Humphrey & Associates, an EVM consulting and training firm, uses Munger’s list to highlight common pitfalls (and one useful trait) encountered on EVM implementations.

In its purest form, earned value management (EVM) enables informed decision-making, as practitioners develop time-phased plans to measure the progress of a project. Data allows professionals to locate the cause of delays or cost overruns. EVM, however, is much more than a science. It requires an understanding of the political agendas and psychology of the client.

Charlie Munger, the long-term Vice Chairman of Berkshire Hathaway and partner of Warren Buffet, offers project managers a valuable list of conceptual errors to avoid.1 His Poor Charlie’s Almanack: The Wit and Wisdom of Charles T. Munger includes a chapter examining what he calls “The Psychology of Human Misjudgment.”2 Munger describes a series of tendencies that highlight common pitfalls (and one useful trait) encountered by EVM practitioners.

1. Reward and Punishment Super-Response. People move toward what is incentivized; they seek the reward. According to Munger, the tendency to move toward a reward can become too strong, and it can distort the information. He reminds us to “fear professional advice when it is especially good for the advisor.” Your EVM decision-making process should include “clearing the minefield” early in the process to make sure you have the right incentives. For example, a long-term program is undermined by incentives that encourage a focus on short-term profits.

2. Deprival Super-Reaction. This is the tendency to feel more pain from a loss than to feel pleasure from an equal gain. How counterproductive is the tendency to avoid risk in carefully-considered decision-making? Without risk, there is no opportunity. We all fight an innate bias against taking a risk or accepting a loss. The idea can be seen in the commonly observed action of throwing good money after bad. Think about the situation where someone says, “I will not be denied, no matter the cost.” Even if the data shows a project is behind schedule or over-budget, corrective action is not taken. You probably do not want that kind of rigid thinking involved in your decision making.

Consider a situation where the wrong incentives and a refusal to accept defeat both align against one of the options being considered in the EVM process. Would that option stand a chance?

3. Liking/Loving. The foundational notion that people seek affection and approval is familiar to us all. This tendency can be a problem when it gets in the way of rational thinking. For example, will people on your team ignore the faults of people they like? How can we fairly evaluate our friends? What about when EVM practitioners favor the products, actions, or ideas of those analysts they like? People may not even consciously understand the underlying bias, so we need to find ways to circumvent this tendency when making a decision.

ON THE PSYCHOLOGY OF HUMAN MISJUDGMENT: CHARLIE MUNGER ON DECISION-MAKINGBy Paul Bolinger with Steven Phillips

1) Charlie T. Munger was born in Omaha, Nebraska, in 1924. After serving with the Army Air Corps in World War II, Munger earned a law degree at Harvard, then became a successful investor. He went into business with Warren Buffet in 1978. Besides his business success and personal friendship with Buffet, Munger became famous for his straight-forward advice to investors. On his connection to Buffet, see Catherine Clifford, CNBC interview, “Warren Buffett Remembers First Meeting. Charlie Munger: ‘We were sort of made for each other,’” February 26, 2018, https://www.cnbc.com/2018/02/26/berkshire-hathaways-warren-buffett-remembers-meeting-charlie-munger.html. 2) Charlie Munger, “The Psychology of Human Misjudgment,” in Charlie Munger, Poor Charlie’s Almanack: The Wit and Wisdom of Charles T. Munger (Walsworth Publishing, 2005). This essay was a revised version of a speech Munger gave at Harvard in 1995.

10 The Measurable News 2018.04 | mycpm.org

3) Wikipedia, “Ben Franklin effect,” last modified on November 23, 2018, https://en.wikipedia.org/wiki/Ben_Franklin_effect.

4. Disliking/Hating. This can motivate people to ignore or devalue the work, opinions, or ideas of those they dislike. This tendency can be strong enough to distort facts or judgments. Perhaps you have worked with someone about whom that you have strong negative feelings about. Can you remember that, and remain aware that you have that bias operating in a decision-making environment? We need to make a neutral playing field in order to get fair EVM analysis for decision-making.

5. Doubt-Avoidance. That is our tendency to fear doubt and our penchant for seeking certainty. This can be a powerful motivator to reach a quick decision, perhaps before all the

“homework” of gathering information has been accomplished. It is human nature to desire certainly but we can understand that hasty decisions may not be the best. Unfortunately, this tendency becomes stronger when we are under pressure to deliver an EVM status report, as managers are tempted to take liberties with the data in order to meet a reporting deadline. Our challenge is to not let this tendency dictate our decision-making process.

The Wisdom of Benjamin Franklin. Benjamin Franklin understood psychology! Very recently a representative from a company called us at Humphreys & Associates and asked for perspectives on the future of the performance management field. We are in the same industry and compete sometimes, so it seemed odd that this was happening. Along with the request for a favor came an offer; an offer to attend a session with others of their clients and associates on the future of the industry during which we would be given the chance to make even more valuable inputs. What was going on? I think it was application of the Ben Franklin effect. Simply stated in Franklin’s own words about a situation when he had to work with an opposition figure. “Having heard that he had in his library a certain very scarce and curious book, I wrote a note to him, expressing my desire of perusing that book, and requesting he would do me the favour of lending it to me for a few days. He sent it immediately, and I return'd it in about a week with another note, expressing strongly my sense of the favour. When we next met in the House, he spoke to me (which he had never done before), and with great civility; and he ever after manifested a readiness to serve me on all occasions, so that we became great friends, and our friendship continued to his death.” 3 In this story notice the use of the words “serve me.” Just asking for and receiving the smallest favor from the man changed the man’s attitude toward Franklin.

Asking a favor of someone with whom we are “on-the-outs” can signal that we consider them to have or to know something valuable. If they respond positively and give us something as slight as their time and attention, we stand a chance of enhancing their opinion of us; our effort to manipulate their opinion could work. Their opinion of us could be enhanced because they then cannot see us as someone simultaneously unworthy and worthy. Each party is forced to recognize the value of the other.

6. Reciprocation. The tendency to reciprocate favors (or slights) and can be a strong subconscious motivator. If the person being manipulated by Franklin, and he considered it manipulation, had thought, “There’s that wily Ben again trying to win me over by asking me for help,” the ploy would not have worked. It must be subconsciously received. This is why contractors cannot buy civil servants lunch or give them gifts. How does this play into EVM decision-making? You need to be aware of the personal dynamics of any situation and to watch out for the subtle ways that others might influence a decision. You need to be especially watchful if the person doing the maneuvering if you believe he or she may not have the best interests of the project at heart.

7. Excessive Self-Regard. We have all known someone who has a very highly-inflated opinion of himself or herself. This person can, and often does, overestimate his/her own knowledge, competence, and capability. These people choose to associate with people like themselves, and they definitely are partial to their own ideas and conclusions. They are dangerous in decision making because they may not be motivated by facts; they may be on that personal voyage known as the ego trip. EVM practitioners must understand the personalities of those on their project who might manipulate data in their favor.

8. Over-Optimism. This is the human tendency to err on the side of optimism. This is a form of selection bias. There are many opportunities for optimism to damage a project. Quickly committing to the impossible could mean eventual failure. An overly aggressive schedule or budget will gain praise in the short-term but will prove impossible to meet. The EVM decision-making process should force all stakeholders to consider independent evaluations of their work. Also, a decision-making process can be badly damaged by the influence of a stakeholder who combines over-optimism and excess self-regard. Using subject matter expert and analytics independent of the project can help bring reality into the process.

11The Measurable News 2018.04 | mycpm.org

9. Kantian Fairness. People tend to give and expect fair treatment. It is something innate and reinforced with something learned. An example would be the way people line up to wait and the way they expect others to obey the unwritten rules of waiting in line. Applying this tendency to decision-making, people expect the decision process to be fair. Being unaware of perceptions of unfairness can be dangerous to the decision-making process as disgruntled stakeholders may hesitate to contribute or even undermine your efforts.

10. Influence from Mere Association. Munger points out that if something good happened to us in the past, we will be influenced by things associated with that good event or outcome. Something associated with a known good thing is not weighed and measured by the same standards as something that is “stand-alone,” or unassociated with past experiences. But the good things we remember were not always key to past successes. If we assume or just “know” something about a previous EVM effort based on company lore, we stand on dangerous ground. We would need to investigate the cause and effect chain for the good outcome to be confident we are dealing with a positive associated influence. The best we can do is be skeptical and challenge things that are presented as associated with previous good outcomes. In God we trust; all others bring data!

11. Social Proof. You might know it by the name “groupthink” or “the herd instinct.” We need to understand that people tend to act and think as the others around them act and think. This tendency can become an issue in a team or group charged with deciding a crucial issue, such as variance reporting. Will individual members have the strength for independent thought and action or will many of them “go along to get along”? The instinct can be much worse if the situation is a high stress one. The herd must react quickly to stress. The movement of the first spooked members could lead to a stampede. Deciders must be coached to avoid being a blind follower and to value independent thought and action. If necessary, the process might have to require mixing the attendance or substituting more independent-minded people.

12. Contrast Mis-Reaction. This can be seen as the reaction toward what is perceived as the lesser of two evils. A good decision-making process does not make false contrasts—the dreaded false dichotomy. We have all been taught that it is good to compare and contrast choices; but that is only useful when enough truth is known about the choices. A manipulative manager might use this tendency for mis-reaction by explaining that there are two outcomes. The first is Outcome A, which is described by the manager in such terms as to leave no doubt that this is a terrible choice. The second one is Outcome B, which is also not good, but appears to be so much better than A. In this way, the unknowing members of the team see the obvious contrast and are drawn to the outcome favored by the manager. To them, the whole process appears above-board. Such an either-or approach precludes consideration of alternatives.

Based on these tendencies, is there a possibility that people in your decision-making processes are being manipulated? Have you ever been manipulated and known it? Have you ever been manipulated but didn’t know it? Of course, that question can only be answered by thinking back now.

13. Curiosity. Munger is not all negative. He points to one beneficial tendency of human nature, curiosity. It is our nature to be curious and that curiosity has brought humankind from the cave dwellers to the high-rise dwellers. If we can harness that tendency in our decision-making process, maybe it will help counter some of the negative and damaging tendencies. Maybe our curiosity will lead us to getting the facts before we decide emotionally.

Let’s postulate a situation where the authority to make a crucial decision is assigned, in a high stress situation, to a weak “groupthink team” with a strong leader who has a bloated opinion of himself. The team has been shown some very limited choices, some of which have been described as distasteful ones. The leader is in denial and trying to avoid loss. He or she wants to force a quick decision. How do you rate the chances of this scenario yielding a decision that will facilitate the long-term success of the project?

There is much more to learn from the thinking of Charlie Munger and others who deal with EVM decision-making in the context of human psychology. Humans are fraught with innate tendencies that, if not blocked, can lead us to make a misjudgment. In the Humphreys & Associates seminar on decision-making, the goal is to develop a “decision-making manifesto” that outlines how your company or department can establish a process that deals with the tendencies toward misjudgment and leads to better decisions.

ProjStream.com | 800.914.1710

Software that works together. Seamlessly.

“Cutting-edge technology in EVM capabilities, customizable software development options, resolution-based technical support and effective tool training. “ Bob Harbaugh, EVM Director, Southwest Research Institute

The absolute go-to company for effective program planning & control solutions for the most complex industry programs

Program Planning – Proposal, Startup, Surge, ReplansProgram Control System Setup & Operations Program Cost ManagementEfficient Earned Value Management (EVM) ComplianceProgram Control Tools & Technology

ClearPlan has built the strongest team of experts in the industryOur experts in program controls make an immediate impact to your teamWe bring ensured compliance when you need it mostTell the clear, winning story in your proposal

Contact Us:[email protected]

888.249.7990

Nationwide Team of ExpertsCost Effective Solutions

The absolute go-to company for effective program planning & control solutions for the most complex industry programs

Program Planning – Proposal, Startup, Surge, ReplansProgram Control System Setup & Operations Program Cost ManagementEfficient Earned Value Management (EVM) ComplianceProgram Control Tools & Technology

ClearPlan has built the strongest team of experts in the industryOur experts in program controls make an immediate impact to your teamWe bring ensured compliance when you need it mostTell the clear, winning story in your proposal

Contact Us:[email protected]

888.249.7990

Nationwide Team of ExpertsCost Effective Solutions

THE MEASURABLE NEWS


2018.04

13

ABSTRACT

ARTIFICIAL NEURAL NETWORKS, DOVETAILING GOALS, AND RAPID SKILL ACQUISITION: WHY SHARING YOUR EXPERIENCE HELPS US ALL(OR–THREE THINGS I REALIZED WHILE BEING HELD IN AN ITALIAN POLICE STATION)By Nathan Eskue

In this article, Nathan uses key research/examples spanning several centuries to show how sharing our experience can benefit everyone—including ourselves—more than keeping it private. He summarizes the three key skills that were shared with him. Each of which was completely life altering—catalysts for an adventure-filled and successful career, the obliteration of limits, and the deep desire to pass it forward.

I take in the scene around me, trying to absorb what I can with each of my senses. It smells of paper, dust, sweat (it’s July, and the forecast is hot, humid, with zero percent chance of effective AC), and if I’m not mistaken, latent fear. The metal chair I’m in feels warm and clammy; designed, it seems, to be uninviting. I hear the loud clacking of old keyboard keys, the wailing cry of a woman somewhere out in the hallway, and the sound of traffic out the window and two floors down. I taste dehydration—I am exhausted, dirty, and pale, but have not been offered anything to drink. I see a desk in front of me, utilitarian when it was at its best, now worn and disgruntled at being in use decades longer than intended; a calendar on the wall apparently sponsored by a tow truck company; and prominent in my view, the two police officers, trying to bridge the language barrier by typing into Google Translate in real time.

I’m in an Italian police station, am a (hopefully) temporary guest, and am trying to keep up with the conversation. I realize that I may be in trouble as one of the officers speaks very rough English after typing into the translator, clearly not understanding the words they pick off the screen. Unfortunately, Google isn’t at its best today, and what I think they are trying to say—indeed, what I hope they are trying to say, is “What crime was committed against you?” (Quale crimine è stato commesso contro di te?). What the officer says, however, is

“What crime was committed by you?” Commence sweating in 3, 2, 1... As I struggle to find a way to defuse this situation, I can’t help but wonder what led me to this moment in time and in particular, to the hot seat I am in. I realize to my surprise that it is--at least somewhat--EVM World 2018. Perhaps it is best if I back up just a bit…

Reflection 1: Sharing your experience will focus your research/experimentation, and can provide strong motivation to complete your milestones on schedule.

I’ve been speaking at conferences for the last four years, and have come to truly love sharing research and the resulting insights I’ve had; equally so, I’ve enjoyed hearing all the new ideas, thoughts or suggestions I’ve never considered, and a realization of shared workplace struggles by the other presenters. To me it feels somewhat like the Olympics might—different groups who may be allies, may be in conflict or in competition, set aside whatever else is happening and for a short time come together in a peaceful and productive way to show the fruits of their labors. In this case however, the goal isn’t friendly competition but rather to be on the same team, even if for a few days. There aren’t proprietary secrets shared, there aren’t nefarious questions being asked. Just a realization that we are all working to solve the same problem—namely, how do you build X for your customer in a way that optimizes scope, schedule, and cost? This equalizer of a problem allows shipbuilders and software designers to speak the same language, share meaningful insights, and build a network of colleagues that work to build each other up. I’ve dedicated my research to three key areas in order to make my own attempt at “solving” the ever-present program management puzzle: Rapid Skill Acquisition (RSA), Artificial Neural Networks (ANN),


and what I call the Dovetail Principle--my own delve into doing more with less. The interesting side effect in planning to share your work is that the conference deadline and its accountability generate a sense of urgency, a reason to not let this “not part of my day job” extra work slip out of the must-do list each day, and a burst of excitement at sharing ideas that may help someone else. In other words, what would very likely stay on the “someday” list actually gets done.

For instance, my research and experience with RSA has been largely driven by preparing results for both conferences and college courses. I live a charmed life in that I have two parallel careers I love: working in the aerospace industry helping to manage large, complex programs; and teaching undergrad/grad students at a niche school called the University of Advancing Technology, using courses I’ve built from scratch that focus on traditional areas such as business, robotics, and math, but also decidedly non-traditional areas. My favorite course subjects are how to think, how to rapid prototype, how to create actionable intelligence by analyzing very messy data, and with RSA—how to learn almost any skill in around 20 hours.

Experience Sharing 1: Rapid Skill Acquisition (RSA)

RSA has taken many forms under many names, but the best methodology I’ve found is that laid out in the book The First 20 Hours by Josh Kaufman. There are few times a single book has changed the direction of my life, and this was one of them. The premise is simply that even though the “10,000 hour” rule developed by Malcom Gladwell (one cannot become an expert in a cognitively complex field until they put in at least 10,000 hours) seems to hold, the difference between “expert” and “pretty good” makes all the difference. Kaufman proposes a hack of sorts: thanks to the exponential nature of the learning curve, with proper planning and focused practice, a person can go from zero to “pretty good” at most skills in about 20 hours. Some of the key pillars of this methodology are:

1. Choose a skill you are passionate about learning. When you add practicing a new skill to your schedule, life conspires against you—best to have a strong passion when the going gets tough.

2. Set a clear and objective performance goal. For example, instead of “I want to play the guitar”, use an objective goal such as “I want to play a full song with few mistakes in front of a group of friends”. This prevents many self-attempts to cheat, and also focuses the specific skills you need.

3. Break down the skill (e.g., playing the guitar) into micro-skills (e.g., tuning, strumming, single note strikes, basic chord positioning, etc.). This is itself a skill that takes time and practice to learn, but can shortcut learning considerably. If you break any skill into micro-skills, you probably already know how to do some of those micro-skills through other experience you already have. For example,

“sprinting” is a micro-skill that is part of many sports—you don’t need to re-learn it when you decide to play a new sport.

4. Learn just enough to identify your goal, identify the key micro-skills needed to meet the goal, remove barriers to practice, find a method for rapid feedback, and that’s it. Don’t let analysis paralysis take hold, and don’t fool yourself into thinking you can read your way into a skill.

5. Practice for 20 hours—around 45-60 continuous minutes a day. This allows repetition, but also the very important sleep cycle that locks in what we’ve learned each day.

And that’s basically the formula that can change your life. Incredibly simple and easy to remember; eye opening in that a great many micro-skills are transferrable (and the more skills you learn, the more micro-skills you possess to learn even MORE skills); and still, at the end of the day, hard work—but effective, rewarding work.

When applied to the industry of program management, it’s interesting that many key skills one could learn do not have an abundance of training materials available online; at least not many sources of free, effective, and easy to digest material. Instead, a key part of learning new project management related skills is to learn from each other; to share our successes and failures; and to suggest new and innovative ways to approach the ever-present problems that we as project managers face.

Back to the police station… The reason that my being here stems back to EVM World 2018 is because I was selected to present my research on RSA development. As a result, I had selected several skills I wanted to acquire during 2018 and documented the process in order to have specific and recent examples of my own RSA journey when I presented at


the conference. My family had planned a vacation to Italy, and I very much wanted to learn the basics of the language. Because of this, I became (I found out once we arrived) the designated tour guide and translator for the trip. And when it came time to tell the police officers why we were in their station, I was the natural, and—due to their not understanding English—the only choice.

So, when the officer asks me in broken English, reading from their screen, “What crime was committed by you?”, I panic—but only for a moment. Fortunately, I have learned enough Italian in the last several months to understand what is actually being said. I answer in Italian, causing both officers to raise their eyebrows as I suggest that Google dropped an extremely important “contro” in the translation, and I would like to stress that I am the victim and not the perpetrator.

They nod, smile in appreciation at my broken but understandable words, and what was moments ago an interrogation is now an interview. I explain that one of our two cars had been broken into, that all our possessions have been stolen, and that it all happened when we parked on a very busy street, in broad daylight, leaving the car for less than ten minutes. They nod, sympathetic, and because I am working hard to explain this in Italian (it’s stretching the RSA goal I had set of being able to have a basic Italian-only conversation), I am sure that they are giving me extra assistance. Our friends’ passports were stolen, and they helped walk us through the process of getting to the embassy to replace them. They searched for about 15 minutes to find the best rental car location. And they spent time making sure that we had what we needed to continue our journey. They didn’t have to do this; in fact, I got the impression that they didn’t normally do this. But their kindness and extra effort to be a helping hand and fellow human above all else went a long way to not only calming our frayed post break-in nerves, but to feeling the overwhelming need to pay it forward the rest of the trip, looking for ways we could help those around us.

Reflection 2: In an age where we can be endless consumers of free information, when the world often consists of judging the quality of information based on many strangers’ compiled ratings, there is still nothing close to replacing the value gained by conversing face-to-face.

The role of the Internet in our lives cannot be overstated. Having ready access to vast amounts of information can speed innovation, discovery, and improvement. However, one key trait the Internet does not have is the ability to quickly and easily discuss an idea in real time, without the restrictions of data input/output, with fellow practitioners and experts in the field. This reflection is thought to be responsible—at least in part—for the surge of innovations that helped usher the Renaissance, the Industrial Age, and even the American Revolution.

In fact, coffee shops of 17th and 18th century London were called “Penny Universities”, because for the price of a penny, one could enter a coffee shop and spend the day discussing news, politics, business, innovation, art—all the key elements of cross-pollination growth. (As a side note, this growth in the spurring of new thought is also because these shops sold caffeinated drinks vs. the often only other sterile form of hydration—beer.).

I am convinced that this type of idea sharing continues to spur new growth and ideas, especially in our industry. We are living in a world where information, education, and a platform to share ideas is more and more accessible and even expected to be free. But the best way to share ideas continues to be face-to-face, during a conference presentation, and perhaps just as effectively, during the lunches and breaks after the presentation, where thoughts are energetically shared and new innovations born. One such idea was born a few years back, when I was in a hotel lobby coffee shop with a few other conference attendees, sufficiently caffeinated and discussing how some people can accomplish so much with the same 24 hours as the rest of us. As we came up with more examples and shared our own successes and failures, I decided that this problem needed to be solved in a structured, repeatable way. I’ve been researching and experimenting consistently since that discussion, and what I’ve found has been fascinating. I shared my findings along with my work on RSA this May, and will continue to hone and simplify the process until it can be used by anyone to

“dovetail” their efforts and create a more effective result.


Experience Sharing 2: The Dovetail Principle

The Dovetail Principle can best be explained through the analogy of spinning plates. For those unfamiliar with the concept, I’ve included a brief history in the references section below.

Imagine your life is made up of a stage, those key activities in your life are each a tall stick and plate, and the performer exerts the effort you put into each of those activities. What happens? If you focus time and effort on a task, it is like spinning a plate. This is great because the momentum created will keep the plate upright for a bit. However, before long you’ll need to give it another spin. Also, each time you spin a given plate, you are choosing NOT to spin every other plate in your life. This results in a limited amount of activities you can focus on in life without one or all of them crashing due to a lack of attention, and if you try to set up one too many plates, the whole mess can end in disaster. Sound familiar?

But what if I were to stack the plates? What if I had a smaller number of sticks, but four or five plates stacked on each stick? It might take a little more effort to spin the stick, but the effect of my effort would be multiplied. Also, my “switching cost” of that non-value-added movement going from one stick to the next is minimized. I would clearly want to do this—but how do I relate this metaphor to my life? The answer is, you relate it in three steps.

1. First decide what plates you want to spin—i.e., what are your goals in life? Start with the big ones, and work your way to the small ones.

2. Next, decide what skills (And how do we learn new skills quickly? Through RSA, of course!) it will take to accomplish each goal. Make sure that part of your efforts each day are working on learning this list of skills (one at a time ideally), and look at each to see what transferable micro-skills you already have.

3. Third, understand that your life becomes a sort of optimization problem. Essentially, your day is made up of a large list of things that need to get done, not enough time to do them all, and hopefully a process in which to prioritize them. If you look at all your potential tasks to work, align each to see how many goals you can

“touch” by working the task, prioritize by the total importance of the touched goals, and factor in the length of time since the last time you’ve spun each goal’s plate (don’t want any of them crashing due to inattention). This task is actually simpler than it sounds, and with practice you find yourself cross-pollinating your life by squeezing as much value out of each task you perform by seeing how it might help you to meet as many of your goals as possible. The more goals you can effectively move forward by executing a single task, the more you are utilizing the Dovetail Principle.

A quick example of this is my choosing to learn Italian for this trip. When I considered taking on the skill, I was realistic in thinking I would likely give up before making true progress, as the fires of day-to-day life seem more critical than a far-off goal—especially a goal that is optional. To be sure I stuck with it—and I was fine with the struggle of learning itself, but knew I’d be tempted to allow other tasks to become priorities—I used the Dovetail Principle.

I examined how not just the skill itself, but learning the skill would impact my goals. I found a number of goals (my plates) that would be spun by my taking on this task: the skill would help wanting to retire abroad, wanting to learn about as many cultures as possible, show my children that the world is so much bigger than our slice of suburbia; the learning of the skill would serve as an example for my RSA course, my Thinking Strategies course, my presentation at EVM World 2018, several other conferences I would be presenting my RSA/Dovetail research, and an example to my children who I’ve been slowly easing into RSA. One spinning effort = 9-10 plates that benefit. By setting up the skill this way, I had no choice but to make it happen because too many important goals were depending on it, so it made its way onto my top 5 to-do list nearly every day. Dovetail Principle in action—aligning many goals with a given task eliminates the need for superhuman willpower.

Back once more to the police station. We left the building with uplifted spirits and renewed hope in mankind. Even after the local branch of the rental car company refused to exchange cars, and we were about 950 miles from our original location, we kept our spirits high (I can’t disclose which car rental company it was, but I can tell you their name might just rhyme with “irrational”…). In fact, I was able to take an inventory of transferrable skills and use them to construct what is arguably an excellent temporary window. Made of trash bags, cardboard, and packing tape (literally all we could find at several nearby shops), the window held up to another 1,500 miles of high speed driving, heavy rain, island living for a week, and more. I’m convinced that when we finally returned the car, the company had to replace the


entire door because they couldn’t remove our very securely installed window. I felt zero pity. If they had only worked with us, only looked at the bigger picture, I think we would have all been better off. And looking back at it now, I realize that some of the greatest strides we are witnessing today are in those industries willing to overlook at least a certain amount of near-term advantage. Instead, they share what they have, get inspiration from others, and create different hybrid innovations without the concern of hiding behind intellectual property and without the fear of overstepping those same legal bounds. In those industries the intellectual property still exists in many forms to ensure companies can stay competitive, but they are open to sharing key thoughts, ideas, research, innovation—knowing that the benefits gained will outweigh the cost and risk.

Reflection 3: Instead of closely guarding our progress from each other, a group that creates an open environment of idea sharing will spur new growth for each member, can spawn new ideas that would not have been developed by any member working alone, and can significantly increase the ability of every group member to better achieve their goals.

Benjamin Franklin was wholeheartedly against patents. Though I understand their utility, and certainly respect the intellectual property of my own company as well as others, I can also understand the problem that Franklin was aiming to solve. In his biography, he states that “As we enjoy great advantages from the inventions of others, we should be glad of an opportunity to serve others by any invention of ours; and this we should do freely and generously.”

Ben was no slouch when it came to inventions, as he was credited with a laundry list of breakthroughs from bifocals to street lighting to swim fins (to clarify, these were separate products—not a great idea to use them together). However, he was convinced that by sharing our ideas with others, we can create literal synergy—where the combined effect is greater than the sum of the parts. Indeed, he wrote about one of his inventions in this context.

He had taken the wood stove and made significant, demonstrable improvements to the design. His friends suggested he patent the design and become rich through licensing. Indeed, he could have certainly done this and would have been applauded for his accomplishments. However, he instead worked hard to convince through evidence that the design was superior; having done that, he made sure to document how to manufacture the stove so that as many homes as possible could benefit from the gained efficiency. More homes had to chop less wood to keep warm, which is great. In fact, I’m sure that in at least one case a life was saved as a result. However, what stemmed from his decision were the numerous innovations and improvements made from Franklin’s design, and done so with the assurance that there would not be the profit-killing legal repercussions.

We’ve actually seen this situation repeated when Tesla made enormous strides in battery technology. Instead of cornering the market, they released the rights of use to their direct competitors because they felt the end goal—a cleaner earth with cleaner cars—was more important than the obvious competitive advantage they had developed. In this modern example, Tesla actually did patent the design. However, they released a statement pledging to not pursue infringement but needed to patent in order to prevent a patent troll from suing after patenting the technology themselves (such is the nastier side of modern IP).

An area I’ve been researching and pushing into has made improvements in the last 10 years that defy the imagination. A big reason for this is the very open nature of its growth during that time: from the access to multiple types of free, open source software such as R and Python; to the key (and openly shared) math breakthroughs from 2008-2012 that make it work today; to the endless examples and applications freely shared to ensure that others are able to stand on the shoulders of giants. The area is artificial intelligence, and in particular, the study and use of an artificial neural network (ANN).

Experience Sharing 3: Artificial Neural Networks

The purpose of an ANN is to find relationships and causality within vast amounts of data. When your computer automatically tags photos with people’s names or descriptions, this is an ANN. When you talk to SIRI, you’re talking to an ANN. And when Amazon delivers a package that you don’t actually recall ordering, but can’t complete because it’s exactly what you needed (I swear this has happened to me), your own personal Santa is actually an ANN.

The artificial neural network is dichotomous because it is mind boggling both in its simplicity (the model for a “perceptron” was introduced in 1958, long before big data or even the


dream of a computer able to actually use one) and its complexity (it took 50 years of technological progress and was still nowhere close to being able to be effective, until a Toronto professor named Geoffrey Hinton finally cracked the code from the mathematical side of the problem). Indeed, the field is full of paradox-riddled elements; you can create an ANN in minutes with a few Google searches and some deft copying/pasting open code. However, training an ANN to be the super brain—one that will help you forecast, find connections, make suggestions, and tease out invisible patterns—takes time, insight, experience, and data science expertise.

An ANN mimics the process of raising a child so much it’s a little creepy. Conception is very easy, but training them over countless hours and across the successes and failures is never ending—and ending up with a productive, mature being is far from assured. I won’t give you my parenting success rate to date, but I can tell you I won’t be writing a parenting book titled “I Cracked the Code!”. ANN’s are the same, and like with your own children, to see it succeed can create joy like nothing else.

For example, let’s say we want to teach an ANN how to do something relatively simple, like recognize the front passenger door of a vehicle. If you were doing this with traditional programming, how would you go about it? For that matter, if you had to leave instructions for a child—a child who doesn’t know what a car is--how to identify that specific door, what would your process look like? Hmm. Maybe it’s not as simple as that… For either case, you would try to think of all the things that help you identify the front passenger door. Maybe that it’s at the front of the car on the right? But what is the front? And what does a vehicle look like? And what if it’s night time? And what if the cars are all different colors, and shapes, and sizes? And what if the photos I’m looking at were taken at different angles, and distances, and apertures, and some are color and some are black and white, and some are actually photos of billboards of cartoon cars, and...

You get the point. You simply can’t think of everything. To do so would probably take your entire life, and by that point you’d have to mention flying cars (I hope). So no, we can’t train an ANN that way, but we also can’t train a human that way. So maybe if we think about how we train a human, we can back into how an ANN works.

Say you have a stack of about 10,000 different photos of cars. You divide the stack into, say, 3,000 – 2,000 – 5,000.

Learning: For the first 3,000 photos, you have marked any that show the front passenger door of a vehicle and where the door is in the photo. The child looks through each, and with each photo their brain rewires the neurons working the problem just a tiny bit. In a sense, they become more biased on what they believe a front passenger door looks like with every new image. Prior assumptions they’ve developed are challenged—where they are wrong they are modified, where they are correct they become more rigid, and new situations provide new context.

Training: Now, the 2,000 photos may or may not have doors in them, and they are not marked. The child has to look at each, use the complex biases they’ve developed, and guess. An adult is watching, and when the child correctly identifies where the door is in the photo (if it’s there at all), the adult praises them. For photos the child gets wrong, the adult corrects the child by showing where the door in the photo is (or states that the photo has no door). The child makes their way through the photos, and even when they get one wrong the adult’s corrections are helping to improve the quality of their biases. They continue to improve, and if they are able to get a total error score of less than some pre-determined threshold, they can move onto the final stage.

Operating: Thanks to some statistical concepts such as the Law of Large Numbers (we might call it “practice makes perfect” when we talk to the child), unless the 5,000 remaining photos contain drastically different photos, the child should be able to identify any front passenger doors on the photos with a predictable error rate that is below the threshold. We might not know for sure if no one had ever marked the photos to see if the child was right, but there is a reasonable assumption that their talent will continue at that rate.

Guess what? The exact same process is how you train an ANN (at least the type of ANN trained to identify things) to tell us what photos have front passenger doors! This is a math-free oversimplification, but you set up an ANN with multiple nodes, and each takes in a certain element of the image. Depending on the bias of each node (which starts as a random number when an ANN is first born), it may contribute to the total weighted score of the neurons. It may do this several times more depending on how complex the ANN


is. Once done, the total weight (combined with another “triggering” type of neuron) will determine if the ANN guesses right or wrong. When learning, it is told and it will use a fancy math process called “back propagation” to figure out for each node what the bias error is. Depending on how drastic the learning curve (another variable created when an ANN is born), the biases are adjusted some amount. The process repeats to train the ANN, and when the ANN reaches an average error below some threshold, it is then “trained” is a similar way that more finely tunes its learning and bias. Finally, it can be given unmarked photos and guess at where the front passenger doors are with reliable accuracy. The accuracy should be checked from time to time to ensure the pictures haven’t changed significantly, but the ANN can now sort through massive amounts of photos and identify specific items (in this case, passenger car doors) without you ever having to describe what a

“car”, a “door”, or a “passenger” are.

The key things to know, at a top level, when diving into the world of ANN’s are:

1. 1. ANN’s need a lot of data to work well. Like, a LOT. Thankfully, we have access to more big data than ever before, and there are many examples and courses online that focus specifically on the data element—how to structure your data, where to find it, and (most helpful in my eyes) how to deal with data that is far from perfect.

2. 2. ANN’s need a lot of computing power. For smaller applications you can task a PC to get the job done, but if the ANN is more complex, additional hardware might be needed. Computers with multiple GPUs (graphics processing units) instead of the standard CPU are becoming more mainstream as this setup is ideal for the parallel processing required for the ANN to crunch its data.

3. 3. ANN’s are not all seeing, all knowing AI’s. If done right, an ANN can do one task very well—much better than a human. However, it takes an expert hand, time, and training to make that happen. The ANN isn’t a magic wand, and there are no shortcuts to it knowing what you need it to do. That said, a properly trained ANN can transform your business in a way that 1,000 analysts working a problem could not.

The hours spent in that Italian police station could not be called “fun.” However, I did appreciate the spurring of different reflections caused by it. No matter how you view the world, we live in an amazing time, in an ever more connected global community, and are enjoying collaboration and inspiration that has never been experienced at any point in history.

Looking ahead, I see that trend, like technology that enables it, continuing to increase at an exponential rate. What we accomplish in the next 5-10 years is the science fiction of today. And through all of that--and indeed, despite all of that—there will always be an absolutely essential need for experts in a given field to come together in person; to share ideas in a focused group setting; and face-to-face, to talk and brainstorm and ponder and inspire for the hours after and in between. This is where those new thoughts are introduced, where partnerships are made, and above all else—where innovation is born.

About the AuthorNathan is a “multipotentialite” (look it up), enjoying both of his careers immensely—as the Material Control Manager for Northrop Grumman Innovation Systems; and as a full-time professor at the University of Advancing Technology teaching business, strategy, thinking, and advanced robotic systems. Nathan has a Master’s of Engineering from Columbia University (data science), an MBA from the University of Arizona, and a BS from the University of Arizona in Management Information Systems, Operations Management, and Marketing.

Referenceshttps://first20hours.com/https://smallbusiness.com/history-etcetera/benjamin-franklin-never-sought-a-patent-or-copyright/Ellis, Aytoun. 1956. The Penny Universities; A History of the Coffee-houses. London: Decker & War-burg.https://www.npr.org/sections/thesalt/2013/04/24/178625554/how-coffee-influenced-the-course-of-historyhttps://en.wikipedia.org/wiki/English_coffeehouses_in_the_17th_and_18th_centurieshttps://www.juggle.org/the-art-of-plate-spinning/https://www.tesla.com/blog/all-our-patent-are-belong-youhttps://torontolife.com/tech/ai-superstars-google-facebook-apple-studied-guy/https://www.wired.com/story/googles-ai-wizard-unveils-a-new-twist-on-neural-networks/https://www.entrepreneur.com/article/283990

forProject Technology, Inc. 4020 N. MacArthur Blvd. Suite 122-304 Irving, Texas 75038 USA Phone: (214) 550-8156 Email: [email protected] Web: www.forproject.com

www.forproject.com

Tired of fighting with your current EVMS Software?

EVMS forProject™ is the easy-to-use, configurable EVMS solution that can help to simplify your EVMS process and reduce the effort required for effective cost/schedule performance measurement and reporting.

Designed to work closely with Microsoft Project, Project Server and Primavera P6, EVMS forProject™ simplifies the process of cost/schedule integration, freeing your users for more important analysis.

EVMS forProject™ is designed to be scalable from simple to complex projects and can be configured to match your unique standards and business rules. Flexible integration tools allow you to integrate with your ERP or financial systems, avoiding errors and reducing the effort required for cost integration.

If you have a formal EVMS requirement, the tool is designed to support the principles of EIA 748 and to generate your required EVMS reports, all in a modern, easy-to-use interface that will save your users time and effort.

Give us a call if you’d like to learn more about how a modern, flexible EVMS solution can help your organization to stop fighting and get more done.

Earned Value Management. Simplified.

website: www.CBTWorkshop.com phone:

Let The CBT Workshop Assist You!

Staff Augmentation

Tools Implementation and Training

Skilled EVMS Planners, Schedulers, Program Control Analysts, Estimators, and EVMS Consultants

Short or Long Term Assignments Contract-to-hire personnel

We have EVMS tools experts who can implement, train, integrate, and migrate EVMS data in the following tools:

Microsoft Project and Project Server Deltek MPM, wInsight, Cobra, Open Plan Primavera

Everyone is Looking for Skilled EVMS Resources…We Have Them

EVMS Services

Web Based EVMS Training

EVMS Gap Analysis System Description Design and Development Desktop Operating Procedures IBR Preparation and Dispositioning CAM Training and Mock Interviews

EVMS Fundamentals and EVMS Advanced training,

Call UsToday!

THE MEASURABLE NEWS


2018.04

21

ABSTRACT

APPLYING EARNED BENEFIT MANAGEMENT THE VALUE OF BENEFITSIF YOU CAN’T TRACK IT, YOU CAN’T MANAGE IT!By Crispin “Kik” Piney, PgMP, PfMP

Kik Piney continues his series of articles on Earned Benefit Management. In this, the second article in the series, he explores the issue of applying the concept to program and portfolio management using a case study to illustrate issues and solutions.

Introduction: Reminder on Benefits MapsIn the first of this series of articles [Piney, 2018a; Piney, 2018b], I presented the basic ideas around program and portfolio. These concepts were illustrated on a simple case study. This introduction provides a brief reminder of these ideas.

Benefits and Benefits MappingAs stated in the earlier article, whereas, for projects, you need to be able to specify precisely what you want to create, for programs as well as for portfolios, the objective is different. The question to be answered in this case is “how can I achieve a specific business or strategic benefit?” The approach for defining the solution is to create a benefits map. The output of this mapping exercise is a logical network that can be read in two directions.

The map illustrates how to make the benefits happen. Once the required benefits have been defined by the strategic sponsor, you need to determine all of the steps that are required in order to identify the component projects required in order to achieve the strategic objectives. The dependencies between these logical steps are quantified with respect to the size of the contribution of the source node to the required result. In conjunction with the forecast value of the strategic objectives, this link information allows the forecast contribution of every node in the benefits map to be evaluated. Comparing the calculated contribution of each component project with its estimated cost provides a measure of its business value: its forecast benefit-cost ratio.

The Case StudyThe business objective of the program in this example is to increase profits for an organization in the area of customer service. For the purpose of the case study, strategic analysis has shown that increased customer satisfaction with after-sales support enhances business results and has the potential for delivering a benefit of €300,000 per annum compared with the current level of business. The steps to achieving this benefit have then been developed from this required strategic outcome all the way across to identifying the projects required. Analysis of this solution indicates that it will also lead to an increase in operational costs amounting to 25% of the corresponding benefit, thereby reducing the net benefit to be achieved by the program. The benefits map for this program, including all of the financial numbers mentioned above is shown in Figure 1. One important point about this case study is that, although the overall figures show a healthy return on investment, one component project (B: Call Handling Tool) costs more to develop than in contributes to the final benefit. The first article, however, explained why its inclusion was justified.

The benefits map provides you with a static view of the forecast result of the completed program. However, the addition of the Earned Benefit concept to benefits mapping provides additional, essential information for tracking the performance of the program during implementation.

This series is by Crispin “Kik” Piney, author of the book Earned Benefit Program Management, Aligning, Realizing And Sustaining Strategy, published by crc press in 2018. Merging treatment of program management, benefits realization management and earned value management, kik’s book breaks important new ground in the program/project management field. In this series of articles, kik introduces some earned benefit management concepts in simple and practical terms.


Page 2 of 7

The business objective of the program in this example is to increase profits for an organization in the area of customer service. For the purpose of the case study, strategic analysis has shown that increased customer satisfaction with after-sales support enhances business results and has the potential for delivering a benefit of €300,000 per annum compared with the current level of business. The steps to achieving this benefit have then been developed from this required strategic outcome all the way across to identifying the projects required. Analysis of this solution indicates that it will also lead to an increase in operational costs amounting to 25% of the corresponding benefit, thereby reducing the net benefit to be achieved by the program. The benefits map for this program, including all of the financial numbers mentioned above is shown in Figure 1. One important point about this case study is that, although the overall figures show a healthy return on investment, one component project (B: Call Handling Tool) costs more to develop than in contributes to the final benefit. The first article, however, explained why its inclusion was justified. The benefits map provides you with a static view of the forecast result of the completed program. However, the addition of the Earned Benefit concept to benefits mapping provides additional, essential information for tracking the performance of the program during implementation.

Figure 1: Complete Benefits Map

The Earned Benefit Concept To understand the Earned Benefit concept, let us start from a known technique: Earned Value (Abba, 2000; Lucas, 2012). The Earned Value Method (EVM) is ideally suited for managing the performance of project delivery. The essence of a project is to create one or more deliverables as specified by the client – whether the client is internal or external to the delivery organization. So, the “value” earned by a project is the value to the delivery organization

Figure 1: Complete Benefits Map

The Earned Benefit ConceptTo understand the Earned Benefit concept, let us start from a known technique: Earned Value (Abba, 2000; Lucas, 2012).

The Earned Value Method (EVM) is ideally suited for managing the performance of project delivery. The essence of a project is to create one or more deliverables as specified by the client – whether the client is internal or external to the delivery organization. So, the “value” earned by a project is the value to the delivery organization which is equal to the price agreed with the customer – this explains the focus of the earned value method on “work performed” and on the corresponding allocated budgets. For programs, however, the

“value” to the business is the potential contribution of the work performed – normally across multiple component projects – to the program’s benefits.

As explained in the introduction, once the algorithm for calculating the contributions has been carried out across the entire benefits map, the contribution of each component project to the total forecast benefit is known. The sum of the contributions of all of the component projects is therefore equal to the total forecast benefit. If any component project is incomplete or missing, its contribution will obviously be reduced proportionally or not occur, and the total benefit of the program will be reduced accordingly. It is just a small conceptual step, now, to define the earned benefit of a component project and, by extension, the earned benefit of the program.

The convention in this case is that, for any component project, its benefit contribution is linearly proportional to its percentage complete as calculated using the standard EVM. The sum of these contributions gives the Earned Benefit of the program. There are some additional concepts to be added once this idea has been fully understood, but the first, major step is to assimilate the basic rules of the Earned Benefit Method (EBM). To make this idea clearer it will now be applied to the case study.

An Earned Benefit ExampleThe business benefits forecast and technical implementation estimates have been used in order to build the quantified benefits map as explained in the first article in this series and shown in Figure 1. The next step is to use the component projects’ schedule forecasts to evaluate the expect implementation performance. This additional data for the case study are as follows:

The program is planned to run from the start of January to the end of June and the component projects are planned accordingly as shown in the Gantt chart (Figure 2).


Page 3 of 7

which is equal to the price agreed with the customer – this explains the focus of the earned value method on “work performed” and on the corresponding allocated budgets. For programs, however, the “value” to the business is the potential contribution of the work performed – normally across multiple component projects – to the program’s benefits. As explained in the introduction, once the algorithm for calculating the contributions has been carried out across the entire benefits map, the contribution of each component project to the total forecast benefit is known. The sum of the contributions of all of the component projects is therefore equal to the total forecast benefit. If any component project is incomplete or missing, its contribution will obviously be reduced proportionally or not occur, and the total benefit of the program will be reduced accordingly. It is just a small conceptual step, now, to define the earned benefit of a component project and, by extension, the earned benefit of the program. The convention in this case is that, for any component project, its benefit contribution is linearly proportional to its percentage complete as calculated using the standard EVM. The sum of these contributions gives the Earned Benefit of the program. There are some additional concepts to be added once this idea has been fully understood, but the first, major step is to assimilate the basic rules of the Earned Benefit Method (EBM). To make this idea clearer it will now be applied to the case study. An Earned Benefit Example The business benefits forecast and technical implementation estimates have been used in order to build the quantified benefits map as explained in the first article in this series and shown in Figure 1. The next step is to use the component projects’ schedule forecasts to evaluate the expect implementation performance. This additional data for the case study are as follows: The program is planned to run from the start of January to the end of June and the component projects are planned accordingly as shown in the Gantt chart (Figure 2).

Figure 2: Gantt Chart for the Case Study

As shown in the Gantt chart, the component project schedules are as follows: A: Call Handling Service run from 1 June to end June B: Call Handling Tool runs from 1 January to end February C: Business Process Analysis runs from 1 March to end May

Let us now use this in a hypothetical implementation, tracking progress across the

Figure 2: Gantt Chart for the Case Study

As shown in the Gantt chart, the component project schedules are as follows: • A: Call Handling Service run from 1 June to end June• B: Call Handling Tool runs from 1 January to end February • C: Business Process Analysis runs from 1 March to end May

Let us now use this in a hypothetical implementation, tracking progress across the lifetime of the program. The planned and actual percentages for each of the component projects in this example are shown in Table 1:

Page 4 of 7

lifetime of the program. The planned and actual percentages for each of the component projects in this example are shown in Table 1:

Table 1: Schedule Planned and Actual Values

By applying EVM and EBM to this example for each monthly reporting period, we can generate the following table (Table 2) from which we can plot the corresponding curves (Figure 3).

Table 2: Earned Value and Earned Benefit Figures

Figure 3: Planned and Earned Value; Planned and Earned Benefit

From Table 2, you can see that the Earned Benefit figures lag behind the Earned Value ones during the first half of the program’s lifetime. Earned Benefit less than Earned Value indicates a negative return on investment (ROI) forecast (i.e., the added value is less

Total EB

Total PB

Total EV

Total PV

July June

May Apr Mar Feb

Jan $0.00

$50,000.00

$100,000.00

$150,000.00

$200,000.00

$250,000.00



Page 4 of 7







Total EB

Total PB

Total EV

Total PV

July June

May Apr Mar Feb

Jan $0.00

$50,000.00

$100,000.00

$150,000.00

$200,000.00

$250,000.00


Page 4 of 7







Total EB

Total PB

Total EV

Total PV

July June

May Apr Mar Feb

Jan $0.00

$50,000.00

$100,000.00

$150,000.00

$200,000.00

$250,000.00



From Table 2, you can see that the Earned Benefit figures lag behind the Earned Value ones during the first half of the program’s lifetime. Earned Benefit less than Earned Value indicates a negative return on investment (ROI) forecast (i.e., the added value is less than the cost) at that point in the program. Unless the reason for it is understood and accepted, this negative ROI could send the wrong message to management, and might result in premature cancellation of the project and loss of the investment to date. In this case study, Figure 3 shows that this loss could amount to the actual spend at the end of March corresponding to a planned value (PV) of $83,250 – over half of the total program budget. As explained earlier, the reason for the negative ROI during this stage is that component B, the Call Handling Tool, delivers on its own less of a benefit than it costs. However, due to synergy effects, by the time the other components start to come online, the total Earned Benefit far outweighs the Earned Value. In the example, the program’s ROI really takes off in the last two months of the program. Earned Benefit forecasting can provide this positive message to management at every status review in order to ensure continued executive support for the program. In order to provide a complete picture of performance to date, progress measurement indicators that are reported should address both Earned Value and Earned Benefit. Some of these indicators are mentioned next.

For the EBM, “percent achieved” is the progress indicator that is conceptually similar to “percent complete” in EVM, and is defined as (<Earned Benefit>/<Earned Benefit at Completion>). Once the schedules for the component projects are known, the Planned Benefit can be calculated based on the Planned Value for any date, in a similar manner to Earned Benefit. The contrast between the Planned Value and Planned Benefit progress indicators for the case study example is shown in Figure 4. For this example, the EVM percent complete figures are more optimistic than the EBM percent achieved ones throughout the program. This difference comes about because the component project with the largest ROI (i.e., A: the Call Handling Service) is the last one to reach completion and provide its contribution to the total benefit. This “late surge” indicates that the risk of failing to achieve the strategic objectives is considerable right up until the last moment. In this way, in addition to enabling business-focussed performance management, the EVM also provides valuable information for managing overall program risk and verifying the program’s compatibility with the organization’s level of risk appetite at any point during the execution of the program. In this case, as just mentioned, management needs to understand that the actual benefit of the program only occurs late on in the implementation phase, and that the financial exposure up until that point is higher than might be expected from looking at the scheduling and planned value forecasts.

Page 6 of 7

Figure 4: The Percent Complete and Percent Achieved Results

This example shows that the approach of tracking and reporting on the figures from both EVM and EBM can provide a complete image of the program status and a powerful means of ensuring the final success of the program. Key Messages The example underlines one important message that applies to all programs: Although EVM is an extremely useful technique for evaluating and forecasting progress towards creating or receiving the result of a project, it only measures the value with respect to the delivery organization or the purchasing function in the receiving organization. Put another way, EVM focusses on what you put into achieving a result. Business management, on the other hand, is interested in the benefit to be gained from the results of the work, so this needs to be measured using the EBM. The addition of EBM to EVM provides a more complete picture of performance across the lifetime of a program. Additional Numbers - Completing the Map The first two articles have explained how to create the benefits map, quantify the dependencies, calculate the contribution of each node to the required benefits, and compare the cost allocation of each component project to its contribution. The way in which these numbers can be used in the Earned Value and Earned Benefit methods for approving, tracking and forecasting the program benefit has been explained. Even with all of these numbers, the map is incomplete until we can fill in the costs that should be allocated to each of the intermediate nodes. So long as these intermediate allocation numbers are unavailable, the viability of the intermediate steps between the component projects and the planned benefits cannot be objectively assessed. However, once the means for calculating these numbers is available, the benefits mapping technique becomes considerably more powerful. For example:

• different scenarios based on the original benefits map can be reviewed and the solution strategy optimized accordingly;

Figure 4: The Percent Complete and Percent Achieved Results

This example shows that the approach of tracking and reporting on the figures from both EVM and EBM can provide a complete image of the program status and a powerful means of ensuring the final success of the program.

Key MessagesThe example underlines one important message that applies to all programs:Although EVM is an extremely useful technique for evaluating and forecasting progress towards creating or receiving the result of a project, it only measures the value with respect to the delivery organization or the purchasing function in the receiving organization. Put another way, EVM focusses on what you put into achieving a result. Business management, on the other hand, is interested in the benefit to be gained from the results of the work, so


this needs to be measured using the EBM. The addition of EBM to EVM provides a more complete picture of performance across the lifetime of a program.

Additional Numbers - Completing the MapThe first two articles have explained how to create the benefits map, quantify the dependencies, calculate the contribution of each node to the required benefits, and compare the cost allocation of each component project to its contribution. The way in which these numbers can be used in the Earned Value and Earned Benefit methods for approving, tracking and forecasting the program benefit has been explained.

Even with all of these numbers, the map is incomplete until we can fill in the costs that should be allocated to each of the intermediate nodes.

So long as these intermediate allocation numbers are unavailable, the viability of the intermediate steps between the component projects and the planned benefits cannot be objectively assessed. However, once the means for calculating these numbers is available, the benefits mapping technique becomes considerably more powerful. For example:

• different scenarios based on the original benefits map can be reviewed and the solution strategy optimized accordingly;

• later adjustments to the solution strategy can be catered for without invalidating all of the earlier analysis. This capability reduces rework and prevents people from “moving the goalposts” to hide performance issues;

• intermediate cost-benefit performance indicators can be defined, based on the achievement of planned capabilities or outcomes.

These ideas will be expanded in the forthcoming articles in this series.

References1. Abba, W. F. (2000). How earned value got to primetime: a short look back and glance ahead.

Paper presented at Project Management Institute Annual Seminars & Symposium, Houston, TX. Newtown Square, PA: Project Management Institute. https://www.pmi.org/learning/library/earned-value-college-performance-management-8909

2. Lukas, J. A. (2012). How to make earned value work on your project. Paper presented at PMI® Global Congress 2012—North America, Vancouver, British Columbia, Canada. Newtown Square, PA: Project Management Institute. https://www.pmi.org/learning/library/make-earned-value- work-project-6001

3. Piney, C. (2018a). Applying Earned Benefit; Introduction to a new Series. PMWorld Journal, February 2018. https://pmworldjournal.net/article/applying-earned-benefit/

4. Piney, C. (2018b). Benefits Maps You Can Count On. PMWorld Journal, March 2018. http://pmworldjournal.net/article/benefits-maps-can-count/

About the AuthorAfter many years managing international IT projects within large corporations, Crispin (“Kik”) Piney, B.Sc., PgMP is now a freelance project management consultant based in the South of France. At present, his main areas of focus are risk management, integrated Portfolio, Program and Project management, scope management and organizational maturity, as well as time and cost control. He has developed advanced training courses on these topics, which he delivers in English and in French to international audiences from various industries. In the consultancy area, he has developed and delivered a practical project management maturity analysis and action- planning consultancy package.

Kik has carried out work for PMI on the first Edition of the Organizational Project Management Maturity Model (OPM3™) as well as participating actively in fourth edition of the Guide to the Project Management Body of Knowledge and was also vice- chairman of the Translation Verification Committee for the Third Edition. He was a significant contributor to the second edition of both PMI’s Standard for Program Management as well as the Standard for Portfolio Management. In 2008, he was the first person in France to receive PMI’s PgMP® credential; he was also the first recipient in France of the PfMP® credential. He is co-author of PMI’s Practice Standard for Risk Management. He collaborates with David Hillson (the “Risk Doctor”) by translating his monthly risk briefings into French. He has presented at a number of recent PMI conferences and published formal papers.

Kik Piney is the author of the book Earned Benefit Program Management, Aligning, Realizing and Sustaining Strategy, published by CRC Press in 2018

26

Sidebar footnotes

The Measurable News 2018.04 | mycpm.org

MEMBERS GET MORE

BECOME A MEMBER NOW!

www.mycpm.org/membership

Voice - Be heard on the CPM blog, in the Measurable News Magazine and as a speaker at one of CPM’s annual events or webinars.

Discounts - Members get discounts on annual confer-ences, free webinars and free PEP Program testing as well as other events conducted by CPM partners.

Network - Join a local chapter for free and connect with colleagues in your area; join our LinkedIn and Twitter forums for information sharing.

Visibility - Stand out and get noticed as an active member of your community.

Credibility - Your reputation matters! Being a part of CPM allows you access to the most cutting edge and up-to-date information in the �eld.

Growth - Grow your knowledge, grow your career; access to the EVM library will greatly expand your resources; Members are allowed the PEP testing for free.

The College of Performance Management (CPM),an independent entity, is the premier organization for earned value

management (EVM) and project planning and controls.

As an international non-pro�t organization, CPM is dedicated to sustained improvement in project and performance management.

THE MEASURABLE NEWS


2018.04

27

ABSTRACT All risk comes from uncertainty. Uncertainty comes in two forms.

• Aleatory uncertainty – from the Latin alea (a single die in Latin) is the inherent variation associated with a physical system or the environment, which arises from an inherent randomness, natural stochasticity, environmental or structural variation across space and time in the properties or behavior of the system under study.

• Epistemic uncertainty – from the Greek επιστηµη (episteme), meaning knowledge of uncertainty due to a lack of knowledge of quantities or processes of the system or the environment. Epistemic uncertainty is represented by the ranges of values for parameters, a range of workable models, the level of model detail, multiple expert interpretations, and statistical confidence.

The paper describes how to identify, analyze, plan, track and control risk on complex system of systems. With this knowledge the reader will be able to assess the maturity of the existing risk management process and determine correct actions needed to increase the probability of program success. Examples of root cause analysis, framing assumptions, high-level sources of risk are provided to guide the reader in applying these to their own programs.

The rationalization for separate risk into epistemic and aleatory is shown and evidence provided that failing to make this separation decreases the probability of program success by undermeasuring the impacts of each risk category.

The paper provides actionable guidance for applying risk management to complex system of system, with inputs, processes, and expected outputs for a variety of risk categories.

INTRODUCTION TO CONCEPTSCost and schedule growth on complex development programs is created when unrealistic technical performance expectations, unrealistic cost and schedule estimates, inadequate risk assessments, unanticipated technical issues, and poorly performed and ineffective risk management handling processes, create project technical and programmatic shortfalls, shown in Figure 1 and documented in a wide range of sources. [22], [23], [30], [37], [53], [59], [61], [72], [73], [98], [104], [105], [106], [114], [120], [122]

Root causes of cost and schedule growth and technical short fall starts with not adequately specifying what Done looks like in Measures of Effectiveness (MOE) for the project’s outcomes, not quantifying uncertainties in these measures, and failing to manage resulting risks to the Measures of Performance (MOP) associated with these MOE’s during execution of the project. [1], [18], [34], [71], [79], [112]

Applying Continuous Risk Management (CRM) [2] provides the needed discipline to produce risk informed actionable information to decision makers to address these issues, by: [17], [34], [35], [106]

• Continuously managing risks produced by uncertainty has shown positive correlations between risk management and increasing the probability of project success. [24], [38], [34], [94]

• Determining which risks are important to deal with in a prioritized model, their interactions, statistical interdependencies, propagation, and evolution. [5], [37], [90], [102]

INCREASING THE PROBABILITY OF PROGRAM SUCCESS WITH CONTINUOUS RISK MANAGEMENTBy Glen Alleman, Tom Coonce, and Rick Price


1 ) In this paper we use the terms Buyer and Provider in a neutral manner. This removes the biases of the words Government and Industry with the hope that the principles presented here can be applied in a broad range of business

• Implementing strategies to handle risk with mitigation, correction, prevention, and avoidance, derived from Root Cause Analysis, to remove and prevent the conditions and actions creating the risks. [50], [57]

Continuous Risk Management increases the Probability of Program Success by: [11]• Preventing problems before they occur with pre-mortems to identify cause, take

corrective and preventive actions to remove the condition and activity of the root cause before the risk becomes an issue. [50], [66], [104]

• Improving product quality by focusing on project objectives by consciously looking for risks that affect Cost, Schedule, and Technical Performance throughout the project’s lifecycle. [24], [80], [83]

• Enabling better use of resources through early identification of problems and providing input to management decisions regarding resource allocation. [34], [82]

• Promoting teamwork by involving personnel at all levels of the project and focusing their attention on a shared vision of the mission to provide a mechanism for achieving the MOE’s and MOP’s as planned.

Uncertainties creating risk must be identified, corrected, or prevented to remove programmatic, technical cost, schedule, and performance shortfalls that impact the Probability of Program Success. [15], [20], [41] Identifying the cause and effect relationships between cost, schedule, and technical risks and the propagation of those risks and their impact on the Probability of Program Success is a Systems Engineering and Program Management process. [39], [107]

Probability of Program Success Reduced by Programmatic Uncertainties

Condition or Action (Cause) Undesirable Outcome (Effect)

Buyer1 fails to define what Done looks like in units of measure meaningful to the decision makers prior to starting and during the project work. [16]

System deliverables don’t provide needed capabilities, effectiveness and performance, arrive too late, and cost more than budgeted.

Different participants on the project impose conflicting demands on project structure and processes. [52]

Participates do what is in their own best interest versus the best interest of product, cost, and schedule.

Budget process forces funding decisions to be made in advance of project decisions. [22], [122]

Budget process encourages undue optimism creating project risk to cost, schedule, and technical performance.

Program managers’ short tenures, limitations in experience, and formal training, create short falls in management skills. [8], [56]

Program manager focused on minimizing risk to Keep the Program Sold, creating the opportunity for career advancement.

Schedule and Cost Margin not defined or related to schedule and cost risk. Schedule margin tasks not assigned to protect end-item deliverables [16], [35], [36], [65]

Lack of validated cost and schedule margin causes the project to under deliver technical, go over budget, and deliver late.

Technical Plan not aligned with Budget and Spend Plans, with missing Schedule Margin and risk buy down plan. [122]

Lack of integrated cost, schedule, technical progress gives wrong performance information of future performance.

Alternative Points of Integration not considered in Master Schedule. [93]

Missing Plan-B means Probability of Success is reduced.

Cost, Schedule, and Technical Uncertainties impact Probability of Program Success without handling plans. [54], [73]

Without a credible risk strategy, there is no credible plan for success.

Missing Resource Loaded Schedule to assure staff, facilities, and other resources available when needed. [84]

Resources are not available when needed, with no credible plan for success.

Propagation of Aleatory and Epistemic Uncertainty not considered in risk profile or handling plans. [5], [42], [75], [117]

Without understanding the dynamics of risk, there is no credible plan to Keep the Program Green. [28]

Table 1 - for programs, these Root Causes (Condition, Action, Outcome) have been shown to reduce the Probability of Program Success. Each are addressed with Corrective and Preventive actions.


2) The structure of this paper is modeled after Beyond Bullet Points, guided by the following framework• The Setting - the current

project not successfully delivering effectiveness and performance needed to achieve the capabilities user paid for.

• The Protagonist – Buyer and Provider Program Managers, supporting Engineering, Program Controls, and Finance and Business Operations staff.

• The Imbalance - Buyer and Provider minimize technical difficulties leading to overlooking principles, processes, and practices that reveal risk to cost, schedule, and technical performance, identified corrective and preventive actions needed to Keep the Program Green.

• The Balance - prior to each major decision milestone, all participants have a clear understanding of what Done looks like in units of measure meaningful to decision makers, informed with analysis of technical, schedule, and cost risks affecting probability of project success.

• The Solution - a Master Plan, with Measures of Effectiveness (MOE), Measures of Performance (MOP), Technical Performance Measures (TPM), and Key Performance Parameters (KPP) for each deliverable in support of Mission or Business Strategy.

RISK MANAGEMENT OVERVIEW

This section provides the framework for reducing risk of cost growth, schedule delay, and technical performance shortfalls on programs in the presence of uncertainty by providing risk informed decisions that increase the Probability of Program Success (PoPS). [11], [84], [85]

Research shows management of complex programs regularly misses opportunities to increase the probability of project success. Symptoms include: Overestimating technology readiness; Underfunding the procurement of systems; Underestimating potential problems; Inadequate risk management at project initiation; Use of unsubstantiated cost estimates; Unaddressed technical problems; Operating with reduced manpower and functional expertise in the project office; Overestimating cost savings for insertion of COTS hardware and software; and Operating in an environment that burs of the traditional roles of independent cost, schedule, technical, and contract performance assessment. [12], [24], [47], [49], [50], [54], [96], [103], [122]

Focusing on risks created by uncertainty (reducible and irreducible), their identification, and corrective and preventive actions needed to address these risks, increases the probability of project success. [103], [119]

Uncertainty is the consequence of all complex systems. It is the lack of needed knowledge of the state of the system in the present and in the future that creates risk. [76], [77] Adaptability, flexibility, and other systems “-illities” are devoid of meaning in a deterministic world. [102] In the non-deterministic (uncertain) world, risk management is a critical success factor for increasing the probability of project success. [99]

Risk Never Occurs Without a Cause, that Cause is Uncertainty - Risk is the Outcome of Uncertainty

A contributing factor to projects successfully delivering capabilities needed to accomplish their mission is determined by how well the integrated data and processes are used to track and manage technical and programmatic performance and reduce risks to that needed performance.

Each performance factor informs one or more of the Five Immutable Principles of Program Success and enables modeling of this probability in units of measure meaningful to the decision makers. [4], [29], [70], [96], [97]

1. What does Done look like in units of measure meaningful to the decision maker?2. What is the Plan and Schedule to reach Done with needed Capabilities, at needed

time, for needed cost?3. What time, money, and resources are needed to reach Done and in what period are

they needed?4. What impediments will be discovered on the way to Done and their corrective or

preventative actions?5. What are the units of measure needed to credibly describe how progress is being

made toward Done?

Risk, created by uncertainty, and its management, is the central paradigm in a range of fields, in engineering, medicine, life sciences, and economics. [100], [126] This paper presents the principles on risk, its impact on the probability of project success, and the preventative and corrective actions needed to increase that probability.2

Framing AssumptionsFraming Assumptions (FA) inform project leaders about key project assumptions, enable discussion of their validity, and establish a context for project assessments. FA is any supposition (explicit or implicit) central in shaping cost, schedule, or performance expectations of the project.

Delays in schedule, cost overruns, and shortfalls in technical capabilities are often traced to difficulties in managing technical risk, initial assumptions, or expectation that are difficult to fulfill, and funding instability. [99]

One purpose of this paper is to identify the explicit and implicit framing assumptions of risk to the cost, schedule, and technical performance that decrease the probability of project performance.


3) Schumpeter, J. A., Capitalism, Socialism and Democracy, Allen & Unwin, 1947, p. 84, which is the "process of industrial mutation that incessantly revolutionizes the economic structure from within, incessantly destroying the old one, incessantly creating a new one"

With this starting point, there are several framing assumptions that needed when applying the Five Immutable Principles. Different agencies will have different approaches to answering the questions posed by the Five Principles. The fundamental assumptions for how risk impacts the answers to each question remains the same. [4]

Immutable Principle Framing Assumption

What does Done look like in units of measure meaningful to the decision makers?

To define Done, there needs to be Measures of effectiveness, Measures of Performance, Technical Performance Measures, and Key Performance Parameters traceable to the Capabilities needed for mission success.

What is the Plan and Schedule to reach Done, at the needed time, for the needed cost, with the needed Capabilities?

Plans are strategies for the successful deliver of the needed capabilities.These plans come in a variety of forms, each described increasing maturity of deliverables,Integrated Master Plan [35]Product Roadmap [35], Narratives in long term agency planning guidance [31], [32]Concept of Operations [65]

What time, money, and resources are needed to reach Done?

Some form of a master schedule showing deliverables is needed, the dependencies between the deliverables and measures of performance, technical performance measures and quantifiable backup data for these deliverables to assess the physical percent complete.

What impediments will be encountered on the way to Done and what are their preventive and corrective actions?

All risk comes from uncertainty in two forms - epistemic (reducible) which can be bought down and aleatory (irreducible) which requires margin.

What units of measure describe progress to plan toward Done?

Physical percent complete must be in some unit meaningful to the decision maker for the technical performance measures, Measures of Performance, and Key Performance Parameters for the project deliverables.

Table 2 - Framing assumptions for the Five Immutable Principles of Program Success.

Seven Risks to success requiring preventive and corrective actions to handle impacts from each risk. [15]

Risk to Success Impacts on the Probability of Program Success

Novelty There is no way to accurately assess if the project will be successful without proper model of the past performance, risk adjusted for future performance, with explicit preventive and corrective actions.

Uncertainty Results from lack of data needed to make reliable predictions of performance requirements, cost or time schedules or other significant project elements.

Complexity Systems have become complex that the parties involved can barely comprehend them. Without some means of modeling and assessing this complexity, no individual person or organization has the range of skills needed to understand the risks and issues involved in the project.

Interdependence No project is an island. Dependencies multiply the number of factors outside the control of the project. Changes in these factors impact the risks to cost, schedule, and technical performance.

Resource Limitations

The common results are when capability, quantity, and provision for support are traded off against cost that result in unintended and unexpected consequences. Without adequate funding, budgetary pressures impact creates cost ceilings, creating tradeoffs between capabilities, quantities, and support.

Creative Destruction3

Firms form, pass through phases of growth and decline, absorb other corporations or are taken over or dismantled, creating disruption to the established knowledge base needed for project success.

Political Constraints

Politics promises more and admits to less, making political constraints arguably the deadliest risk. If adherence to estimates is prime measure of performance, project managers will be pressured to sacrifice real objectives in order to avoid condemnation. All cost and schedule estimates must be probabilistic, risk adjusted, and updated to address the emerging needs of the user community.


4) IDA, MITRE, RAND, NASA, NRO, Naval Postgraduate School, Air Force Institute of Technology, Aerospace Corporation, all have research on the root cause analyses of acquisition problems.

5) Among the many sources that need to be corrected or prevented are the perverse Incentives create unintended and undesirable results contrary to the interests of the incentive makers. [30] Other sources can be identified, with corrections or preventions, but these incentives are backed into the politics and culture of large complex programs in both government and commercial domains

SOURCE OF THE IMBALANCE Risks Identified are rarely realized, risks realized were rarely identified.All unidentified risks are accepted risks. [46]

Research reveals root causes of a project’s failure to deliver the needed capabilities, for needed cost, at needed time, has many sources.4 Primary sources of the imbalance include technical and programmatic complexity and weak implementation of formal risk management. These conditions have been routinely ignored or minimized while not questioning or defining the framing assumptions for cost, schedule, and performance expectations that create the risks. [6], [25], [48], [55], [60], [86] The corrections needed for this imbalance are not addressed in this paper.5 This paper does address the principles, processes, and practices for handling the consequences of the lack of risk management in the Imbalance and the impact on the Probability of Program Success.

FRAMEWORK FOR IMPROVING PROGRAM PERFORMANCEThe starting point for improving the probability of project success is to assure that the Buyer and Provider share an understanding of the measurable descriptions of what the systems must deliver to accomplish the mission.

Every successful system must define unambiguous, risk adjusted, Measures of Effectiveness (MoEs) informed by the Technical Performance Measures (TPM), including the evaluation of all assumptions. Both Buyer and Provider must understand these complexities and evaluate

“what could go wrong” when defining the system’s MOEs.

Once this evaluation is complete, the validated MOEs are included in the Master Plan. [113] During project execution, the Provider must continue to perform risk analysis when responding to the buyer’s RFP in the form of a Master Schedule. Traditional Risk Management is based on these core concepts with models of [28], [110]

• Impact of naturally occurring uncertainties - aleatory, their statistical processes, and margins needed to handle the resulting risk.

• Impact of the risk event modeled as a probabilistic process, corrective or preventative actions, and residual risk.

These probabilistic and statistical processes are assessed with qualitative and quantitative approaches, with an aggregated characteristic to prioritize risk. A combination of probability and impact defined in an aggregate manner informs the decision makers about impacts of risk. Many of these methods independently evaluate the characteristics of risks and focus on the analysis of individual risks. Risks are usually listed and ranked by one or more parameters. Many times, these methods do not consider later influence of risks and cannot represent the interrelation between the risks created by the reducible and irreducible uncertainties and how these risks propagate through the project to create other risks not specified defined in the Risk Register.

Before implementing a risk’s corrective or preventive actions to increase the probability of project success, it is critical to realize:

Uncertainty is about ambiguous knowledge of probabilistic events (Epistemic), variances of naturally varying processes (Aleatory), or emergent uncertainty (Ontological) - all creating risk to the probability of success of the project.Risk Management means Managing in the Presence of these Uncertainties. [33], [45], [69]Predictive indicators are needed to increase the Probability of Program Success [25], [28]

PROBLEM STATEMENT FOR INCREASING PROBABILITY OF SUCCESS IN THE PRESENCE OF RISK

How can uncertainties for the project be identified and classified, with the needed level of confidence, to enable appropriate corrective or preventative actions to improve the probability of successfully completing programs on time, on budget, with the needed mission Capabilities?

The four areas affecting the probability of success in Figure 1 are associated with unmitigated or unanticipated risk to cost, schedule, and technical performance.


Risk Impact on Probability of Program Success

Risk Impact on Probability of Program Success 7

Before implementing a risk’s corrective or preventive actions to increase the probability of project success, it is critical to realize:

Uncertainty is about ambiguous knowledge of probabilistic events (Epistemic), variances of naturally varying processes (Aleatory), or emergent uncertainty (Ontological) ‒ all creating risk to the probability of

success of the project. Risk Management means Managing in the Presence of these Uncertainties. [33], [45], [69]

Predictive indicators are needed to increase the Probability of Program Success [25], [28] Problem Statement for Increasing Probability of Success in the Presence of Risk How can uncertainties for the project be identified and classified, with the needed level of confidence, to

enable appropriate corrective or preventative actions to improve the probability of successfully completing programs on time, on budget, with the needed mission Capabilities?

The four areas affecting the probability of success in Figure 1 are associated with unmitigated or unanticipated risk to cost, schedule, and technical performance.

Figure 1 ‒ Four summary sources of project cost and schedule overruns and technical shortfalls are from a more

detailed assessment of the root cause of project cost, schedule, and performance shortfalls. [62] There are thousands of ways to fail … most have not been explored. [63]

What is Risk? Risk is the effect of uncertainty of objectives. Uncertainty is a state or condition that involves a deficiency of information and leads to inadequate or incomplete knowledge of understanding. In the context of risk

management, uncertainty exists whenever the knowledge or understanding of an event, consequence, or likelihood is inadequate or incomplete

‒ ISO 31000:2009, ISO 17666:2016, and ISO 11231:2010 Risk is Uncertainty that Matters [58]

Risk is the potential consequence of specific outcomes that affect the projects’ ability to meet cost, schedule, and/or technical objectives. Risk has three primary components: [62] ▪ Probability of the activity or event occurring or not occurring, described by a Probability

Distribution Function. ▪ Consequence or effect resulting from the activity or event occurring or not occurring,

described by a Probability Distribution Function.

Figure 1 - Four summary sources of project cost and schedule overruns and technical shortfalls are from a more detailed assessment of the root cause of project cost, schedule, and performance shortfalls. [62]

There are thousands of ways to fail … most have not been explored. [63]

WHAT IS RISK?

Risk is the effect of uncertainty of objectives. Uncertainty is a state or condition that involves a deficiency of information and leads to inadequate or incomplete knowledge of understanding. In the context of risk management, uncertainty exists whenever the knowledge or understanding of an event, consequence, or likelihood is inadequate or incomplete

– ISO 31000:2009, ISO 17666:2016, and ISO 11231:2010Risk is Uncertainty that Matters [58]

Risk is the potential consequence of specific outcomes that affect the projects’ ability to meet cost, schedule, and/or technical objectives. Risk has three primary components: [62]

• Probability of the activity or event occurring or not occurring, described by a Probability Distribution Function.

• Consequence or effect resulting from the activity or event occurring or not occurring, described by a Probability Distribution Function.

• Root Cause (condition and activity) of a future outcome, which when reduced or eliminated, will prevent occurrence, non-occurrence, or recurrence of the cause of the risk.

For the project manager, there are three risk categories that must be identified and handled:

• Technical - risks preventing end items from performing as intended or not meeting performance expectations described by Measures of Effectiveness, Measures of Performance, Technical Performance Measures, and Key Performance Parameters.

• Programmatic - risks that affecting cost and schedule performance within the control or influence of Program Management, through managerial actions of work activities in the Integrated Master Schedule. [36]

• Business - risks originating outside the project or not within the control or influence of the Program Manager.

Uncertainty comes from the lack information to describe a current state or to predict future states, preferred outcomes, or the actions needed to achieve them. [51], [94], [123] This uncertainty can originate from the naturally (randomly) occurring processes of the project (Aleatory Uncertainty). Or it can originate from the lack of knowledge about the future outcomes from the work on the project (Epistemic Uncertainty). [69], [108]

MEASURING PROGRAM SUCCESS

Without a programmatic architecture (cost, schedule, risk management and its impact on the technical architecture), risks and their corrections and preventions to deliver needed technical capabilities cannot be defined, assessed, or measured, reducing the overall probability of project success. [75], [82]


6) “Military Research and Development Policies,” Klein, B. H. Meckling, W. H., and Mesthene, E. G., The RAND Corporation, R-333-PR, December 4, 1958.

7) When a direct measure is available, and objective measure is possible. When predicting an outcome, a subjective measure is used. The likelihood of the possible outcomes can be expressed quantitatively in terms of probabilities to indicate the degree of uncertainty. These probabilities can be objective when the events they predict are the results of random processes.

8) Measures of Effectiveness (MOE), Measures of Performance (MOP), Technical Performance Measures (TPM), and Key Performance Parameters (KPP) are measures of project success.

Measuring progress to plan in the presence of uncertainty, requires probabilistic modeling of the uncertainties of cost, schedule, and technical performance in units of measure meaningful to the decision makers. This principle of project management has been around for decades. It’s time these are put back to work. [111]In the presence of uncertainty, projects have several important characteristics:

• The development of new, operational systems involves combining sub-systems and component technologies to produce a new systems capability. Early in the project two types of information are available - the design basis of the general class of systems and some knowledge about newer or improved technologies to be incorporated into the components or subsystems.

• The development status can be a measure of uncertainty and progress measured by change in this uncertainty. Physical progress to plan can measured by the reduction of uncertainty, when it is reducible, and by the assessment of remaining margin when the uncertainty is irreducible. This process has been described - long ago - as the sequential purchase of information regarding some object about which the state of knowledge is incomplete.6 This knowledge is periodically assessed to confirm risk is being bought down at the planned rate, and the margin is being consumed according to plan.

• The system’s variables defining the performance characteristics must indicate an increasingly mature capability to fulfill a prescribed mission, in the Concept of Operations and other documents, in units described in Measures of Progress to Plan section.

• Acquiring these measures requires a method of systematically analyzing the implications of this knowledge at the subsystem and component levels of the project.

• Since the development of is not a random process, but a directed engineering process, in the presence of uncertainty, subjective probability distributions are needed for measuring this technological uncertainty. When an objective measurement is possible, it will be used to inform the decision maker. When this measure is not directly available, a subjective measure must be made with a model containing a degree of uncertainty.7

MEASURES OF PROGRESS TO PLANTo assess the increasing or decreasing project probability of success for the characteristics described above, units of measure are needed to define this success.8

• Measures of Effectiveness (MoE) - are operational measures of a desired capability whose success is related to the achievement of the mission or operational objectives, evaluated in the operational environment, under a specific set of conditions.

• Measures of Performance (MoP) - are measures that characterize physical or functional attributes relating to the system operation, measured, or estimated under specific conditions.

• Technical Performance Measures (TPM) - are attributes that determine how well a system or system element is satisfying or expected to satisfy a technical requirement or goal.

• Key Performance Parameters (KPP) - represent the capabilities and characteristics so significant that failure to meet them can be cause for reevaluation, reassessing, or termination of the project.

Each of these measures provides visibility to Physical Percent Complete of the development or operational items in the project.

These measures of progress to plan have been developed and applied since the mid 1960’s. [1], [21], [91], [111]The challenge is for the Buyer to connect the Technical Plan, with its MOEs, MOPs, TPMs, and KPPs with the Cost and Schedule Plan for any complex project before issuing the RFP. [71], [96], [99], [101]

THE MOVING PARTS IMPACTING PROBABILITY OF PROGRAM SUCCESSTo increase the probability of project success, processes and data must be connected in a logical manner. Figure 2 describes this data and the connections. Each data element is needed for success, starting with the contents of the Concept of Operations (ConOps). Figure 3 shows the processes connecting this data.

These performance measurement elements are the raw materials providing visibility to the success of the project. The programmatic architecture is as important as the technical architecture of the product or services produced by the project. This programmatic architecture provides the basis of getting the product out the door on the needed date for the needed cost, with the needed Capabilities.




programmatic architecture provides the basis of getting the product out the door on the needed date for the needed cost, with the needed Capabilities.

Figure 2 – the data elements and their interactions needed to increase the probability of project success at with a

Programmatic Architecture for managing the project in the presence of reducible and irreducible uncertainty.

Framework for Technical, Cost, and Schedule Uncertainties Uncertainties have many sources and drivers. Some are model related, some are parameter related.

Parameter related uncertainties come in two categories ‒ aleatoric and epistemic. [71] How Is the Degree of Uncertainty Modeled for the Probability of Program Success?

To answer this question, uncertainties impacting the probability of project success need to be separated into two categories. If the uncertainty can be reduced with work effort, it is labeled Epistemic. If there is a random process creating the uncertainty, no possibility of reducing the uncertainty, it is labeled Aleatory.

Programs containing data and processes shown in Figure 2 are complex. Models that attempt to simulate the risks to project success are many times simplified mathematical representations of complex phenomena. [87] This risk assessment uses probabilistic models for future events, identifying needed corrections and preventions of the outcome a risk created by these uncertainties. This traditional approach to risk analysis and management constructs a Risk Register containing probabilistic events, where the probability of occurrence of these events are reduced with explicit work activities captured in the Risk Register as Handling tasks.

On complex projects there are also statistically random processes affecting work durations, technical attributes, labor and facilities, and cost. These naturally occurring deviations from a desired state, a planned duration, or some technical performance parameter are irreducible. It is useful to separate these two types of uncertainties in the risk model in order to be clear on which uncertainties have the potential of being reduced or not. [70] More importantly, reducible uncertainty, Epistemic, may introduce dependencies between probabilistic risk events, that may not be properly modeled if their character is not correctly identified. [1] Uncertainty can be modeled as knowable in principle, fundamentally unknowable (random), or a combination of the two. [110]

Figure 2 – the data elements and their interactions needed to increase the probability of project

success at with a Programmatic Architecture for managing the project in the presence of reducible and irreducible uncertainty.

FRAMEWORK FOR TECHNICAL, COST, AND SCHEDULE UNCERTAINTIES

Uncertainties have many sources and drivers. Some are model related, some are parameter related. Parameter related uncertainties come in two categories - aleatoric and epistemic. [71] How Is the Degree of Uncertainty Modeled for the Probability of Program Success?

To answer this question, uncertainties impacting the probability of project success need to be separated into two categories. If the uncertainty can be reduced with work effort, it is labeled Epistemic. If there is a random process creating the uncertainty, no possibility of reducing the uncertainty, it is labeled Aleatory.

Programs containing data and processes shown in Figure 2 are complex. Models that attempt to simulate the risks to project success are many times simplified mathematical representations of complex phenomena. [87] This risk assessment uses probabilistic models for future events, identifying needed corrections and preventions of the outcome a risk created by these uncertainties. This traditional approach to risk analysis and management constructs a Risk Register containing probabilistic events, where the probability of occurrence of these events are reduced with explicit work activities captured in the Risk Register as Handling tasks.

On complex projects there are also statistically random processes affecting work durations, technical attributes, labor and facilities, and cost. These naturally occurring deviations from a desired state, a planned duration, or some technical performance parameter are irreducible. It is useful to separate these two types of uncertainties in the risk model in order to be clear on which uncertainties have the potential of being reduced or not. [70] More importantly, reducible uncertainty, Epistemic, may introduce dependencies between probabilistic risk events, that may not be properly modeled if their character is not correctly identified. [1]

Uncertainty can be modeled as knowable in principle, fundamentally unknowable (random), or a combination of the two. [110] Uncertainty must be properly modeled to quantify risk between the inherent variability with sufficient data (Aleatory) and uncertainty from the lack of knowledge (Epistemic).

RISKS, THEIR SOURCES, AND THEIR HANDLING STRATEGIES

Risk identification during early design phases of complex systems is commonly implemented but often fails to identify events and circumstances that challenge project performance. Inefficiencies in cost and schedule estimates are usually held accountable for cost and schedule overruns, but the true root cause is often the realization of programmatic risks. A deeper understanding of frequent risk identification trends and biases pervasive during space system design and development is needed, for it would lead to improved execution of existing identification processes and methods. [95], [96], [97]


9) Aleatory is mentioned 55 times and Epistemic 120 in NASA/SP-2011-3421, Second Edition, December 2011, “Probabilistic Risk Assessment Procedure Guide for NASA Managers and Practitioners.” 10 times for Epistemic and 11 for Aleatory in NASA/SP-2010-576, “Risk-Informed Decision Making Handbook.” [84]

Risk management means building a model of the risk, the impact of the risk on the project, and a model for handling of the risk, since it is a risk, the corrective or preventive action has not occurred yet. [40], [67], [88], [108], [118]

Probabilistic Risk Assessment (PRA) is the basis of these models and provides the Probability of Program Success Probabilities result from uncertainty and are central to the analysis of the risk. Scenarios, model assumptions, with model parameters based on current knowledge of the behavior of the system under a given set of uncertainty conditions. [79]

The source of uncertainty must be identified, characterized, and the impact on project success modeled and understood, so decisions can be made about corrective and preventive actions needed to increase the Probability of Program Success.

Since risk is the outcome of Uncertainty, distinguishing between the types of uncertainty in the definition and management of risk on complex systems is useful when building risk assessment and management models. [53], [57], [67], [79], [124]

• Epistemic Uncertainty - from the Greek επιστηµη (episteme), meaning knowledge of uncertainty due to a lack of knowledge of quantities or processes of the system or the environment. Epistemic uncertainty is represented by the ranges of values for parameters, a range of workable models, the level of model detail, multiple expert interpretations, and statistical confidence. Epistemic uncertainty derives from a lack of knowledge about the appropriate value to use for a quantity that is assumed to have a fixed value in the context of a particular analysis. The accumulation of information and implementation of actions reduce epistemic uncertainty to eliminate or reduce the likelihood and/or impact of risk. This uncertainty is modeled as a subjective assessment of the probability of our knowledge and the probability of occurrence of an undesirable event.

Incomplete knowledge about some characteristics of the system or its environment are primary sources of Epistemic uncertainty.

• Aleatory Uncertainty - from the Latin alea (a single die in Latin) is the inherent variation associated with a physical system or the environment. Aleatory uncertainty arises from an inherent randomness, natural stochasticity, environmental or structural variation across space and time in the properties or behavior of the system under study. [3] The accumulation of more data or additional information cannot reduce aleatory uncertainty. This uncertainty is modeled as a stochastic process of an inherently random physical model. The projected impact of the risk produced by Aleatory uncertainty can be managed through cost, schedule, and/or technical margin.

Naturally occurring variations associated with the physical system are primary sources of Aleatory uncertainty.

There is a third uncertainty found on some projects that is not addressed by this White Paper, since this uncertainty is not correctable.

• Ontological Uncertainty - is attributable to the complete lack of knowledge of the states of a system. This is sometimes labeled an Unknowable Risk. Ontological uncertainty cannot be measured directly.

Ontological uncertainty creates risk from Inherent variations and incomplete information that is not knowable.

SEPARATING ALEATORY AND EPISTEMIC UNCERTAINTY FOR RISK MANAGEMENT

Knowing the percentage of reducible uncertainty versus irreducible uncertainty is needed to construct a credible risk model. Without the separation, knowing what uncertainty reducible and what uncertainty is irreducible inhibits the design of the corrective and preventive actions needed to increase the probability of project success.9

Separating uncertainties increases the clarity of risk communication, making it clear which type of uncertainty can be reduced and cannot be reduced. For irreducible risk, only margin can be used to protect the project from the uncertainty. [14], [69]


As uncertainty increases, the ability to precisely measure the uncertainty is reduced so a direct estimate of risk can no longer be assessed through a mathematical model. While a decision in the presence of uncertainty must still be made, deep uncertainty and poorly characterized risks lead to absence of data and risk models in the project’s business and technical domain. [74]

EPISTEMIC UNCERTAINTY CREATES REDUCIBLE RISKRisk created by Epistemic Uncertainty represents resolvable knowledge, with elements expressed as probabilistic uncertainty of a future value related to a loss in a future period of time. [7], [125] This lack of knowledge provides the opportunity to reduce uncertainty through direct corrective or preventive actions. [68]

Epistemic uncertainty, and the risk it creates, is modeled with the probability that the risk will occur, the time frame in which that probability is active, and the probability of an impact or consequence from the risk when it does occur, and the probability of the residual risk when the handing of that risk has been applied.

Epistemic uncertainty statements define and model these event-based risks:• If-Then - if we miss our next milestone then the project will fail to achieve its business

value during the next quarter.• Condition-Concern - our subcontractor has not provided enough information for us to

status the schedule, and our concern is the schedule is slipping and we do not know it.• Condition-Event-Consequence - our status shows there are some tasks behind schedule,

so we could miss our milestone, and the project will fail to achieve its business value in the next quarter.

For these types of risks, an explicit or an implicit risk handling plan is needed. The word handling is used with special purpose. “We Handle risks” in a variety of ways. Mitigation is one of those ways. To handle the risk, new effort (work) must be introduce into the schedule. We are buying down the risk, or we are retiring the risk by spending money and/or consuming time to reduce the probability of the risk occurring. Or we could be spending money and consuming time to reduce the impact of the risk when it does occur. In both cases actions are taken to address the risk.

Reducible Cost RiskReducible cost risk is associated with unidentified reducible Technical risks, changes in technical requirements and their propagation that impacts cost. [13] Understanding the uncertainty in cost estimates supports decision making for setting targets and contingencies, risk treatment planning, and the selection of options in the management of project costs. Before reducible cost risk can take place, the cost structure must be understood. Cost risk analysis goes beyond capturing the cost of WBS elements in the Basis of Estimate and the Cost Estimating Relationships. This involves:

• Development of quantitative modelling of integrated cost and schedule, incorporating the drivers of reducible uncertainty in quantities, rates and productivities, and the recording of these drivers in the Risk Register.

• Determining how cost and schedule uncertainty can be integrated in the analysis of the cost risk model

• Performing sensitivity analysis to provide understanding of the effects of reducible uncertainty and the allocation of contingency amounts across the project.

Reducible Schedule Risk

While there is significant variability, for every 10% in Schedule Growth there is a corresponding 12% Cost Growth. [16]

Schedule Risk Analysis (SRA) is an effective technique to connect the risk information of project activities to the baseline schedule, to provide sensitivity information of individual project activities to assess the potential impact of uncertainty on the final project duration and cost.

Schedule risk assessment is performed in 4 steps:1. Baseline Schedule - Construct a credible activity network compliant with GAO-16-

89G, “Schedule Assessment Guide: Best Practices for Project Schedule.”2. Define Reducible Uncertainties - for activity durations and cost distributions from

the Risk Register and assign these to work activities affected by the risk and/or the work activities assigned to reduce the risk.

3. Run Monte-Carlo simulations - for the schedule using the assigned Probability


10) Technical debt is a term used in the Agile community to describe eventual consequences of deferring complexity or implementing incomplete changes. As a development team progresses there may be additional coordinated work that must be addressed in other places in the schedule. When these changes do not get addressed within the planned time or get deferred for a later integration, the project accumulates a debt that must be paid at some point in the future.

11) The use of Design Structure Matrix provides visibility and modeling of these dependencies. Many models consider the dependencies as statistic or fixed at some value. But the dependencies are dynamic driven by nonstationary stochastic processes, that evolve as the project evolves.

Distribution Functions (PDFs), using the Min/Max values of the distribution, for each work activity in the IMS.

4. Interpret Simulation Results - using data produced by the Monte Carlo Simulation, including at least: [113]

• Criticality Index (CI): Measures the probability that an activity is on the critical path. [26], [27]• Significance Index (SI): Measures the relative importance of an activity. [115], [116]• Schedule Sensitivity Index (SSI): Measures the relative importance of an activity taking

the CI into account. [92], [115], [116]• Cruciality Index (CRI): Measures the correlation between the activity duration and the

total project duration. [115], [116]

Reducible Technical RiskTechnical risk is the impact on a project, system, or entire infrastructure when the outcomes from engineering development do not work as expected, do not provide the needed technical performance, or create higher than planned risk to the performance of the system. Failure to identify or properly manage this technical risk results in performance degradation, security breaches, system failures, increased maintenance time, and significant amount of technical debt10 and addition cost and time for end item deliver for the project.

Reducible Cost Estimating RiskReducible cost estimating risk is dependent on technical, schedule, and programmatic risks, which must be assessed to provide an accurate picture of project cost. Cost risk estimating assessment addresses the cost, schedule, and technical risks that impact the cost estimate. To quantify these cost impacts from the reducible risk, sources of risk need to be identified. This assessment is concerned with three sources of risk and ensure that the model calculating the cost also accounts for these risks: [47]

• The risk inherent in the cost estimating method. The Standard Error of the Estimate (SEE), confidence intervals, and prediction intervals.

• Risk inherent in technical and programmatic processes. The technology’s maturity, design and engineering, integration, manufacturing, schedule, and complexity. [121]

• The risk inherent in the correlation between WBS elements, which decides to what degree one WBS element’s change in cost is related to another and in which direction. WBS elements within the system have positive correlations with each other, and the cumulative effect of this positive correlation increases the range of the costs.11

Unidentified reducible Technical Risks are often associated with Reducible Cost and Schedule risk.

Aleatory Uncertainty Creates Irreducible RiskAleatory uncertainty and the risk it creates comes not from the lack of information, but from the naturally occurring processes of the system. For aleatory uncertainty, more information cannot be bought nor specific risk reduction actions taken to reduce the uncertainty and resulting risk. The objective of identifying and managing aleatory uncertainty to be preparing to handle the impacts when risk is realized.

The method for handling these impacts is to provide margin for this type of risk, including cost, schedule, and technical margin.

Using the NASA definition, Margin is the difference between the maximum possible value and the maximum expected Value and separate from Contingency. Contingency is the difference between the current best estimates and maximum expected estimate. For systems under development, the technical resources and the technical performance values carry both margin and contingency.

Schedule Margin should be used to cover the naturally occurring variances in how long it takes to do the work. Cost Margin is held to cover the naturally occurring variances in the price of something being consumed in the project. Technical margin is intended to cover the naturally occurring variation of technical products.

Aleatory uncertainty and the resulting risk are modeled with a Probability Distribution Function (PDF) that describes the possible values the process can take and the probability of each value. The PDF for the possible durations for the work in the project can be determined. Knowledge can be bought about the aleatory uncertainty through Reference Class Forecasting and past performance modeling. [19], [43], [124] This new information then allows us to update - adjust - our past performance on similar work will provide information about our future performance. But the underlying processes is still random, and


12) FAR 31.205-7 defines “contingency” as a probable future event or condition arising from presently known or unknown causes, the outcome of which is indeterminable at the present time. In “An Analysis of Management Reserve Budget on Defense Acquisition Contracts,” David Christensen and Carl Templin, 9th Conference on the Society of Cost Estimating and Analysis, June 13-16, 2000.

our new information simply created a new aleatory uncertainty PDF.

The first step in handling Irreducible Uncertainty is the creation of Margin. Schedule margin, Cost margin, Technical Margin, to protect the project from the risk of irreducible uncertainty. Margin is defined as the allowance in budget, programed schedule … to account for uncertainties and risks. [82]

Margin is quantified by:• Identifying WBS elements that contribute to margin.• Identifying uncertainty and risk that contributes to margin.

Irreducible Schedule RiskPrograms are over budget and behind schedule, to some extent because uncertainties are not accounted for in schedule estimates. Research and practice are now addressing this problem, often by using Monte Carlo methods to simulate the effect of variances in work package costs and durations on total cost and date of completion. However, many such project risk approaches ignore the significant impact of probabilistic correlation on work package cost and duration predictions. [9], [64]

Irreducible schedule risk is handled with Schedule Margin which is defined as the amount of added time needed to achieve a significant event with an acceptable probability of success. [5] Significant events are major contractual milestones or deliverables. [91]

With minimal or no margins in schedule, technical, or cost present to deal with unanticipated risks, successful acquisition is susceptible to cost growth and cost overruns. [102]

The Program Manager owns the schedule margin. It does not belong to the client nor can it be negotiated away by the business management team or the customer. This is the primary reason to CLEARLY identify the Schedule Margin in the Integrated Master Schedule. [36] It is there to protect the project deliverable(s). Schedule margin is not allocated to over-running tasks, rather is planned to protect the end item deliverables.

The schedule margin should protect the delivery date of major contract events or deliverables. This is done with a Task in the IMS that has no budget (BCWS). The duration of this Task is derived from Reference Classes or Monte Carlo Simulation of aleatory uncertainty that creates risk to the event or deliverable. [19], [89]

The IMS, with schedule margin to protect against the impact aleatory uncertainty, represents the most likely and realistic risk-based plan to deliver the needed capabilities of the project.12

Cost Contingency

Cost Contingency is a reserved fund held by the Buyer’s Program Manager, added to the base cost estimate to account for cost uncertainty. It is the estimated cost of the “known-unknowns” cost risk that impact the planned budget of the project. This contingency is not the same as Management Reserve, rather this Cost Contingency is not intended to absorb the impacts of scope changes, escalation of costs, and unforeseeable circumstances beyond management control. Contingency is funding that is expected to be spent and therefore is tightly controlled.

Contingency funds are for risks that have been identified in the project12

Irreducible cost risk is handled with Management Reserve and Cost Contingency are project cost elements related to project risks and are an integral part of the project’s cost estimate. Cost Contingency addresses the Ontological Uncertainties of the project. The Confidence Levels for the Management Reserve and Cost Contingency are based on the project’s risk assumptions, project complexity, project size, and project criticality. [33]

When estimating the cost of work, that resulting cost number is a random variable. Point estimates of cost have little value in the presence of uncertainty. The planned unit cost of a deliverable is rarely the actual cost of that item. Covering the variance in the cost of goods may or may not be appropriate for Management Reserve.

Assigning Cost Reserves needs knowledge of where in the Integrated Master Schedule these cost reserves are needed. The resulting IMS, with schedule margin, provides locations in the schedule where costs reserves aligned with the planned work and provides the ability to layer cost reserves across the baseline to determine the funding requirements for the project. This allows the project management to determine realistic target dates for project


13) Apologies to Mulder and Scully and the X-Files.

deliverables and the cost reserves - and schedule margin - needed to protect those delivery dates. [81]

Irreducible Technical Risk

The last 10 percent of a project’s technical performance development work generates one-third of the cost and two-thirds of the problems - Norman Augustine’s 15th Law. [10]

Margin is the difference between the maximum possible value and the maximum expected Value and separate that from contingency is the difference between the current best estimates and maximum expected estimate, then for the systems under development, the technical outcome and technical performance values both carry margin and contingency.

• Technical Margin and Contingency serve several purposes: – Describe the need for and use of resource margins and contingency in system

development. – Define and distinguish between margins and contingency. – Demonstrate that, historically, resource estimates grow as designs mature. – Provide a representative margin depletion table showing prudent resource

contingency as a function of project phase.

For any system, in any stage of its development, there is a maximum possible, maximum expected, and current best estimate for every technical outcome. The current best estimate of a technical performance changes as the development team improves the design and the understanding of that design matures.

For a system in development, most technical outcomes should carry both margin and contingency. Typical spacecraft outcomes include: mass, end-of-life power, average and peak data rate, propellant, and data storage. As designs mature, the estimate of any technical resource usually grows. This is true historically and, independent of exactly why, development programs must plan for it to occur. [78]

The goal of Technical Margin (unlike Cost and Schedule Margin) is to reduce the margins (for example Size Weight and Power) to as close to zero as possible, to maximize mission capabilities. The technical growth and its handling include:

• Expected technical growth - contingency accounts for expected growth – Recognize mass growth is historically inevitable. – As systems mature through their development life cycle, with better understanding of

the design emerging from conceptual to actual designs. – Requirements changes often increase resource use

• Unplanned technical growth - margins account for unexpected growth – Recognize space system development is challenging – Programs encounter “unknown unknowns” with the use of new technology that is

difficult to gauge, that develop into uncertainties in design and execution of the project, all the way to manufacturing variations.

ONTOLOGICAL UNCERTAINTYOn the scale of uncertainty - from random naturally occurring processes (aleatory) to the Lack of Knowledge (epistemic), Ontological uncertainty lies at the far end of this continuum and is a state of complete ignorance. [109] Not only are the uncertainties not known, the uncertainties may not be knowable. While the truth is out there,13 it cannot be accessed because it is simply not known where to look in the first instance. NASA calls it operating outside the experience base, where things are done for the first time and operate in a state of ignorance.

Management of uncertainty is the critical success factor of effective project management. Complex programs and the organizations that deliver them organizations can involve people with different genders, personality types, cultures, first languages, social concerns, and/or work experiences.

Such differences can lead to ontological uncertainty and semantic uncertainty. Ontological uncertainty involves different parties in the same interactions having different conceptualizations about what kinds of entities inhabit their world; what kinds of interactions these entities have; how the entities and their interactions change as a result of these interactions. Semantic uncertainty involves different participants in the same


interactions giving different meanings to the same term, phrase and/or actions. Ontological uncertainty and semantic uncertainty can lead to intractable misunderstandings between people. [44], [99]

When new technologies are introduced into these complex organizations, concepts, principles and techniques may be fundamentally unfamiliar and carry a higher degree of ontological uncertainty than more mature technologies. A subtler problem is these uncertainties are often implicit rather than explicit and become an invisible and unquestioned part of the system. When epistemic and ontological uncertainty represents our lack of knowledge, then reducing the risks produced by this uncertainty requires improvement in our knowledge of the system of interest or avoiding situations that increase these types of uncertainty. To reduce Epistemic and Ontological uncertainty there must be a reduction in the uncertainty of the model of system behavior (ontology) or in the model’s parameters (epistemology).

Data and Processes Needed to Reduce Risk and Increase the Probability of SuccessThe data and processes needed by the Buyer to increase the probability of project success are in Figure 3, starting with data in Figure 2.

Each input data item should be held in the Performance Measurement Baseline (PMB) and supporting documents. Each process should be defined in some form of Command Media or Earned Value Management Systems Description. Each outcome should be used to inform the decision makers on the increasing probability of project success.

For each of the data items and processes that consumes or produces this data, there is risk. Reducible risk and Irreducible risk. Table 3 describes the source of the uncertainty and the Aleatory and or Epistemic Uncertainty associated with the data or process. Table 5 describes the outputs from the processes using the Input data. Each of these outputs has uncertainty as well. Reducible and Irreducible uncertainty which creates risk that must be handled.

Each Input, Process, and Output needs to be assessed for uncertainties, if any, and their impact on the probability of project success. If there is impact, a handling plan or margin needs to be in place to reduce the uncertainty to an acceptable level.



Figure 3 ‒ The data and processes needed from the Buyer to increase the probability of project success. The

connections between the data and the process is representative of the typical project in the space and defense business domain over the lifecycle of a project. These connections and data are the basis of the Business Rhythm.

Each data and process item in Figure 3 has risk created by the Aleatory and Epistemic uncertainties shown Table 3, Table 4, and Table 5.

Table 3 – The risks associated with the Inputs needed to increase the Probability of Program Success. Input Data Needed to Increase Probability of Program Success

Inputs Aleatory Risk Epistemic Risk Needed Customer Capabilities and Requirements

Aleatory uncertainties derived from Reference Classes and other models applied to Capabilities.

Epistemic uncertainties derived from system engineering model of needed Capabilities

Time Phased Available Budget

Variance in spend plan and variances in efficacy of that spend to produce value must have margin to assure sufficient budget is allocated to produce the planned product value for the planned cost.

Probabilistic risks require planned work effort to reduce the uncertainty and buy down the risk at the planned point in the schedule.

Desired Completion Date with Interim Measures of Maturity

Confidence intervals around the delivery dates in the IMS and other planning documents must be produced with modeling processes using both reducible and irreducible uncertainties.

Figure 3 - The data and processes needed from the Buyer to increase the probability of project success. The connections between the data and the process is representative of the typical project in the space

and defense business domain over the lifecycle of a project. These connections and data are the basis of the Business Rhythm.


Each data and process item in Figure 3 has risk created by the Aleatory and Epistemic uncertainties shown Table 3, Table 4, and Table 5.

Input Data Needed to Increase Probability of Program Success

Inputs Aleatory Risk Epistemic Risk

Needed Customer Capabilities and Requirements

Aleatory uncertainties derived from Reference Classes and other models applied to Capabilities.

Epistemic uncertainties derived from system engineering model of needed Capabilities

Time Phased Available Budget Variance in spend plan and variances in efficacy of that spend to produce value must have margin to assure sufficient budget is allocated to produce the planned product value for the planned cost.

Probabilistic risks require planned work effort to reduce the uncertainty and buy down the risk at the planned point in the schedule.

Desired Completion Date with Interim Measures of Maturity

Confidence intervals around the delivery dates in the IMS and other planning documents must be produced with modeling processes using both reducible and irreducible uncertainties.

Available technologies and their maturity assessment

Unaddressed Variances in the maturity of the technologies.

Unmitigated risks in the technology.

Time phased skills mix and their availability

Variances in the availability of the skills and the efficacy of those skills to produce the needed value.

Unaddressed efficacy of staff.

Historical Uncertainties for each Reference Class of Work

Past performance of efficacy of labor absorption.

Past technical performance of products, services, components, or systems.

Table 3 – The risks associated with the Inputs needed to increase the Probability of Program Success.

Processes Needed to Increase Probability of Program Success

Source Aleatory Risk Epistemic Risk

Define Done in Units of Measure Meaningful to the Decision Makers.

Naturally occurring variances in each of the measures - MOE, MOP, TPM, KPP.

Event based uncertainties in each of the measures - MOE, MOP, TPM, KPP.

Develop the Plan and Schedule to reach Done as planned.

For both reducible and irreducible uncertainties, the Plan and Schedule must include the margin and risk reduction work activities. These processes must be captured in the Risk Register, but work needed to deal with them must be shown the IMP/IMS.

Define the Resources needed to reach Done as planned.

For both reducible and irreducible uncertainties, the resource management processes must assure the margin and risk reduction work activities are properly staffed. These processes must be captured in the Risk Register, but work needed to deal with them must be shown the IMP/IMS.

Adjust Plan for reducible and Irreducible risks.

With the identified reducible and irreducible uncertainties, vertical and horizontal traceability must be visible in the IMP/IMS.

Finalize the Risk adjusted measures of progress toward Done.

For each identified uncertainty and resulting risk, assess impact on the MOE, MOP, TPM, and KPP in the IMP/IMS and provide reduction plan or margin to assure risk is reduced as needed to increase the probability of project success.

Table 4 - The risks associated with the Processes needed to increase the Probability of Program Success.


Outputs Data Needed to Increase Probability of Program Success

Output Aleatory Risk Epistemic Risk

Work Breakdown Structure

Identify aleatory risks by WBS element. Identify Epistemic risk by WBS element.

Integrated Master Plan

Identify aleatory risks by IMP element. Identify Epistemic risk by IMP element.

Technical Performance Plan with TPM’s

Define Technical Performance Plan assurance in the presence of aleatory uncertainties.

Define epistemic risk buy down plan for each Technical Performance Measure.

Initial Integrated Master Schedule

The IMS at ATP and possibly IBR will not have sufficient reference class data to a credible assessment of Aleatory uncertainty.

Event based uncertainty are likely in the future, so credible modeling of Epistemic uncertainty requires more maturity.

Adjusted Integrated Master Schedule

As the project proceeds the fidelity of the Reference Class Forecasting data improves.

As knowledge is acquired reduction in the Epistemic takes place - this is the Cone of Uncertainty paradigm.

Time Phased Staffing Plan

Natural variances in the productivity of the staff creates uncertainty and resulting risk.

Event based uncertainty, turnover, reassignment, and other disruption impact productivity.

Schedule Reserve Naturally occurring variances in the work productivity creates risk to completing work as planned

Event based uncertainties associated with the work processes.

Management Reserve

Management Reserve early, in the project is usually based on models built during the proposal. As the project proceeds better models of aleatory uncertainty emerged.

Management Reserve models of epistemic uncertainty mature as the project proceeds.

Risk Register with Mitigation Plans

The Risk Register must contain the range of uncertainties for work in the IMS.

Risk Register must contain the event-based risks, their mitigations, and residual risks contained in the IMS.

Risk Burn Down Plan with Quantified Measures of Risk

Reduction in the aleatory uncertainties modeled as the Cone of Uncertainty, showing the planned risk reduction as the project proceeds.

Measurement of the risk burn down is subject to epistemic uncertainty in the measurement process. As well subject to calibration of the risk ranking in the burn down process.

Table 5 - The risks associated with the outputs of the processes needed to increase the Probability of Program Success

Bibliography

Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information upon it. When we enquire into any subject, the first thing we must do is to know what books have treated of it. This leads us to look at catalogues, and at the backs of books in libraries.

- Samuel Johnson, 1709 - 1784

1. Acker, D. D., “Management Disciplines: Harbingers of Successful Programs,” Defense Systems Management Review, Summer 1979, Volume 2, Number 3, pp. 7-20.

2. Alberts, C. J., e. al., Continuous Risk Management Handbook, Software Engineering Institute, Carnegie Mellon University, 1996.

3. Ali, T., Boruah, H., and Dutta, P., “Modeling Uncertainty in Risk Assessment Using Double Monte Carlo Method,” International Journal of Engineering and Innovation Technology (IJEIT), Volume 1, Issue 4, April 2012.

4. Alleman, G. B., Performance-Based Project Management: Increasing the Probability of Project Success, American Management Association, 2014.

5. Alleman, G. B., Coonce, T. J., and Price, R. A., “Building a Credible Performance Measurement Baseline,” The Measurable News, 2014, Issue 4.

6. Alzaharna, I. T., Seering, W. P., and Yang, M. C., “Exploration of the Use of Design Methods with the Design Structure Matrix for Integrating new Technologies into Large Complex Systems,” Proceedings of the ASME 2012 International Design Engineering Technology Conferences & Computers and Information Engineering Conference, August 12-15, 2012.

7. Apeland, S., Aven, T., and Nilsen, T., “Quantifying Uncertainty under a Predictive, Epistemic Approach to Risk Analysis,”

8. Arena, M. V., et. al., “Managing Perspectives Pertaining to Root Cause Analyses of Nunn-McCrudy Breaches, Volume 4: Program Manager Tenure, Oversight of Acquisition Category II Programs, and Framing Assumptions” RAND MG1171/4, Rand Corporation, 2013.


9. Arízaga, J. F. O, “A Methodology for Project Risk Analysis Using Bayesian Belief Networks Within A Monte Carlo Simulation Environment,” PhD Dissertation, University of Maryland, 2007.

10. Atin, A., “Project Risk Propagation Modeling of Engineering, Procurement and Construction,” Ph. D. Thesis Wayne State University, 2016.

11. ALTESS, “Probability of Program Success Operations Guide,” Acquisition, Logistics, & Technology Enterprise Systems & Services, August 22, 2007.

12. Aven, T. and Krohn, B. S., “A New Perspective of How to Understand, Assess, and Manage Risk and the Unforeseen,” Reliability Engineering & System Safety, Volume 121, January 2014, pp. 1-10.

13. Becerril, L., Sauer, M., and Lindemann, U., “Estimating the effects of Engineering Changes in Early Stage Product Development,” 18th International Dependency and Structural Modeling Conference, August 29-30, 2016.

14. Beer, M., “Alternative Approaches to the Treatment of Epistemic Uncertainties in Risk Assessment,” Institute for Risk and reliability, Leibniz Universität Hannover, 2016.

15. Bennett, F. N., “The Amateur Managers: A Study of the Management of Weapon Systems Programs,” Canberra Papers on Strategy and Defense, No. 67, 1990.

16. Blickstein, I., et. al., “Root Cause Analyses of Nunn-McCurdy Breaches: Volume 2,” Rand Corporation, MG1171z2.

17. Bilardo, V. J., et. al., “Seven Key Principles of Program and Project Success: Best Practices Survey,” NASA/TM-2008-214692, April 2008.

18. Bolten, J. G., Leonard, R. S., Arena, M. V., Younossi, O., and Sollinger, J., “Sources of Weapon System Cost Growth: Analysis of 35 Major Defense Acquisition Programs,” MG-670-AF, Rand Corporation, 2008.

19. Bordley, R. F., “Reference Class Forecasting: Resolving Its Challenge to Statistical Modeling,” The American Statistician, 68:4, 221-229, 2014.

20. Bronson, P., “Correlating Earned Value Management to Technical Performance,” Institute for Defense Analyses, Cost Analysis and Research Division.

21. Browning, T. R., “Adding Value in Product Development by Creating Information and Reducing Risk,” IEEE Transactions on Engineering Management, Vol. 49, No. 4, November 2002.

22. Charette, R., Dwinnell, L., and McGarry, J., “Understanding the Roots of Process Performance Failure,” CROSSTALK: The Journal of Defense Software Engineering, August 2004, pp. 18-24.

23. Clemen, R. T., Making Hard Decisions: An Introduction to Decision Analysis, Second Edition, Duxbury Press, 1996.

24. Conrow, E., Effective Risk Management, 2nd Edition, AIAA, 2003.25. Cooke-Davies, T., Aspects of Complexity: Managing Projects in a Complex World, Project

Management Institute, 2011.26. Covert, R., “Analytic Method for Probabilistic Cost and Schedule Risk Analysis, Final Report,”

prepared by Raymond Covert, for National Aeronautics and Space Administration (NASA) Office of Program Analysis and Evaluation (PA&E) Cost Analysis Division (CAD), 5 April 2013.

27. Covert, R., "Final Report Briefing Analytic Methods for Cost and Schedule Risk Analysis," March 4, 2013.

28. Cox, L. A. T., “What’s Wrong with Risk Matrices?” Risk Analysis, Vol. 28, No. 2, 2008, pp. 497–512.

29. Cox, L. A., “Evaluating and improving risk formulas for allocating limited budgets to expensive risk-reduction opportunities,” Risk Analysis, 32(7), pp. 1244-1252, 2012.

30. Davis, D., and Philip, A. S., “Annual Growth of Contract Costs for Major Programs in Development and Early Production,” Acquisition Policy Analysis Center, Performance Assessments and Root-Cause Analyses, OUSD AT&L, March 21, 2016.

31. Davis, P. K., “Analytic Architecture for Capabilities-Based Planning, Mission-System Analysis, and Transformation,” RAND MR-1513-OSD, 2002.

32. Davis, P. K., “Lessons from RAND’s Work on Planning Under Uncertainty for National Security,” RAND Corporation, TR-1249, 2012.

33. Davis, G. A., Giles, M. L., and Tate, D. M., “Complexity in an Unexpected Place: Quantities in Selected Acquisition Reports,” Institute for Defense Analyses, IDA Paper P-8490, December 2017.

34. Dezfuli, H., et. al., “NASA Risk-Informed Decision Making Handbook,” NASA/SP-2010-576, Version 1.0, April 2010.

35. Dezfuli, H., “NASA’s Risk Management Approach, Workshop on Risk Assessment and Safety Decision Making Under Uncertainty,” Bethesda, MD., September 21-22, 2010.

36. DOD, “Integrated Program Management Report (IPMR), DI-MGMT-81861, June 20, 2012.37. van Dorp, J. R., and Duffey, M. R., “Statistical Dependence in Risk Analysis for Project

Networks using Monte Carlo Methods,” International Journal of Production Economics, 58, pp. 17-29, 1999.

38. Dwyer, M., Cameron, B., and Sazajnfarber, Z., “A Framework for Studying Cost Growth on Complex Acquisition Programs,” Systems Engineering, Volume 18, Issue 6, November 2015, pp. 568-583.

39. Eisner, H., Essentials of Project and Systems Engineering Management, Second Edition, John Wiley & Sons, 2002.

40. Erikson, J., “Analysis of Alternatives (AoA) Handbook: A Practical Guide to the Analysis of Alternatives,: Office of Aerospace Studies, Headquarter Air Force HAF/A5R-OAS, 6 July 2016.

41. Fang, C., Marle, F., and Vidal, L-A., “Modelling Risk Interactions to Re-Evaluate Risk in Project Management,”, 12th International Dependency and Structure Modelling Conference, DSM ’10, 22-23 July 2010.


42. Fang, C., Marle, F., and Xio, M., “Applying Importance Measures to Risk Analysis in Engineering Project Using a Risk Network Model,” IEEE Systems Journal, 2016.

43. Fleishman-Mayer, L., Arena, M., and McMahon, M. E., “A Risk Assessment Methodology and Excel Tool for Acquisition Programs,” RAND Corporation, RR 262, 2013.

44. Fox, S.,” Ontological Uncertainty and Semantic Uncertainty in Global Network Organizations,” VTT Working Papers 102, 2008.

45. Fox, C. R. and Ülkumen, G., “Distinguishing Two Dimensions of Uncertainty,” in Perspectives of Thinking, Judging, and Decision Making, Editors, Wibecke Brun,- Gideon Keren,- Geir Kirkeboen,- Henry Montgomery, Universitetsforlaget; UK ed. Edition, November 28, 2011.

46. Frazier, B., McBride, J., Hsu, A. and Weir, A., “Technical Risk Identification at Program Inception Product Overview,” U.S. Space Program Mission Assurance Improvement Workshop (MAIW), Aerospace Corporation, Dulles, VA, May 8, 2014.

47. Galway, L. A., “Quantitative Risk Analysis for Project Management: A Critical Review,” RAND Corporation, WR-112-RC, February 2004.

48. GAO, “Space Acquisitions: DOD Faces Challenges in Fully Realizing Benefits of Satellite Acquisition Improvements,” GAO-12-563T, March 21, 2012.

49. GAO, “Defense Acquisitions: Assessments of Selected Weapon Programs,” GAO-15-342SP50. Gano, D., Apollo Root Cause Analysis: Effective Solutions to Everyday Problems Every Time,

3rd Edition, Atlas Books, 2008.51. Garner, W. J., Uncertainty and Structure as Psychological Concepts, John Wiley & Sons, 1962.52. Gerstein, D. M. et. al., “Developing a Risk Assessment Methodology for the National

Aeronautics and Space Administration,” RAND Corporation, RR 1537, 2015.53. Grady, J. O., Systems Requirements Analysis, Academic Press, 2006.54. Hamaker, J. W., “But What Will It Cost? The History of NASA Cost Estimating,” Readings in

Program Control, pp. 25-37, 1994.55. Harvett, C. M., “A Study of Uncertainty and Risk Management Practice Relative to Perceived

Project Complexity,” Ph. D. Thesis, Bond University, May 2013.56. H.A.S.C., “Twenty-Five Years of Acquisition Reform: Where Do We Go from Here?” Committee

on Armed Services, House of Representatives, October 29, 2013.57. Helton, J. C. and Burmaster, D. E., “Treatment of aleatory and epistemic uncertainty in

performance assessments for complex systems,” Reliability Engineering and System Safety, vol. 54, no. 2-3, pp. 91–94, 1996.

58. Hillson, D. and Simon, P., Practical Risk Management: The ATOM Method, Management Concepts, 2007.

59. Hofbauer, et al., “Cost and time overruns for major defense Acquisition programs: An annotated brief,” Center for Strategic and International Studies, Washington, DC, 2011.

60. Hubbard, D., The Failure of Risk Management: Why It’s Broken and How to Fix It, John Wiley & Sons, 2009.

61. Hulett, D. T. and Hillson, D., “Branching Out: Decision Trees Offer a Realistic Approach to Risk Analysis,” PM Network, May 2006, p. 43.

62. IDA, “Root Cause of Nunn-McCrudy Breaches - A Survey of PARCA Root Causes Analyses,” 2010-2011, Interim Report, IDA Paper P-4911, August 2012.

63. Jolly S., “Is Software Broken?”, IEEE Fourth International Conference of System of Systems Engineering, Albuquerque, New Mexico June 1, 2009.

64. Karim, A. B. A., “The Development of an Empirical-Based Framework for Project Risk Management,” Ph. D. Thesis, University of Manchester, 2014.

65. Kim, Y., et. al., “Acquisition of Space Systems, Past Performance and Future Challenges,” RAND MG1171z7, Rand Corporation, 2015.

66. Klein, G., “Performing a Project Premortem,” Harvard Business Review, September 2007.67. der Kiureghian, A., “Aleatory or Epistemic? Does it matter?” Special Workshop on Risk

Acceptance and Risk Communication, March 26-27, 2007, Stanford University.68. Kwak, Y. H, and Smith, B. M., “Managing Risks in Mega Defense Acquisition Projects:

Performance, Policy, and Opportunities,” International Journal of Project Management, 27, pp. 812-820, 2009.

69. Laitonen, J. (Project manager) Losoi, H., Losoi, M., and Ylilammi, K., “Tools for analyzing epistemic uncertainties in probabilistic risk analysis Final report,” Mat-2.4177 Seminar on Case Studies in Operations Research, 24 May 2013.

70. Lehman, D., Groenendaal, H., and Nolder, G., Practical Spreadsheet Risk Modeling for Management, CRC Press, 2012.

71. Leveson, N., et/ al., “Risk Analysis of NASA Independent Technical Authority,” Massachusetts Institute of Technology, June 2005.

72. Lorell, M. A., Lowell, J. F., and Younossi, O., “Implementation Challenges for Defense Space Programs,” MG-431-AF, Rand Corporation 2006.

73. Lorell, M. A., Payne, L. A., and Mehta, K. R., “Program Characteristics That Contribute to Cost Growth: A Comparison of Air Force Major Defense Acquisition Programs,” Rand Corporation, RR1761, 2017.

74. Lempert, R. and Collins, M., “Managing the Risk on Uncertain Threshold Responses: Comparison of Robust, Optimum, and Precautionary Responses,” Risk Analysis 27 August 2007, pp. 1009-1026, Rand Corporation.

75. Maier, M. and Rechtin, E., The Art of Systems Architecting, Third Edition, 200976. Marle, F. and Vidal, L-A., “Project Risk Management Processes: Improving Coordination using

a Clustering Approach,” Research in Engineering Design, Springer, 2011, pp. 189-206.77. McManus, H. and Hastings, D., (2005), “A Framework for Understanding Uncertainty and its

Mitigation and Exploitation in Complex Systems”, Fifteenth Annual International Symposium of the International Council on Systems Engineering (INCOSE), 10 – 15 July 2005.


78. NASA, “Margins and Contingency Module Space Systems Engineering,” Version 1.0, Lisa Guerra of NASA’s Exploration Systems Mission Directorate, Department of Aerospace Engineering at the University of Texas at Austin, 2008

79. NASA,” Probabilistic Risk Assessment Procedures Guide for NASA Managers and Practitioners, Version 1.1., August 2002.

80. NASA, “Joint Confidence Level Requirement: Policy and Issues,” NASA/TM-2011-216154, July 2011.

81. NASA, “Probabilistic Schedule Reserve Allocation,” NASA 2013 Cost Symposium, August 29, 2013

82. NASA, “NPR 7120.5E, NASA Space Flight Program and Project Management Handbook,” NASA/SP-2014-3705, September 2014.

83. NASA, “Risk-Informed Decision Making,” NASA/SP-2010-576, Version 1.0, April 2010.84. Navy, “Naval PoPS Guidebook, Guidance for the Implementation of Naval PoPS, A Program

Health Assessment Methodology for Navy and Marine Corps Acquisition Programs,” Version 1.0. Department of the Navy, September 2008.

85. Navy, “Naval PoPS Criteria Handbook, Supplement for the Implementation of Naval PoPS, Version 1.0., Department of the Navy, September 2008.

86. Nemfakos, C., “Observations from Root Cause Analysis,” NPS Acquisition Symposium, May 2012.

87. Nilchiani, R. R. and Puglises, A., “A Complex Systems Perspective of Risk Mitigation and Modeling in Development and Acquisition Programs,” Naval Postgraduate School, April 30, 2016.

88. DePasquale, D. and Charania, A. C., “I-RaCM: A Fully Integrated Risk and Life Cycle Cost Model,” American Institute of Aeronautics and Astronautics, 2008.

89. Pawlikowski, E., Loverro, D., and Cristler, T., “Disruptive Challenges, New Opportunities, and New Strategies, Strategic Studies Quarterly, Maxwell Air Force Base, Spring 2012, pp. 27-54.

90. Perez, E. J., “Air Force Project Risk Management - The Impact of Inconsistent Processes,” Thesis, Air Force Institute of Technology, AFIT-ENV-MS-16-S-047, September 2016.

91. Petković, I., “Risk Management using Dependency Structure Matrix,” Workshop in Opatija, September 2 - 8, 2012, German Academic Exchange Service.

92. PM Knowledge Center, “Schedule Risk Analysis: How to measure your baseline schedule sensitivity?” https://goo.gl/jpMxuy

93. Price, R. A. and Alleman, G. B., “Critical Thinking on Critical Path Analysis,” College of Performance Management, EVM World 2016.

94. Public Works and Government Services Canada, “Project Complexity and Risk Assessment Manual,” May 2015, Version 5.

95. Raiffa, H., Decision Analysis: Introductory Lectures on Choice Under Uncertainty, Addison Wesley, 1968.

96. Razaque, A., Bach, C., Salama, N. Alotaibi, A., “Fostering Project Scheduling and Controlling Risk Management,” International Journal of Business and Social Science, Vol. 3 No. 14, Special Issue, July 2012.

97. Reeves, J. D., “The Impact of Risk Identification and Trend on Space Systems Project Performance,” Ph. D. Thesis Schools of Engineering and Applied Science, The George Washington University, May 19th, 2013.

98. Rendleman, J. D. and Faulconer, J. W., “Escaping the Space Acquisition Death Spiral: A Three Part Series,” High Frontier Journal, Volume 7, Number 4, pp. 51-57, August 2011.

99. Riesch, Hauke, “Levels of Uncertainty,” in Handbook of Risk Theory: Epistemology, Decision Theory, Ethics, and Social Implications of Risk, Rosener, S. and Hillebrand R. Editors, Springer, 2011.

100. Roedler, G. J. and Jones, C., “Technical Measurement: A Collaborative Project of PSM, INCOSE, and Industry, INCOSE-TP-2003-202-01, 27 December 2005.

101. Roedler, G. J., Rhodes, D. H., Schimmoller, H. and Jones, C. “Systems Engineering Leading Indicators Guide,” Version 2.0, MIT, INCOSE, and PSM, January 29, 2010.

102. Rogers, J. L., Korte, J. J., and Bilardo, V. J., “Development of a Genetic Algorithm to Automate Clustering of a Dependency Structure Matrix,” NASA/TM-2006-212279, February 2006.

103. Salado, A. and Nilchiani, R. R., “The Concept of Problem Complexity,” Conference on Systems Engineering Research (CSER 2014), Procedia Computer Science, 28

104. Scheinin, W. “Start Early and Often: The Need for Persistent Risk Management in the Early Acquisition Phases,” Paper to the Aerospace Corporation / Air Force SMC / NASA Seventh Symposium on Space Systems Engineering & Risk Management Conference, 27-29 February 2008.

105. Schwartz M., “Reform of the Defense Acquisition System,” United States Senate Committee on Armed Services, Congressional Research Service, April 30, 2014.

106. SMC, “Space and Missile Systems Center Risk Management Process Guide,” Version 2.0 - 5 September 2014.

107. Sharon, A., de Weck, O. L., Dori, D., “Project Management vs. Systems Engineering Management: A Practitioners’ View on Integrating the Project and Product Domains,” Systems Engineering, 18 December 2010.

108. Siu, N. and Marble, J., “Aleatory vs. Epistemic Uncertainties: Principles and Challenges,” ASME PVP Conference, Baltimore, MD., July 20, 2011.

109. Squair, M., “Epistemic, Ontological, and Aleatory Risk,” Critical Uncertainties (Blog), https://criticaluncertainties.com, 11/10/2009, Retrieved, 5 January 2018.

110. Steward, D., “The Design Structure System: A Method of Managing the Design of Complex Systems,” IEEE Transactions on Engineering Management, August 1918.

111. Timson, F. S., “Measurement of Technical Performance in Weapon System Development

https://goo.gl/jpMxuy

https://criticaluncertainties.com

https://criticaluncertainties.com

Programs: A Subjective Probability Approach,” RAND RM-5207-ARPA, December 1968.112. USAF, “Air Force Instruction 90-802, Risk Management.”113. USAF, “Air Force Instruction 63-101/20-101, Integrated Life Cycle Management,” 9 May 2014.114. USAF, “AF/A5R Requirements Development Guidebook: Volume 1, Implementation of the

JCIDS Deliberate Process,” 2 Feb 2017, Version 8, Directorate of Requirements Integration, AF/A5R-P, Pentagon 5C858.

115. Vanhoucke, M., Measuring Time: Improving Project Performance Using Earned Value Management, Springer, 2009.

116. Vanhoucke, M., Integrated Project Management Sourcebook: A Technical Guide to Project Scheduling, Risk, and Control, Springer, 2016.

117. Vidal, L. and Marle, F., “Modeling project complexity,” International Conference on Engineering Design, 2007.

118. Vrouwenvelder, A.C.W.M., “Uncertainty analysis for flood defense systems in the Netherland,” Proceedings, ESREL, 2003.

119. Walker, W.E., Harremoës, P., Rotmans, J. van der Sluijs, J.P., van Asselt, M.B.A., Janssen P. and Krayer von Krauss, M.P., “Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support,” Integrated Assessment, Volume 4, Issue 1, 2003.

120. Williams, T., “The Nature of Risk in Complex Projects,” Project Management Journal, 48(4), pp. 55-66, 2017.

121. Wood, H. L. and Ashton, P., “The Factors of Project Complexity,” 18th CIB World Building Congress, Salford, United Kingdom, 10 May 2010.

122. Younossi, Obaid et. al., “Improving the Cost Estimation of Space Systems: Past Lessons and Future Recommendations,” RAND Project Air Force, MG-690.

123. Zack, M. H., “Managing Organizational Ignorance, Knowledge Directions, Volume 1, Summer, 1999, pp. 36-49.

124. Zarikas, V. and Kitsos, C. P., “Risk Analysis with Reference Class Forecasting Adopting Tolerance Regions,” in Theory and Practice of Risk Assessment, Chapter: 18, Springer International, June 2015.

125. Zhang, Y., Bai, S., and Guo, Y., “The Application of Design Structure Matrix in Project Schedule Management,” 2010 International Conference on E-Business and E-Government, pp. 2813-2816.

126. Zio, E., “From Complexity Science to Reliability Efficiency: A New Way of Looking at Complex Network Systems and Critical Infrastructures,” International Journal of Critical Infrastructures, January 2007.

EVMLibrary.org:The CPM Online Library

A comprehensive, online library of resources about Earned Value Management (EVM) and related project management topics.

The CPM library now contains well over 2,100 documents, such as articles and presentations, and is growing everyday.

Each document has been catalogued by subject area, author, publication date, keywords, etc.

How You Can Contribute? Do you have topical articles, speeches, or presentations?

Contact Beatrice Barry at [email protected] to have your contributions included in this online resource.

The most complete project performance management and EVMS available in a single system

IPM includes all of the features you need for your entire program management team in one package, eliminating the need to connect multiple third party software solutions. IPM’s integrated cost engine replaces MPM and includes:

(256) 881-1038 | [email protected] | www.opdec.com

Comply with government standards

ANSI 748, OMB Circular No. A-11 Exhibit 300 and 53,

C/SSR DFARS clauses, EVMIG of the DCMA and NASA

Interim Directive to NPR 7120.5C

More efficient licensing for small organizationsIPM is a single licensed web based EVMS

software that is licensed per user.

IPM should be considered if you are…

Cost/ScheduleIntegration

CostManagement

ScheduleManagement

EVM ComplianceReports & Analysis

Wanting the benefits of an

integrated EVMS

“We need something

more robust than Excel

and Microsoft Project*

to get the job done, but

not as complicated or

expensive as other

EVM systems.”

“Since we have to move

away from MPM, we are

interested in exploring

our options.”

With our data conversion utility, it

can take just minutes to convert MPM projects to IPM!

An organization who uses MPM and is seeking a

replacement

New in2018!

VENDORS & SERVICES

P.O. Box 20996Sarasota, FL 34276

Phone: (888) 644-5613

www.CBTWorkshop.com

www.steelray.com

Schedule quality for Project and P6

www.humphreys-assoc.com

Planning every step — Every step as planned™

EVPrep™ Exam Study Guide EVPrep™ Exam Prep Workshops

www.mgmt-technologies.com*Earned Value Professional

Get it now!

Got your ?*

www.sna-software.com

Get Involved In CPM:Become A VolunteerMost people agree that the value of membership in any organization is greatly impacted by one’s involvement in that organization. CPM has many opportunities for you to become engaged in your organization.

You can help with our certificate program, become a leader in our two conferences each year or other volunteer opportunities as they appear on the website. Take the time to read each of the job descriptions and ask questions of the point of contact for each if you are not sure what is expected. PDU’s are available for volunteering in CPM.

These opportunities are listed on our website:www.mycpm.org/membership/volunteer

see our ad on p. 8

see our ad on p. 18

see our ad on p. 4

www.bdo.com

mycpm.org

fl˝˙ˆˇ˘ ˙ ˆ ˙ˆ ˆ ˝˙ˆ · CONTENTS 05 Letter from the Editor(s) 07 Message from the...

Documents

Transcript of fl˝˙ˆˇ˘ ˙ ˆ ˙ˆ ˆ ˝˙ˆ · CONTENTS 05 Letter from the Editor(s) 07 Message from the...