TeamFormation

Turning Experts Into Expert Teams

Developing & Conducting Decision Making GamesDeveloping & Conducting Decision Making Gamesfor for

System OperatorsSystem Operators

EPRIEPRI’’s XIII Annual Power Switching Safety s XIII Annual Power Switching Safety

and Reliability Conferenceand Reliability Conference

Doug HarringtonTeam FormationTeam Formation

E-Mail – teamform@yadtel.net

Teaching Decision Making Without Teaching Decision Making

Decision Making StylesDecision Making Styles• Analytical: Get the RIGHT answer

the first time. Involves use of analytical processes or tools (ex. Checklists , Performance charts, Procedures, Automation)

• Knowledge-Based: Get the BEST answer in available time using Knowledge, Brainstorming, BOGSAT.

• Action-Oriented: Get MOVING.Make a decision and continue to monitor

Analy

tica

l

Kn

ow

led

ge-B

ase

d

Act

ion

-Ori

en

ted

In-Depth

Deliberate

Time Critical

Decision Making StylesDecision Making Styles

Decision making style Decision making style will be impacted by:will be impacted by:

Available timeAvailable timeCriticality of the decision Criticality of the decision Number of people Number of people involvedinvolvedAvailable tools & sources Available tools & sources of information of information Decision making bias Decision making bias

Tim

e

Criticality

Analy

tica

l

Kn

ow

led

ge-B

ase

d

Act

ion

-Ori

en

ted

Generate & implement a possible solution

Put your head in a stream of data

Monitor progress of your action

Look for a familiar pattern

Individual Decision Makingat the Pointy End

RecognitionRecognition--Primed Decision Primed Decision ModelModel

Situation

Cues

generatesgenerates

Patternsthat let that let you you recognizerecognize

Action Scripts

that activatethat activate

to to affect affect

thetheMental

Simulation

which you which you assess byassess by

Mental Models

using yourusing your

Adapted from Recognition-Primed Decisions Model in Gary Klein’s, The Power of Intuition, Doubleday, 2003.

Are You An Expert?Are You An Expert?An expert is someone An expert is someone who has made all the who has made all the mistakes that can be mistakes that can be made in a particular made in a particular

domain.domain.

Characteristics of the Intuitive Characteristics of the Intuitive Decision MakerDecision Maker

Their performance is markedly better than Their performance is markedly better than average.average.They have a good sense of what is going They have a good sense of what is going to happen next.to happen next.They can explain how the current situation They can explain how the current situation has developed.has developed.They are aware of their fallibility.They are aware of their fallibility.They are confident in the face of time They are confident in the face of time pressure and uncertainty.pressure and uncertainty.

They can anticipate problems in time to They can anticipate problems in time to avoid or defuse them.avoid or defuse them.They know how to work around They know how to work around unexpected events.unexpected events.They know the limits of routines and are They know the limits of routines and are not trapped by them.not trapped by them.They are still trying to improve themselves.They are still trying to improve themselves.They can tell you about their mistakes and They can tell you about their mistakes and how they should have done better.how they should have done better.

ExpertiseExpertise

The ability to:The ability to:Maintain situational awareness Maintain situational awareness Make sound decisions Make sound decisions Execute actions required as part of the Execute actions required as part of the crewcrewRecognize the need for Recognize the need for ““course course correctionscorrections””Stay ahead of the situation Stay ahead of the situation

Developing ExpertiseDeveloping ExpertiseDeveloping expertise requires Developing expertise requires pulling the pulling the lessons learned lessons learned from: from:

Past experiences Past experiences SimulationsSimulationsAnalysis of real eventsAnalysis of real eventsDiscussions with expertsDiscussions with experts

And turning the lessons into And turning the lessons into BASIC SKILLSBASIC SKILLS

Consequently, very little time Consequently, very little time (or conscious thought) is (or conscious thought) is required to make a decision required to make a decision or perform an action.or perform an action.

Analy

tica

l

Kn

ow

led

ge-B

ase

d

Act

ion

-Ori

en

ted

Developing ExpertiseDeveloping Expertise

Debrief the Debrief the experience to experience to learn from learn from and improve and improve performanceperformance

Requires that you…

Analy

tica

l

Kn

ow

led

ge-B

ase

d

Act

ion

-Ori

en

ted

““ My concern is simple: I donMy concern is simple: I don’’t mind t mind that computers are getting smarter that computers are getting smarter than us because they are growing in than us because they are growing in intelligence. intelligence. I I dodo mind that theymind that they’’re getting smarter re getting smarter than us by making us stupid. And than us by making us stupid. And thatthat’’s what can happen. Thats what can happen. That’’s the s the trend we have to resist.trend we have to resist.””

Gary Klein in his book, The Power of Intuition, Currency Books, 2003.

How Can You TrainDecision Making Skills?

How Can You TrainDecision Making Skills?

Teach a generic decisionTeach a generic decision--making processmaking process•• Has not been demonstrated to be effectiveHas not been demonstrated to be effective

Rational Choice Modelof Decision Making

Rational Choice Modelof Decision Making

Evaluation Dimensions

A

Total

123

3 2 2 1

4

B C DCAR CHOICES

Rational Choice Model of Decision Making

Rational Choice Model of Decision Making

Limitations:Limitations:

• Requires too much time

• Requires too much data

• Can result in worse performance

• “One-size fits all” method is too weak

• Little value to training

• Requires too much time

• Requires too much data

• Can result in worse performance

• “One-size fits all” method is too weak

• Little value to training

Evaluation Dimensions A

Total

123

3 2 2 1

4

B C D

OptionsEvaluation Dimensions A

Total

123

3 2 2 1

4

B C DA

Total

123

3 2 2 1

4

B C D

Options

How Can You TrainDecision Making Skills?

How Can You TrainDecision Making Skills?

Teach a generic decisionTeach a generic decision--making processmaking process•• Has not been demonstrated to be effectiveHas not been demonstrated to be effective

Provide tools for building expertiseProvide tools for building expertise•• Based on research with experts from a wide Based on research with experts from a wide

variety of domainsvariety of domains

•• Involves the use of Decision Making Exercises Involves the use of Decision Making Exercises and effective Debriefingsand effective Debriefings

Experts vs. Novices in Choosing COAs

An example…Imagine a hiker approaching a rain swollen stream that he Imagine a hiker approaching a rain swollen stream that he needs to cross. The hiker stops and instead of wasting a lot ofneeds to cross. The hiker stops and instead of wasting a lot oftime hiking up and down the river bank trying to find the best time hiking up and down the river bank trying to find the best place to cross he simply stops and studies the rocks in front place to cross he simply stops and studies the rocks in front of him. of him.

The mental process of “connecting the dots” or in this case, mapping a path across the stream, is called satisficing. In other words, the hiker thinks to himself, “before I start looking at other places, either better or worse, first I need to decide if I can make this particular string of rocks work. If so, then it’s good enough to get the job done.”

The mental process of The mental process of ““connecting connecting the dotsthe dots”” or in this case, mapping a or in this case, mapping a path across the stream, is called path across the stream, is called satisficingsatisficing. In other words, the . In other words, the hiker thinks to himself, hiker thinks to himself, ““before I before I start looking at other places, either start looking at other places, either better or worse, first I need to better or worse, first I need to decide if I can make this particular decide if I can make this particular string of rocks work. If so, then itstring of rocks work. If so, then it’’s s good enough to get the job done.good enough to get the job done.””

How Do Experts Learn?How Do Experts Learn?

•• They build an experience base using They build an experience base using deliberate practicedeliberate practice

•• They thoroughly debrief the event looking They thoroughly debrief the event looking for mistakes and lessons learnedfor mistakes and lessons learned

Objectives:Objectives:

Learn how to interview subject matter experts Learn how to interview subject matter experts to attain stories for your Decision Making to attain stories for your Decision Making Games.Games.Turn those stories into valuable learning Turn those stories into valuable learning experiences.experiences.Develop and write Decision Making Games.Develop and write Decision Making Games.Facilitate those games for your operators.Facilitate those games for your operators.Lead a decisionLead a decision--making review. making review.

Why is Time Critical Decision Why is Time Critical Decision Making Different?Making Different?

HighHigh--level reasoninglevel reasoningProblemProblem--solvingsolvingMotor skillsMotor skillsSpeechSpeechMemoryMemoryVisionVisionBalance & coordinationBalance & coordinationBreathing & heart rateBreathing & heart rate

High Level Brain Functions

Low Level Brain Functions

Understanding Brain Function under Stress

The brain shuts down from top to bottom under stress

Under Stress or in Need of Under Stress or in Need of Immediate ActionImmediate Action

Decision Making can becomeDecision Making can becomeMore intuitiveMore intuitiveBased on skills and pattern Based on skills and pattern matchingmatchingUses Uses ““past experiencepast experience””FastFast

What is a DMG?What is a DMG?

LowLow--fidelity, paperfidelity, paper--andand--pencil simulationpencil simulationA dilemma with high levels of uncertaintyA dilemma with high levels of uncertaintyBuiltBuilt--in time pressurein time pressureAllows practice in rapid decision makingAllows practice in rapid decision makingOperators practice communicating intentOperators practice communicating intentOperators analyze and learn from their decisionsOperators analyze and learn from their decisions

Goals of DMGsGoals of DMGs

Expose operators to unfamiliar decision making Expose operators to unfamiliar decision making situationssituationsProvide practice acting under time pressure and Provide practice acting under time pressure and uncertaintyuncertaintyHelp operators build an experience baseHelp operators build an experience baseImprove operatorsImprove operators’’ patternpattern--recognition skillsrecognition skillsProvide practice in communicating plans and Provide practice in communicating plans and intentintent

Benefits of DMGsBenefits of DMGs

Operators improve their ability to make decisions Operators improve their ability to make decisions quickly and accuratelyquickly and accuratelyHelps them see patterns and recognize Helps them see patterns and recognize situations more quicklysituations more quicklyOperators can experiment with different ideasOperators can experiment with different ideasOperators learn from each otherOperators learn from each otherA way to transfer experience from experts to A way to transfer experience from experts to rookiesrookies

Components of a DMGComponents of a DMGDMGs are DMGs are storiesstories that have that have namesnamesThe The backgroundbackground tells the history of the situationtells the history of the situationOrganizational constraintsOrganizational constraints are factors that make the are factors that make the situation even more challengingsituation even more challengingThe The scenarioscenario is where the current story begins and is where the current story begins and decision making is forceddecision making is forcedThe The environmentenvironment describes where you are and what describes where you are and what problems you have been experiencingproblems you have been experiencingResources Resources are what you have available to youare what you have available to you

Creating a DMGCreating a DMG

Select a focusSelect a focusPersonal experiences Personal experiences Upcoming eventsUpcoming eventsLesson points for an upcoming lectureLesson points for an upcoming lectureDecision Requirements Decision Requirements –– judgments that judgments that make complex tasks challengingmake complex tasks challengingPlaces where novices strugglePlaces where novices struggle

Decision Requirements TableIdentify a critical, difficult, and frequent decision or judgment:________________________________________________________________________________________

________________________________________________________________________________________

What Makes This Decision Difficult?

What Cues or Factors Are Used To Make This

Decision?

What Kinds of Errors Are Often Made?

How Would an Expert Make This Decision Differently From a

Novice?

Trouble on the BridgeTrouble on the Bridge

N

Armistan

2

1

3

Dry gully

2nd St.

1st St.

Develop the BackgroundDevelop the Background

Tells the history of the situationTells the history of the situationCurrent status of the gridCurrent status of the gridRecent load trendsRecent load trendsWhat are operators expecting todayWhat are operators expecting todayWeather forecastWeather forecastAny power plants out of serviceAny power plants out of serviceSet the stage of the unfolding storySet the stage of the unfolding story

Select Organizational ConstraintsSelect Organizational Constraints

Can be a critical part of the DMGCan be a critical part of the DMGExtremely detailed or rather ambiguousExtremely detailed or rather ambiguousCan add complexity and realismCan add complexity and realismGives operators practice in struggling with Gives operators practice in struggling with problematic organizational factorsproblematic organizational factors

Develop the ScenarioDevelop the Scenario

Tells the story of the specific situation the Tells the story of the specific situation the operators are encounteringoperators are encounteringDefines the operators specific role and time of Defines the operators specific role and time of day day –– ““You are the Reliability Coordinator and You are the Reliability Coordinator and your team has just assumed the day shift.your team has just assumed the day shift.””Defines the environment such as operatorsDefines the environment such as operators’’location, weather, equipment status, etc.location, weather, equipment status, etc.Resources availableResources availablePresents the dilemma the operators are facingPresents the dilemma the operators are facing

Set the RequirementsSet the Requirements

Defines what the participants must accomplish Defines what the participants must accomplish and in what time frameand in what time frame““You have You have fivefive minutes to assess the situation minutes to assess the situation and make your decision.and make your decision.””Final decision or list of considerations?Final decision or list of considerations?How will the decision be communicated How will the decision be communicated ––verbally, sketches, written decision?verbally, sketches, written decision?

Representation of the EnvironmentRepresentation of the Environment

A written description may be enoughA written description may be enoughA drawing or diagram may be helpfulA drawing or diagram may be helpfulComputer generatedComputer generatedWhere is the participant located in this Where is the participant located in this scenarioscenario

Naming the DMGNaming the DMG

Select a name that will help you and Select a name that will help you and others remember the contentothers remember the contentMay refer to the general situationMay refer to the general situationMay refer to the dilemma the participants May refer to the dilemma the participants face, e.g. face, e.g. ““Trouble on the BridgeTrouble on the Bridge””Be creativeBe creative

DMG _____________________DMG _____________________Background:Background:

Scenario:Scenario:

Requirement:Requirement:

Facilitator InformationFacilitator Information

DMG Quality TestDMG Quality Test

Does your DMG tell an engaging story?Does your DMG tell an engaging story?Does it go from general to specific?Does it go from general to specific?Is there a good level of time pressure?Is there a good level of time pressure?Does it contain uncertainty?Does it contain uncertainty?Are there multiple courses of action?Are there multiple courses of action?Does your DMG force a tough decision?Does your DMG force a tough decision?

Man In The WaterMan In The WaterBackground:Background:

Baldwin Creek Power Plant is owned by Midstate Energy. The plantBaldwin Creek Power Plant is owned by Midstate Energy. The plantconsists of five units rated 220consists of five units rated 220--MW each, for a total generating capacity of MW each, for a total generating capacity of 10501050--MW.MW.The power plantThe power plant’’s forebay is Pinecrest Reservoir, where a single intake s forebay is Pinecrest Reservoir, where a single intake supplies the Baldwin Creek Power Plantsupplies the Baldwin Creek Power Plant’’s five penstocks.s five penstocks.Three 230Three 230--kV lines transmit the Power PlantkV lines transmit the Power Plant’’s power to the city of Mason in s power to the city of Mason in the eastern part of the Midstate Energy balancing authority areathe eastern part of the Midstate Energy balancing authority area. .

ScenarioScenarioIt is a hot August Wednesday. You are the Shift Supervisor on duIt is a hot August Wednesday. You are the Shift Supervisor on duty at the ty at the Midstate Energy Dispatch Center. The NPL is 3830Midstate Energy Dispatch Center. The NPL is 3830--MW.MW.900900--MW is being generated by Baldwin Creek Power plant.MW is being generated by Baldwin Creek Power plant.At 1415 hours the generation dispatcher receives a phone call frAt 1415 hours the generation dispatcher receives a phone call from the om the Pinecrest Reservoir operator and is told that a worker has fallePinecrest Reservoir operator and is told that a worker has fallen into the n into the water at the intake. He immediately hangs up the phone and adviswater at the intake. He immediately hangs up the phone and advises you.es you.What other information do you need before deciding what to do? AWhat other information do you need before deciding what to do? Ask sk specific questions. The facilitator will supply you with the infspecific questions. The facilitator will supply you with the information. ormation.

Requirement:Requirement:Answer the following questions. You have five minutes. Answer the following questions. You have five minutes. What are your immediate concerns?What are your immediate concerns?What would you do?What would you do?

Facilitator Information:Facilitator Information:Answer the shift supervisorAnswer the shift supervisor’’s questions using the following information:s questions using the following information:Additional generation available: Additional generation available:

200 MW unused hydro capacity200 MW unused hydro capacityThree 47Three 47--MW combustion turbines are available in Mason and can be onMW combustion turbines are available in Mason and can be on--line in line in 15 minutes15 minutesYou have 200You have 200--MW of unused onMW of unused on--line thermal generation capacity evenly divided line thermal generation capacity evenly divided between three units. Each can ramp at 3 MW per minute. between three units. Each can ramp at 3 MW per minute.

Midstate Energy has no interruptible load.Midstate Energy has no interruptible load.Midstate Energy is not part of a reserve sharing group.Midstate Energy is not part of a reserve sharing group.Midstate Energy is currently: Midstate Energy is currently:

Selling 250 MW nonSelling 250 MW non--firm energyfirm energySelling 350 MW firm energySelling 350 MW firm energyPurchasing 50Purchasing 50--MW spinning reserveMW spinning reserve

Midstate EnergyMidstate Energy’’s single greatest contingency is loss of a 350s single greatest contingency is loss of a 350--MW thermal unit. MW thermal unit. The load is expected to go up 200The load is expected to go up 200--MW during the next hour. MW during the next hour.

Decision Making ReviewDecision Making ReviewWhat were the goals of the shift supervisor?What were the goals of the shift supervisor?What other questions should the shift supervisor have asked befoWhat other questions should the shift supervisor have asked before making his re making his decision?decision?What other communication should take place during this event?What other communication should take place during this event?Did Midstate Energy have adequate reserves?Did Midstate Energy have adequate reserves?Was there a violation of a NERC Standard?Was there a violation of a NERC Standard?Should the shift supervisor shed load?Should the shift supervisor shed load?

TeamFormation

Turning Experts Into Expert Teams

Debriefing Decision Debriefing Decision Making Games Making Games

The Decision Making The Decision Making ReviewReviewDoug HarringtonTeam FormationTeam Formation

E-Mail – teamform@yadtel.net

The DMG/DMR Gives Participants:The DMG/DMR Gives Participants:

Practice making quick decisionsPractice making quick decisionsFeedback on those decisionsFeedback on those decisionsInsights into othersInsights into others’’ thinkingthinking

Facilitating the Decision Making Facilitating the Decision Making Review (DMR)Review (DMR)

A debriefing tool used with the DMGA debriefing tool used with the DMGIdentify important decisions madeIdentify important decisions madeIdentify important features of the situationIdentify important features of the situationIdentify uncertaintiesIdentify uncertaintiesIdentify difficulties with the critical decisionIdentify difficulties with the critical decisionReflect on the decision making during the Reflect on the decision making during the exerciseexercise

RecognitionRecognition--Primed Decision Primed Decision ModelModel

Situation

Cues

generatesgenerates

Patternsthat let that let you you recognizerecognize

Action Scripts

that activatethat activate

to to affect affect

thetheMental

Simulation

which you which you assess byassess by

Mental Models

using yourusing your

Adapted from Recognition-Primed Decisions Model in Gary Klein’s, The Power of Intuition, Doubleday, 2003.

Decision Making ExerciseDecision Making ExerciseUse the questions below to guide your team through the decision Use the questions below to guide your team through the decision making making exercise.exercise.

What are the critical cues or factors that will What are the critical cues or factors that will influence your decisions?influence your decisions?What are your major concerns about this situation?What are your major concerns about this situation?What is the primary goal, or outcome, you are trying What is the primary goal, or outcome, you are trying to achieve?to achieve?List, in order, the courses of action your team List, in order, the courses of action your team decides to take. (Be prepared to explain your decides to take. (Be prepared to explain your decisions)decisions)What communications will you need to make, and to What communications will you need to make, and to whom?whom?

Lessons LearnedLessons LearnedDuring the Decision Making Review, make a list of lessons learneDuring the Decision Making Review, make a list of lessons learned from our d from our discussion.discussion.

List additional actions other teams took that your List additional actions other teams took that your team may not have considered.team may not have considered.

What, if anything, would you do differently based on What, if anything, would you do differently based on our discussions?our discussions?

What are the lessons we all should take away from What are the lessons we all should take away from this exercise?this exercise?

Why Is the DMR So Important?Why Is the DMR So Important?

Gets the most out of a training exercise or Gets the most out of a training exercise or DMGDMGDigs deeper into the thinking behind the Digs deeper into the thinking behind the decisionsdecisionsParticipants reflect on what went well and Participants reflect on what went well and not so wellnot so wellParticipants talk about what was going on Participants talk about what was going on inside their headsinside their headsParticipants learn from each otherParticipants learn from each other

Decision Making ReviewDecision Making ReviewWhat What

Happened?Happened?(situation (situation ––

cues cues -- patterns)patterns)

What We Did?What We Did?(course(s) of (course(s) of

action)action)

Why?Why?(goal, intent)(goal, intent)

What What Other Other OptionsOptions Did Did

You Consider? You Consider? Why Not Used?Why Not Used?

What Would What Would You Do You Do

Differently Next Differently Next Time?Time?

What were the really tough What were the really tough decisions?decisions?

For each decision:For each decision:Why was this difficult?Why was this difficult?Why did you choose that course of action?Why did you choose that course of action?In hindsight, what cues and patterns should In hindsight, what cues and patterns should you have picked up on?you have picked up on?What one piece of missing information would What one piece of missing information would have helped you the most?have helped you the most?What other actions did you consider?What other actions did you consider?Why did you not choose them?Why did you not choose them?

What cues did you consider when you made this decision?What cues did you miss? Why?What were you thinking about when you made this decision?What information did you use to make the decision?What made you realize this decision had to be made?

Cues and FactorsCues and Factors

What other options did you consider before choosing your course of action?Looking back on the situation, what other options could you have considered?Why did you not choose those other options?

Other OptionsOther Options

Where did you get the information Where did you get the information that helped you make this decision?that helped you make this decision?What information sources did you What information sources did you miss? Why?miss? Why?What other sources were available to What other sources were available to you for critical information (other team you for critical information (other team members, individuals outside the members, individuals outside the team, technologies and mechanical team, technologies and mechanical indicators, and even tools like maps indicators, and even tools like maps or diagrams)?or diagrams)?

Information SourcesInformation Sources

General discussion at the end:General discussion at the end:

From what youFrom what you’’ve learned here, what ve learned here, what would you do differently if you were in a would you do differently if you were in a similar situation again?similar situation again?What did your team have going for them? What did your team have going for them? What was going against them?What was going against them?What will you pay attention to next time to What will you pay attention to next time to help you with this decision?help you with this decision?What are some important lessons learned What are some important lessons learned from this exercise?from this exercise?

Decision Making Review(A Facilitator’s tool for use during DMGs)

• What were the tough decisions?• For each situation assessment:

– What was your initial assessment of the situation?– What specific ques did you use in your assessment?– What ques or factors were missed? Why?– What made the assessment difficult?– What assumptions did you have to make? Why?– What was your greatest uncertainty?– How did you expect the situation to develop?– What impact did time have on your assessment?

• For each course of action:– What was your intent in choosing that course of action?– How did your intention change over time? Why?– What other options did you consider? – Why did you not choose them?

• Summary at the end:– What are some important lessons learned?– How might a novice (or expert) misdiagnose this event?– What would have made this a more challenging scenario?

Decision-Making ReviewBriefly describe the situation:

Initial cues/patterns?Cues/patterns missed? Why?Expectations for what next?Greatest uncertainty?

For each course of action:What was your intent?How did your intent change?Other factors influencing you?Other options considered?

Summary:Lessons learned?