Post on 12-May-2015
description
Evaluation in support of Strategic Learning
Principles of Practice
Tanya BeerCenter for Evaluation Innovation
Strategic Learning ≠ Evaluation
Strategic learning is the use of data and insights from a variety of information-gathering approaches—including evaluation—to inform decisions about strategy.
What counts as “strategy” and how is it created, executed, and adapted?
Challenge 1
Strategic learning is the use of data and insights from a variety of information-gathering approaches to inform decisions about strategy.
What kind of data and intelligence are meaningful for informing strategic decisions?
Challenge 2
Strategic learning is the use of data and insights from a variety of information-gathering approaches to inform decisions about strategy.
Strategic learning is the use of data and insights from a variety of information-gathering approaches to inform decisions about strategy.
How do organizational and group culture, structures, and process support or hinder reflection and the use of data?
Challenge 3
Strategic learning is the use of data and insights from a variety of information-gathering approaches to inform decisions about strategy.
What counts as “strategy” and how is it created, executed, and adapted
What kind of data and intelligence are meaningful for informing strategic decisions?
How do organizational and group culture, structures, and process support or hinder reflection and the use of data?
How do we design and practice evaluation to support strategic learning?
Principles of Evaluation For Strategic Learning
PRINCIPLE 1: Evaluation is positioned as a support for strategy and is seated at the strategy table
PRINCIPLE 2: Evaluation is integrated into initiative development and management and conducted in partnership.
Understanding Strategy
PRINCIPLE 3: Data to inform strategy can come from a wide variety of sources and methods.
PRINCIPLE 4: Evaluation emphasizes context.
PRINCIPLE 5: Evaluation is flexible and timely, and ready for the unexpected.
Principles of Evaluation For Strategic LearningCollecting meaningful data at the right time
Principles of Evaluation For Strategic Learning
PRINCIPLE 6: Program implementers place a high value on evaluation use, and evaluators help to support it.
PRINCIPLE 7: Evaluation must take place within a culture that encourages risk taking, learning, and adaptation.
PRINCIPLE 8: Evaluation is constructivist.
Understanding the drivers of data use
“You don’t plan a strategy, you learn a strategy.”Henry Mintzberg
What will it take for us to make use of data and other intelligence in our strategic decision-making
throughout the lifecycle of a strategy or program?
Tell a story about a time when your team either failed or succeeded at incorporating data or other intelligence into a strategic decision.
Rules: Talk about actual past situations, events, or results. No abstract concepts, interpretations or editorializing,
just what happened!! Tell stories about both successes and failures Listen carefully for (but don’t comment on) similarities,
differences between stories.
20 minutes to tell at least 4 stories.
Groundtruths
Insights
What patterns do you see in the stories?
What insights can we generate about the relationship between past results and what caused them?
Rules: Link insights clearly to groundtruth data (stories)--no ungrounded
assertions!
One person writes insights on a sheet of paper, and another records key insights on stickies by color:
25 minutes to generate insights and record them on stickies
CULTURE/PROCESS
DATA/EVALUTION
STRATEGY/PROGRAMS
OTHER
BREAK 10 minutes
Bring your stickies to us BEFORE you break!
Beginning at 11am, you will have 10 minutes to browse the gallery wall of insights. Make note of
the most compelling ones!
Quickly review the insights you listed and add any others you noticed from the gallery wall.
Given these insights, what hypotheses do you have about what it’s going to take to make better use of data and other intelligence in our strategic decision-making throughout the lifecycle of a strategy or program?
Rules: Use this structure: “In [situation X], if we do [action Y], then
we will produce [result Z].” Avoid vague statements (e.g., “build trust”) and push each
other to instead offer specific actions. One person records hypotheses and the group picks top 3.
20 minutes to generate hypotheses and pick top 3 most compelling.
Hypotheses
Give us your best hypotheses
Most doableMost impactfulMost creative
What will it take for us to make use of data and other intelligence in our strategic decision-making
throughout the lifecycle of a strategy or program?
IN PAIRS, describe upcoming opportunities in your work where you will test 2 or 3 hypotheses.
What is the actual meeting or event where a decision will be made?
What kind of decision(s) will be on the table? What kind of data will the group need to consider? How will you test a hypotheses to improve the group’s
application of the data to the decision?
Rules: Opportunities must be real things on your calendar.Start with what exists already and add new events only when
absolutely necessary.Help your partner think through how to make it happen.
Opportunities
So what was that like?
Act
Observe
ReflectAdjust
Plan
We are pretty good at this
THE WEAK LINK
If we take time, we can do well at this,
too
Action Learning Cycle
From Fourth Quadrant Partners, LLC
Emergent Learning™ Table
Framing Question: What will it take to…? How can we…?
Ground Truth
Insights
Opportunities
Hypotheses
Translate past lessons into a
theory of success
Recallrelevant
experiences and examine data
Learn from experience &
data
Apply new theory to
opportunities
From Fourth Quadrant Partners, LLC
PAST FUTURE
DOING
THINKING
“The capacity to adapt is higher level and of greater value than any specific ‘lesson learned’.”
“Learning in the Thick of It,” HBR
Let’s Eat!
Teams of 4-6
18 minutes
Tallest structure
GOAL: Build the tallest freestanding structure (measured from the table top surface to the top of the marshmallow)
✦The ENTIRE MARSHMALLOW MUST BE ON TOP (no eating it)
✦Use as much or as little of the kit (spaghetti, tape, string, marshmallow
✦You cannot use the paper bag as part of your structure.
✦You can break the spaghetti, rip the tape and string to create new structures.
✦ You cannot hold on to the structure when the time runs out. Those touching or supporting the structure at the end of the exercise will be disqualified.
18 minutes
Where Cognitive Traps Trip Us Up:Removing the Road Blocks to Learning
A bat and ball cost a dollar and ten cents.
The bat costs a dollar more than the ball.
How much does the ball cost?
Source: Shane Frederick
Ball = Bat = 10 cents One dollar
Ball =
5 cents
Bat = + One dollar
5 cents
Why are we talking about this?
You deal with complexity and uncertainty.
How can we help metropolitan regions shape the opportunities available to low-income families for access to good quality jobs
and services, and for development of their assets, particularly quality housing?
How can we reverse the childhood
obesity epidemic by
2015?
How can we achieve high-quality preschool for all 3- and 4-year-olds in our
state??
And you try to reduce or manage it (I said try).
31
Strategies and theories of change help to manage
complexity and uncertainty, but they
tend to be more descriptive than
prescriptive.
32
So evaluation could be enlisted to help.
How can we take advantage of the evolving
political climate?
What is motivating the actors and how do we
reinforce desired behaviors?
How should assumptions about the true cost of
implementation be adjusted?
But unfortunately, we are not rational.
We are hardwired to use heuristics and biases— cognitive traps—when processing information.
They affect:
How we access information
What we pay attention to
What we learn
How we apply it
We need to finish the equation.
Ask the
right questions and get the right
data
Structure the work to enable
regular use of data
+
Effectively process and use the data
+
What are cognitive traps?
36
System 1: Rapid intuitive decisions based on associative memory, images, and emotional reactions
System 2: Monitors the output of System 1 and overrides it when the result conflicts with logic, probability, or some other decision-making rule
Our minds have two processing systems.
The two systems act differently.
System 1 System 2
System 2 is lazy and trips us up, especially when we are busy.
There are all sorts of cognitive traps.
We are focusing on a few relevant to philanthropy and evaluation.
Decision Making
• Anchoring/adjustment
• Conjunction fallacy
• Focusing effect
• Overconfidence effect
• Curse of knowledge
• Ease of recall
• Planning fallacy
Social
• Actor-observer bias
• False consensus effect
• Fundamental attribution error
• Halo effect
• In-group bias
• Projection bias
• Self-serving bias
Memory
• Primacy effect
• Recency effect
• Serial position effect
• Google effect
• Hindsight bias
• Illusory correlation
• Egocentric bias
The availability heuristic makes us overestimate what comes to mind the quickest.
Probability of a real threat?
Confirmation bias makes us favor information that confirms our pre-existing beliefs.
State polls showing consistent leads in key battleground states
Late-October national polls showing gains
Hurricane Sandy performance
More engaged and motivated voters
Large and enthusiastic crowds in swing states
Beliefs that national polls were skewed.
Internal polls showing leads in key states.
Nate Silver’s “Model” of all presidential polling
Escalation of commitment makes it difficult to revisit previous choices.
$2000
Tires
$300
Alternator$220
Water Pump
$150
The Texas sharpshooter fallacy causes us to mistake coincidence for meaningful pattern.
Use devil’s advocacy.
Which consulting firm is better?
Yes Man
All your ideas are
great!
Devil’s Advocate
Here are 10 reasons you
shouldn’t do that.
Remind yourself what you don’t know.
1. Tell me what you know.
2. Tell me what you don’t know.
3. Only then can you tell me what you think.
--Colin Powell
Compare and
contrast across silos.
45
Reason analogically.
Health Education Environment
Get an outsider’s view.
The outsider is more capable of
generalizing across situations, and makes better
estimates and decisions than
the insider.
Outsiders Insiders
47
Let’s put it all together and think about…
…the team meeting.
48
Who is at the table?
Devil’s Advocate
Outsider
49
What are you talking and thinking about?
Wait. What don’t we
know here?
Didn’t our other program area experience a
similar problem?
I disagree. I should probably
speak up.
I want to take over this meeting! But I must control
myself….
50
Flip-Chart Go-Round: Sidestepping Cognitive Traps
Chart 1: One person read aloud the definition of the cognitive trap.
In what kinds of situations or for what kinds of decisions are organizations are particularly vulnerable to the trap?
Chart 2: One person read aloud the definition of the cognitive trap AND the previous team’s responses to the question in round 1.
What concrete things can programmatic staff and leaders do to help sidestep this trap?
Chart 3: One person read aloud the definition of the cognitive trap AND the previous team’s responses to the question in round 1 & 2.
What concrete things can evaluators do to help sidestep this trap?
51
Flip-Chart Go-Round: Sidestepping Cognitive Traps
Chart 1: 10 minutes
1) One person read aloud the definition of the cognitive trap.
2) Discuss:
In what kinds of situations or for what kinds of decisions are organizations are particularly vulnerable to the trap?
3) Record your thoughts on a flipchart as concisely (and legibly) as possible.
52
Flip-Chart Go-Round: Sidestepping Cognitive Traps
Chart 2: 15 minutes
1) One person read aloud the definition of the cognitive trap AND the first group’s answers to the question in the last round.
2) Add additional insights to the first group’s list of types of situations or decisions where this trap is particularly relevant.
3) Discuss:
What concrete things can programmatic staff and leaders do to help sidestep this trap?
4) Record your thoughts on a flipchart as concisely (and legibly) as possible.
53
Flip-Chart Go-Round: Sidestepping Cognitive Traps
Chart 3: 20 minutes
1) One person read aloud the definition of the cognitive trap AND the previous group’s answers to the question in the last two rounds.
2) Add additional insights to the last group’s list of ideas about what concrete things programmatic staff and leaders can do to sidestep the trap.
3) Discuss:
What concrete things can evaluators do to help sidestep this trap?
4) BE READY TO REPORT OUT ON THE FULL CHART!
Where Cognitive Traps Trip Us Up:Removing the Road Blocks to Learning