www.kse.org.ua

Evidence Based ManagementTom Coupé

Kyiv School of Economics

Jeffrey Pfeffer and Robert Sutton (Stanford)

Basic Idea• Decisions based on

• Sound analysis• Data• Research

• Not on• Conventional wisdom• Tradition• Expert opinion.

==> a manager should also be able to think like a researcher

An Example from Medecine

An Example from Business

Pop Tarts

Data Driven Decision Making and Ukraine • Algorithmic trading at major investment bank• Banks and CRM• Retail: loyalty programs• Telecommunication• “We are hiring a Business Insights Manager (BIM) with business analytics being a key source of growth”. (Big Fast Food Chain)

Why does not everybody practice evidence-based management?

• Cognitive Biases

• Changing Power Dynamics

• The market for ideas is messy

Why does not everybody practice evidence-based management? Cognitive Biases• Intuitive Thinking versus reflective thinking (Thaler and Sunstein, Nudge – Kahneman, Thinking Slow and Fast)

Biased decision making

• Confirmation bias  ‘is a tendency to search for or interpret information in a way that confirms one's preconceptions, leading to statistical errors’ (Wikipedia)

• Status Quo bias: ‘a preference for the current state of affairs. The current baseline (or status quo) is taken as a reference point, and any change from that baseline is perceived as a loss’ (Wikipedia)

Behavioral Biases in Investment Decision MakingGervais, 2009

• Overconfidence: managers overestimate the precision of their knowledge and information

Overconfidence distorts investment decisions

Measuring Overconfidence• Malmendier and Tate (2005a, 2008) - CEO own wealth decisions

• Malmendier and Tate (2005b, 2008) - popular press • Ben-David et al. (2008) - forecast of returns • Lin, Hu, and Chen (2005) - forecasts of company earnings

Overconfidence and Investments• Firms that have more cash and rely less on debt financing tend to invest more than other firms, keeping investment opportunities fixed.• More so with overconfident managers?

• Malmendier and Tate (2005a): yes• Glaser, Schäfers, and Weber (2008): not for CFOs, yes for boards.

Overconfidence and Mergers• Malmendier and Tate (2008):

• P(acquisition): +65%• three-day abnormal return: -0.90 percent vs. -0.29 percent)

• $2 to 4 billion loss • Ben-David et al. (2008): overconfident CFOs

• More acquisitions • 1.3 percent lower returns 

Overconfidence and Entrepreneurs• Cooper et al. (1988)

• 35 percent of entrepreneurs say 100 percent probability success

• half the ventures end up failing.

Why does not everybody practice evidence-based management? Changing Power Dynamics

• Authority versus Knowledge

Why does not everybody practice evidence-based management? Too much knowledge

Why does not everybody practice evidence-based management?

• Cognitive Biases• Changing Power Dynamics• The market for ideas is messy

• Next: How to be an evidence-based manager?

How to be an evidence based manager?• 1. Formulate Answerable Questions• 2. Search for the best available evidence• 3. Evaluate the evidence• 4. Integrate this relevant evidence with your managerial experience and organizational concerns. If possible, generate evidence within your context.

• 5. Apply

How to be an evidence based manager?• 1. Formulate Answerable Questions• 2. Search for the best available evidence• 3. Evaluate the evidence• 4. Integrate this relevant evidence with your managerial experience and organizational concerns. If possible, generate evidence within your context.

• 5. Apply

Formulating Answerable Questions• PICOC• Problem or Population – who?• Intervention – what/how?• Comparison – compared to?• Outcome – what outcome?• Context – in what environment?

• Example: I am a consultant who advices a health care company that plans to merge with another company that has a different corporate culture – I wonder whether this difference in corporate culture will affect the outcome of the merger? (based on CEBMA)

To be answerable, you need to know• P: What kind of Population are we talking about? Middle managers, back-office employees, medical staff, clerical staff?

• P/C: And how is the assumed cultural difference assessed? Is it the personal view of some managers or is it measured by a validated instrument?

Extra Info Provided• According to the board the objective of the merger is to integrate the back-office of the two health care organizations (ICT, finance, purchasing, facilities, personnel administration, etc.) in order to create economy of scale. The front offices and primary process of the two organizations will remain separate.

• The cultural difference is not objectively assessed (it is the perception of the senior managers of both organizations).

Already better• P =back office employees • I = merger, integration back office • C = status quo• O = economy of scale • C = healthcare

Answerable Question• P = back office employees • I = merger, integration back office • C = status quo• O = economy of scale

• Employee productivity?• Job satisfaction?• Return on investment? • Market share?• Organizational commitment?

• C = healthcare

How to be an evidence based manager?• 1. Formulate Answerable Questions• 2. Search for the best available evidence• 3. Evaluate the evidence• 4. Integrate this relevant evidence with your managerial experience and organizational concerns. If possible, generate evidence within your context.

• 5. Apply

Searching for evidence• Own Experience+• Experience of Competitors+• Academic literature• Case studies • News • Google

How to be an evidence based manager?• 1. Formulate Answerable Questions• 2. Search for the best available evidence• 3. Evaluate the evidence• 4. Integrate this relevant evidence with your managerial experience and organizational concerns. If possible, generate evidence within your context.

• 5. Apply

Evaluate the evidence• Know analytical tools and concepts

• Econometrics, statistics!• Criteria to evaluate information

• Causality• Sample Size• Sample Selection• Explicit about shortcomings

Causality vs Correlation

The Simpsons: "Much Apu About Nothing“Homer: Not a bear in sight. The

"Bear Patrol" is working like a charm!

Lisa: That's specious reasoning, Dad.

Homer: [uncomprehendingly] Thanks, honey.

Lisa: By your logic, I could claim that this rock keeps tigers away.

Homer: Hmm. How does it work? Lisa: It doesn't work; it's just a

stupid rock! Homer: Uh-huh. Lisa: But I don't see any tigers

around, do you? Homer: (pause) Lisa, I want to

buy your rock.

Possible Explanations for Correlation

Causality versus correlation

•Bain: ‘our clients outperform the market 3 to 1’

Big Samples

•How do Belgians look like?• ‘case studies’

Non-Random Samples - Selection• Selection by Researcher (reporter)

• ‘In Search of Excellence: Lessons from America’s Best-Run Companies’

• Self-Selection of Respondents

How to be an evidence based manager?• 1. Formulate Answerable Questions• 2. Search for the best available evidence• 3. Evaluate the evidence• 4. Integrate this relevant evidence with your managerial experience and organizational concerns. If possible, generate evidence within your context.

• 5. Apply

Exercise – what is the best book title?• Super Crunchers – why thinking by numbers is the new way to be smart

• The End of Intuition - why thinking by numbers is the new way to be smart

How to be an evidence based manager?• 1. Formulate Answerable Questions• 2. Search for the best available evidence• 3. Evaluate the evidence• 4. Integrate this relevant evidence with your managerial experience and organizational concerns. If possible, generate evidence within your context.

• 5. Apply

Google’s Quest to Build a better Boss• What makes a good manager?• 10000 observations 8 rules • Trade off ‘technical expertise’ vs ‘being accessible’

• Improved performance: 75% of worst performing managers significantly improved

Google’s Research on Successful teams• “Over two years we conducted 200+ interviews with Googlers

(our employees) and looked at more than 250 attributes of 180+ active Google teams. … Who is on a team matters less than how the team members interact, structure their work, and view their contributions.

• Psychological safety: Can we take risks on this team without feeling insecure or embarrassed?

• Dependability: Can we count on each other to do high quality work on time?

• Structure & clarity: Are goals, roles, and execution plans on our team clear?

• Meaning of work: Are we working on something that is personally important for each of us?

• Impact of work: Do we fundamentally believe that the work we’re doing matters?”

Evidence Based Management and Organizational Design• Be committed to "fact based" decision making• build a culture in which people are encouraged to tell the truth

• encourage experimentation and learning by doing.• Look for the risks and drawbacks in what people recommend 

Exercise – bank deposits• You are the CEO of a bank and you want to increase the amount of money clients deposit on their savings account – what do you do?

Exercise – Glass Windows• You are the CEO of a company that installs glass windows – your employees all are paid a fixed wage. Should you switch to a flexible wage, where wage and performance are linked?

Important Lessons• 1. Decision making should be based on analysis• 2. Use the criteria to evaluate knowledge• 3. Use own data whenever possible • 4. Design your organization in such a way that 1 becomes possible

The Best Question After this Talk?• Capgemini Survey - on average, business performance improvements of 26% in processes where big data analytics have been applied

• Brynjolfson et al (2011): ‘We examine whether firms that emphasize decision making based on data and business analytics (“data driven decision making” or DDD) show higher performance. Using detailed survey data on the business practices and information technology investments of 179 large publicly traded firms, we find that firms that adopt DDD have output and productivity that is 5-6% higher than what would be expected given their other investments and information technology usage. Furthermore, the relationship between DDD and performance also appears in other performance measures such as asset utilization, return on equity and market value.’

Conclusion I• Practice Evidence Based Management

• Decision making should be based on analysis• Use the criteria to evaluate knowledge• Use own data whenever possible • Design your organization in such a way that 1 becomes possible

Conclusion II• Who Will Benefit?

• Companies that create value from their data• Companies that help business to create value from their data

• Those who combine managerial skills with an understanding of data analytics!