The Human Side of Metrics

Dennis Opacki, CISSP QDSP

dopacki@covestic.com

Covestic, Inc.

Metricon 1.0

Vancouver, Canada

August 1, 2006

slide 2

How do organizations use metrics?

Most purposes cited reduce to: (Hauser & Katz 1998)

• Metrics enable firms to take stock of where they are, and help them plan for the future.

• Metrics provide estimates of future performance.

• Managers use metrics to allocate assets and select strategies.

• Metrics form the basis for bonuses and promotions.

Even measures derived from automated systems reflect and influence human behavior.

slide 3

Common metrics pitfalls

Goal displacement (Kerr, 1975)

• Means becomes the ends in themselves.

• People seek rewards to the exclusion of all non-rewarded behavior.

Unintended consequences• Metrics affect decisions even if those decisions inadvertently

sacrifice long-term benefits. (Hauser & Katz, 1998)

• Broken reward systems encourage behavior organizations are trying to discourage. (Kerr, 1975)

Passive or active resistance• Staff will find ways to avoid metrics they oppose. (Neal, 2006)

• Zone of indifference – “a person can and will accept a communication as authoritative only when… at the time of his decision, he believes it to be compatible with his personal interests as a whole.” (Kerr, 1975)

slide 4

Factors contributing to failure

Evolutionary psychology (Economist, 2005)

• Humans are hard-wired not for logic, but for detecting injustice.

• People will even forego their own self-interests to punish those behaving unfairly.

Social psychology (Neal, 2006)

• Locus of control • Internal – Sense of self-reliance – “master of own destiny”

• External –Sense of helplessness – external parties are in control

• Fundamental attribution errors – incorrectly assuming bad behavior is irrational or malicious

• Diffusion of responsibility – bystander effect• Everyone believes someone else is better suited to act.

• This can occur if responsible parties lacks authority to affect change.

• Reciprocity norm – What’s in it for me?

slide 5

Factors contributing to failure (cont.)

Behavioral Economics (Kahneman, 2003)

• Intuition vs. reason – people don’t think very hard• Intuition uses heuristics, prototypes and predictions – fast and parallel.

• Reason is slow and single-threaded.

• Natural attributes enhance intuition – size, color, valence.

• Good/bad can be substituted for any attribute.

• Default options have a natural advantage.

• Prospect theory• People anticipate emotions associated with changes in state.

• People are willing to ignore scope when presented with definite solutions to emotional problems.

• Framing• How we present situations greatly influences outcomes.

• Multiple frames increase the chance of specific outcomes.

slide 6

Improving chances for success

Select a few good metrics• Focus on attributes and scales that people gauge intuitively -

good/bad, like/dislike, percentile.• Use a small set of metrics to keep time pressure from driving

cognitive processes from reason to intuition. (Kahneman, 2003)• Survey employees to understand which behaviors are actually

being rewarded. (Hauser & Katz, 1998)• Express metrics in dollars; research shows this improves peoples’

ability to assess probability. (Kahneman, 2003)• Use clear metric names and descriptions to increase accessibility

of metrics’ traits. (Kahneman, 2003)

Present them well• Don’t ignore entertainment value. (Pijpers & Montfort, 2006)

• Drill-down is key to getting middle-management buy-in. (Gartner, 2005)• Slicing and dicing – bells and whistles

• Use common visualizations to improve mental prototyping.• Give bad news first - peak/end rule (Kahneman, 2003)

slide 7

Works Cited

[1] Buytendijk, F. and Gassman, B. (2005). Management Update: Just Give Me a CPM Dashboard. Gartner whitepaper.

[2] Hauser, J. R. and Katz, G. M. (1998) Metrics: You are what you measure! European Management Journal, 16(5).

[3] Kerr, Steven (1975).On the Folly of Rewarding A, While Hoping for B. Academy of Management Journal. 18(4).

[4] Kahneman, Daniel (2003). Maps of Bounded Rationality: Psychology for Behavioral Economics. The American Economic Review. 93(5).

[5] Neal, Russ. (2006). Social Psychology Variables that Contribute to Resistance to Security Assessment Findings. Information Systems Security, 15(1).

[6] No Author Cited, (2005). Survey: The concrete savannah. The Economist, 377(8458).

[7] Pjipers, G.G.M. and van Montfort, K. (2006). An Investigation of Factors that Influence Senior Executives to Accept Innovations in Information Technology. International Journal of Management. 23(1).