Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making
-
Upload
indeedeng -
Category
Technology
-
view
730 -
download
0
Transcript of Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making
![Page 1: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/1.jpg)
Data-Driven Off a CliffAnti-patterns in evidence-based decision making
Ketan Gangatirkar & Tom Wilbur
![Page 2: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/2.jpg)
Data-Driven Off a CliffAnti-patterns in evidence-based decision making
Ketan Gangatirkar & Tom Wilbur
![Page 3: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/3.jpg)
I helppeopleget jobs.
![Page 4: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/4.jpg)
Indeed is the #1 job site worldwide
![Page 5: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/5.jpg)
![Page 6: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/6.jpg)
Headquartered in Austin, Texas
![Page 7: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/7.jpg)
We have tons of ideas
![Page 8: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/8.jpg)
We have tons of bad ideas
![Page 9: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/9.jpg)
Occasionally, we have good ideas
![Page 10: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/10.jpg)
It’s hard to tell the difference
![Page 11: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/11.jpg)
What helps people get jobs?
![Page 12: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/12.jpg)
The only reliable way is to see what works
![Page 14: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/14.jpg)
We set up experiments
![Page 15: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/15.jpg)
We collect results
![Page 16: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/16.jpg)
We use the data to decide what to do
![Page 17: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/17.jpg)
We’ve used data to make good decisions
![Page 18: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/18.jpg)
But having data is not a silver bullet
![Page 19: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/19.jpg)
We’ve also used data to make bad decisions
![Page 20: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/20.jpg)
Science is hard
![Page 21: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/21.jpg)
![Page 22: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/22.jpg)
ProblemRunning an experiment can ruin the experiment
![Page 24: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/24.jpg)
![Page 25: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/25.jpg)
Change Effect on productivity
Brighter light UP
Dimmer light UP
Warmer UP
Cooler UP
Shorter breaks UP
Longer breaks UP
![Page 26: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/26.jpg)
Change Effect on productivity
Brighter light UP (temporarily)
Dimmer light UP (temporarily)
Warmer UP (temporarily)
Cooler UP (temporarily)
Shorter breaks UP (temporarily)
Longer breaks UP (temporarily)
![Page 27: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/27.jpg)
Change Effect on productivity
Brighter light UP (temporarily)
Dimmer light UP (temporarily)
Warmer UP (temporarily)
Cooler UP (temporarily)
Shorter breaks UP (temporarily)
Longer breaks UP (temporarily)
![Page 28: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/28.jpg)
ProblemStatistics are hard
![Page 30: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/30.jpg)
Simpson’s Paradox
![Page 32: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/32.jpg)
Using data is more than just statistics
![Page 33: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/33.jpg)
+ + + +
=
Good math. Bad idea.
![Page 34: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/34.jpg)
Bad practices can undermine good math
![Page 35: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/35.jpg)
You don’t need me to teach youto be bad at math
![Page 36: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/36.jpg)
I’ll teach you to be bad at everything else
![Page 37: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/37.jpg)
Anti-Lesson 01
Be impatient
![Page 38: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/38.jpg)
p-value is the standard measure ofstatistical significance
![Page 39: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/39.jpg)
p-value is by measurement, not experiment
![Page 40: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/40.jpg)
If you check results on Monday,that’s one measurement
![Page 41: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/41.jpg)
If you check results on Tuesday,that’s another measurement
![Page 42: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/42.jpg)
![Page 43: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/43.jpg)
Got the result you want?
![Page 44: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/44.jpg)
Declare victory!
![Page 45: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/45.jpg)
Move quickly! Becauseresults and p-values can shift fast
![Page 46: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/46.jpg)
of “winning” A/B tests stopped earlyare false-positives
80%
http://bit.ly/1LtaLkV
![Page 47: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/47.jpg)
Anti-Lesson 02
Sampling is easy
![Page 48: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/48.jpg)
Beware the IEdes of MarchStory
![Page 49: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/49.jpg)
Building Used Cars Search
![Page 50: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/50.jpg)
Shoppers specifying price, mileageor year do better
![Page 51: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/51.jpg)
Nudge shoppers to specify price,mileage or year
![Page 52: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/52.jpg)
![Page 53: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/53.jpg)
+3% conversion
![Page 54: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/54.jpg)
After rollout, conversion > +3%
![Page 55: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/55.jpg)
Why?
![Page 56: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/56.jpg)
We’d taken a shortcut in ourtest assignment code
X
![Page 57: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/57.jpg)
Users on oldest browsers got ignored
![Page 58: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/58.jpg)
Distorted sample Distorted results
![Page 59: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/59.jpg)
Anti-Lesson 03
Look only at one metric
![Page 60: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/60.jpg)
If a little bit is good, a lot is great
![Page 61: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/61.jpg)
![Page 62: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/62.jpg)
Indeed has a heartStory
![Page 63: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/63.jpg)
![Page 64: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/64.jpg)
![Page 65: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/65.jpg)
❤ > ★ ?
![Page 66: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/66.jpg)
+16% Saves on search results page
![Page 67: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/67.jpg)
Everyone ❤s ❤s!
![Page 68: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/68.jpg)
❤s everywhere!
![Page 69: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/69.jpg)
![Page 70: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/70.jpg)
![Page 71: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/71.jpg)
Hearted
![Page 72: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/72.jpg)
![Page 73: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/73.jpg)
Not so fast
![Page 74: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/74.jpg)
Did ❤ help people get jobs?
![Page 75: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/75.jpg)
❤ jobs: +16%Clicks: no change
Applies: no changeHires: no change
![Page 76: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/76.jpg)
I helppeople❤ jobs.
![Page 77: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/77.jpg)
Upsell teamStory
![Page 78: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/78.jpg)
We formed an “upsell team”and measured their results
![Page 79: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/79.jpg)
+ =
Success measure
![Page 80: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/80.jpg)
It’s working!Upsells
![Page 81: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/81.jpg)
So why isn’t revenue moving?Overall Revenue
![Page 82: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/82.jpg)
+ 0 -
![Page 83: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/83.jpg)
= ⅓+⅓
-⅓
![Page 84: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/84.jpg)
What you measure is what you motivate
![Page 85: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/85.jpg)
Redefine success to include all outcomes
![Page 86: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/86.jpg)
Upsell Team revenue +200%
![Page 87: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/87.jpg)
Anti-Lesson 03: Reloaded
Look at all the metrics
![Page 88: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/88.jpg)
![Page 89: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/89.jpg)
![Page 90: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/90.jpg)
It's better for them. Is it better for us?
![Page 91: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/91.jpg)
Job applications: UpJob clicks: Down
Recommended Jobs traffic: UpJob views: Sideways
New resumes: UpReturn visits: Down
Logins: UpRevenue: Down
(and it goes on…)
![Page 92: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/92.jpg)
We didn’t really know what we wanted
![Page 93: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/93.jpg)
Too much noise from too many metrics
![Page 94: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/94.jpg)
![Page 95: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/95.jpg)
I helppeopleget jobs.
![Page 96: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/96.jpg)
Anti-Lesson 04
Be sloppy with your analysis
![Page 97: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/97.jpg)
We engineer features rigorously
![Page 98: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/98.jpg)
SpecificationSource control Code review
Automated testsManual QA
MetricsMonitors
...
![Page 99: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/99.jpg)
But analysis…
![Page 100: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/100.jpg)
Bad analysis won’t take down Indeed.com
![Page 101: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/101.jpg)
200 million job seekers don’t careabout our sales projections
![Page 102: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/102.jpg)
So we don’t try as hard with analysis code
![Page 103: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/103.jpg)
SpecificationSource control Code review
Automated testsManual QA
MetricsMonitors
...
![Page 104: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/104.jpg)
DublinersStory
![Page 105: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/105.jpg)
Indeed reports on economic trends
![Page 106: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/106.jpg)
South Carolinians wanted to move to Dublin
![Page 107: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/107.jpg)
Dublin?
![Page 108: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/108.jpg)
No, the other one
![Page 109: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/109.jpg)
Incorrect IP location mapping
![Page 110: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/110.jpg)
IP blocks for South Carolinagot reallocated to London, England
![Page 111: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/111.jpg)
Worse things can happen
![Page 112: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/112.jpg)
Growth and DebtStory
![Page 113: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/113.jpg)
“Growth in a Time of Debt”Carmen Reinhart and Kenneth Rogoff
2010
![Page 114: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/114.jpg)
Public debt > 90% GDPleads to slower economic growth
![Page 115: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/115.jpg)
Governments made policy based on this
![Page 116: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/116.jpg)
![Page 118: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/118.jpg)
Genetic MutationStory
![Page 120: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/120.jpg)
SEPT2 to a geneticist is Septin 2
![Page 121: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/121.jpg)
SEPT2 to Excel is 42615
![Page 122: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/122.jpg)
![Page 123: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/123.jpg)
Does your company use spreadsheets?
![Page 124: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/124.jpg)
How do you know they’re correct?
![Page 125: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/125.jpg)
Under-spending Advertisers
Story
![Page 126: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/126.jpg)
Employer budgets ran outbefore the end of the day
![Page 127: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/127.jpg)
So no evening job seekers saw the jobs
![Page 128: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/128.jpg)
How big was this missed opportunity?
![Page 129: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/129.jpg)
Clicks received1260Out of budget time20:00
% of day w/o budget0.1667
Potential clicks1260 / (1 - 0.1667) = 1512Missed clicks1512 * 0.1667 = 260
Missed Clicks Report
Dear Customer,
You got 1,260 clicks yesterday.Your daily budget ran out at 8:00pm.
If you funded your budget through the whole day, you’d get another260 clicks - a +20% improvement!Get More Clicks
![Page 130: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/130.jpg)
Assumption
100
75
50
25
00:00 2:00 4:00 6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00
Missed = 260 clicks (+20%)
0:00
![Page 131: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/131.jpg)
Reality
100
75
50
25
0
Missed = 100 clicks (+8%)
2:00 4:00 6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 0:000:00
![Page 132: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/132.jpg)
Naive analysis bad recommendation
![Page 133: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/133.jpg)
Anti-Lesson 05
Only look for expected outcomes
![Page 134: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/134.jpg)
Zero results pages from misspelled locations
![Page 135: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/135.jpg)
![Page 136: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/136.jpg)
![Page 137: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/137.jpg)
Goals: fewer ZRPs, more job clicks
![Page 138: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/138.jpg)
Zero-results pages
-2.7%
![Page 139: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/139.jpg)
Job clicks
+8%
![Page 140: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/140.jpg)
+1,410%
Ad revenue
![Page 141: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/141.jpg)
![Page 142: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/142.jpg)
+1,410%
Ad revenue
![Page 143: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/143.jpg)
![Page 144: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/144.jpg)
ads
![Page 145: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/145.jpg)
![Page 146: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/146.jpg)
![Page 147: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/147.jpg)
![Page 148: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/148.jpg)
![Page 149: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/149.jpg)
Ad revenue after fix
![Page 150: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/150.jpg)
Treatment onhomepage
Effect onsearch page
![Page 151: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/151.jpg)
Anti-Lesson 06
Metrics, not stories
![Page 152: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/152.jpg)
I helppeopleget jobs.
![Page 153: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/153.jpg)
How do I know if people got jobs?
![Page 154: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/154.jpg)
I need employers to tell me
![Page 155: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/155.jpg)
![Page 156: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/156.jpg)
One employer hired 4500 people in 45 minutes!
![Page 157: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/157.jpg)
![Page 158: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/158.jpg)
Nope
![Page 159: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/159.jpg)
Accurate recording of outcomes helps us
![Page 160: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/160.jpg)
It doesn’t help employers
![Page 161: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/161.jpg)
They don't care about usingthe product “right”
![Page 162: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/162.jpg)
Go away!
![Page 163: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/163.jpg)
There is no “user story”
![Page 164: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/164.jpg)
Right metrics + wrong story = wrong conclusion
![Page 165: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/165.jpg)
Anti-Lesson 06: Parte Deux
Story over metrics
![Page 166: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/166.jpg)
Stories are seductive
![Page 167: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/167.jpg)
Even incorrect stories are seductive
![Page 168: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/168.jpg)
Taste BudsStory
![Page 169: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/169.jpg)
Taste map
![Page 170: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/170.jpg)
Totally wrong
![Page 171: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/171.jpg)
Every bite you eat proves it’s wrong
![Page 172: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/172.jpg)
People still believe it
![Page 173: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/173.jpg)
Job AlertsStory
![Page 174: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/174.jpg)
![Page 175: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/175.jpg)
Success for emails is well understood
![Page 176: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/176.jpg)
New subscriptions: GoodEmail opens: Good
Clicking on stuff: GoodUnsubscribing: Bad
![Page 177: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/177.jpg)
I helppeopleget emails.
![Page 178: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/178.jpg)
I helppeopleget jobs.
![Page 179: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/179.jpg)
What does job seeker success look like?
![Page 180: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/180.jpg)
01Search for jobs
![Page 181: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/181.jpg)
02Sign up for alerts
![Page 182: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/182.jpg)
03Click on some jobs
![Page 183: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/183.jpg)
04Apply to some jobs
![Page 184: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/184.jpg)
05Get a job!
![Page 185: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/185.jpg)
06Unsubscribe from emails
![Page 186: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/186.jpg)
People with new jobs don't need job alerts
![Page 187: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/187.jpg)
The standard story for email fails here
![Page 188: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/188.jpg)
Light and Dark ReduxStory
![Page 189: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/189.jpg)
![Page 190: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/190.jpg)
It’s a persuasive story
![Page 191: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/191.jpg)
But the original study was flawed
![Page 192: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/192.jpg)
Hawthorne Revisited
… the variance in productivity could be fully accounted for by the fact that the lighting changes were made on Sundays and therefore followed by Mondays when workers’ productivity was refreshed by a day off.”
https://en.wikipedia.org/wiki/Hawthorne_effect
![Page 193: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/193.jpg)
We con people with stories
![Page 194: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/194.jpg)
We con ourselves with stories
![Page 195: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/195.jpg)
Anti-Lesson 07
Believe in yourself
![Page 196: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/196.jpg)
Believing in yourself can be good
![Page 197: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/197.jpg)
“My startup will succeed.”
![Page 198: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/198.jpg)
Often it’s bad
![Page 199: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/199.jpg)
“I’d never fall for a scam like that.”
![Page 200: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/200.jpg)
“I knew it all along.”
![Page 201: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/201.jpg)
“I’m too smart to make that mistake.”
![Page 202: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/202.jpg)
Every story of mistakes is deceptive
![Page 203: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/203.jpg)
We tell stories with 20/20 hindsight
![Page 204: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/204.jpg)
When we live the story, we live in the fog
![Page 205: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/205.jpg)
You won’t think you’re making a mistake
![Page 206: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/206.jpg)
Search your past for mistakes
![Page 207: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/207.jpg)
Painful, embarrassing mistakes
![Page 208: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/208.jpg)
If you didn’t find any, you’re exceptional
![Page 209: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/209.jpg)
Either you’re making mistakes you find
![Page 210: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/210.jpg)
Or you’re making mistakes you don’t find
![Page 211: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/211.jpg)
How do you defend against mistakes?
![Page 212: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/212.jpg)
The first step is admitting you have a problem
![Page 213: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/213.jpg)
There are 174 cognitive biases[citation needed]
![Page 214: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/214.jpg)
Data can help you make better decisions
![Page 215: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/215.jpg)
Or more confidently make bad decisions
![Page 216: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/216.jpg)
Data can’t make you a better decision-maker
![Page 217: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/217.jpg)
Good data + bad decision-maker = bad decision
![Page 218: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/218.jpg)
Our anti-lessons teach youhow to use data badly
![Page 219: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/219.jpg)
Do the opposite to do better
![Page 220: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/220.jpg)
Lesson 01Lesson 02Lesson 03Lesson 04Lesson 05Lesson 06Lesson 07
Be patientSampling is hardFocus on a few, carefully chosen metricsBe rigorous with your analysisWatch out for side effectsUse metrics and storiesPlan for fallibility
![Page 221: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/221.jpg)
Learn from our mistakes
![Page 222: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/222.jpg)
Be prepared for your own
![Page 223: Data-Driven off a Cliff: Anti-Patterns in Evidence-Based Decision Making](https://reader035.fdocuments.us/reader035/viewer/2022062903/58ed1a111a28ab8e208b469d/html5/thumbnails/223.jpg)
Learn MoreEngineering blog & talks http://indeed.tech
Open Source http://opensource.indeedeng.io
Careers http://indeed.jobs
Twitter @IndeedEng