[@IndeedEng] Data-Driven Product Design

191
Data-Driven Product Design J Christopher Garcia Chris Hyams

TAGS:

description

@IndeedEnd April: Wednesday, April 24, 2013 Video available: http://youtu.be/Y6NI8y21xhg Advertising pioneer John Wanamaker famously said, "Half the money I spend on advertising is wasted; the trouble is I don't know which half." At Indeed, our experience shows that more than half of product features are not only wasted but actually have a negative impact on user experience. Thanks to rigorous A/B testing we know precisely which half, so we can ensure that every feature we add delivers measurable value. With 100 million monthly unique visitors and 3 billion monthly searches, our data provides empirical answers to (nearly) all of our product questions. In this talk, we will cover: * a motivation for why we A/B test everything * the tools and systems we have developed to measure and analyze test results and the key metrics we focus on, and * multiple case studies illustrating how years of testing has shaped Indeed's design and product thinking. We will include specific examples of how small, overlooked changes have impacted our metrics in significant and unexpected ways, and how implementing simple "no-brainer" updates on our site decreased key business metrics and caused us to re-evaluate our solutions. This talk is for designers, product managers, and engineers from companies of any size who want to understand how data-driven design makes better products. Speakers J Christopher Garcia is the User Experience Lead at Indeed. Chris ensures Indeed's products are easy to use for millions of people all over the world. He has focused on user experience at Indeed for over 6 years. As the company has grown from start-up underdog to the world's leading job search engine, he has contributed to every major product Indeed offers. Chris Hyams is SVP of Product & Engineering at Indeed. Prior to Indeed, Chris was founder of B-Side, an Austin-based company that provided digital distribution, marketing, and analytics technology for the film industry. Prior to B-Side, he was VP of Engineering at Trilogy Software.

Transcript of [@IndeedEng] Data-Driven Product Design

Page 1: [@IndeedEng] Data-Driven Product Design

Data-DrivenProduct Design

J Christopher GarciaChris Hyams

Page 2: [@IndeedEng] Data-Driven Product Design

DesignJ Christopher Garcia

User Experience Lead, Indeed

Page 3: [@IndeedEng] Data-Driven Product Design
Page 4: [@IndeedEng] Data-Driven Product Design
Page 5: [@IndeedEng] Data-Driven Product Design
Page 6: [@IndeedEng] Data-Driven Product Design

2008

Page 7: [@IndeedEng] Data-Driven Product Design

2013

Page 8: [@IndeedEng] Data-Driven Product Design
Page 9: [@IndeedEng] Data-Driven Product Design
Page 10: [@IndeedEng] Data-Driven Product Design
Page 11: [@IndeedEng] Data-Driven Product Design

114 tests244 variations

84,000 combinations

Page 12: [@IndeedEng] Data-Driven Product Design
Page 13: [@IndeedEng] Data-Driven Product Design
Page 14: [@IndeedEng] Data-Driven Product Design
Page 15: [@IndeedEng] Data-Driven Product Design

Direct observation can be inefficient

Page 16: [@IndeedEng] Data-Driven Product Design

Before and afterdoesn't work

Page 17: [@IndeedEng] Data-Driven Product Design

Traffic doesn't look like this

Page 18: [@IndeedEng] Data-Driven Product Design

Hourly traffic looks like this...

Page 19: [@IndeedEng] Data-Driven Product Design

...and daily traffic looks like this

Page 20: [@IndeedEng] Data-Driven Product Design

...and monthly traffic looks like this

Page 21: [@IndeedEng] Data-Driven Product Design
Page 22: [@IndeedEng] Data-Driven Product Design
Page 23: [@IndeedEng] Data-Driven Product Design

Local maximum

Page 24: [@IndeedEng] Data-Driven Product Design
Page 25: [@IndeedEng] Data-Driven Product Design
Page 26: [@IndeedEng] Data-Driven Product Design
Page 27: [@IndeedEng] Data-Driven Product Design
Page 28: [@IndeedEng] Data-Driven Product Design
Page 29: [@IndeedEng] Data-Driven Product Design
Page 30: [@IndeedEng] Data-Driven Product Design
Page 31: [@IndeedEng] Data-Driven Product Design
Page 32: [@IndeedEng] Data-Driven Product Design

What we've learnedUI Principles

Page 33: [@IndeedEng] Data-Driven Product Design

Clickable items should look clickable

Page 34: [@IndeedEng] Data-Driven Product Design
Page 35: [@IndeedEng] Data-Driven Product Design
Page 36: [@IndeedEng] Data-Driven Product Design
Page 37: [@IndeedEng] Data-Driven Product Design
Page 38: [@IndeedEng] Data-Driven Product Design
Page 39: [@IndeedEng] Data-Driven Product Design

Users read

Page 40: [@IndeedEng] Data-Driven Product Design
Page 41: [@IndeedEng] Data-Driven Product Design
Page 42: [@IndeedEng] Data-Driven Product Design
Page 43: [@IndeedEng] Data-Driven Product Design
Page 44: [@IndeedEng] Data-Driven Product Design
Page 45: [@IndeedEng] Data-Driven Product Design
Page 46: [@IndeedEng] Data-Driven Product Design
Page 47: [@IndeedEng] Data-Driven Product Design

a word is worth a thousand icons

Page 48: [@IndeedEng] Data-Driven Product Design
Page 49: [@IndeedEng] Data-Driven Product Design
Page 50: [@IndeedEng] Data-Driven Product Design
Page 51: [@IndeedEng] Data-Driven Product Design
Page 52: [@IndeedEng] Data-Driven Product Design
Page 53: [@IndeedEng] Data-Driven Product Design
Page 54: [@IndeedEng] Data-Driven Product Design

+14%more search results pages

Page 55: [@IndeedEng] Data-Driven Product Design

BEWARE MULTIPLE CHANGES

Page 56: [@IndeedEng] Data-Driven Product Design
Page 57: [@IndeedEng] Data-Driven Product Design
Page 58: [@IndeedEng] Data-Driven Product Design
Page 59: [@IndeedEng] Data-Driven Product Design
Page 60: [@IndeedEng] Data-Driven Product Design
Page 61: [@IndeedEng] Data-Driven Product Design

Searches: -2.0%Job Clicks: -1.7%Saved jobs: -7.6%

Page 62: [@IndeedEng] Data-Driven Product Design
Page 63: [@IndeedEng] Data-Driven Product Design
Page 64: [@IndeedEng] Data-Driven Product Design
Page 65: [@IndeedEng] Data-Driven Product Design
Page 66: [@IndeedEng] Data-Driven Product Design
Page 67: [@IndeedEng] Data-Driven Product Design

Searches: -1.8%Job Clicks: -0.8%Saved jobs: 0%

Page 68: [@IndeedEng] Data-Driven Product Design
Page 69: [@IndeedEng] Data-Driven Product Design

Really?

Page 70: [@IndeedEng] Data-Driven Product Design

Really.

Page 71: [@IndeedEng] Data-Driven Product Design
Page 72: [@IndeedEng] Data-Driven Product Design
Page 73: [@IndeedEng] Data-Driven Product Design
Page 74: [@IndeedEng] Data-Driven Product Design
Page 75: [@IndeedEng] Data-Driven Product Design
Page 76: [@IndeedEng] Data-Driven Product Design
Page 77: [@IndeedEng] Data-Driven Product Design
Page 78: [@IndeedEng] Data-Driven Product Design
Page 79: [@IndeedEng] Data-Driven Product Design
Page 80: [@IndeedEng] Data-Driven Product Design
Page 81: [@IndeedEng] Data-Driven Product Design
Page 82: [@IndeedEng] Data-Driven Product Design
Page 83: [@IndeedEng] Data-Driven Product Design
Page 84: [@IndeedEng] Data-Driven Product Design

ProductChris Hyams

SVP, Product & Engineering

Page 85: [@IndeedEng] Data-Driven Product Design

I help people get jobs.

Page 86: [@IndeedEng] Data-Driven Product Design

What's best for the job seeker?

Page 87: [@IndeedEng] Data-Driven Product Design

How do we know what's best for job seekers?

Page 88: [@IndeedEng] Data-Driven Product Design

Test and measure EVERYTHING

Page 89: [@IndeedEng] Data-Driven Product Design

Measurement

Page 90: [@IndeedEng] Data-Driven Product Design

1. Log everything

Page 91: [@IndeedEng] Data-Driven Product Design

jobsearch indexabredistimeacmetimeaddltimeadscadsdelayadsibadscbadsiboostojcboostojibsjcbsjcwiabsjibsjindappliesbsjindappviewsbsjrevbsjwiackcntckszcountsctkagectkagedaysdayofweekdcpingtimedomTotalTimeds-mpo

dsmissdstimefeatempfjfreekwacfreekwarevfreesjcfreesjrevfrmtimegalatdelayiplatiplongjslatdelayjsvdelaykwackwacdelaykwaikwarevkwcntlacinsizelacsgsizelmstimempotimemprtimenavTotTimendxtime

ojcojclongojcshortojcwiaojiojindappliesojindappviewsojwiaoocscpageprcvdlatencyprimfollowcntprvwojiprvwojlatprvwojopentimeprvwojreqradscradsirecidlookupbudgetrectimeredirCountredirTimerelfollowcntrespTimereturnvisitrojc

rojirqcntrqlcntrqqcntrrsjcrrsjirrsjrevrsavailrsjcrsjirsusedrsviableserpsizesjcsjcdelaysjclongsjcntsjcshortsjcwiasjisjindappliessjindappviewssjrevsjwiasllatsllong

sqcsqisugtimesvjsvjnostarsvjstartadsctadsitimetimeofdaytotcnttotfollowcnttotrevtottimetsjctsjcwiatsjitsjindappliestsjindappviewstsjrevtsjwiaunqcntvpwacinsizewacsgsize

Page 92: [@IndeedEng] Data-Driven Product Design

acmepageacmereviewmodacmeserviceacmesessionadclickadcrequestadcrevadschanneladsclickadsenseclickadveadvtagghttpaggjiraaggjobaggjob_waldorfaggsherlockaggsourcehealthagstimingapiapijsvapisearcharchiveindexarchiveindex_shingled_testbincarclicksclickclickanalyticscobranddctmismatchdrawdupepairsdupepairs_minidupepairs_olddupepairsalldupepairsall_miniejcheckeremilyops

feedbridgeglobalnavgooglebot_organichomepageimpressionindeedapplyjhstjobalertjobalertorganicjobalertsearchjobalertsponsoredjobexpirationjobexpiration2jobexpiration3jobprocessedjobqueueblockjobsearchjssquerykeywordAdlocsvclucyindexermainmechanicalturkmindyopsmobhomepagemobilmobilemobileorganicmobilesponsoredmobrecjobsmobsearchmobviewjobmyindeedmyindfunnelmyindpagemyindrezcreatemyindsessionoldopsesjasx

organicorgmodelorgmodelsubsetorgmodelsubset90passportaccountpassportpagepassportsigninramsaccessrecjobsrecommendserviceresumedataresumesearchrexcontactsrexfunnelreximpressionrexsearchrezSrchSearchrezalertrezalertfunnelrezfunnelrezjserrrezsrchrequestrezviewsearchablejobsseosessionsjmodelsponsoredsysadappinfosysadapptimingtestndxtestndx1testndx2tmpusrsvccacheusrsvcrequestviewjobwebusersignin

Page 93: [@IndeedEng] Data-Driven Product Design

16 million jobs100 million UV

3 billion searches

Page 94: [@IndeedEng] Data-Driven Product Design

1TB per day

Page 95: [@IndeedEng] Data-Driven Product Design

Everything you ever wanted to know about logging but were afraid to ask

Logrepo: Enabling Data-Driven Decisions

https://engineering.indeed.com/blog/2012/11/logrepo-enabling-data-driven-decisions/
https://engineering.indeed.com/blog/2012/11/logrepo-enabling-data-driven-decisions/
https://engineering.indeed.com/blog/2012/11/logrepo-enabling-data-driven-decisions/
https://engineering.indeed.com/blog/2012/11/logrepo-enabling-data-driven-decisions/
Page 96: [@IndeedEng] Data-Driven Product Design

2. Index data

Page 97: [@IndeedEng] Data-Driven Product Design

3. Build tools toquery indexes

Page 98: [@IndeedEng] Data-Driven Product Design

[another tech talk]

Page 99: [@IndeedEng] Data-Driven Product Design

Tools answer questions

Page 100: [@IndeedEng] Data-Driven Product Design

Mobile searches per hour inJP vs. UK?

Page 101: [@IndeedEng] Data-Driven Product Design

Resume creation by country?

Page 102: [@IndeedEng] Data-Driven Product Design

Impressions in job alerts byemail domain?

Page 103: [@IndeedEng] Data-Driven Product Design

Percent of app downloads fromiOS, Android, Windows?

Page 104: [@IndeedEng] Data-Driven Product Design

How quickly does a datacenter take on traffic after a failover?

Page 105: [@IndeedEng] Data-Driven Product Design

Test EVERYTHING

Page 106: [@IndeedEng] Data-Driven Product Design

TextVisual designLayoutControl flowNew featuresAlgorithms [also another tech talk]

Everything

Page 107: [@IndeedEng] Data-Driven Product Design

5% 5%grp:bluebtntst0 grp:bluebtntst1

split:

log:

Page 108: [@IndeedEng] Data-Driven Product Design

Measure

Page 109: [@IndeedEng] Data-Driven Product Design

statistical relevance

Page 110: [@IndeedEng] Data-Driven Product Design

p-value

Page 111: [@IndeedEng] Data-Driven Product Design

Big changes require less data

Conversion Improvement p-value Trials

10% +30% 0.038 1,000

source: http://go.indeed.com/jyusg

http://go.indeed.com/jyusg
Page 112: [@IndeedEng] Data-Driven Product Design

Small changes require more data

Conversion Improvement p-value Trials

10% +30% 0.038 1,000

10% +1% 0.042 750,000

source: http://go.indeed.com/ommgd

http://go.indeed.com/ommgd
Page 113: [@IndeedEng] Data-Driven Product Design
Page 114: [@IndeedEng] Data-Driven Product Design

Product Principles

Page 115: [@IndeedEng] Data-Driven Product Design

There's no such thing as a no-brainer

Page 116: [@IndeedEng] Data-Driven Product Design
Page 117: [@IndeedEng] Data-Driven Product Design

-30%

Page 118: [@IndeedEng] Data-Driven Product Design

Corollary: if a test fails,it means the test failed

Page 119: [@IndeedEng] Data-Driven Product Design

-30%

Page 120: [@IndeedEng] Data-Driven Product Design
Page 121: [@IndeedEng] Data-Driven Product Design

+25%

Page 122: [@IndeedEng] Data-Driven Product Design

Little things can makea big difference

Page 123: [@IndeedEng] Data-Driven Product Design
Page 124: [@IndeedEng] Data-Driven Product Design

organic

Page 125: [@IndeedEng] Data-Driven Product Design

sponsored

Page 126: [@IndeedEng] Data-Driven Product Design

+34%

Page 127: [@IndeedEng] Data-Driven Product Design

Corollary: be skepticalof big changes

Page 128: [@IndeedEng] Data-Driven Product Design

Does color really matter that much?

Page 129: [@IndeedEng] Data-Driven Product Design

a A

a B

a C +34%

+10%

Page 130: [@IndeedEng] Data-Driven Product Design

Are people just boredof blue?

Page 131: [@IndeedEng] Data-Driven Product Design

+29% +22%

Page 132: [@IndeedEng] Data-Driven Product Design

Will the effect wear off?

Page 133: [@IndeedEng] Data-Driven Product Design
Page 134: [@IndeedEng] Data-Driven Product Design

Get it right,then do it right

Page 135: [@IndeedEng] Data-Driven Product Design
Page 136: [@IndeedEng] Data-Driven Product Design

Location backfill: job clicks

+8%

Page 137: [@IndeedEng] Data-Driven Product Design

Location backfill: search time

+1,500%

Page 138: [@IndeedEng] Data-Driven Product Design

Location backfill: performance fix

+1,500%

Page 139: [@IndeedEng] Data-Driven Product Design

Location backfill: performance fix

+1,500% +0%

Page 140: [@IndeedEng] Data-Driven Product Design

Location backfill: job clicks

+8%

Page 141: [@IndeedEng] Data-Driven Product Design

Location backfill: clicks after fix

+8% +9%

Page 142: [@IndeedEng] Data-Driven Product Design

Measure everything(aka, Beware of unintended consequences)

Page 143: [@IndeedEng] Data-Driven Product Design

April Fools' 2010

Page 144: [@IndeedEng] Data-Driven Product Design
Page 145: [@IndeedEng] Data-Driven Product Design
Page 146: [@IndeedEng] Data-Driven Product Design
Page 147: [@IndeedEng] Data-Driven Product Design

Organic clicks

+1.8%

Page 148: [@IndeedEng] Data-Driven Product Design

Job alert signups

-17%

Page 149: [@IndeedEng] Data-Driven Product Design

Revenue

+15.5%

Page 150: [@IndeedEng] Data-Driven Product Design

Revenue

+15.5%

Page 151: [@IndeedEng] Data-Driven Product Design
Page 152: [@IndeedEng] Data-Driven Product Design
Page 153: [@IndeedEng] Data-Driven Product Design
Page 154: [@IndeedEng] Data-Driven Product Design
Page 155: [@IndeedEng] Data-Driven Product Design

Job alert signups: top v. bottom

-17% +6%-21%

Page 156: [@IndeedEng] Data-Driven Product Design

Corollary: attention is a zero-sum game

(usually)

Page 157: [@IndeedEng] Data-Driven Product Design

SearchClickRevenueEmail signupResume create...

Relative value of actions

Page 158: [@IndeedEng] Data-Driven Product Design
Page 159: [@IndeedEng] Data-Driven Product Design

Different for everyone

Page 160: [@IndeedEng] Data-Driven Product Design

Location auto-complete

Page 161: [@IndeedEng] Data-Driven Product Design

Zero-results page

Page 162: [@IndeedEng] Data-Driven Product Design
Page 163: [@IndeedEng] Data-Driven Product Design
Page 164: [@IndeedEng] Data-Driven Product Design

Zero-results pages

-2.7%

Page 165: [@IndeedEng] Data-Driven Product Design

Organic clicks

+0%

Page 166: [@IndeedEng] Data-Driven Product Design

Clicks w/ auto-complete

+8%

Page 167: [@IndeedEng] Data-Driven Product Design

Keyword ad revenue

+1,410%

Page 168: [@IndeedEng] Data-Driven Product Design

Keyword ad revenue

+1,410%

Page 169: [@IndeedEng] Data-Driven Product Design

keyword ads

Page 170: [@IndeedEng] Data-Driven Product Design
Page 171: [@IndeedEng] Data-Driven Product Design
Page 172: [@IndeedEng] Data-Driven Product Design
Page 173: [@IndeedEng] Data-Driven Product Design
Page 174: [@IndeedEng] Data-Driven Product Design

Keyword ad revenue after fix

Page 175: [@IndeedEng] Data-Driven Product Design

Measure everything,every time

Page 176: [@IndeedEng] Data-Driven Product Design
Page 177: [@IndeedEng] Data-Driven Product Design

Getting Started

Page 178: [@IndeedEng] Data-Driven Product Design
Page 179: [@IndeedEng] Data-Driven Product Design
Page 180: [@IndeedEng] Data-Driven Product Design
Page 181: [@IndeedEng] Data-Driven Product Design

Fewer tests

Page 182: [@IndeedEng] Data-Driven Product Design

Focus on big wins

Page 183: [@IndeedEng] Data-Driven Product Design

Conversion Improvement p-value Trials

10% +30% 0.038 1,000

10% +1% 0.042 750,000

source: http://go.indeed.com/ommgd

http://go.indeed.com/ommgd
Page 184: [@IndeedEng] Data-Driven Product Design

Straw man

Determine # of trials in a weekDetermine minimum improvementOne new test each weekIf success, roll out; if not, kill & move on

Page 185: [@IndeedEng] Data-Driven Product Design

What We Get

Page 186: [@IndeedEng] Data-Driven Product Design

Quality

Page 187: [@IndeedEng] Data-Driven Product Design

Risk tolerance

Page 188: [@IndeedEng] Data-Driven Product Design

Culture of measurement

Page 189: [@IndeedEng] Data-Driven Product Design

(Empirically)better products

Page 190: [@IndeedEng] Data-Driven Product Design

fin

Page 191: [@IndeedEng] Data-Driven Product Design

Data Driven Product Design

Data Driven Product Design

User Driven Product Strategy

User Driven Product Strategy

[@IndeedEng Talk] Diving deeper into data-driven product design

[@IndeedEng Talk] Diving deeper into data-driven product design

@IndeedEng: Tokens and Millicents - technical challenges in launching Indeed around the world

@IndeedEng: Tokens and Millicents - technical challenges in launching Indeed around the world

Experiment driven product - LAST 2014

Experiment driven product - LAST 2014

Test Driven Product: Applying Test Driven Thinking to the Product World

Test Driven Product: Applying Test Driven Thinking to the Product World

Data Driven Product Development

Data Driven Product Development

Data-Driven Product Development

Data-Driven Product Development

ONTO-PDM: product-driven ONTOlogy for Product Data ... · ONTO-PDM: product-driven ONTOlogy for Product Data Management interoperability within manufacturing process environment Panetto

ONTO-PDM: product-driven ONTOlogy for Product Data ... · ONTO-PDM: product-driven ONTOlogy for Product Data Management interoperability within manufacturing process environment Panetto

Data Driven Product Innovation - quaero.com · Data Driven Product Innovation Case Study April 2017. Data Driven Ad Product Innovation 2 Case Study Introduction The digital ad ecosystem

Data Driven Product Innovation - quaero.com · Data Driven Product Innovation Case Study April 2017. Data Driven Ad Product Innovation 2 Case Study Introduction The digital ad ecosystem

[@IndeedEng] Redundant Array of Inexpensive Datacenters

[@IndeedEng] Redundant Array of Inexpensive Datacenters

ELECTRONIC PART OBSOLESCENCE DRIVEN PRODUCT …escml.umd.edu/Papers/AgingAircraft.pdf · Sandborn, Electronic Part Obsolescence Driven Product Redesign Optimization If a product requires

ELECTRONIC PART OBSOLESCENCE DRIVEN PRODUCT …escml.umd.edu/Papers/AgingAircraft.pdf · Sandborn, Electronic Part Obsolescence Driven Product Redesign Optimization If a product requires

5 principles for product driven growth

5 principles for product driven growth

Customer-Manager Driven Product-Manager Driven

Customer-Manager Driven Product-Manager Driven

[@IndeedEng] Large scale interactive analytics with Imhotep

[@IndeedEng] Large scale interactive analytics with Imhotep

Systems-driven product development

Systems-driven product development

Data-driven product design

Data-driven product design

Data driven product management

Data driven product management

User story driven product development process

User story driven product development process

Simulation Driven Product Development - uni … · Simulation Driven Product Development Albrecht Gill ... Golf Club/Ball Contact ... – User elements & materials

Simulation Driven Product Development - uni … · Simulation Driven Product Development Albrecht Gill ... Golf Club/Ball Contact ... – User elements & materials