Data-DrivenProduct Design

J Christopher GarciaChris Hyams

DesignJ Christopher Garcia

User Experience Lead, Indeed

2008

2013

114 tests244 variations

84,000 combinations

Direct observation can be inefficient

Before and afterdoesn't work

Traffic doesn't look like this

Hourly traffic looks like this...

...and daily traffic looks like this

...and monthly traffic looks like this

Local maximum

What we've learnedUI Principles

Clickable items should look clickable

Users read

a word is worth a thousand icons

+14%more search results pages

BEWARE MULTIPLE CHANGES

Searches: -2.0%Job Clicks: -1.7%Saved jobs: -7.6%

Searches: -1.8%Job Clicks: -0.8%Saved jobs: 0%

Really?

Really.

ProductChris Hyams

SVP, Product & Engineering

I help people get jobs.

What's best for the job seeker?

How do we know what's best for job seekers?

Test and measure EVERYTHING

Measurement

1. Log everything

jobsearch indexabredistimeacmetimeaddltimeadscadsdelayadsibadscbadsiboostojcboostojibsjcbsjcwiabsjibsjindappliesbsjindappviewsbsjrevbsjwiackcntckszcountsctkagectkagedaysdayofweekdcpingtimedomTotalTimeds-mpo

dsmissdstimefeatempfjfreekwacfreekwarevfreesjcfreesjrevfrmtimegalatdelayiplatiplongjslatdelayjsvdelaykwackwacdelaykwaikwarevkwcntlacinsizelacsgsizelmstimempotimemprtimenavTotTimendxtime

ojcojclongojcshortojcwiaojiojindappliesojindappviewsojwiaoocscpageprcvdlatencyprimfollowcntprvwojiprvwojlatprvwojopentimeprvwojreqradscradsirecidlookupbudgetrectimeredirCountredirTimerelfollowcntrespTimereturnvisitrojc

rojirqcntrqlcntrqqcntrrsjcrrsjirrsjrevrsavailrsjcrsjirsusedrsviableserpsizesjcsjcdelaysjclongsjcntsjcshortsjcwiasjisjindappliessjindappviewssjrevsjwiasllatsllong

sqcsqisugtimesvjsvjnostarsvjstartadsctadsitimetimeofdaytotcnttotfollowcnttotrevtottimetsjctsjcwiatsjitsjindappliestsjindappviewstsjrevtsjwiaunqcntvpwacinsizewacsgsize

acmepageacmereviewmodacmeserviceacmesessionadclickadcrequestadcrevadschanneladsclickadsenseclickadveadvtagghttpaggjiraaggjobaggjob_waldorfaggsherlockaggsourcehealthagstimingapiapijsvapisearcharchiveindexarchiveindex_shingled_testbincarclicksclickclickanalyticscobranddctmismatchdrawdupepairsdupepairs_minidupepairs_olddupepairsalldupepairsall_miniejcheckeremilyops

feedbridgeglobalnavgooglebot_organichomepageimpressionindeedapplyjhstjobalertjobalertorganicjobalertsearchjobalertsponsoredjobexpirationjobexpiration2jobexpiration3jobprocessedjobqueueblockjobsearchjssquerykeywordAdlocsvclucyindexermainmechanicalturkmindyopsmobhomepagemobilmobilemobileorganicmobilesponsoredmobrecjobsmobsearchmobviewjobmyindeedmyindfunnelmyindpagemyindrezcreatemyindsessionoldopsesjasx

organicorgmodelorgmodelsubsetorgmodelsubset90passportaccountpassportpagepassportsigninramsaccessrecjobsrecommendserviceresumedataresumesearchrexcontactsrexfunnelreximpressionrexsearchrezSrchSearchrezalertrezalertfunnelrezfunnelrezjserrrezsrchrequestrezviewsearchablejobsseosessionsjmodelsponsoredsysadappinfosysadapptimingtestndxtestndx1testndx2tmpusrsvccacheusrsvcrequestviewjobwebusersignin

16 million jobs100 million UV

3 billion searches

1TB per day

Everything you ever wanted to know about logging but were afraid to ask

Logrepo: Enabling Data-Driven Decisions

2. Index data

3. Build tools toquery indexes

[another tech talk]

Tools answer questions

Mobile searches per hour inJP vs. UK?

Resume creation by country?

Impressions in job alerts byemail domain?

Percent of app downloads fromiOS, Android, Windows?

How quickly does a datacenter take on traffic after a failover?

Test EVERYTHING

TextVisual designLayoutControl flowNew featuresAlgorithms [also another tech talk]

Everything

5% 5%grp:bluebtntst0 grp:bluebtntst1

split:

log:

Measure

statistical relevance

p-value

Big changes require less data

Conversion Improvement p-value Trials

10% +30% 0.038 1,000

source: http://go.indeed.com/jyusg

Small changes require more data

Conversion Improvement p-value Trials

10% +30% 0.038 1,000

10% +1% 0.042 750,000

source: http://go.indeed.com/ommgd

Product Principles

There's no such thing as a no-brainer

-30%

Corollary: if a test fails,it means the test failed

-30%

+25%

Little things can makea big difference

organic

sponsored

+34%

Corollary: be skepticalof big changes

Does color really matter that much?

a A

a B

a C +34%

+10%

Are people just boredof blue?

+29% +22%

Will the effect wear off?

Get it right,then do it right

Location backfill: job clicks

+8%

Location backfill: search time

+1,500%

Location backfill: performance fix

+1,500%

Location backfill: performance fix

+1,500% +0%

Location backfill: job clicks

+8%

Location backfill: clicks after fix

+8% +9%

Measure everything(aka, Beware of unintended consequences)

April Fools' 2010

Organic clicks

+1.8%

Job alert signups

-17%

Revenue

+15.5%

Revenue

+15.5%

Job alert signups: top v. bottom

-17% +6%-21%

Corollary: attention is a zero-sum game

(usually)

SearchClickRevenueEmail signupResume create...

Relative value of actions

Different for everyone

Location auto-complete

Zero-results page

Zero-results pages

-2.7%

Organic clicks

+0%

Clicks w/ auto-complete

+8%

Keyword ad revenue

+1,410%

Keyword ad revenue

+1,410%

keyword ads

Keyword ad revenue after fix

Measure everything,every time

Getting Started

Fewer tests

Focus on big wins

Conversion Improvement p-value Trials

10% +30% 0.038 1,000

10% +1% 0.042 750,000

source: http://go.indeed.com/ommgd

Straw man

Determine # of trials in a weekDetermine minimum improvementOne new test each weekIf success, roll out; if not, kill & move on

What We Get

Quality

Risk tolerance

Culture of measurement

(Empirically)better products

fin