Rel

ated

Wor

k

Met

hodo

logy

Res

ults

Mea

surin

g th

e Ef

fect

iven

ess

of D

ispl

ay A

dver

ting:

A

Tim

e Se

ries

App

roac

hJo

el B

araj

as*1

, Ram

Ake

lla1 ,

Mar

ius

Hol

tan2

, Jai

mie

Kw

on2 ,

Bra

d N

ull2

1 UC

San

ta C

ruz,

San

ta C

ruz

CA

, US

A2 A

OL

Res

earc

h, P

alo

Alto

CA

, US

A

Ref

eren

ces

Intr

oduc

tion

•O

nlin

e di

spla

y ad

verti

sing

is a

n ar

ea o

f rap

id g

row

th a

nd c

onse

quen

tly

of g

reat

inte

rest

as

a m

arke

ting

chan

nel.

•R

ecen

t stu

dies

sho

wth

at d

ispl

ay a

dver

tisin

g of

ten

trigg

ers

onlin

e us

ers

to s

earc

hfo

r inf

orm

atio

n ab

out c

omm

erci

al p

rodu

cts

[1].

•E

vent

ually

, man

y of

thes

e us

ers

perfo

rm e

ither

onlin

e co

nver

sion

s at

th

e ad

verti

ser's

web

site

or o

fflin

e co

nver

sion

s at

a p

hysi

cal s

tore

.

Met

hodo

logy

[1]A

tlas.

Eng

agem

ent m

appi

ng: A

new

mea

sure

men

t sta

ndar

d is

em

ergi

ngfo

r adv

ertis

ers,

200

8. W

hite

Pap

er.

[2]Z

oubi

n G

hahr

aman

i and

Geo

ffrey

E. H

into

n. P

aram

eter

est

imat

ion

for

linea

r dyn

amic

al s

yste

ms.

Tec

hnic

al re

port,

199

6.

[3]G

oogl

e. C

PApe

rfom

ance

tren

ds o

n th

e G

oogl

e di

spla

yne

twor

k, 2

010.

W

hite

Pap

er.

[4]R

anda

ll Le

wis

and

Dav

idR

eile

y. R

etai

ladv

ertis

ing

wor

ks!m

easu

ring

the

effe

cts

of a

dver

tisin

gon

sal

es v

ia a

con

trolle

dex

perim

ent o

nYa

hoo!

, 20

09. W

hite

Pap

er.

[5]M

ike

Wes

t and

Jef

f Har

rison

. Bay

esia

nfo

reca

stin

g an

d dy

nam

ic m

odel

s (2

nd e

d.).

Spr

inge

r-Ve

rlag

New

York

, Inc

., N

ewYo

rk, N

Y, U

SA

, 199

7.

[6]G

eorg

eC

asel

laan

d R

oger

L. B

erge

r. S

tatis

tical

Infe

renc

e(2

nd e

d.).

Dux

bury

, 200

2.

Onl

ine

disp

lay

ad s

how

n to

a us

er

•O

ur g

oal i

s to

mea

sure

the

effe

ctiv

enes

s of

dis

play

adv

ertis

ing

whe

n us

ers

are

expo

sed

to m

ultip

le a

dver

tisin

g ch

anne

ls.

Use

r is

exp

osed

tom

ultip

le a

dver

tisin

g ch

anne

ls in

tim

e

Eve

ntua

lly, t

he u

ser

perf

orm

s co

mm

erci

al a

ctio

ns

•If

a us

erpe

rform

s a

com

mer

cial

act

ion,

how

shou

ldth

e ad

verti

ser

attri

bute

cre

dit f

or th

e co

nver

sion

acr

oss

thes

e m

ultip

le c

hann

els

and

med

ia im

pres

sion

s?

•Th

is is

par

ticul

arly

impo

rtant

for t

he C

ost-

Per

-Act

ion

CPA

busi

ness

mod

el w

hen

deci

ding

how

to a

lloca

te

reso

urce

s ac

ross

adv

ertis

ing

chan

nels

.

•C

PApe

rform

ance

on

a m

onth

ly le

vel [

3].

Agg

rega

te s

tudy

bas

edju

st o

n th

e ob

serv

ed

aver

age

perfo

rman

ce

•A

stud

y ab

out u

ser

feat

ures

and

per

form

ance

on

com

mer

cial

act

ions

is

per

form

ed in

[4].

How

ever

, the

stu

dy w

as d

one

with

a s

mal

l set

of

user

s an

d ad

verti

sers

to h

ave

wel

l-def

ined

con

trol a

nd te

st g

roup

s.

•O

ur m

ain

goal

is to

mea

sure

the

effe

ctiv

enes

s of

cam

paig

ns o

n a

daily

leve

l. Th

is is

impo

rtant

for s

hort

cam

paig

ns a

nd to

pro

vide

dy

nam

ic o

nlin

e pe

rform

ance

est

imat

es.

•W

e fo

llow

a tim

e-se

ries

appr

oach

to d

ecom

pose

the

daily

num

ber

of

sale

s of

a g

iven

pro

duct

into

a w

eekl

y se

ason

al c

ompo

nent

and

a

trend

com

pone

nt u

sing

Dyn

amic

Lin

ear

Mod

els

base

don

Kal

man

fil

terin

g an

d sm

ooth

ing

idea

s [5

].

Cam

paig

n D

urat

ion

Day

s w

ith p

ositi

ve

sign

ifica

ntch

ange

WN

ww

G

VN

Fy

tt

tt

tt

tt

,0~

,0~

' 1

7/

2cos

/2

sin/

2sin

/2

cos

10

11

T

TT

TT

GG

Tp

2/,

,0,1,0,1,0,1

'T

Tp

GG

Gblockdiag

GF

Num

ber o

f com

mer

cial

act

ions

at t

imet

tyS

tate

evo

lutio

nof

the

mod

el c

ompo

nent

s (s

toch

astic

pro

cess

)t

•B

y fix

ing G

and F,

we

mod

el th

e se

ries

evol

utio

n as

a li

near

com

bina

tion

of a

sea

sona

l and

a tr

end

com

pone

nt [5

]•

Our

goa

l is

to d

esea

sona

lize

the

serie

s to

ass

ocia

te th

e tre

nd w

ith a

se

t of m

arke

ting

cam

paig

ns.

•W

e fin

dth

e M

LE o

f the

var

ianc

es Φ

= (V,W

)thr

ough

the

EM

alg

orith

m[2

] and

sm

ooth

the

serie

s to

ana

lyze

the

trend

com

pone

nt.

•W

e m

easu

re th

e ef

fect

iven

ess

of c

ampa

igns

by

com

parin

g sa

les

occu

rrin

gdu

ring

the

days

of t

he c

ampa

ign

fligh

t to

a ba

selin

e.

•W

e te

st s

tatis

tical

sig

nific

ance

of th

e ch

ange

in th

e tre

nd c

ompo

nent

ag

ains

t the

bas

elin

e.

•95

% c

onfid

ence

inte

rval

s ar

e us

ed to

det

ect a

n in

crea

se, d

ecre

ase,

orno

cha

nge

in th

e tre

nd.

E-step:

,

|log

|:1

:1,

|:1

:1T

Ty

iy

PE

Qi

TT

i

iQ

|

max

arg

1M-step:

Kal

man

Filt

erin

gan

d S

moo

thin

gO

ptim

izat

ion

to u

pdat

eth

e va

rian

ces (W

,V)

i

Q

|

1

i

*

0

iT

Ty

P,

|:1

:1

Cam

paig

n E

valu

atio

n•

We

use

the

prev

ious

day

of t

he s

tart

of a

cam

paig

nas

the

base

line.

•To

eva

luat

e th

e pe

rform

ance

of t

hese

cam

paig

ns, w

e de

fine

four

succ

ess

crite

ria.

1.A

vera

ge p

ositi

ve in

crea

se in

sal

es d

urin

g th

e ca

mpa

ign

2.In

crea

se o

r no

sta

tistic

al s

igni

fican

t cha

nge

insa

les

onev

ery

day

ofth

e ca

mpa

ign.

3.M

ore

days

with

incr

easi

ng s

ales

than

with

decr

easi

ng s

ales

.4.

Mor

e da

ys w

ithin

crea

sing

sal

es o

r no

sig

nific

ant c

hang

eth

an w

ithde

crea

sing

sal

es.

•W

e es

timat

e th

e av

erag

e ch

ange

of s

ales

dur

ing

the

cam

paig

ndu

ratio

n in

the

trend

com

pone

nt.

•W

e us

eth

e de

lta m

etho

dto

app

roxi

mat

e th

e di

strib

utio

nof

the

aver

age

chan

ge in

sal

es r

espe

ct to

the

base

line

•Th

e ap

prox

imat

ion

is b

ased

on th

e Ta

ylor

serie

s de

com

posi

tion

of th

e fu

nctio

n of

the

rand

omva

riabl

es [6

].

Dis

cuss

ion

•M

ain

goal

:–

Mea

sure

the

impa

ct o

f adv

ertis

ing

on s

ales

nea

ran

d lo

ng te

rm.

•D

iffic

ultie

s:–

Adv

ertis

ers

typi

cally

use

all

avai

labl

ech

anne

ls (

TV, r

adio

, prin

t, on

line)

and

ther

e ar

e m

any

vend

ors

with

in e

ach

chan

nel.

–D

ata

is ty

pica

lly n

ot in

tegr

ated

acro

ss c

hann

els

or e

ven

with

in

chan

nels

and

thus

the

idea

of f

ollo

win

g th

e on

line

clic

k-st

ream

beha

vior

of w

eb u

sers

is g

ener

ally

not

pos

sibl

e.•

Ach

ieve

men

tsan

d G

aps:

–H

igh

leve

l stu

dy a

cros

s se

vera

l tho

usan

ds o

f cam

paig

ns–

We

dono

t hav

e a

deta

iled

unde

rsta

ndin

g of

the

exac

t goa

ls o

f ea

ch c

ampa

ign.

–

Com

parin

g sa

les

agai

nst t

he s

ales

gen

erat

ed d

urin

g so

me

perio

d pr

ior

to th

e ca

mpa

ign

mig

ht n

ot a

lway

s be

app

ropr

iate

(th

e ca

mpa

ign

may

focu

s on

driv

ing

offli

ne s

ales

for i

nsta

nce

orla

unch

ed o

ff-se

ason

).–

Neg

ativ

esa

les

chan

ges

as m

easu

red

in th

is s

tudy

do

not

nece

ssar

ily m

ean

that

the

cam

paig

n fa

iled.

Res

ults

Cri

teri

on

1C

rite

rion

2

Cri

teri

on

3C

rite

rion

4

Suc

cesf

ul

Cam

paig

ns (%

)43

%65

%73

%84

%

Ave

rage

Incr

ease

of

Sal

es: S

ucce

ssfu

l C

ampa

igns

74%

42%

40%

31%

•D

istri

butio

n of

the

aver

age

chan

ge in

sal

es b

y ca

mpa

ign.

•

This

is b

roke

n do

wn

into

cam

paig

ns w

ith p

ositi

ve, n

egat

ive

and

no

stat

istic

ally

sig

nific

anta

vera

ge c

hang

e.

•W

e an

alyz

e 1,

635

cam

paig

nsfo

r429

pro

duct

sdu

ring

the

perio

d fro

m J

uly

1st, 2

009

to J

uly

31st

, 201

0.

•Th

e fo

llow

ing

tabl

es p

rese

nts

the

succ

essf

ul c

ampa

igns

ac

cord

ing

to th

e ab

ove

crite

ria.

Ong

oing

and

Fut

ure

Wor

k•

Inco

rpor

ate

the

num

ber

of im

pres

sion

s se

rved

as

adve

rtisi

ng a

ctio

ns in

a

mar

ketin

g ca

mpa

ign.

•D

oes

the

num

ber

of im

pres

sion

s ha

ve

an im

pact

to p

redi

ct th

e nu

mbe

rof

sa

les?

1

11

11

b

b

N tt

cc

c

N

1

11

11

b

b

N tt

cc

E

EE

NE

c

c

cc

c

N tt

bc

b

N tt

tbc

b

N tt

cb

N tj

tj

it

cN

E

E

NE

E

NE

11

12

31

11

24

1

2

11

22

1

11

11

,cov

2var

,cov

2var

var

cc

cE

N

var

,~

024681012141618

Percentage of Campaigns

[-50%

, -40

%]

[-40%

, -30

%]

[-30%

, -20

%]

[-20%

, -10

%]

[-10%

, 0%

][0%, 1

0%][10

%, 2

0%][20

%, 3

0%][30

%, 4

0%][40

%, 5

0%][50

%, 6

0%][60

%, 8

0%][80

%, 1

00%

][100%

, 150

%]

[150%

, 200

%]

[200%

, 250

%]

Sal

es P

erce

ntag

e C

hang

e

Dis

trib

utio

n of

Ave

rage

Cha

nge

in S

ales

Per

cent

age

for

All

Cam

paig

nsS

epar

ated

into

Pos

itive

, Neg

ativ

e an

d N

o-S

igni

fican

t Cha

nge

Com

pone

nts

Ori

gina

l Dis

trib

utio

nP

ositi

ve C

hang

eN

o S

igni

fican

t Cha

nge

Neg

ativ

e C

hang

e

* Con

tact

Aut

hor