Advertising Quality Science

Advertising Quality Science Mounia Lalmas

This talk 4-year effort across research, engineering and product at Yahoo to measure the quality of ads served on Gemini, Yahoo native advertising network Not just measuring but taking actions to improve user experience as well as providing feedbacks to advertisers

à no deep learning but large scale predictive analytics Focus of the talk: the post-click experience on native ads

à the quality of the landing page… if it is seen

3

The advertising world is fun J

Online advertising is big business

Values in $billions

Advertising is how Yahoo (and many other Internet companies) makes money… and what keeps Yahoo services free for its customers

90% of Yahoo’s revenue is from advertising: 2016 search advertising revenue – $2.67B (52% of total revenue) 2016 display including native advertising – $1.98B (38% of total) Scale: billion ads served daily

(Source: Yahoo 2016 10k annual report)

Online advertising is about connecting supply & demand

Search Native Display Video

Brand Direct Response

Yahoo own & operated sites Publisher Partners

SUPPLY (publishers)

DEMAND (advertisers)

Advertising (ad) quality science

Develop predictive models that characterise the quality of ads shown to and clicked by users.

Maximise revenue and guaranteeing ROI to advertisers without negatively impacting user experience.

Publishers

Advertisers

Users Ad inventory

ad network

Being able to help advertisers improve the quality of their ads

Ad Quality: Scope

Non-intentional Ad quality

Intentional Ad compliance

Major shift in how users access the Internet

comScore2015 UK Internet users

Native advertising

(Source:Sharethrough.com&IPGMediaLabStudy:Na;veAdver;sementEffec;veness)

Visually Engaging

Higher user attention

Higher brand lift

Social sharing

The quality of the post-click experience

the quality of the landing page on mobile

The post-click experience: Dwell time

dwell time

proxy of post-click ad quality experience

predictive post-click ad quality models

ad quality ratings & recommendations

accidental clicks identification

proxy of accidental clicks

Metric

Ad serving

User engagement

Advertiser ROI

dwell time






Metric

Ad serving

User engagement

Advertiser ROI

The post-click experience journey

Quality of the post-click experience Best experience is when conversion happens No conversion does not mean a bad experience

Proxy metric of post-click experience: dwell time on the ad landing page

tad-click tback-to-publisher

dwell time = tback-to-publisher – tad-click

Positive post-click experience (“long” clicks) has an effect on users clicking on ads again

dwell time






Metric

Ad serving

User engagement

Advertiser ROI


Optimise for high quality ads

Estimating P(hq|click) = quality score

P(dwell time > t)

Build predictive models that predict if an ad is of high quality = predicted dwell time above a given threshold t

➔  high quality = high dwell time

revenue = 𝓕 (bid, CTR, quality) P(hq|click)

logistic regression, gradient descent boosting, random forest, survival random forest

Landing page features

●  window_size ●  view_port ●  media_support

content

mobile support

requested information multimedia

mobile optimized

out-going connectivity

interactivity

textual content

in-coming connectivity

●  description ●  keywords ●  title

meta information

●  num_forms ●  num_input_radio ●  num_input_string ●  ...

readability multimedia

significant effect of text readability and page structure

A/B testing

dwell time increased by 20% bounce rate decreased by 7%

revenue = bid x CTR x quality

(Lalmas etal, 2015; Barbieri, Silvestri & Lalmas, 2016)

dwell time






Metric

Ad serving

User engagement

Advertiser ROI


Landing page rating: Low, Average or High

landing pages quality score q

…

L H

L and H are customisable: e.g., LOW=[0,25%), AVG=[25%,75%], HIGH=(75%,100%]

2 cut-off points (L, H) that divide distribution of quality scores q into 3 regions:

-  LOW: q < L -  AVG: L <= q <= H -  HIGH: q > H

(L, H)

ad rating q LOW

Improving landing pages Exploiting the features for recommending improvements

mobile optimized

out-going connectivity

interactivity textual content

in-coming connectivity

meta information

readability

multimedia

●  num_forms ●  num_input_radio ●  num_input_string ●  ...

interactivity ●  mediannum_forms ±ε ●  mediannum_input_radio ±ε ●  mediannum_input_string ±ε ●  ...

for each feature compute median and confidence interval

for each ad feature compute the distance from

the confidence interval

given an ad

num_input_radio num_forms num_input_string ...

There might be too few/much textual content

There might be too few/many entities

There might be too few/many images

LandingPageContentn.ofwords

n.ofWikipediaen;;es

n.ofimages

LandingPageLayoutheight/width

resizability(fittomul;plescreensize)

LandingPageStructuren.ofdrop-downmenus

n.ofcheckboxes

n.ofinputstrings

LandingPageReadabilitycontentsummarizability

The height/width of the landing page might be too small/large The landing page might not be adapted to different screen sizes

There might be too many drop-down menus

There might be too many checkboxes

There might be too much information requested from the users

The textual content might be further summarised to make it more readable

Examples of recommendations

dwell time






Metric

Ad serving

User engagement

Advertiser ROI


peak on app X

●  accidental clicks do not reflect post-click experience

●  not all clicks are equal

app X

The quality of a click on mobile apps peak on app Y

dwell time distribution of apps X and Y for a given ad

app Y

dwell time






Metric

Ad serving

User engagement

Advertiser ROI


Fitting the data into a mixture model

The number of mixture components is determined using the BIC criterion which selects the model that fits best the data while avoiding overfitting

Time period 1 Time period 2 (after UI change)

Bayesian information criterion (BIC)

bouncy clicks

accidental clicks

Accidental clicks threshold for app X

Min

1st Quartile

Median

Mean

3rd Quartile

Max

Distribution of the medians as computed on the first component of each ad

Applications -  discount accidental clicks

using economics models -  train click models

discarding accident clicks -  input to UI design

ads with all three components

dwell time






Metric

Ad serving

User engagement

Advertiser ROI

Ad quality: The post-click experience journey

Acknowledgments: Marc Bron, Ayman Farahat, Andy Haines, Miriam Redi, Gabriele Tolomei, Guy Shaked, Ke (Adam) Zhou, Fabrizio Silvestri, Michele Trevisiol, Ben Shahshahani, Puneet M Sangal and many others

Advertising Quality Science

Internet

Transcript of Advertising Quality Science