-

Target Modeling Validation Draft V.2

Prepared for the Council for Research Excellence by Gerard Broussard, Principal, Pre-Meditated Media May 25, 2017

2

TABLE OF CONTENTS

FOREWORD 3

EXECUTIVE SUMMARY 4

SCALING CONSUMER TARGET AUDIENCES 7 Modeled Consumer Targets: How Close Are They to Actual Target? 7 Modeling Procedures: Limited Transparency 8

IN-MARKET TEST DESIGN 8

TEST EXECUTION DETAIL 10 AIG Travel Insurance Consumer Journey 10 Media Selection, Media Target, Placement and DSP Selection 10 Truth File Preparation and Match Back 11 Online Survey 12 Back-end metrics 13

CAMPAIGN MODELED TARGET DELIVERY BY DSP 13

DSP MATCH & PERFORMANCE RESULTS 16 Performance Metric Summary and Insights 20

SURVEY RESULTS – DSP TARGET AUDIENCE PROFILES 21 Survey Metric Summary and Insights 25

QUESTIONS TO ASK TARGETING PARTNERS 26

STUDY EXECUTION INSIGHTS 27

3

FOREWORD

“ANA applauds the work of the CRE and the participation of AIG to learn more about the reliability of using modeled consumer data to inform media targeting decisions in advertising. In the digital advertising ecosystem, the lack of transparency in modeling adds to the various issues that have caused some advertisers to reconsider their digital investments. And to the extent that modeling becomes more prevalent in television, it's going to become an increasingly important issue there as well. Reliability in modeled data is critical to help advertisers regain trust and optimize their advertising spend.” Bill Duggan, Group Executive Vice President, Association of National Advertisers “At its core, the Council for Research Excellence (CRE) aims to conduct studies that help the marketplace develop responsible and credible research products that can be used to unearth great insights and facilitate honest trade. The burgeoning use of Big Data in the marketplace creates great opportunities to model audiences and targets with great precision – but only if data and techniques are both credible and reliable. The tools and techniques that are applied in Big Data analytics today are still largely unique and foreign to media marketplace decision-makers and often obscured from evaluation in black box, proprietary approaches. This compromises executives’ ability to properly assess service providers and their products. This project, conducted by the CRE’s Big Data Committee, is meant to help shine a light on variations that currently exist in the audience modeling sector so that the industry can begin to understand how to best use modeling capabilities to achieve superior business results.” Stacey Schulman, Executive Vice President, Katz Media Group; Chairperson, CRE Big Data Committee

4

EXECUTIVE SUMMARY

Background

Modeled Consumer Data Has Become the New Normal Combining media audience data with consumer transactions to inform media targeting and measure ad effectiveness has dramatically accelerated during the past several years. Modeling has become a significant element within these data integrations as no one data source provides comprehensive representation of the consumer population. In many cases, probabilistic methods are used to identify consumers that meet marketers’ targeting criteria. Modeling is Prevalent in Digital Lookalike modeling, for example, is a common digital marketing practice used to identify and advertise to people that resemble marketers’ most desired consumer segments Modeled Targets in TV Expected to Rise The use of modeled data in the TV space is likely to become more pervasive as advanced targeting deals are now being offered by major TV network groups. For example, Twentieth Century Fox, Turner and Viacom have combined to form Open AP to standardize advanced TV target data sources and definitions. 3rd party data enrichment providers may use some form of modeling to develop consumer target segments used in advanced TV targeting. More Modeling Transparency is Needed Not enough is known about how consumer models are constructed as well as how accurately they depict intended target consumers; many third party data providers are not forthcoming regarding techniques and validation procedures

Study Overview

• Lookalike modeling is a common digital marketing practice used to identify and then

advertise to people that resemble marketers’ most desired consumer segments.

• Some form of this modeling will likely become more prevalent for TV, where behavioral

advanced targeting implementation is accelerating.

• Purpose – The goal of this research was to determine how closely modeled/lookalike targets

reflect marketers’ true target audience.

• Method – AIG Travel Insurance participated in an in-market digital display advertising test of

modeled targets executed across six demand supply platforms (DSPs)

o Each DSP created modeled target cookies from an anonymized seed file sourced

from AIG Travel Insurance customer records

o Each DSP served 3mm unique modeled target ad impressions during a 5-week

period

5

Measuring Modeled Target Quality

Three target measurement points were used to gauge modeled target quality:

1. Modeled cookie match back – to a hold-out database of AIG Travel Insurance customer

records

2. Performance metrics –AIG Travel Insurance website back-end measures including ROAS

(return on ad spend) APV (average policy value), visit rate and conversion to policy sale

3. Target profile surveys – consumers reached by test campaigns provided demographics and

recent travel activity

Findings and Insights (results are white labeled for confidentiality)

• All DSP Modeled Targeting Eclipsed AIG Performance Benchmarks Despite Wide Swings in

Match Rates - In general, lookalike-targeted ads were effective, easily surpassing the AIG

Insurance ROAS benchmark by at least 3X. These results are an indicator of the strength of

modeled targeting relative to AIG’s experience with standard, in-market, non-programmatic

digital display placement. However, two other drivers likely contributed to ROAS lift that

were specific to the test campaign (versus the AIG benchmark): 1) ad frequency capping of

one time per unique cookie and 2) ad-retargeting of AIG Travel Insurance website visitors.

• Modeled Target Composition Was Substantially Higher Than Population Benchmarks

Survey results indicated that 71% of consumers exposed to any DSP advertising took a trip

during the past year versus 51% of respondents reported in MRI’s Fall 2016 Survey of the

American Consumer. This ratio was more than double for air travelers, a key customer

segment: 46% across the DSPs compared a 21% MRI benchmark.

• DSP 3 Modeling Excels

• Match rate and performance - DSP 3’s match rate was nearly double (185 Index) the

average, a strong indication that DSP 3-modeled cookies came closest to resembling

AIG customers and why DSP 3 was the leader in several other target validation

benchmarks including ROAS (166 index), APV (115 index) and volume of policies sold

(145 index).

• Survey results – Respondents who were served ads by DSP 3 possessed the highest

levels of desirable AIG Travel Insurance target characteristics including travel

incidence, balanced age distribution, female skew, household income and likelihood

to purchase travel insurance online.

• Parity ROAS, Despite Low Match Rates - DSPs with the lowest match rates still managed to

achieve ROAS on par with others. Low AIG Travel Insurance match rates paired with strong

ROAS suggests these DSPs were credited with significant sales from consumers with no prior

AIG purchase history.

• Modeling Performance Drivers - DSP 3’s compelling results invite further investigation of

their modeling processes and procedures to better understand why their approach excelled

above the others. Conversely, how did other DSP modeling techniques differ from DSP 3’s?

6

Ad tech firms will generally provide surface-level descriptions of their modeling

methodology but deeper disclosures are rare.

• Key Questions for Targeting Partners - While engaging modeled targeting partners, it is

advised to inquire about their experience in the ad category under study, modeling

processes, data quality and lookalike model validation procedures. Discussing these items

will help to get a sense of the likelihood that their modeled target solutions will resemble a

marketer’s target audience.

• Results Perspective - This study was conducted for one brand within the insurance category.

Results would certainly differ for brands within other categories according to variations in

purchase cycles, consumer tracking abilities, verifiable customer truth files and selection of

DSPs. The ad frequency cap of 1x per cookie likely drove ROAS performance by reducing

high-frequency saturation of consumers that might occur during standard AIG digital display

ad placement. Re-targeting of AIG Travel Insurance website visitors was another factor in

boosting campaign ROAS. From a macro view, campaign duration and ad weight levels also

come into play when assessing results. Had the test run for several months versus five

weeks or at higher ad impression levels, the ROAS may have started to recede in the

direction of the AIG benchmark.

• Study Execution Insights – Bottlenecks and delays were encountered during the course of

this study and the learning from these experiences is valuable for future research of this

type. For example, establishing privacy clearance should come first; two prospective

participants pulled out when their in-house legal team passed on the project, weeks after

initial discussions had progressed. On another front, several weeks transpired before

discovering that two of three prospective market research survey partners could not meet

sample size goals, after initially claiming they could. Finally, from a project management

perspective, frequent status updates across the multiple companies involved in the research

served to ensure that AIG, media agency Partners & Simon and matching company

LiveRamp were all in sync.

7

SCALING CONSUMER TARGET AUDIENCES

Modeled Consumer Targets: How Close Are They to Actual Target?

It is common practice for marketers to leverage sales and other transactional information in their consumer databases to acquire more customers who resemble the people who have been successfully acquired. By creating lookalike models of current customers, they aspire to increase their customer base. This practice is executed when digital ad tech firms create lookalike models of consumer targets based on a seed file of anonymized customer records combined with their own cookie pooling of online and offline purchase behavior. In most of these cases, not enough is known about how accurately models depict an advertisers’ consumer target or the process for building the models. No industry audits or standards currently exist to authenticate modeled targets or at least facilitate disclosure about the process for their creation. There is a need to understand the extent to which online target descriptors are reflective of actual attributes of marketers’ most desired consumers, especially when lookalike modeling is deployed for scaling target audience reach in the digital space. And the recent flurry of data integrations in the TV space suggest that some form of lookalike modeling is likely to take root to inform advertising placement on the first screen. For example, an advertisers’ first-party customer profile and sales data could be used as a starting point to model TV viewers that most closely resemble existing customers. Since the volume of lookalike modeling is likely to increase in the future, this study seeks to answer the following critical questions:

• To what extent do modeled lookalike targets depict a marketer’s true consumer target?

i.e., What portion of modeled target consumers are considered in-target vs. out-of-

target?

• To what extent does modeled target quality vary across ad tech firms that provide

modeled/lookalike targeting services in the digital space?

• Does more accurate modeled targeting translate into superior performance on driving

transactions?

• What are the critical questions to ask modeled targeting vendors to understand the

likelihood that their solutions will reflect desired consumer targets while improving

advertising performance? This line of inquiry may also serve as an opportunity to create

broader industry standards regarding data quality and model validation.

8

To answer these questions, the CRE embarked on this project in January, 2016 with the following goals: 1) Understand the procedures for creating modeled targets and their impact on targeting accuracy. 2) Gain knowledge about the accuracy of various modeling approaches taken by a cross section of ad tech firms (DSPs).

Modeling Procedures: Limited Transparency

The plan for achieving the first goal was to conduct in-depth interviews with eight ad networks or DSPs regarding procedural details for creating modeled targets. The hope was to gain intelligence and understanding regarding the similarities and differences across the variety of lookalike offerings in the marketplace. From a broader perspective, transparency around data quality and modeling techniques would serve as useful input for formulating industry standards for future research needs. Multiple attempts were made to schedule interviews, however, only one firm agreed to participate. Two firms declined the interview to avoid disclosing proprietary information, while the balance of those contacted simply did not communicate a reason at all, their silence perhaps signaling the same need for limited disclosure. Given this lack of participation, focus was turned to the second goal which was to collect insights about the accuracy of different modeling approaches. A modeled-target, results-based in-market test, was deployed in order to achieve this second objective. The balance of this report provides detailed coverage of study execution and findings.

IN-MARKET TEST DESIGN

The goal of the test was to determine to what extent modeled targets reflected an advertisers’ consumer target audience, ideally executed in a live, marketplace environment. The test design focused on determining the accuracy rate of modeled target cookies in identifying the advertiser’s consumer target segment across six demand supply platforms (DSP) targeting

9

partners. With the assistance of the Association of National Advertisers (ANA), the CRE recruited AIG Travel Insurance, an insurance policy brand geared for consumer travelers and sold direct through the AIG Travel Insurance website, mail or phone or through various resellers. AIG Travel Insurance was an appropriate fit for the test as the brand met the following test criteria:

• Privacy policy requirements - permission for using customer data to create and

advertise to lookalike modeled targets, including compliance with database matching

privacy rules.

• Customer data base size - millions of customer records to enable sufficient sample size

for creating a lookalike seed file and holdout sample

• Active digital display advertising - digital display campaign currently in market

• Technological tracking capability - campaign tracking to report modeled-target

advertising impact for each DSP

• Staff commitment – time and resources of internal team, third-party matching firm and

media agency

Three measurement points were used to gauge test campaign modeled target accuracy and

performance across the six DSPs:

1) Truth file match rates – campaign modeled cookies to AIG Travel Insurance customer

records

2) Performance metrics – e.g., ROAS (return on advertising spend), APV (average policy

value)

3) Online survey - demographic and recent travel activities

10

TEST EXECUTION DETAIL

AIG Travel Insurance Consumer Journey

Typical AIG Travel Insurance consumers are people who travel for personal reasons and buy protective life insurance for the trip. The act of buying travel insurance starts with trip planning and the purchase of transportation to the long-distance destination. Next, consumers deciding to buy travel insurance have available a number of ways they can make the purchase. For example, the options for buying policies for air travel, the most common travel insurance purchase, include: through the airline, a travel agent, credit card company or direct from the insurance company. The in-market modeled target test focused on direct sales transacted on the AIG Travel Insurance web site, the destination consumers reached by clicking on an AIG Travel Insurance display ad.

Media Selection, Media Target, Placement and DSP Selection

Display advertising was chosen for the test within browser-based desktop and laptop environments to enable adequate tracking and placement control. The mobile platform was not used given the challenges of tracking users within apps and the fact that the vast majority of mobile inventory is purchased programmatically, which would limit the ability to equivalize ad weight levels and ad frequency across the DSPs. One media agency, Partners and Simon (P&S), was selected to execute the test to control for any variations in price negotiations and general administrative procedures when interacting with the six DSPs. DSPs were selected that had experience creating modeled targets from anonymized customer seed files as well as prior work with P&S. Each DSP was sent an RFP that stated AIG Travel Insurance’s need to gain insights into the speed at which modeled target consumer impressions build over time to inform execution of future

11

full-scale modeled target ad programs. DSPs were also sent a seed file of anonymized customer IDs from which they would create lookalike targets and a media target:

• Frequent travelers, taking 3 or more trips a year, primarily for personal or

leisure reasons

• Purchased travel insurance online

• High income households of $75K+

• Female

Targeting partners possessed no prior knowledge of a multi-DSP test to ensure that modeled targets would be developed in a natural marketplace environment, minimizing any special treatment or process biases. P&S instructed each of the six DSPs to run 3 million unique modeled target impressions with a frequency cap of one ad exposure during a four-week period, which was the estimated time for all DSPs to achieve the 3-million unique cookie goal. A frequency cap of one ad per unique user was deployed to increase the likelihood that a sufficient portion of people reached by the test campaign would also be found in AIG Travel Insurance’s CRM truth file. So, for example, serving ads to 3 million unique cookies one time rather than 500,000 six times improved the chances of matching back to the CRM truth file. In standard practice, frequency caps are usually higher, e.g., three exposures and are used to minimize over saturation of consumers, thereby reducing ad waste.

Truth File Preparation and Match Back

A CRM truth file was created to provide a measurement point for assessing how closely the six DSPs’ modeled cookies resembled the AIG Travel Insurance customer profile. The assumption was that high match rates between DSP modeled cookies and AIG’s Travel Insurance customer hold-out sample would provide an indication of superior modeling performance. Third-party data provider LiveRamp matched AIG Travel Insurance customer records to LiveRamp’s consumer database, achieving a 40% overlap between the data sets. Only one matching partner was used in order to control for variations in test campaign matches that would occur if a mix of match partners were used across the DSPs. LiveRamp’s anonymized IDs are commonly used by DSPs and other ad tech firms for linking customer transaction data with digital media activity. Approximately 17% of matched customer records were assigned to a seed file and sent to each DSP, to be used for modeling lookalike ad impressions. The remaining 83% of records were designated to a hold-out group to be matched a second time against the unique impressions served during the test campaign, to provide a measure of each DSP’s modeling performance.

12

AIG Travel Insurance sent each DSP partner the same seed file of anonymized LiveRamp customer cookie IDs. The DSPs matched the seed file against their internal cookie data to identify other cookies possessing demographic and behavioral profiles similar to the seed file. On the January 3rd, 2017 start date, the DSPs began to run their modeled target cookies and all DSPs achieved the 3 million unique impression target goal on February 7th, 2017. After the test campaign, each DSP’s modeled target cookies were matched back to the AIG Travel Insurance hold-out file of customers. Match rates were calculated by dividing the number of unique cookies that matched back to the AIG Travel Insurance holdout group by the 3 million unique impressions served. Match rate performance was used as an indicator for the strength of modeling approach for reaching AIG Travel Insurance target.

Online Survey

A sample of 677 persons, divided among the six DSP campaign groups, provided information about their demographics, travel activity and purchase of travel insurance. This self-reported data was used as another measurement point to determine the portion of respondents to be classified in- or out-of-target and included:

• Age, gender, income, residential zip code

• Travel activity during the past 12 months

o Air, train, ship/boat, auto, other

• Travel insurance purchase during past 12 months

o Direct from insurance company, transportation company, agent, etc.

Research firm Millward Brown recruited respondents from its consumer panel and administered the surveys in coordination with media agency P&S. Tracking tags were used to create six unique campaigns so that modeled target consumer profiles could be created from survey results and compared across the DSPs.

13

Back-end metrics

Performance measures collected from the AIG Travel Insurance website provided further insight into the quality of each DSP’s modeling. Media agency P&S tagged the six test campaigns with a unique pixel so that modeled cookies could be tracked and correctly attributed to each DSP and report the following metrics:

• ROAS (Return on Advertising Spend)

• Average policy value sold

• Conversion rate to policy sale

• Purchaser descriptors – Age, number of travelers, destination, time to trip, etc.

AIG Travel Insurance deployed a rules-based attribution model to credit each DSP with site visit activity, including policy quotes and sales.

CAMPAIGN MODELED TARGET DELIVERY BY DSP

This section provides an overview of modeled target ad delivery across the DSPs, including the timing and scheduling patterns for achieving the 3 million unique modeled impression goal. All DSP names have been white labeled (DSP 1, DSP 2 . . . DSP 6) so that test results will have no marketplace impact on DSPs that were monitored. As a further step to maintain DSP anonymity, the DSP number designations in this section will not tie directly to those found in Study Findings. Ad Delivery Timing - All DSPs achieved the 3 million unique modeled impression goal within a 5-week period starting January 3rd. The pace of impression accumulation was similar for four of six DSPs, however, DSPs 4 and 6 followed extremely different patterns from the rest:

• DSP 4 – Gradual build - After a campaign start delay, DSP 4 claimed that their modeling

technique required about two weeks of campaign data gathering to fully develop their

lookalike targets. The daily rate of ad impressions served increased dramatically toward

the latter half of January and DSP 4 finished seven to ten days later than the others, on

average.

• DSP 6 – Rapid acceleration - DSP 6 met 97% of the goal on day 3 of the test campaign,

contending that they were eager to demonstrate how quickly they could scale modeled

target reach; the DSP also claimed to have access to a large portion of internet cookies.

14

DSP Ad Frequency – All DSPs were instructed to maintain a frequency cap of one ad per cookie to increase reach and support higher cookie match rates with the AIG Travel Insurance hold-out sample. Gross ad impressions for DSPs 2, 4 and 6 surpassed the 3 million unique goal by sizable margins with much of this impression over delivery likely due to cookie expirations and different modeling approaches taken. DSP 2 amassed nearly twice the number of unique counts because it operated under on a Cost-Per-Click basis, buying the amount of impressions necessary to meet performance goals while also accumulating 3 million unique modeled impressions. Ultimately, differences in ad frequency across DSPs appeared to have no impact on performance-based measures. Some DSPs exceeded the frequency cap. For example, DSP 1, a cost-per-click network delivered additional ad impressions to meet performance-based goals while DSP 4 did not impose the frequency cap during the first few days of the test campaign.

15

DSP Ad Impression Overlap - Cookie overlap was low across DSPs, with the exception of DSPs 5 and 6. 27.5% of DSP 6 cookies were in common with those of DSP 5 while nearly a third of DSP 6 cookies overlapped with DSP 5.

16

DSP MATCH & PERFORMANCE RESULTS

Three measurement points were used to gauge test campaign modeled target accuracy and

performance across the six DSPs:

1) Truth file match rates – campaign modeled cookies to AIG Travel Insurance customer

records

2) Performance metrics – e.g., ROAS (return on advertising spend), APV (average policy

value)

3) Online survey - demographic and recent travel activities

This section focuses on the variation in match rates combined with AIG Travel Insurance

performance measures (numbers 1 and 2 above) to provide an indication of DSP model quality.

Survey findings will then provide a view of how closely the DSPs’ modeled targets resembled the

AIG travel Insurance target. The following definitions provide a reference for match rate and

performance results:

DSP Match Rate - number of modeled ad impressions that match to AIG Travel Insurance holdout sample expressed as a percent of unique ad impressions served, for each DSP ROAS (Return on Advertising Spend) - total value of AIG Travel Insurance policies sold divided by the cost of media purchased; equal media investment across DSPs APV (Average Policy Value) - the average value of policies sold for each DSP Visit Rate - number of visits divided by the number of gross ad impressions for each DSP Visit-to-Sale Conversion Rate – the portion of total visitors that purchased an AIG Travel Insurance policy Test Attribution Window – All DSPs reached the 3 million modeled impression goal by February 7th, however, the AIG Travel Insurance website performance results were tracked for an additional 28 days, until March 7th, to capture the residual impact of advertising and follow up site visits. All results have been indexed to maintain the proprietary nature of the data for the AIG Travel Insurance business. DSP labels, e.g., DSP 1, DSP 2 . . . DSP 6 are consistently referenced throughout the rest of this report. Match Rate - Match rates varied widely across the six DSPs, with DSP 3 achieving nearly 2X (185 index) the norm, followed by DSP 4 at 141 index. These high-performing DSPs were more effective at modeling AIG Insurance Travel customers than others, particularly DSPs 2 and 5. But low match rates were not a definitive indicator of modeling quality; pairing match rates with other AIG Travel Insurance website measures demonstrates parity model performance in the content that follows.

17

ROAS (Return on Advertising Spend) - DSP 3’s stand-out match rate was complemented by the highest ROAS. ROAS of the remaining five was in parity, despite the variation in match rates and the same media dollars invested across all DSPs. For example, match rate indexes for DSP 2 (57) and DSP 5 (14) were far below the 6-DSP average, however, their ROAS was on par with all others except for DSP 3.

ROAS versus AIG Travel Insurance Benchmark - All DSP lookalike modeling solutions dramatically improved ROAS compared to an AIG Travel Insurance benchmark of non-programmatic, non-modeled, non-retargeted display placement. Modeled target ROAS surpassed the AIG Travel Insurance benchmark by at least 2X, with DSP 3 at nearly 5X margin.

18

AIG is now considering increasing its display advertising budget using a modeled approach since ROAS results signal room to grow the business profitably.

APV (Average Policy Value) and Number Policies Sold - DSP 3 was credited with generating the most policy sales (45% above norm) at the highest average price (+15% above norm); these exceptional price and volume levels combined to drive DSP 3 ROAS 66% above the average. All other DSPs’ metrics hovered around the norm.

APV versus AIG Travel Insurance Benchmark - APV for all DSPs was moderately above the AIG Travel Insurance non-programmatic display benchmark, except for DSPs 4 (134 Index) and 6 (125 Index); this implies that brisker policies sales through more precise targeting contributed

19

more to driving ROAS than higher policy values, which provided a more secondary lift versus the AIG Travel Insurance benchmark.

Visit Rate and Conversion to Sale - Volume of traffic and policy sales conversion rate provided additional insights regarding the distinct paths for achieving parity ROAS across the DSPs. For example, the sheer volume of visitors sourced from DSPs 1 and 2 bolstered ROAS, even though their sales conversion rates were at or below average levels. Conversely, DSPs 5 and 6 drove fewer visitors, but with a higher concentration of in-market insurance buyers; these DSPs registered the highest conversion to policy sale.

20

Performance Metric Summary and Insights

• All DSP Modeled Targeting Outpaced AIG Performance Benchmarks Despite Wide

Swings in Match Rates. In general, lookalike-targeted ads were effective, easily

surpassing the AIG Travel Insurance ROAS benchmark by at least 2X. These results are

an indicator of the strength of modeled targeting relative to AIG’s experience with

standard digital placement.

• DSP 3 Produced the Best Model DSP 3’s match rate was nearly double (185 Index) the

average, a strong indication that DSP 3-modeled cookies came closest to resembling the

AIG customers in the seed file provided to all DSPs. These strong match results offer an

indication of why DSP 3 was the leader in several other target validation benchmarks

including ROAS (166 index), APV (115 index) and volume of policies sold (145 index).

• Parity ROAS Despite Low Match Rates - DSPs with the lowest match rates still managed

to achieve ROAS on par with others, most likely because their ads reached more

consumers with no past AIG online purchase history, but who were in-market for travel

insurance. This would explain why match rates with the AIG Insurance hold-out file

were low. Lookalike ads served by these DSPs likely attracted customers of competing

AIG brands, driving them to the AIG Insurance site where they made a purchase.

• Visitor Volume vs. Visitor Quality - There was a performance “wash” of sorts between

DSPs regarding their path to ROAS on the AIG Travel Insurance website. Some DSPs

generated high traffic volume with low sales conversion rates. Other DSPs produced

lower site traffic but those visitors converted at a very high rate.

• Perspective: Scaling ROAS Results - This test was primarily designed to observe

variations in targeting profiles and performance across lookalike DSP solutions in the

marketplace. The modeling appears to have contributed to the dramatic ROAS lift

versus the AIG benchmark, however, two other test campaign ROAS drivers came into

play: ad frequency capping and re-targeting AIG Travel Insurance website visitors with

ads.

O Ad frequency capping - Test campaign ad frequency was capped at one time per

modeled cookie versus the typically-higher AIG display benchmark ad frequency

of three or more ads per cookie. Frequency capping is used to improve

advertising impact by reducing wasteful high-frequency saturation of the same

consumers.

O Website visitor ad re-targeting - Visitors driven to the AIG Travel Insurance

website from the test campaign were re-targeted with additional advertising

that likely drove more policy sales. The AIG benchmark includes no re-targeted

advertising activity.

21

From a macro view, campaign duration and ad weight levels also come into play when

assessing results. Had the test run for several months versus five weeks or at higher ad

impression levels, the ROAS may have started to recede in the direction of the AIG

benchmark.

SURVEY RESULTS – DSP TARGET AUDIENCE PROFILES

677 people exposed to the test campaigns completed online surveys and provided information on their demographics, travel activity and travel insurance purchases during the past year; no survey questions were included to measure the AIG Travel Insurance ads’ branding impact. This survey data afforded another checkpoint for measuring how closely the people that were served modeled target ads resembled AIG Travel Insurance’s consumer target:

• Frequent travelers, taking 3+ long distance trips for personal or leisure reasons

• Purchased travel insurance online

• Household income $75k+

• Female

Modeling Target Composition Exceeded Travel Benchmarks - Past travel activity is considered a barometer for future behavior and potential business for AIG Travel Insurance. Millward Brown survey results indicated that 71% of consumers exposed to any DSP advertising took a trip during the past year versus 51% of respondents reported in MRI’s Fall 2016 Survey of the American Consumer. Meanwhile, 46% of respondents used air travel, the primary transportation mode for travel insurance purchasers, which is more than 2X incidence (21%) found in MRI’s survey. Both advantages demonstrate the ability of the DSP modeling to identify a significantly higher concentration of potential travelers than is found in the population at large.

22

Since trip takers form a core universe for AIG Travel Insurance targeting, the balance of survey insights will concentrate on differences in trip-taker demographics and insurance purchases across DSPs. Throughout this series of survey metrics, DSP 3, the leader in match rate, ROAS, value and volume of policies sold, continues to demonstrate its superior modeling through survey results. DSP 3 - Trip-Taker Leader - The higher the trip-taking incidence, the more likely that consumers reached through modeled target cookies are in-market for travel insurance. DSP 3 came out on top with the highest portion (75%) of respondents taking a trip during the past year, while DSP 2 was lowest (64%). All other DSPs hovered more closely to the 70% mark.

23

Age Varied Across DSPs, DSP 3 evenly balanced - Age profiles across the six DSPs were distinctly skewed. For example, about a third of DSPs’ 1 and 2 respondents were over 65 while DSPs 4, 5 and 6 were decidedly younger, under 45 years old. DSP 3 possessed the most balanced composition across the age groups, suggesting more opportunities to target travelers of all ages.

DSP 3 - Highest Women Concentration - Across all DSPs, 65% of respondents were female, which is consistent with the AIG Travel Insurance target’s female skew. DSP 3’s 80% female composition, however, was significantly higher than all others.

24

DSP 3 - Highest Income – DSP 3 respondents were by far the most upscale with 37% living in homes with $100K+ income compared to the 24% DSP average, suggesting more discretionary income for travel. All other DSP household income fell around the six-DSP average.

DSP 3 - Most Likely to Purchase Insurance Online – Across all DSPs, 60% of respondents who bought any travel insurance claimed to have purchased it online. This figure was 86% for DSP 3, however, and provides more evidence of DSP 3’s advantage for modeling the AIG Travel Insurance target. While the sample size for this finding is relatively low, it provides compelling support for the quality of DSP 3’s modeling when viewed with all other DSP 3 test results.

25

DSP 3 – More Air Travel – AIG Travel Insurance policies are purchased primarily for air travel so a DSP’s ability to target consumers who are likely to book airline trips is another measure of modeling strength. The DSPs were relatively even in this regard, although DSP 3 registered the highest air travel rate (69%) among trip takers.

Survey Metric Summary and Insights

• Modeled Target Composition Was Dramatically Higher Than Population Benchmarks

Survey results indicated that 71% of consumers exposed to any DSP advertising took a trip

during the past year versus 51% of respondents reported in MRI’s Fall 2016 Survey of the

American Consumer. This ratio was more than double for air travel: 46% across the DSPs

compared to a 21% MRI benchmark

• DSP 3’s consumer profile demonstrated further evidence of superior modeling

DSP 3 survey respondents consistently possessed desirable AIG Travel Insurance consumer

target characteristics:

• Most likely to have taken a long-distance trip in the past year (75%) vs the six-DSP

norm, a predictive indicator of future travel

• Highest female skew, consistent with the AIG Travel Insurance target

• Most evenly-distributed age profile relative to other DSPs, some which skewed

markedly older or younger

• Highest $100,000+ household income by a wide margin, implying more

discretionary resources for travel

• Highest likelihood to purchase insurance online

26

Questions to Ask Targeting Partners

While engaging modeled targeting partners, it is advised to inquire about their modeling

processes, data quality and model validation procedures. Discussing these items will help to get

a sense of how likely modeled target solutions will improve advertising performance. Below is a

list of questions for current and prospective modeled targeting partners:

• Please identify the sources and types of data that your firm uses for implementing online

targeting, e.g., cookie pools sourced from Blue Kai, demographic profiles sourced from

Acxiom, etc.

• Please describe the process of integrating these data sources for implementing your

targeting solution

• Please provide detail about your data recency and frequency of updates for each of the

sources included in your solution

• Please provide detail on the collection timeframe for all types of data. E.g. Jan - Oct 2016

• Please provide detail on the representativeness or completeness of any transactional data

that informs the targeting capability that you provide. E.g., credit card vs. cash transaction

• Please describe the technique(s) used for modeling lookalikes to scale targeting

• Please describe the procedure(s) in place for model validation. Are the procedures

performance-based? E.g., test ad response. Are they target-descriptive? E.g, validated

against a truth file data set.

• How often are targeting models validated?

• To what extent does the modeled target solution resemble the originally described target

audience?

• Please describe any "truth" data sets that are used to modeled target consumers.

27

STUDY EXECUTION INSIGHTS

Challenges were encountered during the course of this study and the learning from these experiences is considered valuable for future research of this type. First priority: privacy clearance – Before embarking on any market/media research project that requires access to customer data, confirm that the use of this data is compliant with the marketer’s privacy guidelines. While advertising/marketing and data/research personnel were the primary advertiser points of contact for the design and execution of the CRE study, privacy and legal experts held the final word on the use of customer data. Time and effort could have been conserved if privacy had weighed in on the test during the proposal stage of the project. Below are two types of such scenarios encountered during the advertiser recruitment phase of the project:

• The marketing champion – this refers to persons who are enthusiastic about pushing the

initiative forward by gathering internal alignment at the company without making privacy

the first hurdle to overcome. In one instance, the CRE’s marketing and advertising contacts

were on board to participate in the test, but weeks into the recruitment process privacy

personnel confirmed that the use of the customer data would have been in violation of their

policy.

• Privacy policy shift – In another instance, an advertiser participant was several weeks into

the actual process of setting up the test but pulled out when they became aware that their

privacy policy had changed to more conservative European standards. In retrospect, it

would have made sense for the point of contact to get in touch with privacy personnel

during the early stages of the project to see if any changes would be anticipated regarding

permissible uses of customer data.

Truth file creation - Probably the most challenging aspect of the CRE project was establishing a truth file that could be used to fairly benchmark modeling performance across the six DSPs. At issue was the ability to use data that none of the DSPs had access to from their previous modeling endeavors. The CRE first attempted to deploy one or two Data Enrichment Providers (DEPs) to serve as truth file sources, since some DEPs have access to a rich source of authenticated consumer records originating from their credit rating services. The test called for one truth file DEP source to be used as a consistent benchmark across all DSPs for a single advertiser. While this approach ensured uniformity of truth file consumer records, it also introduced the possibility that some DSPs who had previously partnered with the truth file DEP would hold unfair advantage when matching modeled cookies back to the truth file. In the chart below, DSP 1 holds a matching advantage if DEP X is the exclusive truth file source; DSP 6 benefits if DEP Y is selected as the sole truth file source. This potential imbalance in match rates ultimately led to establishing a truth file from a marketer’s customer records, which no DSPs had previous access to.

28

Selecting a survey partner: meeting sample size requirements – Surveys were administered during the test campaign to determine how closely the people who were served modeled cookies met target audience criteria. Three market research companies were engaged in the RFP process. Since each DSP was budgeted to serve 3 million unique impressions, it was imperative that a survey partner have access to a large pool of panelists to ensure a minimum of n=100 sample for each DSP. During the first wave of RFP responses, all three candidates indicated that they could meet sample size goals. Upon further inquiry, however, it was discovered that the panel size of two of the three firms was not sufficient for achieving the desired sample counts without substantially increasing the project media budget, i.e., to buy more ad impressions. Each firm had the technical ability to execute the study but there was a chasm of difference regarding survey panel size. The odds of selecting a survey partner that could not meet sample size objectives within the given media budget was two out of three, based on information contained in the first wave of proposals. Much credit should be given to the CRE Big Data Committee for requesting extensive drill-down information on all potential partner capabilities to identify the survey partner with a sufficient number of panelists. Technology IQ and communication– This project required the involvement of more than ten different companies, each with their own expertise and perspective of process and execution:

• 1 advertiser (AIG)

• 1 matching company (LiveRamp)

• 1 media agency (Partner & Simons)

• 6 demand supply platforms (DSPs)

• 1 survey partner (Millward Brown)

• 1 project manager (Pre-Meditated Media on behalf of CRE)

It is critically important to note differences in level of understanding regarding processes, nomenclature and project continuity. Some examples:

Processes - a customer match-back of this type had not been executed at AIG prior to this project. Consequently, a substantial amount of time was required for LiveRamp to

29

establish the process and for the approach to be articulated and understood by all parties in the mix. Nomenclature - the term “holdout” group was not a familiar name to many involved; most were more familiar with “control” that is used in experimental designs. Continuity – during the course of the project, there were many procedural interruptions sourced from delays in operational readiness due to permissions and preparation required to enable the matching capability. Status meetings were regularly held to ensure that all participants were updated and advised of their roles in the process. This continuous communication ensured a smooth execution, once the test campaign was executed.