Persuasive Message Evaluation with Google AdWords
-
Upload
marco-guerini -
Category
Technology
-
view
684 -
download
3
description
Transcript of Persuasive Message Evaluation with Google AdWords
![Page 1: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/1.jpg)
Persuasive Message Evaluation with Google AdWords
1
Marco Guerini Carlo Strapparava Oliviero Stock
![Page 2: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/2.jpg)
Introduction
2
![Page 3: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/3.jpg)
3
Problems • Evaluating systems and theories about persuasion is a
bottleneck: expensive and time consuming experiments.
• In lab settings results are biased by subjects awareness
of being tested.
designing the experiment, inventing cover stories, recruiting a sufficient number of subjects, making them take part to the experiment, dispense questionnaire (pre-post), gather data, analyze them.
![Page 4: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/4.jpg)
Crowdsourcing - Mechanical Turk? Mechanical Turk used for behavioral studies (Mason and Suri, 2010) to overcome part of these limitations - like subject recruitment.
4
We believe that it is not the right instrument to assess behavioral changes, and, more generally, persuasive effects…
![Page 5: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/5.jpg)
Requirements
1. Studies as ecological as possible (real, even if controlled, scenarios).
2. Subjects must not be aware of being under observation.
3. Subjects must not be biased by external rewards.
5
![Page 6: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/6.jpg)
6
![Page 7: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/7.jpg)
Proposed Approach
Guerini M., Strapparava C. & Stock O. “Evaluation Metrics for Persuasive NLP with Google AdWords”. In Proceedings of LREC2010, May 2010.
7
![Page 8: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/8.jpg)
8
Approach • Goal: develop “cheap and fast” methodologies to
measure the persuasiveness of communication. • Solution: tweaking and using existing commercial tools
for advertising on the web, such as Google AdWords.
Conducting field experiments with thousands of subjects for a few dollars in a few hours
!
![Page 9: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/9.jpg)
9
Adwords Description • Google AdWords is Google advertising
program. It lets advertisers display their ads only to relevant audiences by means of keyword-based contextualization on the Google network, divided into:
• Search network: Google search pages, search sites etc.
• Content network: news pages, topic specific websites, blogs and other
![Page 10: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/10.jpg)
10
Put your test messages on the Web.
![Page 11: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/11.jpg)
11
AdWords Metrics Identifying the performance (persuasiveness) of each ad from various perspectives:
• CTR: the number of clicks divided by the number of times an ad has been displayed. Indicates the initial impact.
• Web Analytics: insights on landing site: number of visited pages, time spent on the site, etc. Indicates interest/attitude generated.
• Conversion Rate: how many user turned into conversions (e.g. buying a product, download a brochure). Indicates message complete success.
• ROI: The more relevant (from a research pov) the action the user performs, the higher the value.
![Page 12: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/12.jpg)
AdWords Targeting Targeting options for different evaluation scenarios:
• Language message impact in different languages.
• Location message impact in different cultures sharing the same language.
• Keyword matching message impact with users having different interests.
• Placements message impact on people with different values –democrats/republican sites.
12
![Page 13: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/13.jpg)
AdWords Tuning To facilitate statistically meaningful exp. (subj randomness, etc.) some tuning needed:
• Ad serving (optimize vs rotate) • Ad delivery (standard vs. accelerated) • …
No longer necessary.
Since our first experiments some proposal have been implemented in AdWords, under the tool AdWords Campaign Experiments (ACE)
13
![Page 14: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/14.jpg)
Experiments
14
![Page 15: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/15.jpg)
15
Experimental Setup • Pilot experiments to test irony, using a real
promotion campaign. • Experiments based on a between-subject
design with two conditions, one ad for each condition:
• Control condition, with a neutral message. • Experimental condition with ironical message. • χ2 test used on CTR.
![Page 16: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/16.jpg)
16
Experimental Design
Two experiments with img based ads on the content network (promoting the book series).
Two experiments with text based ads on the search network (promoting a book).
![Page 17: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/17.jpg)
Stimuli Material • “Diversamente Occupati” (tr. “differently
employed”) mocks the Italian “diversamente abile” (tr. “differently able”), a politically correct word for handicapped persons.
• “Diversamente Occupati” ironically refers to temporary employed (i.e. “differently employed”) people, generally considered a less than ideal state of employment.
17
![Page 18: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/18.jpg)
18
Experiments with text based ads
Context: Keyword “Precario”
Dura%on Cost Subjects
24 h 1,2 € -‐ CPC 0,04 € ~ 3.000
Dura%on Cost Subjects
24 h 1,5 € -‐ CPC 0,04 € ~ 4.000
![Page 19: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/19.jpg)
19
Results 1st Experiment - CTR (1A) = 1,7% CTR (1B) = 0,5% • Test significant condition 1A “Sei Precario?” (tr. “Are you
temporarily employed?”) performs three times better than 1B “Sei Diversamenteoccupato?”.
• Possible bias: given keyword matching the word “Precario” in condition 1A is automatically highlighted by Google.
2nd Experiment - CTR (2A) = 1,2% CTR (2B) = 0,8% • Test not significant first experiment biased by highlighting. • 2B “Sei DiversamenteOccupato?” outperformed other conditions (1A,
1B, 2A) w.r.t. permanence of users on landing site (~3 min vs ~1 min). • Capitalizing letter “O” in “DiversamenteOccupati” helped users spelling
the word and get the irony - that induced more interest in the subjects.
![Page 20: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/20.jpg)
20
Dura%on Cost Subjects
12 h 7 € -‐ CPC 0,60 € ~ 4.200
Dura%on Cost Subjects
1 week 2 € -‐ CPC 0,20 € ~ 12.000
Experiments with image based ads
Context 1: a site of job offers Context 2: the whole network
![Page 21: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/21.jpg)
21
Results 3rd Experiment - CTR (C1) = 0,5% CTR (C2) = 0,1% • Test significant, condition C1 performs five times better than C2. • Should we conclude that the img has a negative impact on the
message?
4th Experiment - CTR (C1) = 0,16% CTR (C2) = 0,08% • Test not significant In the previous experiment the img did not
have a negative impact by itself, rather it had a “positive” impact in cutting uninterested users, by helping them in “disambiguating” the message (users were looking for a job, not for amusement).
![Page 22: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/22.jpg)
Conclusion • AdWords is an appropriate context for
evaluating persuasive messages.
• AdWords proved to be very accurate, flexible and fast, far exceeding our expectations.
• The ACE tool bring us a step further.
22
![Page 23: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/23.jpg)
Further details in: Guerini M., Strapparava C. & Stock O. “Evaluation Metrics for Persuasive NLP with Google AdWords”. In Proceedings of LREC2010.
23
![Page 24: Persuasive Message Evaluation with Google AdWords](https://reader033.fdocuments.us/reader033/viewer/2022060117/5585721cd8b42a472c8b4c65/html5/thumbnails/24.jpg)
24