MESSAGE TESTING

Elsie Puig MMC5427 — Research Methods in Digital Communication July 13, 2016

for digital communications

MESSAGE TESTINGin a nutshell

http://blog.hubspot.com/agency/transforming-market-intelligence-to-intelligent-marketing#sm.0001731d8khl3d2apce2fj5p1mxsa

SOURCES

Market intelligence not only helps you spend effectively but it helps you improve the quality of your message. Message testing is not a new concept but not all approaches are equal. By quantifiably measuring the choices made during a message testing study, you can begin to identify the weight of individual messaging elements.This type of market intelligence enables you to predict purchase intent and ultimately the success of your concept or marketing message.“

— Isaiah Adams, Optimization Group

In our line of work, we’re constantly reminded that words have a profound impact on how we form our opinions and preferences. Whether its politics or policy, litigation, sales or marketing, putting a method behind the words you choose to use can make the difference between winning and losing.“ — Megan O’Hara, DialSmith

http://www.dialsmith.com/brands-marketing/word-games-putting-a-method-behind-the-words-you-choose

Message Testing Methods

•

http://her.oxfordjournals.org/content/23/3/467.fullSOURCES

A/B Testing and Multivariate Testing A/B testing is comparing two versions of a web page, or ad, to see which one performs better. With multivariate testing (unlike the A/B testing), there are no limits to the number of elements that you can vary.

Concept Testing Concept testing is the process of using quantitative methods and qualitative methods to evaluate consumer response to a product idea, or marketing campaign, before launching it to the market. Cognitive Response Testing Cognitive response testing (CRT), a form of qualitative research, allows the interviewer to probe for deeper understanding of comprehension and language by asking participants to paraphrase items, discuss thoughts or emotions that come to mind and offer suggestions for improvement.

Conjoint Analysis Conjoint (trade-off) analysis is one of the most widely-used quantitative method. It is used to measure preferences for product features, to learn how changes to price affect demand for products or service, and to forecast the likely acceptance of a product if brought to market.

Other MethodsFocus Groups

Interviews

Dial Testing

Word Associations

Sentence Completion

Memory/Recall

https://www.sawtoothsoftware.com/products/conjoint-choice-analysis/conjoint-analysis-software#tourism

Conjoint Analysis Example

•

Cognitive Response Example

•

Concept Testing

•

Google Content Experiments

•

Content Experiments uses a somewhat different approach than standard A/B and multivariate testing. Content Experiments uses an A/B/N model. You're not testing just two versions of a page as in A/B testing, and you're not testing various combinations of components on a single page as in multivariate testing. Instead, you are testing up to 10 full versions of a single page, each delivered to users from a separate URL.

More Info

What are we testing?

•

Message TypesCritical Case Sampling Choose content elements that are most significant in meeting your project goals. For example, if your project involves getting people to sign up to your newsletter, you should test the messaging of the sign-up form or content on the landing page.

Common Case Sampling Choose a representative content sample. If the goal of your campaign, or website, is to raise awareness on a particular public health issue and to influence people’s healthcare decisions, test the content for a typical healthcare topic.

http://www.uxmatters.com/mt/archives/2009/12/testing-content-concepts.phpSOURCES

Concepts

Logos and Slogan Associated Imagery

Ads (Online or Offline) Calls-To-Actions

Value Proposition Website and Content

Features and Benefits Promotional Messages

What do we need to know?

•

Message testing can scale all the way from an individual conversation with a handful of different people in your target audience, to a large-scale research effort involving surveys, competitive analysis, and focus groups. The key to getting message testing right is asking the right questions and making sure your questions aren’t inadvertently shaping the answers.“ - Meghan Keaney Anderson, HubSpot

Is our messaging focused on the right things? Say to the participant you're interviewing, "I'm going to list a series of features for an imaginary product. At the end of the list, I want you to tell me which of the features stood out to you as something you might want to know more about. If nothing stands out, say 'none.'" Then list the features in an even tone. Change the order for different interviews.

Does our product name resonate with people? Start out by providing a scenario that explains what the product does. In describing it, don't use any of the words you're already thinking about using for the product name. When you've finished laying out the scenario, ask the interviewee to tell you what words he or she would use to describe that functionality. Record what they say.

How will the message differentiate us from the competitors? Ask your interviewees: What other types of [product category] have you tried? What were the strengths of each? What were the weaknesses? If you were advising a friend or client, in what situations would you recommend one company's product over the other?

Will the message be perceived as consistent with what people have come to expect from your company? Consider the following as you're message testing: Do those words align well with how they describe the product you're introducing? Are they fairly consistent with how media or local associations view you?

http://blog.hubspot.com/blog/tabid/6307/bid/33981/The-5-Questions-You-Should-Ask-to-Nail-Your-Product-Messaging.aspx?source=Blog_Email_[The%205%20Questions%20You%20&buffer_share=03883&utm_source=buffer#sm.0001731d8khl3d2apce2fj5p1mxsa

SOURCES

The Non-Case Study: Groupon

•

The Message ”The people of Tibet are in trouble, their very culture in jeopardy. But they still whip up an amazing fish curry. And since 200 of us bought on Groupon.com, we’re getting $30 worth of Tibetan food for just $15 at Himalayan restaurant in Chicago.”

Link https://www.youtube.com/watch?v=pOwJOcp-Mxk

Case Study: Groupon

•Main Takeaway

Testing your messaging is important, especially when stakes are high and mistakes are costly

The Backlash After all the criticism, Groupon stopped airing the commercial and issued several apologies, but it was too late. The damage had been done, and news spread faster on social media. This could had been avoided if message testing had been performed.

https://www.visioncritical.com/7-critical-questions-ask-when-doing-ad-testing/ SOURCES

Case Study: Hyundai

•

The Message In preparation for the 2012 Super Bowl, Hyundai Motor Co. tested its commercial Think Fast many months in advance. Early versions included a conversation in which the older man, discussing business, made condescending remarks about “a blonde from Albuquerque.” Respondents found the older man offensive, so the South Korean car maker restructured the ad to have the older man chat about how to succeed in business, making him more amiable.

https://www.youtube.com/watch?v=mdVIirBUIn0

Link

https://www.visioncritical.com/7-critical-questions-ask-when-doing-ad-testing/SOURCES

Case Study: Hyundai

•

The Method Steve Shannon, vice president of marketing for Hyundai's American division, says that while Hyundai has used focus-group testing to help it select and shape its Super Bowl ads for several years this year it went even further and put its two

big game spots through an additional focus group test that was done online.

Main Takeaway Many companies test their ads before they are released to the public to prevent unexpected reactions and ensure their message is clearly communicated.

https://www.visioncritical.com/7-critical-questions-ask-when-doing-ad-testing/SOURCES

Case Study: National Coalition for Sexual Health

•

The Message The Communications Action Group (CAG) of the National Coalition for Sexual Health (NCSH) drafted a series of messages designed to motivate individuals to take action to protect and improve their sexual health.

LinkMessage Testing Report PDF

Case Study: National Coalition for Sexual Health

•

The Method Testing and refinement of draft sexual health messaging was conducted in two phases: Small group discussions and a national online panel review.

Message Refinement The first round of testing did reveal that some messages needed refinement and clarification. One of the revised messages was a statement in support of the sexual health benefit of having “positive relationships.” In multiple sessions participants reacted negatively to the inclusion of the words “when wanted” in the statement “Experience love, intimacy and affection, when wanted.” To these individuals, “when wanted” suggested an unwillingness or even resistance to experiencing love and affection.

Reference ListAdams, I. (2013, November 13). Transforming market intelligence to intelligent marketing. Retrieved from http://blog.hubspot.com/agency/transforming-market-intelligence-to-intelligent-marketing#sm.0001731d8khl3d2apce2fj5p1mxsa

Anderson, M. K. (2012, December 27). The 5 questions you should ask to nail your product messaging. Retrieved from http://blog.hubspot.com/blog/tabid/6307/bid/33981/The-5-Questions-You-Should-Ask-to-Nail-Your-Product-Messaging.aspx?source=Blog_Email_%5BThe+5+Questions+You+&buffer_share=03883&utm_source=buffer#sm.0001731d8khl3d2apce2fj5p1mxsa

Beggs, T. (2013, December 15). 7 critical questions to ask when doing ad testing. Retrieved from https://www.visioncritical.com/7-critical-questions-ask-when-doing-ad-testing/

Jones, C. (2009, December 21). Testing content concepts. Retrieved from http://www.uxmatters.com/mt/archives/2009/12/testing-content-concepts.php

Lapka, C., Jupka, K., Wray, R., & Jacobsen8, H. (2007, November). Health Education Research. Retrieved from http://her.oxfordjournals.org/content/23/3/467.full

O'Hara, M. (2015, February 15). Dialsmith. Retrieved from http://www.dialsmith.com/brands-marketing/word-games-putting-a-method-behind-the-words-you-choose

Pasha, L., & Lattery, K. (2007). Cutting through the clutter - message optimization research. Retrieved July 10, 2016, from http://www.maritztravel.com/~/media/Files/MaritzDotCom/White%20Papers/Research/Cutting-Through-the-Clutter.pdf

Results of Qualitative and OnlinMessage testing to support a sexual health campaign. (n.d.). Retrieved from http://nationalcoalitionforsexualhealth.org/data-research/body/NCSH-Message-Report-FINAL-09-17-2013.pdf

Survey software & conjoint analysis - what is conjoint analysis? (n.d.). Retrieved from https://www.sawtoothsoftware.com/products/conjoint-choice-analysis/conjoint-analysis-software#tourism Vranica, S. (2012, February 3). Advertisers practice for super bowl. Retrieved from http://www.wsj.com/news/articles/SB10001424052970203920204577196962118348008