Evaluating Social Media Interventions

Jason T. Siegel & Eusebio M. AlvaroClaremont Graduate University

Acknowledgements

• Centers for Disease Control• Mario A. Navarro, M.A. (CGU)

What is social media?• “The term social media refers to activities

among people gathered online who share information using conversational media that make it easy to create and share content in the form of words, pictures, videos, and audios.”

(Neiger et al., 2012)

What is social media?• Social networking sites (e.g., Facebook)• Creativity works sharing sites:

• Video (e.g., YouTube)• Photos (e.g., Flickr)• Music (e.g., Napster)• Intellectual property sharing sites (e.g.,

Creative Commons)

What is social media?• User-sponsored blogs (e.g., Cnet.com)• Company-sponsored web-sites• Company sponsored causes (e.g., Dove’s

campaign for Real Beauty)• Invitation-only social networks networking sites

(e.g., Asmallworld.net)

What is social media?• Business networking sites (LinkedIn)• Collaborative websites (Wikipedia)• Virtual Worlds (Second Life)• Podcasts• News delivery sites

(see Mangold & Faulds, 2009)

Why is social media unique?

• Offers a low cost delivery modality.• Allows for ongoing monitoring of the message

and how the message is being spread. • Opens a window into the lives of the target

audience. • It expands the directionality of communication.

Social media and health

Social Media and Evaluation

• “Despite the expanding use of social media, little has been published about its appropriate role in health promotion, and even less has been written about evaluation.”

(Neiger et al 2012 )

Can social media be evaluated?

• Yes. Good design principles still apply• Awareness is not change• Correlation is not causation• Views of a Facebook page are not necessarily

equated to behavior change.

What are you looking to do?• Program development

• Greene and colleagues (2011) used “wall posts” to diabetes Facebook sites to develop a campaign.

• Needs assessment• Scanfeld, Scanfeld, & Larson (2010)

analyzed Twitter posts to identify inaccurate beliefs associated with use of antibiotics.

What are you looking to do?• Process Evaluation

• Nguyen et al (2013) assessed relative impact of Facebook site components on participant interactions with a site distributing sexual health info (FaceSpace).

• Summative Evaluation***• Napolitano and colleagues (2013) assessed

whether a Facebook page plus texts and feedback lead to greater weight loss than a Facebook page only condition or a control?

What are you looking to change?

• Awareness• Attitudes• Interaction• Behavior

Awareness• Refers to the number of people who were

exposed to social media efforts.

Awareness• Examples:

• The goal of the campaign was to promote and increase awareness of LIVESTRONG and the LIVESTRONG navigation services(Justice-Gardiner et al., 2012). They compared print and media campaign to print, media, plus social media.

Awareness• Evaluating Awareness

• Traditional Methods• New Media Methods

Awareness• Traditional Methods

• Interviews• Focus Groups• Phone Surveys• Online Surveys

Awareness• Traditional Outcomes

• Ad recall (Broockman & Green, 2013)

• Awareness (Justice-Gardiner et al., 2012)

Awareness• New Media Methods

• Web hits (google analytics)• Total visits (Justice-Gardiner et al., 2012).

• Total page views• Total unique visitors• Total repeat view

• Number of video views (George et al., 2013)

• Twitter followers (Abroms & Lefebvre, 2009)

• Facebook friends or likes (Alhabash et al., 2013)

• Demographics of people visiting the websites (Pedrana et al., 2013)

Attitudes• An evaluative response to a specific stimuli

(attitude object).

Attitudes• Alhabash and colleagues (2013) used

status updates for a fictitious anti-cyberbullying organization to assess different messages.• Positive messages resulted in more positive

message evaluations and stronger anti-cyberbullying attitudes and viral behavioral intentions.

AttitudesAttitudes, Attitudes, Attitudes: • Attitudes towards a behavior.• Attitudes toward a website.• Attitudes towards a brand. • Attitudes towards a group of people (e.g.,

smokers).

Attitudes• Traditional Methods

• Interviews• Focus Groups• Phone Surveys• Online Surveys

Attitudes• Measuring attitudes with surveys

• Likert-type items (e.g., strongly disagree/strongly agree)

• Semantic Differential (e.g., good/bad, useful/useless)

• Open-ended questions

Interaction• Any communication between a user and the source

(social media site) or between a user and another user. • “interdependent message exchange”

(Burgoon et al., 2002)

• Interaction as a dyadic construct• True virtual “conversations” (vs. one-way info)

• Occurs within the social media universe• Counts of postings, conversations, shares, re-tweets,

etc.

Interaction• Examples:

• Nguyen et al (2013) evaluated reach of and engagement with FaceSpace – a sexual health promotion site. Peaks in site interactions (comments and likes) coincided with new video postings.

Interaction• Affordances: Structural aspects of a medium

that facilitate interaction (Burgoon et al., 2002)

• Contingency (tailored responses to queries)• Participation (active users)• Synchronicity (real-time message exchange)• Proximity (geographic dispersion)• Nonverbal richness (much contextual info)

Interaction (Measures)• Comments on videos

• YouTube comments: public library site (Vucovich et al., 2013)

• Posts (Facebook, Pinterest, etc)• FaceSpace: sexual health (Nguyen et al., 2013)

• Blog posts: “heart truth” heart disease & females (Taubenheim et al., 2008)

Interaction (Measures)• Twitter retweets

• Retweets – how to measure (Bruns & Burgess, 2012)

• Responses to source’s questions• Survey – PA via Facebook (Valle et al., 2013)

• Measure response (yes/no, frequency, rate)(not content of response)

Interaction (Measures)• Uploads

• Upload of non-user-created content (existing)• User-generated content

• Obama campaign: 1,792 user-developed videos compared to 329 videos developed for McCain (Abroms & LeFebvre, 2009)

• Participation in “Private” forums, private messages between users• MyBarakObama users interacted to set up

own events (Abroms & LeFebvre, 2009)

Interaction ‐ Continuum• Study of E-Health sites (Bacigalupe & Askari, 2013)

Measure of “collaborative potential”• Very Low Users only consume content• Low + Users comment on content• Mod + Users modify content• High + Users create content• Very High + Users develop virtual

community (“Prosumers”)

Facebook friend network

(Bull et al., 2012)

Twitter “authority value”

(Cataldi et al., 2010)

Behavior• Actions targeted for change by the social

media outreach/intervention.• Example: Web intervention for veteran with

problem drinking and PTSD symptoms (Brief et al., 2013).

Behavior• Program behavior vs. Health behavior

• Program behavior : Enacting site-sponsored actions, participation in site-sponsored events, etc. • Example:

• Health behavior: Traditional health outcomes targeted by an intervention.• Example:

Behavior• On-line behavior

• Targeted actions take place on a specific social media site or outside that site (but still online)

• Off-line behavior• Targeted actions motivated by a specific

social media site but occurring in the real world

Behavior• On-line behavior

• MyBarackObama (private site)1.5M volunteers, 100,000 user-organized events

• Viral sharing intentions (Alhabash et al., 2013)

To measure VBI, participants indicated their agreement/disagreement with five statements about whether the status update is worth sharing with others, whether they would recommend it to others, like it, share it, and comment on it on Facebook (1 = strongly disagree; 9 = strongly agree).

Behavior• Off-line behavior - Program Behavior

• Participation in program-organized eventsNapolitano et al., 2012 – Weight loss program72% - 88% RVSPs to events

• Cavallo et al., 2012 – Physical Activity studyParticipants post PA behavior on wall

Behavior• Outcome measures:

• Condom use (Bull et al., 2012)

• Self-reported physical activity (Cavallo et al.,2012)

• Reduction in Chlamydia cases (Jones et al., 2012)

• Number of people recruited (Lohse, 2013)

Design Considerations• What will you compare?

• Social Media versus nothing• Social media versus comparison

• Social Media vs. Traditional Campaign• Social Media vs. Social Media plus• Social Media A vs. Social Media B.

Design Considerations• How will you compare?

• True Experiment• Pre-test?

• Quasi-Experiment• Pre-test? • Multiple Pre-test?• Comparison Group?• Consistent Intervention Presence?

• Case Study

Twitter “trend” via keyword(earthquake vs. computer)

(Cataldi et al., 2010)

Design Considerations• Good design principles still apply

• Awareness is not change.• Correlation is not causation.• Views of a social media views are necessarily equated

to behavior change. • Consider possible confounds.

Summary• Social media offers exciting possibilities.• Evaluation of social media offers exciting

possibilities as well—particularly for summative evaluation.

• Evaluation of social media is in its infancy. • Fundamental rules of good design still apply.

ReferencesAbroms, L. C., & Lefebvre, R. (2009). Obama’s wired campaign: Lessons for public health communication. Journal

Of Health Communication, 14, 415-423. doi:10.1080/10810730903033000Alhabash, S., McAlister, A. R., Hagerstrom, A., Quilliam, E., Rifon, N. J., & Richards, J. I. (2013). Between likes and

shares: Effects of emotional appeal and virality on the persuasiveness of anticyberbullying messages on Facebook. Cyberpsychology, Behavior, And Social Networking, 16, 175-182.

Bacigalupe, G., & Askari, S. F. (2013). E-Health innovations, collaboration, and healthcare disparities: Developing criteria for culturally competent evaluation. Families, Systems, & Health, 31, 248-263. doi:10.1037/a0033386

Brief, D. J., Rubin, A., Keane, T. M., Enggasser, J. L., Roy, M., Helmuth, E., & ... Rosenbloom, D. (2013). Web intervention for OEF/OIF veterans with problem drinking and PTSD symptoms: A randomized clinical trial. Journal Of Consulting And Clinical Psychology, 81, 890-900. doi:10.1037/a0033697

Broockman, D. E., & Green, D. P. (2013). Do online advertisements increase political candidates’ name recognition or favorability? Evidence from randomized field experiments. Political Behavior, online first, 1-27. doi:10.1007/s11109-013-9239-z

Bruns, A., & Burgess, J. (2012). Researching news discussion on Twitter: New methodologies. Journalism Studies, 13, 801-814.

Bull, S. S., Levine, D. K., Black, S. R., Schmiege, S. J., & Santelli, J. (2012). Social media–delivered sexual health intervention: A cluster randomized controlled trial. American Journal Of Preventive Medicine, 43, 467-474. doi:10.1016/j.amepre.2012.07.022

Burgoon, J. K., Bonito, J. A., Ramirez, A., Dunbar, N. E., Kam, K., & Fischer, J. (2002). Testing the interactivity principle: Effects of mediation, propinquity, and verbal and nonverbal modalities in interpersonal interaction. Journal of Communication, 52, 657-677.

Cataldi, M., Di Caro, L., & Schifanella, C. (2010, July). Emerging topic detection on Twitter based on temporal and social terms evaluation. In Proceedings of the Tenth International Workshop on Multimedia Data Mining (p. 4). ACM.

Cavallo, D. N., Tate, D. F., Ries, A. V., Brown, J. D., DeVellis, R. F., & Ammerman, A. S. (2012). A social media–based physical activity intervention: A randomized controlled trial. American Journal Of Preventive Medicine, 43,527-532. doi:10.1016/j.amepre.2012.07.019

George, D. R., Dellasega, C., Whitehead, M. M., & Bordon, A. (2013). Facebook-based stress management resources for first-year medical students: A multi-method evaluation. Computers In Human Behavior, 29,559-562. doi:10.1016/j.chb.2012.12.008

ReferencesGreene, J. A., Choudhry, N. K., Kilabuk, E., & Shrank, W. H. (2011). Online social networking by patients with

diabetes: A qualitative evaluation of communication with Facebook. Journal Of General Internal Medicine, 26, 287-292. doi:10.1007/s11606-010-1526-3

Jones, K., Baldwin, K. A., & Lewis, P. (2012). The potential influence of a social media intervention on risky sexual behavior and Chlamydia incidence. Journal Of Community Health Nursing, 29, 106-120. doi:10.1080/07370016.2012.670579

Justice-Gardiner, H., Nutt, S., Rechis, R., McMillan, B., & Warf, R. (2012). Using New Media to Reach Hispanic/Latino Cancer Survivors. Journal of Cancer Education, 27, 100-104.

Lohse, B. (2013). Facebook Is an effective strategy to recruit low-income women to online nutrition education. Journal of Nutrition Education and Behavior, 45, 69-76.

Mangold, W. G., & Faulds, D. J. (2009). Social media: The new hybrid element of the promotion mix. Business horizons, 52, 357-365.

Napolitano, M. A., Hayes, S., Bennett, G. G., Ives, A. K., & Foster, G. D. (2013). Using Facebook and text messaging to deliver a weight loss program to college students. Obesity, 21 25-31. doi:10.1002/oby.20232

Neiger, B. L., Thackeray, R., Van Wagenen, S. A., Hanson, C. L., West, J. H., Barnes, M. D., & Fagen, M. C. (2012). Use of social media in health promotion: Purposes, key performance indicators, and evaluation metrics. Health Promotion Practice, 13, 159-164. doi:10.1177/1524839911433467

Nguyen, P., Gold, J., Pedrana, A., Chang, S., Howard, S., Ilic, O., & ... Stoove, M. (2013). Sexual health promotion on social networking sites: A process evaluation of The FaceSpace Project. Journal Of Adolescent Health, 53, 98-104. doi:10.1016/j.jadohealth.2013.02.007

Pedrana, A., Hellard, M., Gold, J., Ata, N., Chang, S., Howard, S., & ... Stoove, M. (2013). Queer as Fk: Reaching and engaging gay men in sexual health promotion through social networking sites. Journal Of Medical Internet Research, 15, 121-136.

Scanfeld, D., Scanfeld, V., & Larson, E. L. (2010). Dissemination of health information through social networks: Twitter and antibiotics. American journal of infection control, 38, 182-188.

Taubenheim, A. M., Long, T., Smith, E. C., Jeffers, D., Wayman, J., & Temple, S. (2008). Using social media and internet marketing to reach women with The heart truth. Social Marketing Quarterly, 14, 58-67. doi:10.1080/15245000802279433

Valle, C. G., Tate, D. F., Mayer, D. K., Allicock, M., & Cai, J. (2013). A randomized trial of a Facebook-based physical activity intervention for young adult cancer survivors. Journal Of Cancer Survivorship, 7, 355-368. doi:10.1007/s11764-013-0279-5

ReferencesVucovich, L. A., Gordon, V. S., Mitchell, N., & Ennis, L. A. (2013). Is the time and effort worth It? One library's

evaluation of using social networking tools for outreach. Medical Reference Services Quarterly, 32, 12-25.

Contact Information

Jason T. Siegel Eusebio M. Alvaro(jason.siegel@cgu.edu) (eusebio.alvaro@cgu.edu)

Co-Directors, Health Psychology and Prevention Science InstituteSchool of Social Science, Policy, and Evaluation

Claremont Graduate University