The Fake News Spreading Plague: Was it Preventable?

Eni MustafarajDepartment of Computer Science

Wellesley CollegeWellesley, MA, USA

emustafaraj@wellesley.edu

Panagiotis Takis MetaxasDepartment of Computer Science

Wellesley CollegeWellesley, MA, USA

pmetaxas@wellesley.edu

ABSTRACTIn 2010, a paper entitled “From Obscurity to Prominencein Minutes: Political Speech and Real-time search” [8] wonthe Best Paper Prize of the Web Science 2010 Conference.Among its findings were the discovery and documentationof what was termed a “Twitter-bomb”, an organized effortto spread misinformation about the democratic candidateMartha Coakley through anonymous Twitter accounts. Inthis paper, after summarizing the details of that event, weoutline the recipe of how social networks are used to spreadmisinformation. One of the most important steps in sucha recipe is the “infiltration” of a community of users whoare already engaged in conversations about a topic, to usethem as organic spreaders of misinformation in their ex-tended subnetworks. Then, we take this misinformationspreading recipe and indicate how it was successfully usedto spread fake news during the 2016 U.S. Presidential Elec-tion. The main differences between the scenarios are the useof Facebook instead of Twitter, and the respective motiva-tions (in 2010: political influence; in 2016: financial benefitthrough online advertising). After situating these eventsin the broader context of exploiting the Web, we seize thisopportunity to address limitations of the reach of researchfindings and to start a conversation about how communi-ties of researchers can increase their impact on real-worldsocietal issues.

Keywordsfake news; misinformation spreading; Facebook; Twitter;Google

1. INTRODUCTION

1.1 The Anatomy of a political Twitter-BombIt was January 15, 2010, when a group of nine Twit-

ter accounts, all created within an interval of 15 minutes,with names such as “CoakleyAgainstU”, “CoakleyCatholic”,etc. These were accounts with names starting with the

Figure 1: The journalist John Carney (at that timewith CNBC), received one of these “reply-tweets”,which he retweeted adding a comment expressinghis surprise, “<Wow! Political Tweetbots!–JC>”,because this was an unknown phenomenon at thattime on Twitter. Carney deleted the URL of theoriginal tweet.

name of the democratic candidate Martha Coakley who wasrunning in the Special Election for the Massachusetts U.S.Senate seat. The newly minted accounts sent 929 tweetsaddressed to 573 unique users in the course of 138 min-utes. All the tweets contained a URL to the same websitehttp://coakleysaidit.com, (also registered on January 15,2010), that showed video and audio from a speech by MarthaCoakley, taken out of context, to advance the false claim thatshe is against the employment of Catholics in the emergencyroom.

The nine accounts were sending a tweet per minute andrepeating more or less the same content aAS reasons tobe flagged as a spamming account. Twitter discovered theautomated tweets and consequently suspended all nine ac-counts. Their existence and their misinformation attackwould have gone unnoticed had it not been for one fortunatecircumstance: we were collecting all tweets containing thenames aAIJcoakleyaAI and aAIJbrown,” for Martha Coak-ley and Scott Brown respectively who were the two candi-dates for the senate election, in real-time during the weekleading to the election. But, the tweets sent by these anony-mous accounts were not simple tweets, they were so-called“reply tweets”, tweets directed to particular users. Why?Because a new account on Twitter doesn’t have any follow-ers. Tweets sent by such an account will not be read byanyone. Thus, addressing the tweets to a particular usermakes it likely that the tweet will be read. But to whatuser do you send the tweet, out of the millions that are onTwitter? This is when a common spamming technique on

arX

iv:1

703.

0698

8v1

[cs

.SI]

20

Mar

201

7

Twitter comes in handy: reply to users who used certaindesired keywords in their tweets, that is, to users alreadyattuned to the topic. Our analysis of the recipients of these“reply tweets” revealed that 96% of them had been tweetingabout the MA senate race in the four-hour interval betweenthe time the anonymous accounts were created and whenthey started to send the “reply tweets”. Almost 25% of theusers who received a tweet (143 out of 573) retweeted themessage. A screenshot from one of the retweets is shown inFigure 1. We chose to show this tweet, because the user isa well known journalist1 and experienced user who joinedTwitter on March 2007. His surprise at the message indi-cates the novelty of this technique at the time. The retweetshad the effect that the followers of the retweeters were likelyexposed to the misinformation, which they would have notseen otherwise. This is because the messages didn’t includehashtags, a common way to group together tweets about atopic. Our estimation of the audience size, based on the fol-lowers of the retweeters, amounted to 61,732 Twitter users.

1.2 A recipe for spreading misinformation onTwitter

All the facts presented in the previous subsection werepart of [8]. What we didn’t do in that paper was to summa-rize our findings in an easily memorable recipe, which con-tains the steps used by the propagandists in spreading theirmisinformation on Twitter. We’re providing this recipe forthe first time in this paper.

Step 1 Register a domain name for a new website,for example: http://coakleysaidit.com

Step 2 Create anonymous accounts, for example:CoakleySaidWhat, etc.

Step 3 Identify a community of users interestedin the topic, for example, the MA SenateElection race.

Step 4 Target members of this community withmessages, for example, reply to users pro-viding link to website.

Step 5 Wait for members of community to spreadmessage via retweets in their organic sub-networks.

Table 1: A recipe for spreading misinformation onTwitter via a “Twitter-Bomb”.

Our discovery attracted the attention of both journalistsand other researchers. A team at Indiana University, headedby Fil Menczer, developed Truthy 2, a system that collectsTwitter data to analyze discourse in near real-time [13]. Inaddition, our team at Wellesley developed Twitter Trails 3, asystem that can be used to monitor the spreading of rumorson Twitter [7]. This focus on Twitter is justified by the factthat it provides APIs for researchers to collect and analyzeits data, as well as the public nature of conversations onTwitter. Both these features are missing on Facebook (notentirely, but they are severely limited), thus, only Facebookemployees are able to study them. As evidence, see [2].Meanwhile, researchers not affiliated with the company havealmost no opportunities to study information spreading on

1John Carney, https://twitter.com/carney.2Truthy, now known as OSoMe, http://truthy.indiana.edu3TwitterTrails.com, http://twittertrails.com

Facebook, especially that of rumors, hoaxes, and recentlyfake news, a topic to which we turn our focus now.

The term “fake news” refers to lies presented as news, thatis, falsehoods online formatted and circulated in such a waythat a reader might mistake them for legitimate news arti-cles. “Fake news” has been around since ancient times, butinformation technology has made it possible to produce andconsume it on a massive scale. Such articles appear on a vari-ety of little known websites, then turn a profit by competingfor clicks as advertisements on social media sites. In order tobe successful in attracting user attention, they present a fakestory of political nature, religious nature, or anything withan emotional appeal. Typically, fake news stories, whichmay or may not have some remote connection to reality, areplanted on social media sites using provocative titles andimages. “Click bait” attracts the attention of unsuspectingsocial media users who click on links to these stories think-ing they are visiting a legitimate news site. These engagedusers are drawn in by the emotional appeal, while fake newsproviders get a share of advertising money from each click.

1.3 Spreading Fake News on FacebookAfter the surprise results of the 2016 U.S. Presidential

Election, the American media directed its ire at Facebookand Google, as in this NY Times piece [20] written by theEditorial Board, on November 19, 2016:

Most of the fake news stories are produced byscammers looking to make a quick buck. Thevast majority of them take far-right positions.But a big part of the responsibility for this scourgerests with internet companies like Facebook andGoogle, which have made it possible for fake newsto be shared nearly instantly with millions ofusers and have been slow to block it from theirsites.

This criticism is only partly correct. Facebook had beenworking toward fixing (or containing) the spread of hoaxeson the site at least since January 2015, almost two years be-fore the election [12]. They defined a hoax as a form of NewsFeed spam report that includes scams (“Click here to win alifetime supply of coffee”), or deliberately false or mislead-ing news stories (“Man sees dinosaur on hike in Utah”). Aswe can notice from this definition, in 2015, the phrase “fakenews” wasn’t being applied yet to the kind of false storiesthat flooded Facebook in the weeks before the election.

However, since it was difficult for independent researchersto know the extent to which Facebook users were affectedby this issue, everything continued more or less as before,and Facebook was alone in its fight. This changed in early2016, when the online publication BuzzFeed took an intereston Facebook’s unsuccessful efforts to deal with the problem.In an article published in April 2016, BuzzFeed proclaims:“it is the golden age of fake news” [5]. The article revealsthat BuzzFeed had conducted a study of fake news that wasspreading via nine known fake news sites such as the Na-tional Report, Huzlers, or Empire News, using the servicesof the company Crowdtangle, specialized in measuring so-cial engagement. The findings revealed that while trafficfor these sites had gone down for a while during 2015, inearly 2016, it had started picking up again. The article alsointerviewed Allen Montgomery – a fake identity for JestinColer, the creator of a factory of fake news websites, as [19]

Figure 2: Screenshot from Google search resultsabout Martha Coakley on Jan 12, 2010. Notice inhighlighted red, the tweets attacking Coakley. Thiswas a finding from [8] on how Google was inadver-tently giving premium space to political propagan-dists, in an effort to have “fresh” and relevant searchresults.

revealed after the election. Coler’s interview reveals some ofthe tricks of the trade of fake news, and points out why hebelieves he can win over Facebook:

Coler believes Facebook is fighting a losingbattle, doomed to fail no matter what it does.“They can shut down National Report. Of coursethey can do that,” he said. “But I could have 100domains set up in a week, and are they going tostop every one of those? Are they now going toread content from every site and determine whichones are true and which ones are selling you a lie?I don’t see that happening. That’s not the waythat the Internet works.” [19]

Despite this sounding of alarm bells by BuzzFeed (as earlyas April 2016), things got only worse with fake news on Face-book. We counted at least 25 articles published on the topicof fake news from April to November 2016 on BuzzFeed,culminating with the story of “How teens in the Balkansare duping Trump supporters with fake news”, published onNovember 3, 2016 and followed up by the related piece on“How Macedonian spammers are using Facebook groups tofeed you fake news”. These two articles provide details aboutone of the fake news factories operated by young people inthe small town of Ceres, Macedonia, that targeted Facebookusers in the United States. After reading these news articles

Figure 3: Screenshot from Google search resultsabout Donald Trump on Jan 23, 2017. In addition tothe many sections to the page (such as the “featuredsnippet” on the right column), notice how the tweetsshown above the fold belong to Trump himself.

(and others on the same topic), we noticed the clear similar-ities to the process that lead to the Twitter-bomb againstMartha Coakley in 2010. In fact, we are able to map thesteps in the two recipes one to one, as shown in Table 2.

Step 1 Register web domains for lots of relatedwebsites, with catchy names such as:http://TrumpVision365.com, see [17].

Step 2 Create Facebook accounts of fictitiouspeople, e.g, Elena Nikolov or AntonioMarkoski, see [18].

Step 3 Identify and join a Facebook group abouta political candidate, e.g., “Hispanics forTrump”or“San Diego Bernicrats”, see [18].

Step 4 Target members of the Facebook groupwith posts, by linking to the fake newswebsite stories, see [18].

Step 5 Wait for members of the group to spreadthe fake news in their organic subnetworks,by sharing and liking it.

Table 2: The recipe for spreading “fake news” onFacebook in the wake of the 2016 U.S. Presidentialelection. It contains the same steps as the recipeshown in Table 1.

This similarity should not be surprising. Once a spam-ming technique has been proven successful, it is easily repli-cated, since the knowledge about its working is also sharedon the Internet. What should surprise and worry us is thefact that researchers and web platform developers may alsoknow about such techniques, but they do little to warn andeducate the public of the consequences. It is also unfortu-nate that tech companies who have been used to facilitatethis practice, do not act pro-actively or effectively in stop-ping it. As an example of ineffective action we refer to theway Facebook handled the accusation that its news verifica-tion was not balanced. We discuss it in a next section.

2. FROM PROPAGANDA TO FAKE NEWSWe should not give the reader the impression that on-

line propaganda started with Twitter-bombs or Facebookfake news. In fact, it is much older than Social Media, itis as old as the Web itself. However, before the develop-ment of search engines it was not easy for propaganda todiscover you. Search engines made it easy for propagandiststo spread their message using techniques we now call WebSpam [6]. Advertisers, political activists and religious zealotsof all kinds have been busy modifying the structure of theWeb in an effort to promote their own biased results overthe organic, unbiased results. The Search Engine Optimiza-tion (SEO) industry has grown out of this effort, sometimesusing unethical techniques to promote their message, andsearch engines have been continuously evolving to fend offthese attacks.

In much of the first decade of the new millennium, searchengines tried to defend against Web Spam, but the spam-mers were successful in circumventing their defenses by usingfirst “Link Farms” and later “Mutual Admiration Societies”,collections of web sites that would intentionally link to eachother in order to increase everyone’s PageRank [6]. Evenwhen Google was reportedly using up to 200 different sig-nals to measure quality [14], professional SEOs would man-age to get sites like JC Penney’s at the top of search results[15]. Google’s ingenious solution to the problem of “unfaircompetition for high placement on a page” was the introduc-tion of the advertising model of AdWords and AdSense thatgave spammers an opportunity to make money while follow-ing rules. That seemed to work for a while. However, thesefinancial incentives were so lucrative that they provided areason for anyone to have a presence on the Web, especiallyif that Web presence managed to attract clicks and thus,ad dollars. This led to “click bait” and to the creation ofads masquerading as outrageous (fake) news stories, as wediscussed in the previous section.

But as Search Engines and Social Media evolve, so do thepropagandistic techniques. The rise of three recent methodsof spamming in the wake of the U.S. presidential elections,“fake news”, “featured snippets”manipulation [4], and“auto-completion revelations” [3] are the latest chapters in spread-ing propaganda through search engines and social media sothat it will find you. As a community of researchers we needto embrace the challenge of documenting and understand-ing these phenomena as well as finding ways to make theseissues known to the platforms providers. Journalists andnon-specialists also need to be informed, as they sometimesgive credence to conspiracy theories, confusing the public[21].

3. RESEARCH THAT INFORMS DESIGNIt is important for researchers, journalists and web users

to continue to pay attention to the information and misinfor-mation they encounter on the Web, be it on Google, Twitter,or Facebook. In this section, we discuss how research resultsand their publicizing lead over time to changes in the designfeatures of these systems, addressing the exhibited weak-nesses.

3.1 The Evolution of Real-Time Search Re-sults

The central finding that gave the title to [8] was the ma-nipulation of Google real-time search results through repe-

tition of Twitter posts by accounts (real users or bots) sup-porting a particular candidate. In December 2009 (just onemonth before the MA special election for the U.S. Senateseat in 2010), Google followed Bing in introducing “real-time search results”, placing social media messages in one ofthe top positions search results for a relevant search query.These messages came mostly from Twitter, which had aneasy-to-use API to pull the tweets. Tweets appearing inthe search results were those that had been recently posted.That created the opportunity for political propagandists toexploit the search rankings creating a Twitter-enabled GoogleBomb. As we documented in our paper, the manipulatorswere repeating the same messages, something also allowedby Twitter, over and over to increase the volume of conver-sation about a particular topic and keep it fresh for searchengines to include in their real-time results. Repetition ofa message would be annoying to the propagandist followers,but the target was not their followers’ feed; it was Googleand Bing, and through them, the public.

We can see these highly-placed tweet messages from ran-dom Twitter accounts in a screen shot for Martha Coakley’ssearch results, taken in January 2010 (Figure 2). During2010, Google eventually recognized that giving anonymoussocial media accounts a premium spot in its search resultswas not in line with its goals for reliable information and fora few years this feature disappeared. However, it has comeback again, but in a different format: when searching for aperson, it will pull up tweets from their timeline, as opposedto tweets about them, as exemplified in Figure 3. This isa great improvement, because it prevents actors who havean interest to promote their adversarial messages about anindividual or product to receive an unearned spot at the topof the search results.

3.2 The Evolution of RetweetingIn [8], we included the following observation at the end of

Section 4:

Our experiments with Google real-time searchhave shown that, even though Google doesn’t dis-play tweets from users that have a spammer sig-nature, it does display tweets from non-suspectedusers, even when these are retweets coming fromspammers. Thus, simply suspending spammingattacks is not sufficient. There should be somemechanism that allows for retroactively deletingretweets of spam and some mechanism that la-bels some Twitter users as enablers of spam.

At that time (in 2010), Twitter didn’t have an easy wayto quote a tweet and it allowed users to edit the originaltweet text when retweeting, as the tweet shown in Figure 1indicated. That design feature turned out to be very prob-lematic, among others for the reason mentioned in the quoteabove: deleted spam tweets lived in the retweets of otherTwitter users. However, it was also problematic becauseusers often were purposefully changing the meaning of thetext they were retweeting [10]. Most of this re-purposing waspossible via third-party applications that were very popularin the early years of Twitter. These applications were shutdown over the years and nowadays Twitter doesn’t allow theediting of a tweet that is being retweeted. Additionally, ifthe original is deleted, the retweet is also deleted, while ina quoted retweet, the text “This tweet is unavailable.” will

Figure 4: Facebook recently moved into implement-ing a system of warning towards sharing news itemsthat have been disputed.

show in place of the deleted tweet.

3.3 The Evolution of Fake News on FacebookThe proliferation of fake news on Facebook achieved new

levels once Facebook made a big change in how its algorithmfor the Trending News feature worked. Before August 2016,when this change took effect, the Trending News feature wasbeing curated by human editors, who filtered out unreliablesources and chose neutral or balanced sources for breakingnews stories. However, when the online tech blog Gizmodoposted an article in May 2016 [11], in which former employ-ees of the Trending News lamented anti-conservative bias,Facebook reacted hastily and clumsily. Worried about po-tential lawsuits for suppressing freedom of speech, it firedits team of human editors and replaced them with MachineLearning algorithms. Given their prior research on rumorcascades on facebook [2] one would expect that technicalexperts had a better understanding on the limitations of AIalgorithms, but maybe they were not involved in the deci-sion [1]. It didn’t take long after that change for fake news tostart achieving Trending News status, as BuzzFeed reportedon August 30, 2016 [16]. Despite BuzzFeed’s relentless re-porting on the fake news plague throughout the pre-electionseason, the rest of the media and the public didn’t tune ininto this conversation until after the election.

Facebook initially disputed that it had a fake news prob-lem, claiming that it accounts for only 1% of the news sto-ries. However, the company changed course under the in-creased and sustained public pressure and introduced newfeatures in its interface and the algorithms to address theissue [9].

One important feature that has rolled out recently is thelabeling of news posts as“Disputed”via fact-checking, third-party providers such as Snopes or PolitiFact. The screenshot in Figure 4 is an example of this feature in action. Inaddition to adding this label, Facebook warns users with analert box before they try to share a disputed story, but theyare still allowed to share [9].

It remains to be seen how this new feature will affect fakenews spreading. Again, access to Facebook data that couldallow independent researchers to evaluate the effectivenessof such interventions. Without it, our understanding ofchanges in human behavior correlated with or caused bychanges in the socio-technical platforms they inhabit, willbe limited. This is a reason of concern for our research com-munities.

4. DISCUSSIONWhat is the moral of the story? In the past, researchers

were the ones discovering and documenting the misuse andabuse of socio-technical platforms by the hands of dubiousactors with dubious agendas. [8] is only one such example.However, our discovery was possible only because we werecollecting data in real-time, after having noticed some evi-dence of foul play. When we think about Twitter’s approachto combating spammers, it seems reasonable that tweetscreated by misinformation-spreading accounts are automat-ically deleted and retracted from the entire network, oncethe accounts are suspended. However, the downside of suchan approach is that it makes it impossible for researchersand fact-checkers to go back in time and study the origin ofmisinformation campaigns and the mechanism for spreadingthem. That is a severe limitation to research. The problembecomes even more pronounced in the content of fake newsspreading on Facebook. Most Facebook groups are privateand if they are the source for starting certain cascades offake news, outside researchers cannot observe them in theirearly stages, missing crucial information that would lead totheir understanding.

Thus, it is not surprising that in the current situationcreated by the fake news plague, researchers didn’t play aleading role in their discovery. It were journalists and notresearchers in academia or Facebook and Google who raisedconcerns, but were not heard. This is worrisome. Facebook,by replacing humans with algorithms, has played a crucialrole in fueling the fake news spreading phenomenon. Simi-larly, the ease with which Google enables earning ad moneyfor page impression provided the financial incentives for thecreation of the fake news industry.

In light of what we know so far, here is our open ques-tion to the research communities interested in informationretrieval:

in the context of the now omnipresent, web-based,socio-technical systems such as Facebook, Google,and Twitter, what tasks should be performed byhumans and what tasks by algorithms?

Our communities should lead the way in providing answersto this question.

5. REFERENCES[1] S. Frenkel. Renegade facebook employees form task

force to battle fake news. BuzzFeed,http:// bzfd.it/ 2mITL31 , 2016-11-14.

[2] A. Friggeri, L. Adamic, D. Eckles, and J. Cheng.Rumor Cascades. In Proc. of ICWSM, 2014.

[3] Q. Hardy and M. Richtel. Don’t ask? internet stilltells. New York Times, http:// nyti.ms/ 2nyWnSx,year = 2012-11-21.

[4] A. Jeffries. Google’s featured snippets are worse thanfake news. The Outline, http:// bit.ly/ 2n5vgB2 ,2017-03-05.

[5] A. Kantrowitz. Facebook wanted a fight against fakenews. it got one. Facebook Newsroom,http:// bzfd.it/ 2nLCmsl , 2015-01-20.

[6] P. Metaxas. Web Spam, Social Propaganda and theEvolution of Search Engine Rankings. Lecture NotesBIP, Springer-Verlag, 2010.

[7] P. T. Metaxas, S. Finn, and E. Mustafaraj. Usingtwittertrails.com to investigate rumor propagation.CSCW’15 Companion, http://bit.ly/2n1CkMH, pages69–72, 2015.

[8] P. T. Metaxas and E. Mustafaraj. From obscurity toprominence in minutes: Political speech and real-timesearch. In Proceedings of the WebSci10: Extending theFrontiers of Society On-Line, http:// bit.ly/ 2n1IEUC ,April 2010.

[9] A. Mosseri. News feed fyi: Addressing hoaxes and fakenews. Facebook Newsroom, http:// bit.ly/ 2mzv8pe,2016-12-15.

[10] E. Mustafaraj and P. T. Metaxas. What editedretweets reveal about online political discourse. InAAAI Workshop on Analyzing Microtext,http:// bit.ly/ zWSeMF , 2011.

[11] M. Nunez. Former facebook workers: We routinelysuppressed conservative news. Gizmodo,http:// bit.ly/ 2ngF2k5 , 2016-05-09.

[12] E. Owens and U. Weinsberg. News feed fyi: Showingfewer hoaxes. Facebook Newsroom,http:// bit.ly/ 2mBUnIF , 2015-01-20.

[13] J. Ratkiewicz, M. Conover, M. Meiss, B. Goncalves,S. Patil, A. Flammini, and F. Menczer. Truthy:Mapping the spread of astroturf in microblog streams.In WWW’11, 2011.

[14] Google Keeps Tweaking Its Search Engine, 2007.URL: http://nyti.ms/2n5JjGx [accessed: 2007-06-03].

[15] D. Segal. The dirty little secrets of search. New YorkTimes, http:// nyti.ms/ 2nXfSE0 , 2012-02-12.

[16] C. Silverman. Facebook must either innovate or admitdefeat at the hands of fake news hoaxsters. BuzzFeed,http:// bzfd.it/ 2nMvfia, 2016-08-30.

[17] C. Silverman and L. Alexander. How teens in thebalkans are duping trump supporters with fake news.BuzzFeed, http:// bzfd.it/ 2mC6tBm, 2016-11-03.

[18] C. Silverman and L. Alexander. How macedonianspammers are using facebook groups to feed you fakenews. BuzzFeed, http:// bzfd.it/ 2mzvCM0 , 2016-11-08.

[19] L. Sydell. We tracked down a fake-news creator in thesuburbs. here’s what we learned. NPR,http:// n.pr/ 2nuHN1T .

[20] The Editorial Board. Facebook and the digital viruscalled fake news. New York Times, 2016-11-19.

[21] C. Timberg. Could google rankings skew an election?new group aims to find out. Washington Post,http:// wapo.st/ 2mNa0wY , 2017-03-14.