Cross-Lingual Cross-Platform Rumor Verification Pivoting onMultimedia Content

Weiming Wen1, Songwen Su1, Zhou Yu1

1University of California, Daviswmwen@ucdavis.edu

Abstract

With the increasing popularity of smart de-vices, rumors with multimedia content becomemore and more common on social networks.The multimedia information usually makes ru-mors look more convincing. Therefore, find-ing an automatic approach to verify rumorswith multimedia content is a pressing task.Previous rumor verification research only uti-lizes multimedia as input features. We proposenot to use the multimedia content but to findexternal information in other news platformspivoting on it. We introduce a new featuresset, cross-lingual cross-platform features thatleverage the semantic similarity between therumors and the external information. Whenimplemented, machine learning methods uti-lizing such features achieved the state-of-the-art rumor verification results.

1 Introduction

Social network’s unmoderated nature leads to thespread and emergence of information with ques-tionable sources. With the increasing popularityof the social media, we are exposed to a plethoraof rumors. Here we borrow the rumor definitionfrom DiFonzo and Bordia (2007) as unverified in-formation. Unmoderated rumors have not onlycaused financial losses to trading companies butalso panic for the public (Matthews, 2013). Es-pecially if rumors contain multimedia content, thepublic generally accepts the multimedia informa-tion as a “proof of occurrence” of the event (Sen-car and Memon, 2009). Readers usually don’thave time to look through similar events acrossdifferent platforms to make an informed judgment.Therefore, even if a credible platform, such asCNN, has debunked a rumor, it can still go viralon other social media platforms.

Intuitively, people believe fake rumors wouldcontain fabricated multimedia content. Boididou

et al. (2015b) used forensics features for detectingmultimedia fabrication to verify rumors. However,these features did not lead to noticeable improve-ment. We suspect that this is because un-tamperedmultimedia content can still convey false informa-tion when paired with fake news from a separateevent. For example, Figure 1 shows one fake poston MH 370 that used a real video about US Air-ways Flight 1549. Inspired by the fact that readers

Figure 1: A video of US Airways Flight 1549 was bor-rowed by news on Malaysia Airlines Flight 370.

tend to search related information covered by dif-ferent media outlets to garner an objective view,we propose to verify rumors pivoting on multime-dia content to tackle such problems. Compared tokeywords, searching information pivoting on thevisual content is more effective and accurate.

In order to access information from differentplatforms easily, we created a new rumor verifica-tion dataset by expanding a Twitter rumor datasetto include webpages from different social me-dia platforms using search engines. Previous ru-mor verification datasets are mainly monolingual,such as English (Derczynski et al., 2017) or Chi-nese (Wu et al., 2015). However, textual informa-tion in the native language where the rumor hap-pened can be more helpful when it comes to ver-

arX

iv:1

808.

0491

1v2

[cs

.CL

] 2

8 A

ug 2

018

ifying worldwide rumors. Therefore, we not onlyindexed English webpages by searching Googlewith images but also included Chinese webpagesvia Baidu.

We next introduced our cross-lingual cross-platform features which capture the similarityand agreement among rumors with posts fromdifferent social media. We built an automaticverification model using the proposed featuresand achieved the state-of-the-art performance onthe MediaEval 2015’s Verifying Multimedia Use(VMU 2015) dataset (Boididou et al., 2015a) uti-lizing information from Google.

Collecting and annotating rumors in foreignlanguages is difficult and time-consuming, espe-cially for languages with low rumor verificationlabeling. Finding out an automatic way to verifythose rumors in an unsupervised way is also mean-ingful. Since our cross-lingual cross-platform fea-tures are adaptable to rumors in different lan-guages, we demonstrated that these features couldtransfer learned knowledge by training on one lan-guage and testing on another. Such cross-lingualadaptation ability is especially useful for predict-ing rumors that have low annotation resource withavailable annotated rumors in languages such asEnglish.

We published our code and dataset on GitHub1.

2 Related work

Previous research has utilized multimedia infor-mation for rumor verification in various ways.Zampoglou et al. (2015); Jin et al. (2015); Boi-didou et al. (2015b) verified rumors by leverag-ing forensic features which are extracted to en-sure the digital images are not tempered (Sencarand Memon, 2009). However, none of these stud-ies found such information useful for rumor ver-ification on a Twitter-based multimedia dataset.Jin et al. (2017) incorporated image features us-ing a pre-trained deep convolutional neural net-work (Krizhevsky et al., 2012) on the extendedTwitter-based multimedia dataset. Although theimage features improve their results, their frame-work cannot outperform methods other than mul-timodal fusing networks. One possible reason isthat the multimedia content in fake rumors is bor-rowed from another real event and usually theircontent corresponds to the text of the rumors. Inthis case, the image itself is real but not real in

1https://github.com/WeimingWen/CCRV

the context of the fake news. We thus proposeto leverage the multimedia information by findingthe agreement and disagreement among posts thatare from different social media platforms but sharesimilar visual contents.

The agreement between rumors and their com-ments is used heavily in automatic verification.Mendoza et al. (2010) declared that fake rumorstended to have more people question their validity.Later, Qazvinian et al. (2011) first annotated com-ments on tweets as supporting, denying or query-ing, and then used such stance information in theclassification to leverage the “wisdom of crowds”.Recently, the best performing system (Enayet andEl-Beltagy, 2017) in RumourEval shared task atSemEval 2017 (Derczynski et al., 2017) also usedsuch information. However, the crowd is not al-ways wise. For example, Starbird et al. (2014)suspected the correctness of public opinions inrumors, pointing out some certain fake news re-ceived more support than questions. In our work,instead of using the “wisdom of crowds”, we usedthe knowledge from different news platforms toassist rumor verification.

Computational journalism (Cohen et al., 2011)exploits external knowledge widely. Diakopouloset al. (2012) first leveraged information from reli-able sources in the context of journalism. They de-veloped a tool for journalists to search for and as-sess sources in social media around breaking newsevents. Ciampaglia et al. (2015) utilized factualknowledge bases, such as Wikipedia, to assess thetruth of simple statements. Shao et al. (2016) de-signed a system for tracking rumors on differentplatforms, which is probably the closest work toours. However, they did not utilize cross-platforminformation for rumor verification. Our proposedmethod is able to leverage information on any plat-form to verify rumors as long as it has both textualand multimedia information.

3 CCMR dataset

We created a cross-lingual cross-platform multi-media rumor verification dataset (CCMR) to studyhow to leverage information from different me-dia platforms and different languages to verifyrumor automatically. CCMR consists of threesub-datasets: CCMR Twitter, CCMR Google, andCCMR Baidu.

CCMR Twitter is borrowed from VMU 2015dataset (Boididou et al., 2015a). There are 17

ID Event Twitter Google BaiduReal Fake Real Fake Others Real Fake Others

01 Hurricane Sandy 4,664 5,558 1,836 165 203 693 134 29102 Boston Marathon bombing 344 189 619 54 49 317 55 1603 Sochi Olympics 0 274 139 132 76 64 124 5304 MA flight 370 0 310 143 65 115 80 59 3105 Bring Back Our Girls 0 131 29 42 37 2 6 406 Columbian Chemicals 0 185 35 2 26 19 1 007 Passport hoax 0 44 24 0 2 16 0 408 Rock Elephant 0 13 3 17 0 4 2 1409 Underwater bedroom 0 113 1 58 0 0 37 1310 Livr mobile app 0 9 0 4 11 0 0 011 Pig fish 0 14 3 13 4 1 12 712 Solar Eclipse 140 137 40 64 39 0 10 9113 Girl with Samurai boots 0 218 2 52 6 2 48 014 Nepal Earthquake 1,004 356 257 60 107 159 19 8115 Garissa Attack 73 6 60 0 3 36 1 016 Syrian boy 0 1,786 4 1 3 0 0 017 Varoufakis and zdf 0 61 2 0 18 0 0 0

Total 6,225 9,404 3,197 729 699 1,393 508 605

Table 1: CCMR dataset statistics.

events containing fake and real posts with imagesor videos shared on Twitter. We created CCMRGoogle and CCMR Baidu by searching Googleand Baidu indexed webpages that share similarmultimodal content with CCMR Twitter. The up-per part of Figure 2 shows the collection pro-cess. We searched Google with every image (URLfor video) in CCMR Twitter to get English web-pages. Then we indexed those webpages to formCCMR Google. Similarly, we searched Baidu toget Chinese webpages and created CCMR Baidu.Two human annotators manually annotated bothdatasets. The annotation is for better analysis anddataset quality control. None of it is utilized dur-ing our feature extraction process. Annotatorswere asked to label collected webpages based ontheir title and multimedia content. If they are notenough to tell fake news from real news, the web-page is labeled as “others”. The Cohen’s kappacoefficient for two annotators is 0.8891 in CCMRGoogle and 0.7907 in CCMR Baidu. CCMR has15,629 tweets indexed by CCMR Twitter (Twit-ter), 4,625 webpages indexed by CCMR Google(Google) and 2,506 webages indexed by CCMRBaidu (Baidu) related to 17 events. The webpagesfrom Google and Baidu are in English and Chineserespectively. The statistics of the CCMR datasetwith respect to each event is listed in Table 1.

3.1 Observation in Annotation

We observe that 15.7% of webpages are fake inCCMR Google while 20.3% in CCMR Baidu. Wespeculate that this is because all events in CCMRdataset took place outside China. Chinese web-

pages searched via Baidu are thus more likely tomistake the information. In the manual annota-tion process, we found that many images are actu-ally borrowed from other events, which confirmsour assumption. Another interesting observationis that webpages indexed by Baidu tend to havemore exaggerating or misleading titles to attractclick rates. We labeled such webpages as fake ifthey also convey false information through theirmultimedia content.

We also found that news in different lan-guages has different distributions concerning thesubtopics of the event. For example, in theBoston marathon bombing, Baidu indexed Chi-nese reports generally put more emphasis on aChinese student who is one of the victims, whileGoogle indexed English reports cover a widerrange of subtopics of the event, such as the pos-sible bomber. This phenomenon is understandableas social media from a specific country usually fo-cus more on information related to their readers.

4 Framework Overview

Figure 2 describes the overview of our framework.After collecting CCMR dataset in Section 3, wefirst performed Twitter rumor verification leverag-ing Google in Section 6 as shown in the bottomleft of the figure. We extracted cross-lingual cross-platform features for tweets in CCMR Twitterleveraging webpages from CCMR Google (TFG).Section 5 discusses the automatic construction ofthis feature set. We then use the features to verifyrumors automatically.

We then performed Twitter rumor verification

Figure 2: The information flow of our proposed pipeline. TFG represents the cross-lingual cross-platform featuresfor tweets leveraging Google information, while TFB is similar but leverages Baidu information instead. BFGmeans cross-lingual cross-platform features for Baidu leveraging Google information.

leveraging Baidu in Section 7 to test if our methodcan verify rumors by borrowing information fromdifferent languages and platforms. This experi-ment is meant to demonstrate that our method islanguage and platform agnostic. Such an advan-tage also enables our method to use one languageinformation to predict in another language (seethe experiment in Section 8). We extracted cross-lingual cross-platform features for tweets leverag-ing webpages from CCMR Baidu instead (TFB)and used it to verify tweets in Section 6.

In Section 8, we performed Baidu rumor ver-ification via transfer learning to test the cross-lingual adaptation ability of the cross-lingualcross-platform features. We treated Chinese web-pages in CCMR Baidu as rumors and empiricallyverified them via transfer learning. We extractedcross-lingual cross-platform features for Baiduwebpages leveraging Google (BFG) in Section 6.Since BFG and TFG are both cross-lingual cross-platform features leveraging Google, we adopted

the classifier pre-trained with TFG on CCMRTwitter to verify webpages in CCMR Baidu usingBFG, under the assumption that tweets and web-pages follow a similar distribution.

Although we labeled webpages in CCMRGoogle and CCMR Baidu, we did not leveragethe annotation here because annotation is time-consuming and using annotation information isnot generalizable to other datasets.

5 Cross-lingual Cross-platform Features

We propose a set of cross-lingual cross-platformfeatures to leverage information across differentsocial media platforms. We first embed both therumor and the titles of the retrieved webpages into300-dimension vectors with a pre-trained multi-lingual sentence embedding. It is trained usingEnglish-Chinese parallel news and micro-blogsin UM-Corpus (Tian et al., 2014). We encodeEnglish-Chinese parallel sentences with the sameword dictionary, as they share some tokens such as

URLs and punctuation. We then use a two-layerbidirectional gated recurrent unit (GRU) (Choet al., 2014) to generate hidden states. We obtainthe embedding by averaging the hidden states ofthe GRU. A pairwise ranking loss is used to forcethe cosine distance between embeddings of pairedsentences to be small and unpaired sentences to belarge. We train our multilingual sentence embed-ding on 453,000 pairs of English-Chinese parallelsentences and evaluated it on another 2000 sen-tence pairs. Our published code includes the im-plementation details of the multilingual sentenceembedding.

After obtaining the embeddings of the rumorand the titles of the retrieved webpages, we fur-ther calculate the distance and agreement featuresbetween these embeddings to create a set of cross-lingual cross-platform features. In total, there are10 features, two for distance features and eight foragreement features.

5.1 Distance Features

We compute the cosine distances between the em-beddings of the target rumors and the titles of theretrieved webpages. The distance indicates if therumor has similar meaning with the retrieved web-pages that have similar multimedia content. Wecalculate the mean and variance of the distance.

The mean of the distance indicates the aver-age similarity between a rumor and its correspond-ing webpages from other platforms. A high valuein mean suggests that the rumor is very differ-ent from the retrieved information from other plat-forms. We suspect that rumors with this propertyhave a higher probability of being fake. Becausethe rumor might have borrowed the image fromanother event that was covered in the retrieved in-formation. Meanwhile, the variance indicates howmuch these retrieved webpages are different fromeach other. A high variance indicates that the mul-timedia information is used by different events oris described in different statements. So the eventor the statement the rumor covers could be fake.

5.2 Agreement Features

We first pre-train an agreement classifier on astance detection dataset provided by the FakeNews Challenge2. This dataset provides pairs ofEnglish sentences with their agreement annota-tions in “agree”, “disagree”, “discuss” and “unre-

2http://www.fakenewschallenge.org/

lated”. Figure 3 shows four example body texts ofa headline corresponding to each type of annota-tion in the dataset. During the training process, weembed the sentences in the Fake News Challengedataset using our pre-trained multilingual sentenceembedding. We then concatenate the embeddingsof the sentence pair as the input. We use a multi-layer perceptron to pre-train our agreement classi-fier. We randomly select a balanced developmentset containing 250 pairs for each label (1000 in to-tal) and train our agreement classifier on the restof 74,385 pairs. The agreement classifier achieves0.652 in the macro-averaged F1-score on the de-velopment set. Our published code also includesthe details.

Figure 3: An example headline and its body texts of theFake News Challenge dataset.

We calculate the agreement features using themean and variance of the prediction probabilitybetween the rumor and all the retrieved webpages.There are in total four agreement labels. There-fore, we have eight agreement features in total.Agreement features capture information about ifthe rumor’s statement agrees with the correspond-ing information in other platforms. Besides beingable to gain similar benefits as distance features,our agreement features also capture the case thatthe information stance is portrayed differently bydifferent resource rumors. Conflicting informationwill also be an indicator of fake news.

6 Rumor Verification LeveragingCross-platform Information

We extracted cross-lingual cross-platform featuresfor tweets in CCMR Twitter leveraging Google(TFG) and evaluated the effectiveness of TFG onrumor verification tasks.

We proposed a simple multi-layer perceptronclassifier (MLP) to leverage the extracted features.MLP has two fully-connected hidden layers of20 neurons with ReLU as the activation function.Each layer is followed by a dropout of 0.5. Weevaluated TFG in two settings: task and event.1) The task setting used event 1-11 for trainingand event 12-17 for testing according to (Boidi-dou et al., 2015a). 2) The event setting evaluatedthe model performance on a leave-one-event-outcross-validation fashion. F1-score is used for eval-uation metric. Since both collecting source rumorsfrom Google and doing feature extraction for agiven tweet can be done automatically, it is fair tocompare the performance of our model with base-lines described below.

6.1 Baselines

We adopted three best performing models in theVMU 2015 task as our baselines:

UoS-ITI (UoS) (Middleton, 2015) uses a naturallanguage processing pipeline to verify tweets. It isa rule-based regular expression pattern matchingmethod. It ranks evidence from Twitter accordingto the most trusted and credible sources.

MCG-ICT (MCG) (Jin et al., 2015) is an ap-proach including two levels of classification. Ittreats each image or video in the dataset as a topicand uses the credibility of these topics as a newfeature for the tweets. They used the tweet-basedand user-based features (Base), such as the num-ber of hashtags in the tweet or the number offriends of the user who posted the tweet.

CERTH-UNITN (CER) (Boididou et al., 2015b)uses an agreement-based retraining scheme. Ittakes advantage of its own predictions to combinetwo classifiers built from tweet-based and user-based features (Base). Besides features providedby the task, it included some additional featuressuch as the number of nouns in tweets and trustscores of URLs in tweets obtained from third-party APIs.

Method F1-Task F1-EventUoS-ITI 0.830 0.224MCG-ICT 0.942 0.756CERTH-UNITN 0.911 0.693TFG 0.908 0.822BFG 0.810 0.739Combo 0.899 0.816

Table 2: The task and event settings performance.

ID UoS MCG CER TFG TFB01 0.658 0.594 0.718 0.715 0.70402 0.007 0.494 0.745 0.557 0.44803 0.057 0.882 0.595 0.956 0.82204 0.538 0.826 0.717 0.856 0.67805 0.000 0.988 0.947 0.969 0.95606 0.555 0.949 0.916 0.912 1.00007 0.000 1.000 0.475 0.989 0.98908 0.000 0.870 1.000 0.960 1.00009 0.000 0.772 0.996 1.000 0.99610 0.000 0.615 0.821 0.875 -11 0.000 0.963 0.000 0.963 0.66712 0.000 0.655 0.754 0.677 0.65613 0.000 0.954 0.795 0.998 0.85014 0.000 0.330 0.419 0.409 0.43015 0.000 0.130 0.156 0.145 0.15416 1.000 0.990 0.999 0.996 -17 1.000 0.827 1.000 0.992 -Avg 0.224 0.756 0.693 0.822 0.739

Table 3: F1-scores for each event.

6.2 Results

We describe the task setting results in Table 2,and detailed per-event results in Table 3. Al-though TFG does not achieve the highest F1-scorein the task setting, it is mainly due to the split ofthe dataset. More than half of the tweets in thetest set do not have images. Thus we can onlyleverage cross-platform information by searchingvideos’ URLs, which results in less accurate cross-lingual cross-platform features. In the event set-ting, which has a more fair comparison, TFGoutperformed other methods with a big margin(p<0.001). It is surprising to see that only 10 fea-tures extracted from external resources indexed bysearch engines leveraged by a simple classifier canbring such a big performance boost.

To further explore the quality of the cross-lingual cross-platform features, we calculated thePearson correlation coefficient (PCC) between

each feature with respect to the tweet’s label (fakeor real). We evaluated both TFG and features usedby baseline models. Table 4 lists the top six fea-tures with the highest absolute PCC values. A pos-itive value indicates this feature positively corre-lates with fake news. We can see four out of thetop six features are cross-lingual cross-platformfeatures. The variance of the unrelated probability(unrelated variance) has the highest score, whichfurther validates our design intuition that tweetsmight convey false information when they havedifferent agreement with all other webpages thatshared similar multimedia content. The secondfeature, “distance var” is also highly correlatedwith fake news. This result supports our hypothe-sis that if there is a large information dissimilarityacross different platforms, there is a high probabil-ity of fake information involved. The only featurefrom baselines (56 features in total) in the top sixfeatures is whether a tweet contains an exclama-tion mark or not (containsExclaimationMark).

Feature PCCunrelated variance 0.306distance variance 0.286agree variance 0.280discuss mean -0.231unrelated mean 0.210containsExclamationMark 0.192

Table 4: Top six features correlated with fake news.

6.3 Analysis

We found that Google webpages usually cover acomplete set of different information. There areusually both fake news and real news that debunkthese fake ones. As a result, there is a big in-formation variance among all those webpages’ ti-tles, which is captured by the cross-lingual cross-platform features. Therefore, TFG performs muchbetter than baselines in a number of events, suchas Event 03 (Sochi Olympics). The statistics of theCCMR dataset, described in Table 1, also supportsour observation that the labels of posts in CCMRGoogle are distributed more evenly compared toother media sources.

However, the F1-score of TFG is very low inEvent 15 (Garissa Attack). Gunmen stormed theGarissa University College in this event. We an-alyzed the Google webpages’ titles that share thesame image in this event. Although some titles

are related to the event, more of them are talk-ing about completely unrelated information suchas “Daily Graphic News Sun 20th Oct, 2013 —GhHeadlines Total News ...”. This webpage’s ti-tle only shows its published date and the nameof the website. Such noise hurt the performanceof the cross-lingual cross-platform features. Sincewe did not perform any manual labeling or filter-ing, sometimes the crawled webpages can be mis-leading. To analyze the prevalence of such noise,we randomly picked 100 Google webpages and100 Baidu webpages from CCMR and counted thenumber of noisy posts. The ratio of noise is 22%in Google and 18% in Baidu. However, even withsuch noise, our proposed methods can still outper-form current state-of-the-art methods.

7 Rumor Verification LeveragingCross-lingual Information

We tested if our cross-lingual cross-platform fea-tures are able to leverage external informationfrom another language for rumor verification. Wesimply replaced the Google webpages with Baiduwebpages to extract features for tweets (TFB), be-cause we have a pre-trained multilingual sentenceembedding that can project Chinese and English toa shared embedding space. We used the same clas-sifier, MLP to evaluate the performance of TFBwith both baselines and TFG.

7.1 Results

Experiment results using the task setting areshown in Table 2 and the detailed per-event resultsare listed in Table 3. Similar to the problem inTFG, we can not obtain any Chinese webpages re-lated to events such as Syrian boy, Varoufakis andzdf, which cover most tweets in the test set. Thosemissing features make TFB perform poorly in thetask setting. However, TFB performs better thantwo of the baselines in the event setting. If we ex-clude events without Baidu webpages (event 10,16 and 17), the average F1-score of UoS, MCGand CER are 0.130, 0.732 and 0.660, which areall lower than TFB’s. The performance of TFBproves that our method can be generalized acrosslanguages or platforms.

To further test the robustness of our cross-lingual cross-platform features, we also examinedif it would still work when leveraging external in-formation that contains different languages. Weextracted the cross-lingual cross-platform features

for tweets leveraging Google and Baidu webpagestogether (Combo) and accessed the performanceof Combo using MLP similarly. The performanceof Combo is also listed in Table 2. Since Combowould contain noise introduced from combiningwebpages indexed by different search engines, it isnot surprising that Combo performs slightly worsethan TFG extracted from Google webpages whichalready cover a wide range of information solely.However, Combo performs much better than TFBwhich only leverages Baidu webpages. It provesthat our cross-lingual cross-platform features arerobust enough to utilize combined external infor-mation from different languages and platforms.

7.2 Analysis

We checked the actual titles of webpages fromCCMR in certain events to analyze the reason forTFB’s worse performance exhaustively. We foundthat those Baidu webpages’ titles often talk aboutsubtopics different from the target rumor’s onTwitter, while Google webpages are more related.For example, the F1-score of TFB is much lowerthan TFG’s in event 02 (Boston Marathon bomb-ing). This performance corresponds to our obser-vation in Section 3 that Baidu webpages mainlyfocus on a subtopic related to the Chinese studentinstead of other things discussed on Twitter.

8 Low-resource Rumor Verification viaTransfer Learning

We extracted cross-lingual cross-platform featuresof webpages in CCMR Baidu leveraging informa-tion from Google (BFG). Then we applied Trans-fer, MLP in Section 6 trained on the whole CCMRTwitter using TFG, to verify those webpages. Be-cause this pre-trained model is for binary classi-fication, only webpages labeled as real or fake inCCMR Baidu are involved.

Since webpages in CCMR Baidu do not sharethe same features with tweets, such as the numberof likes and retweets, we adopted a random selec-tion model as our baseline. It would predict a ru-mor as real or fake with the same probability. Wecompared the performance of the Transfer modelwith this baseline on each event. F1-score is alsoused for evaluation metric.

8.1 Results

Table 5 lists the detailed results of our transferlearning experiment. We achieved much better

performance compared to the baseline with statis-tical significance (p<0.001), which indicates thatour cross-lingual cross-platform feature set can begeneralized to rumors in different languages. Itenables the trained classifier to leverage the infor-mation learned from one language to another.

ID Event Random Transfer01 Hurricane Sandy 0.247 0.28702 Boston Marathon bombing 0.230 0.28403 Sochi Olympics 0.555 0.75204 MH flight 370 0.407 0.53605 Bring Back Our Girls 0.500 0.92306 Columbian Chemicals 0.000 0.10007 Passport hoax 0.000 0.00008 Rock Elephant 0.000 0.50009 Underwater bedroom 0.577 0.97210 Livr mobile app - -11 Pig fish 0.375 1.0012 Solar Eclipse 0.571 0.88913 Girl with Samurai boots 0.559 0.92514 Nepal Earthquake 0.227 0.21115 Garissa Attack 0.125 0.05916 Syrian boy - -17 Varoufakis and zdf - -

Avg 0.312 0.531

Table 5: Rumor verification performance on theCCMR Baidu, where - indicates there is no webpagein that event.

8.2 AnalysisIn event 11 (Pig fish), Transfer achieves muchhigher performance than the random baseline.Generally, Baidu webpages’ titles are semanticallydifferent from tweets. However, in this particularevent, the textual information of those titles andtweets are semantically close. As a result, mod-els learned from English rumors can easily workon Chinese rumors, which is helpful for our trans-fer learning. Figure 4 shows three Twitter-Baidurumor pairs with similar meaning in this event.

Transfer obtains pretty low F1-scores in event07 (Passport hoax). The annotation conflict causedits weak performance. This event is about a Childdrew all over his dads passport and made his dadstuck in South Korea. During the manual annota-tion process, we found out that it is a real eventconfirmed by official accounts according to onenews article from Chinese social media3, whileCCMR Twitter labeled such tweets as fake. SinceTransfer is pre-trained using Twitter dataset, it isnot surprising that Transfer achieves 0 in F1-scoreon this event. The annotation conflict also bringsout that rumor verification will benefit from utiliz-ing cross-lingual and cross-platform information.

3http://new.qq.com/cmsn/20140605/20140605002796

Figure 4: Example parallel rumors in the Pig fish event.

9 Conclusion

We created a new multimedia rumor verificationdataset by extending a multimedia Twitter datasetwith external webpages from Google and Baiduthat share similar image content. We designeda set of cross-lingual cross-platform features thatleverage the similarity and agreement betweeninformation across different platforms and lan-guages to verify rumors. The proposed featuresare compact and generalizable across languages.We also designed a neural network based modelthat utilizes the cross-lingual cross-platform fea-tures and achieved state-of-the-art results in auto-matic rumor verification.

ReferencesChristina Boididou, Katerina Andreadou, Symeon Pa-

padopoulos, Duc-Tien Dang-Nguyen, Giulia Boato,Michael Riegler, and Yiannis Kompatsiaris. 2015a.Verifying multimedia use at mediaeval 2015. In Me-diaEval.

Christina Boididou, Symeon Papadopoulos, Duc-TienDang-Nguyen, Giulia Boato, and Yiannis Kompat-siaris. 2015b. The certh-unitn participation@ veri-fying multimedia use 2015. In MediaEval.

Kyunghyun Cho, Bart Van Merrienboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, HolgerSchwenk, and Yoshua Bengio. 2014. Learningphrase representations using rnn encoder-decoderfor statistical machine translation. arXiv preprintarXiv:1406.1078.

Giovanni Luca Ciampaglia, Prashant Shiralkar, Luis MRocha, Johan Bollen, Filippo Menczer, and

Alessandro Flammini. 2015. Computational factchecking from knowledge networks. PloS one,10(6):e0128193.

Sarah Cohen, James T Hamilton, and Fred Turner.2011. Computational journalism. Communicationsof the ACM, 54(10):66–71.

Leon Derczynski, Kalina Bontcheva, Maria Liakata,Rob Procter, Geraldine Wong Sak Hoi, and ArkaitzZubiaga. 2017. Semeval-2017 task 8: Rumoureval:Determining rumour veracity and support for ru-mours. arXiv preprint arXiv:1704.05972.

Nicholas Diakopoulos, Munmun De Choudhury, andMor Naaman. 2012. Finding and assessing socialmedia information sources in the context of journal-ism. In Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems, pages 2451–2460. ACM.

Nicholas DiFonzo and Prashant Bordia. 2007. Rumor,gossip and urban legends. Diogenes, 54(1):19–35.

Omar Enayet and Samhaa R El-Beltagy. 2017.Niletmrg at semeval-2017 task 8: Determining ru-mour and veracity support for rumours on twitter. InProceedings of the 11th International Workshop onSemantic Evaluation (SemEval-2017), pages 470–474.

Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang,and Jiebo Luo. 2017. Multimodal fusion with re-current neural networks for rumor detection on mi-croblogs. In Proceedings of the 2017 ACM on Mul-timedia Conference, pages 795–816. ACM.

Zhiwei Jin, Juan Cao, Yazi Zhang, and YongdongZhang. 2015. Mcg-ict at mediaeval 2015: Verify-ing multimedia use with a two-level classificationmodel. In MediaEval.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hin-ton. 2012. Imagenet classification with deep con-volutional neural networks. In Advances in neuralinformation processing systems, pages 1097–1105.

Christopher Matthews. 2013. How does one fake tweetcause a stock market crash. Wall Street & Markets:Time.

Marcelo Mendoza, Barbara Poblete, and CarlosCastillo. 2010. Twitter under crisis: Can we trustwhat we rt? In Proceedings of the first workshop onsocial media analytics, pages 71–79. ACM.

Stuart Middleton. 2015. Extracting attributed veri-fication and debunking reports from social media:Mediaeval-2015 trust and credibility analysis of im-age and video.

Vahed Qazvinian, Emily Rosengren, Dragomir RRadev, and Qiaozhu Mei. 2011. Rumor has it: Iden-tifying misinformation in microblogs. In Proceed-ings of the Conference on Empirical Methods in Nat-ural Language Processing, pages 1589–1599. Asso-ciation for Computational Linguistics.

Husrev T Sencar and Nasir Memon. 2009. Overview ofstate-of-the-art in digital image forensics. In Algo-rithms, Architectures and Information Systems Secu-rity, pages 325–347. World Scientific.

Chengcheng Shao, Giovanni Luca Ciampaglia,Alessandro Flammini, and Filippo Menczer. 2016.Hoaxy: A platform for tracking online misinfor-mation. In Proceedings of the 25th InternationalConference Companion on World Wide Web,pages 745–750. International World Wide WebConferences Steering Committee.

Kate Starbird, Jim Maddock, Mania Orand, PegAchterman, and Robert M Mason. 2014. Rumors,false flags, and digital vigilantes: Misinformationon twitter after the 2013 boston marathon bombing.iConference 2014 Proceedings.

Liang Tian, Derek F Wong, Lidia S Chao, PauloQuaresma, Francisco Oliveira, and Lu Yi. 2014.Um-corpus: A large english-chinese parallel corpusfor statistical machine translation. In LREC, pages1837–1842.

Ke Wu, Song Yang, and Kenny Q Zhu. 2015. False ru-mors detection on sina weibo by propagation struc-tures. In Data Engineering (ICDE), 2015 IEEE 31stInternational Conference on, pages 651–662. IEEE.

Markos Zampoglou, Symeon Papadopoulos, and Yian-nis Kompatsiaris. 2015. Detecting image splicing inthe wild (web). In Multimedia & Expo Workshops(ICMEW), 2015 IEEE International Conference on,pages 1–6. IEEE.