A Computer Aided Content Analysis of Online Reviews

Fall 2011 Journal of Computer Information Systems 43

A Computer AIdedContent AnAlySIS oF onlIne revIewS

lAkIShA l. SImmonS SumAlI ConlonIndiana State University The University of MississippiTerre Haute, IN 47809 University, MS 38677

SurmA mukhopAdhyAy Jun yAng The University of Mississippi The University of Houston-Victoria University, MS 38677 Victoria, TX 77901

Received: September 3, 2010 Revised: February 21, 2011 Accepted: March 17, 2011

AbStrACt

Published information extraction (IE) research generally dis-cusses the technological improvement of IE software, yet little addresses the valuable application of such technology. IE and content analysis together enhance the rigor of qualitative research through computer based content analysis. We use CAINES to il-lustrate an effective application of IE and computer aided con-tent analysis on eWOM reviews. CAINES is a system based on linguistic techniques that can extract and analyze online movie reviews. In this research we demonstrate how CAINES is able to analyze the most important factor a moviegoer considers when rating a movie online. CAINES extracted and analyzed movie re-views from 18 action and adventure movies consisting of 20,679 individual reviews. By using CAINES we were able to showthat storyline is most important to moviegoers and they tend to leave more positive than negative reviews. Our results also re-veal that reviewers mainly discuss their personal evaluation of the movie, rather than discouraging or encouraging readers to see the movie. keywordS: Computer aided text analysis; electronic word-of-mouth (eWOM); content analysis; movie reviews

1. IntroduCtIon

Published information extraction (IE) research generally dis-cusses the technological improvement of IE software, yet little addresses the effective application of such technology. Web 2.0, the read-write Web of blogs and content sharing, has cre-ated an environment where consumers can share their opinions with wide audiences with the click of the mouse. Word-of-Mouth (WOM) has been traditionally conducted in a face-to-face off-line fashion. With substantial advances in electronic communi-cation and increased use of the Internet, the term eWOM is now used to describe electronic WOM [30]. IE software, coupled with computer aided content analysis, can help managers intheir decision making better than ever before. Organizationsare now moving from pure economic analysis to sentiment analy-sis (i.e. opinion mining) of online news and social networkingposts [45]. Thirty-five percent of Wall Street investment firmsnow perform sentiment analysis of unstructured online news reports, editorials, company Web sites, blog posts and evenTwitter messages [6]. In some instances, firms use information

extraction programs to not only interpret news, but also to trades stocks based on it. In eWOM, members of electronic forums or social networks can discuss products and services or comment on news stories. These discussions can lead to encouraging or discouraging pur-chasing decisions. Positive or negative eWOM reviews can fi-nancially impact a product manufacturer, service provider, or hosting organization. Managers must find ways to quickly read and process comments left by consumers. It is difficult and time consuming to identify trends and interpret comments individu-ally. This study illustrates how CAINES (Content Analyzer and Information extraction System) can be used by practitioners to effectively and efficiently acquire eWOM comments and interpret them for decision making. Our contribution to information system research is to illus-trate how to use a natural language processing system to enhance the rigor of qualitative research through computer based content analysis. Research has suggested that in order to help managers make better decisions, new information that is relevant for deci-sion-making must be available [9, 32, 54]. Web 2.0 is full of new and relevant information. We use CAINES to demonstrate an ef-fective application of IE on openly available eWOM in the theatre movie industry. In this study, we extract over 20,000 movie re-views from an Internet based movie forum and assign the reviews to a positive or negative valence construct. We then analyze the sentiments of the reviews. Using CAINES, we aim to find out consumer’s opinions of movies by analyzing what drives the valence of their com-ments based on their actual phrases. This is an important liter-ature contribution because computer technology lessens thehuman resource and increases the reliability and speed of sen-timent analysis. Some researchers have conducted this type of content extraction manually. For example, one study had in-dependent judges manually read and code over 12,000 movie reviews [33]. Manually reading and coding eWOM mes-sages is an extremely tedious task that constrains the amountof data that can be studied. Other studies have only analyzed movie ratings because looking inside the content of the re-views was too overwhelming [20]. Therefore, by using IE and computer aided text analysis, we can drastically increase the amount of data analyzed and reduce the effort and risk of mis-take in coding reviews. In this study, we use CAINES toidentify the important factors that movie goers are concerned

44 Journal of Computer Information Systems Fall 2011

about from their written reviews. Our research asks twoquestions:

RQ1: By using CAINES, what is the most important fac-tor a consumer considers when they rate their overall opinion of a movie on an online review Web site?

RQ2: By examining their exact linguistic phrases, do consumers write more positive or negative online reviews and what do they like or dislike about the movies they write about?

The remainder of the paper is organized as follows. In the next section we review the literature on computer techniques used to extract and analyze unstructured web data and the literature of eWOM in the movie industry. Next, we describe our data collec-tion procedure and discuss the empirical testing methodology in Section 3. We present our empirical and content analysis results in Section 4 with a discussion of the results in Section 5. We then conclude with contributions and future work.

2. relAted work

2.1 Information extraction

Information extraction is automatic extraction of structured information such as entities, relationships between entities, and attributes describing entities from unstructured sources such as text corpus or text documents [44]. Information extraction, like text summarization, machine translation, and natural language interface is a sub-field of natural language processing. Extract-ing useful information from Web documents is an intricate task because Web documents differ dramatically in their structure, format, and quality of information. One of the main challenges is extracting structured data from unstructured documents [10]. Managing unstructured data has many interesting research challenges. A wealth of work on managing structured data that has been carried out by the database community [19]; like how noisy information can depreciate the efficiency of the retrieval mechanism. Some unstructured text studies have evaluated IE systems. Peng and McCallum [39] studied the application of in-formation extraction in searching literature and hiring decisions. Yates et al. [55] presented a technical study on ‘open’ informa-tion extraction, directly from the web without any human input. Chang et al. [8] surveyed major web data extraction approaches and compared them on three dimensions: task domain, the auto-mation degree, and the techniques used. Riloff et al. [44] found potentially false results during information extraction in sentenc-es that hold subjective language (such as opinions, emotions, and sentiments). They found that subjectivity analysis improves the accuracy of information extraction systems. KnowItAll was the first published system to extract data from Web pages that was unsupervised, domain-independent, and on a large-scale [25]. KnowItAll is a system that answers the call of the tedious process of extracting large collections of facts from the Web. In an experiment KnowItAll ran for four days on a single machine and 54,753 facts were extracted through its seeded on-tology and extraction rules. Conlon, Hale, Lukose, and Strong [11] created FIRST (Flex-ible Information extRaction SysTem) to extract financial informa-tion from The Wall Street Journal (WSJ) using a training set of documents from the WSJ to build a knowledge base. The FIRST system relied on a service-oriented framework with information

retrieval (IR) and information extraction (IE) components. The IR component retrieves source documents and the IE component analyzes the documents and inputs the information into a data template. We add to the growing literature using content analysis and IE to extract and analyze large amounts information in less time than traditional systems.

2.2 Content analysis & linguistics

The large volume of written documents available to be pro-cessed and the repetitiveness of coding made the computer a natu-ral ally of the content analyst [31]. Content analysis is used to study the content of communication. It involves examining theo-retical definition and empirical measurement [38]. The goal of content analysis is to create systematic and objective criteria for transforming written text in highly reliable data that can be ana-lyzed for the symbolic content of communication [50]. Content Analysis has been applied to newspaper editorials [37], novels [26], and recorded speeches [47]. In information systems research, Son, Tu, and Benbasat [51] identified 11 trust-building measures. They performed content analysis on 100 B2B e-marketplaces to understand the usage of the 11 trust-building measures. In man-agement literature, Short et al. [48] examined differences be-tween family and nonfamily firms on the entrepreneurial orienta-tion using content analysis of shareholder letters from S&P 500 firms. In his book, Holsti [28] discussed that in spite of the di-versity of the definition of the content analysis, there is a broad agreement in the requirements such as objectivity, system, and generality. The method and application of content analysis is changing, especially with the development of computer software and simulation. Before we can begin to analyze any text documents, we must first understand the lexical nature of the text. In English gram-mar, “patterns” are sentences, the “lexicon” consists of words, and the “constraints” are the rules of English syntax [41]. Ininformation retrieval systems, a lexicon consists of all wordsrecognized by the system, their grammatical categories, syno-nyms, and any associations with database objects and forms[1]. Natural language provides reliable and valid indicators of personality, cognitive processes, and social processes [40]. The association link between an array of words can be empirically defined in terms of their lexical co-occurrence, a lexicon, or semantic analysis [5]. Therefore, a lexicon of words indicativeof a specific construct could be developed. Bardi, Calogero,and Mullen [5] showed in their study where a value lexiconwas developed on the basis of the Schwartz [46] value theory to extract lexical indicators of values (e.g. power, achievement) from texts (i.e. newspapers). Their basic process was to come up with three words for each value/construct to create a value lexicon then search ‘pages’ (Internet or newspaper) for all three words (more reliable than one word) to show evidence of that value/construct. Aggarwal, Vaidyanathan, and Venkatesh [3] proposed a method based on lexical semantic analysis of discovering brand-descriptor associations in online search engines such as Google. By examining associations between brands and carefully selected adjectives/descriptors, one can go beyond merely counting text content to uncover the meaning of the content. Lexicons have been created in management to expose some of the pitfalls and traps that plague post-modern managers [13]. For example, Cady and Hardalupas [7] tried to create a lexicon to describe the phenomena of major organizational change. They searched 2,168 issues of 15 publications and gathered an exten-


sive list of terms that describe organizational change. Their results revealed the top 5 mentioned terms representing organizational change could be found in 82% of the relevant articles.

2.3 ewom towards movies

Literature suggests people read reviews for four main reasons. Bailey [4] finds that the principal motivating factor for consumers to visit product review Web sites was to use it as an additional source of information (35.5%), to receive assurance or reassur-ance that they were making a good choice (27.6%), wanted to know what other consumers were thinking (17%) or used the website as a primary source of information prior to a product pur-chase (10.5%).

Other research suggests an eWOM review is generally written to either recommend or discourage others from buying a particu-lar product [52] or in our case discourage or encourage others to see a particular movie. In online movie forums, moviegoers talk about elements of a movie, such as casting, story, directing, vi-sual effects, and technical aspects [56]. They also express their positive or negative judgment about the movie and the different elements. Stimpert et al. [53] compared and contrasted several online movie review studies and showed that some elements im-pact box office revenues. Quality of the film was shown to be a positive factor towards box office success in all three studies.

Litman and Kohl [34] find that information that reduces uncer-tainty about a motion picture seems to be correlated with financial success. For example, superstar actors are associated with suc-cessful pictures as well as positive film critic ratings. The power of a star to drive box office sales is known as “star power” and can mean big profits for a film [21]. Many studies support the influ-ence of critics’ reviews irrespective of the debate of whether this influence is positive or negative [23]. Moviegoers also frequently use film critics as sources of information. A study by Simmons [49] showed that a third of moviegoers chose a film which has a favorable review.

More recently, the storyline of a movie has been found to di-rect the box office success of a movie. Eliashberg and Shugan [23] empirically find that a good storyline is the foundation for a successful movie production. They analyzed themes, scenes, and emotions from movie scripts and compared the information to the box office revenues. An extraction technique was used to gather information from a spoiler (an extensive summary of the storyline written by movie viewers after they watched the movie): (1) semantics, (2) bag-of-word models, (3) genre, and (4) content analysis. Human judges read the movie spoilers and answered a predetermined set of 22 questions to help pick up on genre infor-mation.

Before the Internet, the exchange of opinions among the view-ers was very limited mainly because of the geographical barrier [34]. The Internet allowed moviegoers to exchange their views about movies through movie forums such as Yahoo! Movies. Though marketing plays a significant role in a movie’s opening weekend, consumers word-of-mouth can be frequently cited as the single key factor that sets the long-term success of motion pic-tures [16]. Dellarocas et al. [15] found that in contrast to “offline” word-of-mouth communities where opinions may “disappear into thin air” online communities maintain a persistent, public record of all posted opinions. Hennig-Thurau et al. [27] report that con-sumers articulate themselves online in an eWOM approach due to a consumer’s desire for social interaction, economic incentive, concern for other consumers, and the potential to enhance their

own self-worth. Thus, a consumers’ satisfaction or dissatisfaction may be displayed through their eWOM desire for economic in-centives and their concern for other consumer’s.

CAINES helps us to efficiently examine consumer reviews and identify their concerns. We want to know if consumers write more positive or negative comments and their justification for their positive or negative review. Understanding the important factors leading to a consumer’s positive or negative review could help the producers to better prepare for the next motion picture release.

3. methodology And dAtA

3.1 CAIneS methodology: Information extraction

FIRST (Flexible Information extRaction SysTem) [12] ex-tracted financial information from many sources such as The Wall Street Journal (WSJ) and the Electronic Data-Gathering, Analy-sis, and Retrieval system (EDGAR). CAINES (Content Analyzer and INformation Extraction System) is the extended system of FIRST, and it is designed to analyze unstructured general online documents. The system analyzes texts using syntactic and seman-tic techniques. Several portions of the system include knowledge from domain experts. The system can handle analyzing word fre-quency, co-occurrence information, and extracting unstructured text to create an explicit information dataset. The extracted in-formation can be inserted to data warehouses, templates or slots, processed by data mining systems, and used in business intelli-gent tasks. CAINES was built using Perl, MySQL database manage-ment system, and a knowledge engineering approach. Perl, a free text processing language with numerous modules, is a popular programming language for system designers and web develop-ers. It is dependable at creating, managing, and extracting infor-mation from the Web using its Library for World Wide Web in Perl (LWP). It also supports regular expressions. CAINES has a part-of-speech tagger, noun phrase extractor, lexicon, rule base, and an extraction sub-system. The general system architecture of CAINES is shown in Figure 1. The part-of-speech tagger tags each word in the sentences while the noun phrase extractor produces a list of noun phrases from the document sets (reviews). We use Lingua::En::Tagger, available at http://search.cpan.org/~acoburn/Lingua-EN-Tagger-0.16/Tagger.pm to do these two tasks. Lingua::En::Tagger is a probability based, corpus-trained tagger that assigns part-of-speech (POS) tags to English text based on a lookup dictionary and a set of probability values. In information extraction, the terms that appear as noun phrases and verb phrases help us to identify the subjects and the objects of the key verbs that we are trying to find. Thus, in general, the rule for identifying the subject and object of an act is:

for each row in the portion of the report from which we want to extract information

if the key terms are a verb phrase then return the noun phrase immediately ap pear ing prior

the verb phrase as an actor and return the noun phrase immediately following the verb

phrases as an object end ifend for


The lexicon we used contains the terms that we believe im-portant to this research (e.g. action packed, boring). They can be noun or verb phrases. The noun phrases are listed by the Lingua::En::Tagger while the verb phrases are found from the terms that appear after the subject noun phrases. The noun modifiers can be found as adjectives appearing in front of them (e.g., “great pictures,” “very boring movie,” “pretty good acting”). All of these terms can be found from analyzing the Key Word In Context

(KWIC) [36] file which will be discussed next in the retrieval and analysis.

3.1.1 Data retrieval and output

CAINES downloaded each eWOM review (via Perl LWP)and placed the written comments in a text file for further anal-ysis. CAINES collected the eWOM reviews from the Ya-

FIgure 1: System architecture of CAIneS

FIgure 2. example of one online movie review


hoo! Movies Web site (Figure 2). Yahoo! Movies is one of themost popular movie websites, and its data source has beenused by many studies [e.g. 35, 20]. We chose all the action-adventure genre movies released in the summer of 2007 (May4, 2007-August 31, 2007). The final dataset of Web extracted content contains 18 movies with 20,679 individual reviews(Table 1).

Once CAINES had extracted the text of the reviews from the Web and imported the text file into the database, the first output file could be derived. Using all of the reviews, CAINES generated a Key Word In Context (KWIC) [36] file by putting each word of a review into a field in the MySQL database. For each sentence, every word is put in a field in the first row. The next row contains

every word in that sentence but the first word was removed. This process continues until the last word appears in the first column. The sentence, “The story was excellent and the acting was awe-some,” for example, would be put in the database as shown in Table 2. The KWIC file was useful in our content analysis and will be discussed in the next section.

A second output file from CAINES, a Microsoft Excel file, consisted of ID number, grade categories (overall, story, acting, directing, and visual), review title, reviewer name, review date, and movie title for each review. The file was in Microsoft Excel format. We were able to use this file as input into the Statistical Package for the Social Sciences (SPSS) where we conducted em-pirical analysis.


3.2 methodology: Content analysis

We used an inductive content analysis approach to create positive and negative constructs to measure the data in the movie reviews. Inductive content analysis is often conducted by first ex-amining patterns in data and then seeking to make sense of those patterns [e.g., 42]. There are several psychological studies that provide checklists of adjectives that relate to various human emo-tions that can help us in understanding the valence of the reviews. Russell and Carroll [43] illustrated the concept of ‘bipolarity’ in positive and negative affect. Some examples are elated vs. de-pressed, satisfied vs. dissatisfied and many more. Diener and Em-mons [18] separated affect words in two broad categories’ such as pleasant and unpleasant. The affect categories for pleasant are: happy, joy, pleased, enjoyment/fun, glad, delighted, and content-ed. The categories for unpleasant are: angry, fear/anxiety, frus-trated, depressed, annoyed, sad, and gloomy. These words help us understand mood categories as we begin to analyze our movie reviews for content.

In order to identify terms (word and phrases) to help us con-duct inductive content analysis, we need to look at the words and phrases in the movie reviews. Using the KWIC file, we sorted the terms in some of the columns. This helps us see the collections of terms that we are interested in with their frequencies. Table 3 shows some sample rows that contain the term “excellent” in column 4.

3.2.1 Overall opinion

Two independent raters were trained to view the KWIC filesof the reviews of the 18 movies. The raters were trained toanalyze the KWIC files row by row of a subset of 15% of thereviews (about 300 reviews) and document words and phrases that signified a positive or negative connotation for that review narrative. The raters, who are familiar with movie reviews, then worked together and analyzed a complete movie KWIC file to ensure they agreed on what signified a positive or negativeword. The raters then worked independently to compile listsfor the positive and negative lexicons. After the analysis, theraters met to review the keywords. The raters agreed that ofthe 207 words generated by their independent work, 66 bestcaptured moviegoer sentiments. The raters combined all of the positive terms to create the ‘positive’ lexicon and all of thenegative terms to create the ‘negative’ lexicon (see Table 4 for the

lexicons generated). We used Holsti’s [28] method to demonstrate interrater reliability.

PAO = 2A/nA + nB

PAO stands for the proportion of agreement observed, A is the number of agreements between the two raters, and nA and nB are the number of words coded by the two raters. Holsti doesn’t purport a threshold for high reliability but coefficients above .75 are generally indicative of high reliability [48, 24]. Based on this formula, we observed reliability of .94 demonstrating acceptable reliability between our two raters. We attribute our high rankings to the extensive training and “dry run” coding sessions.

3.2.2 Likes and dislikes

In regards to research question 2, we use content analysis to examine linguistic phrases and determine if customer satisfac-tion is displayed through more positive keywords and phrases or negative keywords. We applied our lexicons to the movie reviews for the following five movies: Spider-Man 3, Shrek the Third, Fantastic Four: Rise of the Silver Surfer, The Bourne Ultimatum, and Live Free or Die Hard (listed in Table 5). Only movies that had our lexicon of keywords in at least 4 of the 5 movies were used in the analysis. For example, the phrase ‘very poor’ from our negative lexicon was represented in Fantastic Four, Live Free or Die Hard, Shrek the Third, and Spiderman 3, but not Bourne Ultimatum. To ensure reliability when coding movie reviews, we used computer-aided content analysis. The computer-aided technique is preferred since many unstructured texts can be ana-lyzed with perfect reliability and without bias in a shorter period of time than manual coding. Specifically, we relied on CAINES. In less than 10 seconds, CAINES was able to extract our lexi-con of terms from a movie. CAINES is attractive because it has been used in previous research analyzing documents from several online sources including web blogs, customer reviews, business and government reports, and online news articles. In addition, the system can efficiently handle analyzing word frequency, co-oc-currence information, and extracting unstructured text to create explicit information.

3.3 methodology: empirical analysis

The eWOM reviews from our 18 movies were sorted bystory, acting, directing, and visual to determine which factor


was most important. The Yahoo! Movies review system usesa thirteen letter grade scale from F to A+. We chose positive reviews (those with grade of B or higher) and negative reviews (those with grade D or lower) to explore the driving forces be-hind those extreme reviews. To answer our first researchquestion that through information extraction, we could findthe most important criteria for consumers, we used the letter grades from our dataset of 18 movies (see Table 1). From the 20,679 reviews, we correlated the grades of each of four cate-gories: story, acting, directing, and visuals with the overallcategory ranking. Since the ratings are in letter grade format,we coded each possible grade, from N/A to A+, to a 0 to 13 score in SPSS 17.0.

4. reSultS

4.1 what consumers feel is most important

For our first research question, to find out what is most im-portant to moviegoers we conducted empirical analysis of the letter grades of all 18 movies. Our empirical analysis shows us that among the four elements, story is the most important and directing is the second most important in correlation to the over-all grade of the movie. Overall grade is correlated with story at a level of .700 (p<.01), the highest correlation with overall grade. Acting and visual take the 3rd and 4th places based on correlation to the overall grade of the movie (Table 6). Based on these re-


sults, moviegoers feel storyline is most important when forming an overall opinion of a movie.

Deeper content analysis allowed us to reveal, in the moviego-ers own words, what they liked most about the storyline. In Live Free or Die Hard, some of the sentiments are “has a good story line and you don’t have to see the first two”, “a very good story-line, superb cinematography”, “Good story, great action and pure kick a** entertainment”, “movie ROCKS!!! Good story, great special effects”, “Good story, good action, plenty of humor, Must see!”, and “GOOD STORYLINE, AND IT WAS AN ACTION PACKED THAT WAS WORTH SEEING”. We next tested our research question concerning customer sat-isfaction. We used CAINES to analyze the content of the reviews by our positive and negative lexicons. External validity of our content analysis was confirmed by the five focal movies from our set with a large number of comments to examine our constructs of interests (see Table 7). After using our computer aided text analy-sis, we were able to determine frequencies in which our positive and negative keywords and phrases appeared in each of our five movies. For example, out of the 944 reviews analyzed for Shrek the Third, there was a mean of 212.61 instances of positive and 42.07 negative keywords.

4.2 positive lexicon

Frequency words and phrases from the positive mood lexi-

con are listed in Table 8. These words indicate some of themost popular ways to express their sentiments towards a moviein our sample. Interestingly, we find that phrases that wouldencourage others to see the movie are not as frequently citedas words simply evaluating the movie. For example, words such as “great,” “fun,” and “better than the first two” round out the top five with frequencies of 2661, 1943, 1674 respectively. Words indicating more recommendation sentiments fall into the lower quartile. Key recommendation words such as “must see” (158), “watch this movie” (98), “worth watching” (41), and “hard to beat” (6), have lower frequencies than the more evaluativecomments. The word “good” was the highest ranked word even after sub-tracting instances where “not” was the word in front of good. The movie Fantastic 4 ranked as one of the highest in terms of occur-rences and statements like “She is a good actress who wasn’t used right.” Some statements were repeated verbatim such as, “a good comic book adaption”, “a good comic book movie”, “a good family film”, and “a good family movie”. At least ten times, the phrase “a good job” was used to say what an actor, director, the film itself, or cast did a good job, such as “of creating the effects”, “on the silver surfer”, with his character”, and “of retelling the classic”. We considered “action packed” to be a positive phrasesbecause of the ways in which it was used in context of the


reviews. In the movie The Bourne Ultimatum consumers men-tioned, for example, “Overall, it’s an action packed and en-joyable movie,” “amazing and intense action packed entry,”action packed from beginning to end,” “definite must see,”“action packed,” something a veteran Bond fan such as myself enjoys”, “Action packed, thrilling, and great visuals”, “Action packed . . . thrilling . . . the things true Bourne fans have come to love”.

4.3 negative mood lexicon

Frequency words and phrases from the negative mood lexicon are listed in Table 9. In this set of analysis, similar to the positive words, we find that phrases that would discourage others from seeing the movie, such as “do not see” (12 occurrences) are not as frequently cited as words simply evaluating the movie, such as “boring” (319 occurrences).

Table 8Evidence of language representing positive comments


The term long was the most frequent keyword, with an ini-tial total of 590. A deeper content analysis revealed that over100 of those in Live Free or Die Hard were in relation to anactor in the movie with the last name “Long.” Other comments including the word ‘long’ were “dragged on too long.” In Shrek the Third reviewers mentioned how “long the lines were” andthat the movie was “2 hours long,” “movie went on too long,”and therefore several people said to wait to see the movie onceit is on DVD. In contrast, some thought Shrek the Third was“too short.”

The online reviewers state that a movie, such as stated in Shrek the Third, with “predictable plotlines,” “predictable quotes,” are “predictable climax” and considered negative attributes of a mov-ie. “Boring,” “stupid,” “poor,” and “terrible” were common words used in negative comments.

To empirically test that we have distinct positive and nega-tive keyword lexicons we conducted a one-sample t test. CAINES

was used to count each occurrence of each lexicon keyword in all of the movie reviews (words in Table 8 and 9). The average of the total occurrences of each lexicon made up the positive and negative constructs. Table 10 displays our average word count, standard deviation, correlations, and t statistic for each of our two constructs. We compared each construct with a one-sample t test with a test statistic of zero. A test statistic of zero would indicate no evidence of positive language in a movie review. Positive mov-ie review keywords were detected in our sample of five movies (mean=134.33, t=4.590, p<.05). In the negative category we also received favorable results (mean=65.44, t=4.177, p<.01). Thus, there was statistically significant evidence that distinct positive and negative keyword constructs were derived from our overall set of data. Therefore, our research question concerning the de-tection of positive and negative reviews and sentiments could be answered with CAINES. Results show moviegoers wrote more positive than negative reviews.

Table 9Evidence of language representing negative comments


5. dISCuSSIon

Our findings suggest that using a system such as CAINES can drastically increase the amount of web-based text thatcan be analyzed and decrease the amount of human depen-dency that some researchers have faced [e.g. 35]. From aninformation systems researcher point of view, CAINES isthe system of choice when the goal is to reduce human effortin analyzing unstructured text in a variety of information ex-traction and retrieval projects. Using an unbiased system suchas CAINES will yield reliable and repeatable results versususing humans to code reviews. This project reduced the timeto categorize and analyze over 20,000 reviews. CAINES pro-cesses for less than 10 seconds for each movie. For example,in less than 10 seconds, CAINES was able to filter through all 8,626 reviews for Spider-Man 3 and extract all pertinent key-words. Studies in the motion picture industry have found that many factors, such as star power and critics movie reviews, seem to be important to financial success [22, 34]. Our results clearly show significant support that storyline drives eWOM reviews of box office movies. Furthermore, reviewers do tend to encourage or discourage others from seeing the movie. We were also able to focus more on the actual human linguis-tics, thoughts, feelings, and mood of the moviegoers whereas other studies have not been able to do this. For example, the con-sumers in our sample express more evaluative comments such as “better than the first two” in their reviews and phrases such as “do not see” as means of discouraging others from going to see the movie. Based on our frequency analysis of such phrases, more evaluative comments were left, yet the recommendation phrases were still profound. Our information extraction and content analysis followed a mainly inductive approach where two independent raters extrapo-lated words from the reviews and then devised lexicons for posi-tive and negative movies. However, deductive content analysis could be used in future content analysis research. In a deductive approach, the researchers would create the appropriate lexicon for the constructs apriori, before examining any of the narratives. This would potentially increase the use of information systems and lessen human effort. Future researchers can use our lexicon of positive and negative terms in their work analyzing box office movies, without going through the labor of creating a new lexicon. Our lexicon had high interrater reliability and encompasses exact feelings and thoughts of moviegoers. Our study was more exploratory in nature and the inductive analysis gave us the power to focus on the specific con-sumer words. We also were careful to ensure that there were no duplicate entries by any one user for a particular movie by analyz-ing usernames.

6. ConCluSIon Our contribution to the information system research is in how we use computer technology to extract over 20,000 movie reviews from a Web 2.0 based forum and automatically assign the reviews based on a positive or negative valence construct. Using our tech-nique we found that consumers leave more positive than negative comments overall and those reviews were driven by the storyline. This is an important contribution because computer technology lessens the human resource as some researchers have conducted this type of categorization manually. Future research in this area would be to further identify the motivations for moviegoers to write reviews. Managerial implications are plentiful in our research since eWOM can be so substantial. For example, 81 percent of partici-pants in an online survey said their perceptions are influenced to a positive extent by WOM in business to consumer environments [2]. Hung and Li [29] showed that in a consumer-to-consumer environment, eWOM may trigger variety-seeking and excessive buying to an informed consumer, but eWOM may facilitate selec-tive buying to an uninformed consumer. Therefore, it is important for managers to keep abreast of feedback in eWOM communities and learn from the comments. Negative messages in a social network environment are shared at a drastically higher rate to an average of 17 people versus an average of 11 people for positive experience messages [17]. It’s also been shown that name changing and reregistering is fairly easy to do [14]. Therefore, negative customers can be difficult to get rid of in a virtual environment so it’s best to create strong relationships with the positive customers. Negative statements can spread fast and organizations must mitigate these situations as quickly as possible to prevent a negative reputation and lower economic benefit. Comparing to relying on employees to read and code reviews one at a time, managers can use CAINES to quickly filter online comments, identify negative feedbacks, and take re-actions correspondingly.

reFerenCeS

[1] Adam, N., Gangopadhyay A. and Clifford J. “A form-based approach to natural language query processing,” Journal of Management Information Systems, 11(2), 1994, pp. 109-135.

[2] Allsop, D., Bassett B., and Hoskins J. “Word-of-Mouth Re-search: Principles and applications,” Journal of Advertising Research, 47 (4), 2007, pp. 398-411.

[3] Aggarwal, P., Vaidyanathan R., and Venkatesh A. “Using Lexical Semantic Analysis to Derive Online Brand Posi-tions: An Application to Retail Marketing Research,” Jour-nal of Retailing, 85(2), 2009, pp. 145-158.

[4] Bailey, A. A. “Consumer Awareness and Use of Product


Review Websites,” Journal of Interactive Advertising, 6 (1), 2005, pp. 90-108.

[5] Bardi, R., Calogero B., and Mullen A. “New Archival Ap-proach to the Study of Values and Value — Behavior Rela-tions: Validation of the Value Lexicon,” Journal of Applied Psychology, 93(3), 2008, pp. 483-497.

[6] Bowley, G. “Computers that trade on the news,” The New York Times, 2010, retrieved January 24, 2010 from http://www.nytimes.com/2010/12/23/business/23trading.html?_r=3.

[7] Cady, S. and Hardalupas L. “A Lexicon for Organizational Change: Examining The Use Of Language In Popular, Prac-titioner, And . . .” Journal of Applied Business Research, 15(4), 1999, pp. 81.

[8] Chang, C., Kayed M., Girgis M.R., and Shaalan K.F. “A sur-vey of web information extraction systems,” IEEE Transac-tions on Knowledge and Data Engineering, 18(10), 2006, pp. 1411-1428.

[9] Changchit, C. “An Investigation Into The Feasibility Of Using An Internet-Based Intelligent System To Facilitate Knowledge Transfer,” Journal of Computer Information Systems, 43(4), 2003, pp. 91-99.

[10] Chen, H. Web retrieval and mining, Decision Support Sys-tems 35 (1), 2003, pp. 1-5.

[11] Conlon, S., Hale J., Lukose S., and Strong J. “Information extraction agents for service oriented architecture using web service systems: A framework,” Journal of Computer Infor-mation Systems, 48(3), 2008, pp. 74-83.

[12] Conlon, S., Lukose S., Hale J., and Vinjamur A. “Automati-cally Extracting and Tagging Business Information for E-Business Systems Using Syntactic and Semantic Analysis,” Semantic Web Technologies and eBusiness: Virtual Organi-zation and Business Process Automation. A. F. Salam and Jason Steven edited. Idea Group Inc, 2007, pp. 101-126.

[13] Cooksey, R., Gates G., and Pollock H. “‘Unsafe’ business acts and outcomes: A management lexicon,” Business Hori-zons, 41(3), 1998 pp. 41.

[14] Dellarocas, C. “The Digitization of Word of Mouth: Promise and Challenges of Online Feedback Mechanisms,” Manage-ment Science, 49(10), 2003, pp. 1407-1424.

[15] Dellarocas, C., Zhang X. M., and Awad N.F. “Exploring the value of online product reviews in forecasting sales: The case of motion pictures,” Journal of Interactive Marketing, 21(4), 2007, pp. 23-45.

[16] De Vany, A. and Walls D. “Uncertainty in the movie indus-try: Does star power reduce the terror of the box office?,” Journal of cultural economics, 23 (4), 1999, pp. 285-318.

[17] Dietz, S. “Online Influence and the Tech-fluentials,” Pre-sented at WOMMA’S Measuring Word of Mouth Confer-ence July 13, 2005.

[18] Diener, E. and Emmons R.A. “The independence of posi-tive and negative affect,” Journal of Personality and Social Psychology, 47, 1984, pp. 1105-1117

[19] Doan, A., Naughton J.F., and Ramakrishnan R. “Informa-tion extraction challenges in managing unstructured data,” SIGMOD Record 37(4), 2008, pp. 14-20.

[20] Duan W., Gu B., and Whinston A. “Do online reviews mat-ter? An empirical investigation of panel data,” Decision Sup-port Systems, 45, 2008, pp. 1007-1016.

[21] Elberse, A. “The Power of Stars: Do Star Actors Drive the Success of Movies?”, Journal of Marketing, 71(4), 2007, pp. 102-120.

[22] Eliashberg, J., Hui S.K., and Zhang Z.J. “From story line to box office: A new approach for green-lighting movie scripts,” Management Science, 53(6), 2007, pp. 881-893.

[23] Eliashberg, J. and Shugan S.M. “Film critics: Influencers or predictors?,” The Journal of Marketing, 61(2), 1997, pp. 68-78.

[24] Ellis, L. “Research methods in the social sciences,” Madi-son, WI:WCB Brown & Benchmark, 1994.

[25] Etzioni, O., Cafarella M., Downey D., Popescue A.M., Shaked T., Soderland S., Weld D. S., and Yates A. “Meth-ods for domain-independent information extraction from the web: an experimental comparison,” AAAI-Conference On Artificial Intelligence, 2004, pp. 391-398.

[26] Griswold, W. “American character and the American novel: An expansion of reflection theory in sociology of the lit-erature,” American Journal of Sociology, 86, 1981, pp. 740-765.

[27] Hennig-Thurau, T., Gwinner K.P., Walsh G., and Gremler D. “Electronic word-of-mouth via consumer-opinion plat-forms: What motivates consumers to articulate themselves on the internet?,” Journal of Interactive Marketing, 18(1), 2004, pp.38-52.

[28] Holsti, O. R. Content Analysis for the Social Sciences and Humanities, Addison Wesley (Current Publisher: Perseus Publishing), 1969.

[29] Hung, K. and Li, S.Y. “The Influence of eWOM on Virtual Consumer Communities: Social Capital, Consumer Learn-ing, and Behavioral Outcomes,” Journal of Advertising Re-search 47(4), 2007, pp. 485-495.

[30] Keller, E. “Unleashing the Power of Word of Mouth: Creat-ing Brand Advocacy to Drive Growth,” Journal of Advertis-ing Research, 47(4), 2007, pp. 448-452.

[31] Krippendroff, K. Content Analysis: An introduction to its methodology, Sage Publication, Edition.2, 2003.

[32] Lee, J. and Park, M. S. “Integration and Composition of Web Service-Based Business Processes,” Journal of Com-puter Information Systems, (44:1), 2003, pp. 82-92.

[33] Litman, B.R. and Ahn H. “Predicting Financial Success of Motion Pictures, Motion Picture Mega-Industry,” Needham Heights, MA: Allyn & Bacon, 1998.

[34] Litman, B.R. and Kohl L.S., “Predicting financial success of motion pictures: The’80s experience,” Journal of Media Economics 2(2), 1989, pp 35 – 50.

[35] Liu, Y. “Word-of-mouth for movies: Its dynamics and im-pact on box office revenue,” Journal of Marketing, 70 (3), 2006, pp. 74-89.

[36] Luhn, H. P. “Keyword-in-context index for technical litera-ture (KWIC index),” American Documentation 11, 1960, pp. 288-295.

[37] Namenwirth, J.Z. “Marks of distinction: An analysis of Brit-ish mass and prestige newspaper editorials,” American Jour-nal of Sociology, 74, 1969, pp. 343-360.

[38] Nunnally, J. and Bernstein I. Psychometric Theory, Mc-Graw-Hill; 3 edition 1994.

[39] Peng, F. and McCallum A. “Information extraction from re-search papers using conditional random fields,” Information Processing and Management, 42, 2006, pp. 963-979.

[40] Pennebaker, J. W. and King L. A., “Linguistic styles: Lan-guage use as an individual difference,” Journal of Personal-ity and Social Psychology, 77, 1999, pp. 1296–1312.

[41] Pentland, B. T. “Grammatical models of organizational pro-cesses, Organization Science,” 6(5), 1995, pp. 541-556.


[42] Potter, J.W. and Levine-Donnerstein D. “Rethinking validity and reliability in content analysis,” Journal of Applied Com-munication Research, 27, 1999, pp. 258-284.

[43] Russell, J.A. and Carroll J.M. “On the bipolarity of positive and negative affect,” Psychological Bulletin, 125(1), 1999, pp. 3-30

[44] Riloff, E., Wiebe J., and Phillips W. “Exploiting subjectivity classification to improve information extraction,” Proceed-ings of the National Conference on Artificial Intelligence, 2009.

[45] Saenz, A. “Wall street computers read the news before trad-ing,” The New York Times. Retrieved January 23, 2010 from http://singularityhub.com/2011/01/03/wall-street-comput-ers-read-the-news-before-trading/.

[46] Schwartz, S. H. “Universals in the content and structure of values: Theoretical advances and empirical tests in 20 coun-tries,” In M. Zanna (Ed.), Advances in experimental social psychology, New York: Academic Press, 25, 1992, pp. 1–65.

[47] Seider, M.S. “American big business ideology: A content analysis of executive speeches,” American Sociological Re-view, 39, 1974, pp. 802-815.

[48] Short, J.C., Payne G.T., Brigham K. H., Lumpkin G.T., and J. C. Broberg, “Family Firms and Entrepreneurial Orienta-tion in Publicly Traded Firms: A Comparative Analysis of the S&P 500,” Family Business Review, 22(1), 2009, pp 1-11.

[49] Simmons, J.“ A ‘Thumbs up’ Pulls in the Audience,” Wall Street Journal, 1994, Section B.

[50] Singleton, R.A. and Straits B.C. Approaches to social re-search, (4th edition), New York: Oxford University Press, 2005.

[51] Son, J., Tu L., and Benbasat I. “A descriptive content analy-sis of trust-building measures in B2B electronic market-places,” Communications of AIS (18), 2006, pp. 2-51.

[52] Stephens, N. and Gwinner K., “Why Don’t Some People Complain? A Cognitive-Emotive Process Model of Con-sumer Complaint Behavior,” Journal of the Academy of Marketing Science 26(3), 1998, pp. 172-189.

[53] Stimpert, J.L., Laux J.A., Marino C., and G. Gleason, “Fac-tors Influencing Motion Picture Success: Empirical Review And Update,” Journal of Business & Economics Research 6(11), 2008, pp 39-52.

[54] Sugumaran, V. “An Agent-Based Knowledge Management Framework for the E-Commerce Environment,” Journal of Computer Information Systems, (42:5), 2002, pp. 63-73.

[55] Yates, A., Cafarella M.J., Banko M., Etzioni O., Broadhead M., Soderland S. “TextRunner: open information extraction on the web, Proceedings of Human Language Technolo-gies,” The Annual Conference of the North American Chap-ter of the Association for Computational Linguistics, 2007, pp. 25-26

[56] Zhuang, L., Jing F., and Zhu X.Y. “Movie review mining and summarization,” Proceedings of the 15th ACM interna-tional conference on information and knowledge manage-ment, 2006, pp 43-50.

Copyright of Journal of Computer Information Systems is the property of International Association for

Computer Information Systems and its content may not be copied or emailed to multiple sites or posted to a

listserv without the copyright holder's express written permission. However, users may print, download, or

email articles for individual use.

A Computer Aided Content Analysis of Online Reviews

Documents

Transcript of A Computer Aided Content Analysis of Online Reviews