Expert Systems with Applications 37 (2010) 6182–6191

Contents lists available at ScienceDirect

Expert Systems with Applications

journal homepage: www.elsevier .com/locate /eswa

DASA: Dissatisfaction-oriented Advertising based on Sentiment Analysis q

Guang Qiu a,*, Xiaofei He b, Feng Zhang a, Yuan Shi a, Jiajun Bu a, Chun Chen a

a Zhejiang Key Laboratory of Service Robot, College of Computer Science, Zhejiang University, Hangzhou 310027, Chinab College of Computer Science, Zhejiang University, Hangzhou 310027, China

a r t i c l e i n f o a b s t r a c t

Keywords:Contextual advertisingAdvertising keyword extractionAd selectionSentiment analysisSyntactic parsing

0957-4174/$ - see front matter � 2010 Elsevier Ltd. Adoi:10.1016/j.eswa.2010.02.109

q This work is supported by China National Ke(2008BAH26B00).

* Corresponding author. Tel.: +86 571 87953955.E-mail address: qiuguang@zju.edu.cn (G. Qiu).

Online advertising has become one of the major revenue sources of today’s Internet ecosystem. The mainadvertising channels used to distribute textual ads are sponsored search and contextual advertising. Herewe consider the problem of contextual advertising, i.e. associating ads with a Web page. Most of previouswork only focuses on topical relevance of ads whereas the consumer attitudes are ignored. In this paper,we propose a novel advertising strategy, called Dissatisfaction-oriented Advertising based on SentimentAnalysis (DASA), to simultaneously improve ad relevance and user experience. Specifically, by using syn-tactic parsing and sentiment dictionary, we propose a rule based approach to extract topic words of opin-ion sentences associated with negative sentiment, which are regarded as the advertising keywords. Wealso design a prototype system for product information submission for the sake of ad selection. We takeinto account the consumer attitudes and promote the competitors of those products with which the con-sumers are not satisfied. The experimental results on advertising keyword extraction and ad selectionhave demonstrated the effectiveness of the proposed approach.

� 2010 Elsevier Ltd. All rights reserved.

1. Introduction their investigation (Mccoy, Everard, Polak, & Galletta, 2007). In

The growth of the Internet signified a dawn of a new age foradvertising. Compared with that on traditional media like newspa-per and television, advertising on the Internet (online advertising)boasts advantages including wide coverage, low cost and so on.Driven by these unique advantages of online advertising and therapid development of World Wide Web, advertisers have paidincreasing attention to this novel and impressive advertising med-ia. As mentioned in (Broder, Fontoura, Josifovski, & Riedel, 2007),the total Internet advertiser cost in US alone in 2006 is estimatedat over 17 billion dollars with a growth rate of almost 20% yearover year which means more than 29 billion dollars in 2009.

The large investment in online advertising market providesmain funding for the majority of Internet companies. For someWeb sites, especially those for-profit non-transactional ones, reve-nue from advertising maintains not only current performance butalso future development. Therefore, it seems that these advertis-ing-relied Web sites should get ads published as many as possibleto gain high revenue from advertisers. However, Bhargava et al.have show in their work (Bhargava & Feng, 2002) that advertisingwould create a disutility to Web users, consequently reducing sites’market share and consumer-side revenue, further the profits fromadvertisers. Also, Mccoy et al. have drew a similar conclusion in

ll rights reserved.

y Technology R&D Program

other words, from the perspective of consumers, advertisingshould be constrained or even boycotted. The tradeoff betweenfinancial revenue and market share triggers the emergence of rel-evant advertising to emphasize the relevance between ads andWeb pages for the sake of consumers.

Broder et al. categorize relevant advertising (textual ads)according to the distribution channels into sponsored search (SS)and contextual match (CM) (Broder et al., 2007). SS aims to placeoriginating query driven ads on the result pages from a Web searchengine. This kind of advertising is used by major Web search en-gines like Google, Yahoo! and Microsoft Live Search. CM aims toplace the commercial ads in the content of generic Web pages.Some major Web search engines also provide such service andthere are also many smaller players. Fig. 1 gives illustrative exam-ples for these two types of advertising. The primary focus of thispaper is on CM which is also referred to as contextual advertising.

The dominant strategy of most of current sponsored search andcontextual match is the keyword targeted marketing (Ribeiro-Neto, Cristo, Golgher, & Moura, 2005). Previous approaches to tar-geted/relevant advertising consider short-term and long-terminterests, however suffer from privacy issues (Wang et al., 2002).In keyword targeted marketing, keywords are extracted fromsearch queries (in the case of SS) or Web pages (in the case ofCM) to match against those keywords associated with ads providedby advertisers. It would be important to note that the privacy is-sues can be avoided in this kind of strategy.

The problem of placing relevant ads using the keyword targetedmarketing strategy (mostly for contextual advertising) has

Fig. 1. Examples of sponsored search and contextual match. (a) Yahoo! search results for query ‘‘Honda automobiles”, in which we can see an ad of ‘‘Honda Autos” on the right.(b) A snapshot of the home page of automotiveforums.com. There are ads provided by Google adsense on the left.

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191 6183

received a lot of attentions recently (Broder et al., 2007; Chakrab-arti, Agarwal, & Josifovski, 2008; Lacerda et al., 2006; Ribeiro-Netoet al., 2005; Wu & Bolivar, 2008; Yih, Goodman, & Carvalho, 2006).These approaches only focus on topical relevance between ads andWeb pages, which may not be optimal in some situations. Forexample, for a blog written by someone who is complaining aboutthe safety of Honda Accord, the ads related to Honda Accord are goodmatches if we simply consider the topical relevance. However, theconsumer reading this blog may have a negative feeling about Hon-da Accord and therefore may not click the ads. Thus, it might bedesirable to place the ads of other automobiles with high safetystandard.

In this paper, we propose a novel ad targeting mechanism,called Dissatisfaction-oriented Advertising based on SentimentAnalysis (DASA), to address the above mentioned problem. Our ba-sic idea is to combine traditional keyword relevance matching withsentiment analysis (also known as opinion mining). In fact, senti-ment analysis of texts is a kind of non-topical text analysis tech-nique which is orthogonal to the topical based technique (Esuli &Sebastiani, 2005). By sentiment analysis, we aim to discover theauthor’s underlying sentiment when he/she is writing the text,and eventually determine his/her attitude towards the mentioned

topics. This way, we are able to promote ads which not only relateto the topics of the Web pages but also meet the consumers’ poten-tial information needs. Naturally, in the example given above, gi-ven the author’s negative attitude towards the safety of HondaAccord, we may consider promoting automobiles which have high-er safety standard like Volvo.

The rest of the paper is organized as follows. Section 2 discussesprevious work on contextual advertising and sentiment analysis. InSection 3, we provide an overview of our DASA advertising strat-egy, along with some discussions on application issues. In Sections4 and 5, we describe our approaches for advertising keywordextraction and ad placement in details, respectively. In Section 6,we describe our experimental methodology and data sets, anddemonstrate experimental results with discussions. Finally, weprovide some concluding remarks in Section 7.

2. Related work

In this section, we provide a brief description of related work oncontextual online advertising and sentiment analysis which is thekey technique employed in our approach.

6184 G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

2.1. Contextual online advertising

Contextual advertising is a major type of online advertising, inwhich ads are placed on Web pages according to their content.Published literature on contextual advertising is still limited. Onereason is that it is an emerging research area; in addition, commer-cial secrets also limit publications of research achievements. Inexisting researches, some reduce the problem to keyword match-ing (Wu & Bolivar, 2008; Yih et al., 2006), while others inspect rel-evance of the entire text on the Web page (Broder et al., 2007;Chakrabarti et al., 2008; Lacerda et al., 2006; Ribeiro-Neto et al.,2005).

In the first line, Yih et al. (2006) emphasize the extraction ofappropriate advertising keywords in contextual advertising andpropose a classification based approach. That is, by using docu-ments manually annotated with advertising keywords, they applylogistic regression to train a classifier to classify a keyword as anadvertising keyword or not. The features used to represent key-words include linguistic features, html metadata, TFIDF, querylog, etc. A similar approach is recently proposed by Wu and Bolivar(2008) which adopts linear regression to determine if a keyword isan advertising one. The features used in their work are similar tothose in (Yih et al., 2006). It would be important to note that, inour work, we develop a novel strategy to extract advertising key-words in which no training process is required.

In the second line, previous work on relevance measures be-tween Web pages and ads can be roughly categorized into syntac-tic based and semantic based approaches. Ribeiro-Neto et al.provide a syntactic solution to ad placement based on the cosinesimilarity measuring between Web page and ad vectors (Ribeiro-Neto et al., 2005). They also propose a solution to solve the imped-ance problem between the vocabulary of ads and Web pages. Intheir follow-up work (Lacerda et al., 2006), they design a rankingstrategy for displaying ads according to their relevance by effec-tively leveraging all individual features using genetic program-ming. They aim to optimize overall precision and minimize thenumber of misplacements. The semantic approach proposed in(Broder et al., 2007) stems from the linguistic phenomena of poly-semy that a word has different meanings in different semantic con-texts, which is contrary to the impedance problem in (Ribeiro-Netoet al., 2005). They use the proximity of the ad and page classes asthe semantic score. Their experimental results show that semanticscore is significant as to improving the ad quality.

However, all previous approaches above focus only on the top-ical relevance and ignore the sentiment information between thelines. In this work, we propose a novel strategy which considersboth the topic relevance and sentiment coherence between adsand Web pages.

2.2. Sentiment analysis

Another work related to our approach is sentiment analysis.Current researches on sentiment analysis mainly focus on topic re-lated issues (Hu & Liu, 2004; Lu & Zhai, 2008; Mei, Cai, Zhang, &Zhai, 2008; Mei & Ling, 2007) and sentiment classification (Ding& Liu, 2007; Hatzivassiloglou & McKeown, 1997; Pang & Lee,2004; Turney, 2002; Turney & Littman, 2003). Here, we only sum-marize some of previous approaches on topic modeling and wordsentiment classification, which are the most closely related toour work.

In the work of (Mei & Ling, 2007), Mei et al. propose a novelprobabilistic model called Topic-Sentiment Mixture to capturethe mixture of topics and sentiment simultaneously. A topic modeland two sentiment models are defined in their work based on lan-guage models to model the probabilistic distribution of words indifferent topics and sentiment polarities. The work by Lu and Zhai

(2008) studies the problem of how to integrate a well-written ex-pert review about an arbitrary topic with many ordinary opinionsexpressed in a text collection. They propose a semi-supervised to-pic modeling approach to cast the expert review as a prior in theprobabilistic topic model PLSA and fit the model to the textcollection.

Hatzivassiloglou et al. are the earliest to tackle the problem ofdetermining the semantic orientation of sentiment words(Hatzivassiloglou & McKeown, 1997). Their proposed method triesto predict the orientation of subjective adjectives by analyzingpairs of adjectives extracted from a large unlabeled documentset. The underlying intuition is that the act of conjoining adjectivessubjects to linguistic constraints on the orientation of the adjectiveinvolved. Turney et al. adopt a different methodology which re-quires little linguistic knowledge (Turney & Littman, 2003). Theyfirst define two minimal sets of seed terms as descriptive of thecategories Positive Sp and Negative Sn. Then they compute thepoint-wise mutual information (PMI) of the target term t with eachseed term ti as a measure of their semantic association.

Sentiment analysis of sentences is a major building block of ourproposed approach. The specific task here is to extract the topicwords of sentences that the contained sentiment words are about.In our work, we propose a rule based approach to accomplish thisextraction task.

3. Overview of DASA advertising strategy

Our contextual advertising strategy DASA works as follows:

(1) Extract topic words as the candidate topics that ads may beabout from the Web pages (the traditional way to promoteads);

(2) Determine consumers’ sentiment on the extracted topicwords and select those towards which consumers have neg-ative sentiment as the final advertising keywords;

(3) Match the advertising keywords with appropriate ads.

In other words, an ad is considered to be relevant to a Web pageonly if it is associated with the topic words towards which con-sumers have negative attitudes. The most crucial steps of our ap-proach are topic word extraction and its sentiment classification,which eventually lead to advertising keyword selection. Fig. 2shows the architecture of DASA.

3.1. Application issues

There are several application issues we have to address beforethe detailed elaboration of our approach.

Since our advertising approach depends on the analysis of con-sumers’ attitudes, the first issue is that whether there exists largescale Web environment containing plenty of personal opinionswhere ads can be placed. The answer to this problem is User Gen-erated Content (UGC), which is also known as Consumer GeneratedMedia (CGM), referring to various kinds of media content producedby consumers. It has been characterized as ‘‘conversational media”which is a two-way process encouraging the publishing of users’own content and commenting on others’ (http://www.en.wikipe-dia.org/wiki/User-generated content). Typical applications includeforums and blogs, which have already been prevailing Web appli-cations on the Internet (http://www.businessweek.com/globalbiz/content/jan2007/gb20070109559223.htm). Therefore, UGC is idealenvironment for our advertising strategy.

The second issue is the targeted audience of our advertising. InUGC, there are two types of users participating in Web activities:writers and readers of the content. When a writer is complaining

Fig. 2. Illustration of the workflow of DASA. TWn means the topic word extracted from Web pages by the topic word extraction technique. AKk is the final advertising keywordselected from the topic words by sentiment classification technique. These two steps form the advertising keyword extraction phase as described by the dashed rectangle.

Fig. 3. Different relations between sentiment word S and topic word T. (a–c) are thethree kinds of direct relations. (d–g) illustrate the four kinds of indirect relations.

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191 6185

about some product in his/her article, our advertising schemetends to place ads which are about rival products. This can poten-tially help the writer find a new product that satisfies his/herneeds. Besides, this scheme also provides alternatives to the read-ers who have similar opinions with the writer. When there are nopersonal comments or only positive opinions contained in the arti-cle, traditional advertising strategy can be applied.

4. Advertising keyword extraction

As stated above, we reduce the task of advertising keywordextraction to topic extraction and sentiment classification sub-tasks, i.e. to identify topics towards which consumers have nega-tive attitudes. Often the consumers are more concerned aboutthe topics on which they have personal comments. Therefore, weonly consider the sentences which contain sentiment words. Andthese sentences are referred to as opinion sentences throughout thispaper. Our extraction approach can be summarized in three steps:

(1) Identify the opinion sentences in the texts;(2) Extract the topics in the opinion sentences given the labeled

sentiment words;(3) Select advertising keywords from the extracted topic words.

4.1. Identify opinion sentences

In this step, as we define opinion sentences as sentences thatcontain sentiment words, the identification task can be tackledby discriminating if any of the contained words are sentimentones. In our work, we employ a dictionary based approach to judgeif a word is a sentiment word. The dictionary used in our work isthe General Inquirer (http://www.wjh.harvard.edu/inquirer/)which consists of 1892 negative words such as ‘‘abandon”, ‘‘abuse”,‘‘poor”, and 1563 positive ones such as ‘‘able”, ‘‘appreciate”, ‘‘good”.

4.2. Extract topic words of opinion sentences using rules

Given opinion sentences identified from the texts and the con-tained sentiment words as well, topic words associated with thesentiment words can be extracted. Here we propose our solutionusing pre-set rules based on syntactic parsing of sentences.

We take advantage of Minipar (http://www.cs.ualberta.ca/lin-dek/minipar.htm) as the syntactic parsing tool, which uses thedependency grammar (Tesniere, 1959). After parsing, words in asentence are assigned with corresponding syntactic categoriesand linked to each other by certain relations. In dependency gram-

mar, the relation between two words A and B can be described asA(or B) depends on B(or A). There are also other complicated rela-tions between A and B. For example, A depends on another word Cwhile C depends on B. We define two categories to summarize allthe relations, which are also illustrated in Fig. 3.

Definition 1. Direct relation (DR): A direct relation means oneword depends on the other one directly or they both depend on athird word directly. In the context of sentiment word S and topicword T, this relation means that S(or T) depends on T(or S), or S andT both depend on another word H.

Definition 2. Indirect relation (IDR): An indirect relation means oneword depends on the other one through another word or they bothdepend on a third word indirectly. In the context of sentimentword S and topic word T, this relation means that S(or T) dependson T(or S) through another word H, or S(or T) depends on a word H1which depends on the same word H which T(or S) depends on.

Fig. 4. Algorithm for extracting topic word.

6186 G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

Note that in IDR, there may be more than one H or H1 in thedependency path. In our work, we only consider the one word sit-uation as compact dependency relations would guarantee the asso-ciation between S and T with high certainty.

Given the seven relations between S and T, we design our ruleswhich are able to capture the relations as well as the word syntac-tic categories. The specific rules are listed in Table 1.

In our rule representations, S(T, H1, H)-Category and S(T, H1,H)-Relation denote the syntactic category and relation of the wordS(T, H1, H), respectively. We also assign priority Pi (i = 1,...,7) toeach rule in case that more than one rule can be applied to theextraction task. In the seven rules, R1 is assigned with the highestpriority P1 and R2 to R7 get lower priorities in descending order.We set the priority for each rule by considering the certainty ofthe association between S and T that the corresponding relationcan guarantee.

For the sake of simplicity, we only consider the topologicalstructures implied by these rules, ignoring most of the detailedinformation of relation and syntactic category. One constraint isthat the category of topic word is limited to N (assigned by Mini-par) which means the part-of-speech of the topic word should benoun. Note that Minipar parses all pronouns and numbers also asN. We do not consider the noun phrases in our current work, sincethe state-of-the-art phrase identification is still not accurate en-ough. There are also some exceptions we need to exclude, suchas the conj relation in R1 and R2 which means the two relatedwords are of equal syntactic role in the sentence.

Using these rules and their priorities, we are now able to extractthe topic word in the opinion sentence. Take R1 for example, giventhe sentiment word S parsed as S-Category, word with category T-Category and relation S-Relation with S is taken as the topic word Tof S with priority P1. The algorithm shown in Fig. 4 elaborates howto extract topic word T using these rules.

4.3. Select advertising keywords

Clearly not all the above extracted topic words can be taken asadvertising keywords directly as explained in the introduction sec-tion. We should identify users’ attitudes towards the topics andthen select appropriate topic words as advertising keywords. Fromthe perspective of sentiment analysis, the implied sentiment of thetopic word is classified to be positive or negative based on thepolarity of the related sentiment word. As stated in the very begin-ning, advertising on topics towards which consumers have nega-tive attitudes is more helpful and targeted to users. Therefore,we take topic words which correspond to negative sentimentwords as the advertising keywords.

5. Appropriate ad placement

After the advertising keywords are obtained through topicextraction and sentiment classification, we can select appropriate

Table 1Rules defined in our work. S(T, H1, H)-Category and S(T, H1, H)-Relation mean thesyntactic category and relation of the word S(T, H1, H), respectively. We use theserules to extract topic words in opinion sentences.

No. Rules

R1 S-Category < S-Relation > T-CategoryR2 T-Category < T-Relation > S-CategoryR3 S-Category < S-Relation > H-Category < T-Relation > T-CategoryR4 S-Category < S-Relation-T-Crelation > T-CategoryR5 T-Category < T-Relation-S-Crelation > S-CategoryR6 S-Category < S-Relation-H1-Relation > H-Category < T-Relation > T-CategoryR7 T-Category < T-Relation-H1-Relation > H-Category < S-Relation > S-Category

ads that are relevant to these keywords and simultaneously meetconsumers’ needs. As stated above, our DASA ad placement strat-egy only concerns topics towards which consumers have negativeattitudes.

In real world applications, advertising keywords can be brandnames (such as BMW in the sentence ‘‘I do not like BMW”), aswell as composed features (such as safety in the sentence ‘‘Iam not satisfied with the safety of Honda Accord”). Therefore, dif-ferent strategies should be adopted to select corresponding adsfor these two situations. In our work, we propose to promoterivals’ products when the advertising keywords are brand namesand select products which have excellent performance in certainfeatures when the advertising keywords are feature names. Incase of multi-candidates, WTP (willingness to pay) (Feng, Bharg-ava, & Pennock, 2003) can be employed to determine which oneto place.

In order to select appropriate brands efficiently for the two sit-uations, we need to know which two brands are rivals and whichfeatures a brand is good at. In traditional advertising systems, anad is always represented by its bid phrases, self-explanatory title,short description and URL of the target Web page. These piecesof information can only provide a general description about theproduct. However, we argue that ad description should be moretargeted and expressive in terms of its advantages over its rivalsso that the ad-network can be aware of products’ advantages andconsequently select appropriate ads to meet consumers’ needs. Inour work, we design a browser based system to construct a knowl-edge base for different products. Fig. 5 is a snapshot of the system.Any manufactories or dealers can provide their product informa-tion through the system. Currently, we take automobile domainas an example. Note that our approach can also be applied otherdomains.

The product information is stored in XML files as shown inFig. 6. The element brand in the node AUTO is the brand name ofthe auto, such as BMW, BENZ, etc. The element name in the nodeMODEL is the model name, such as Z, X serials of BMW. The nodeRIVAL stores brand names of rivals and node ADVANTAGE_FEATUREcontains the outstanding features of the model. An example is that750li of BMW may be labeled as rival of S600 of BENZ. Anotherexample is that Volvo has excellent performance in safety. Thus,we can promote ads of S600 when the consumer is complainingabout 750li and place ads of Volvo when he is unsatisfied withthe safety feature of Honda Accord.

Fig. 5. Snapshot of our system for product information submission. Manufactories or dealers can provide advantageous feature information and rivals of their productsthrough this system.

Fig. 6. XML format for product information storage.

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191 6187

6. Experiments and results

Three experiments are conducted to evaluate the performanceof our approaches to topic word and advertising keyword extrac-tion and ad placement. We begin with a description of the evalua-tion corpus.

Table 2Statistical information for the three data sets for D#0, D#1 and D#2.

D#0 D#1 D#2

128,425 articles 3783 opinion sentences 647 articles

6.1. Evaluation corpus

Since our advertising approach considers not only topical rele-vance but also consumers’ opinions, we use Web forums as theevaluation corpus. Particularly, we choose the popular auto forumautomotiveforums.com as the source data in our experiments. In theforums, the users’ posts are organized as one thread if they all re-spond to the same post. We crawl and store these posts in unit ofthreads, i.e. store all the posts of one thread in a single file. We callthis file an article hereafter. Let D#0 denote the set of articles. An-other two data sets are constructed from D#0 for the first twoexperiments respectively, D#1 for topic word evaluation and

D#2 for advertising keyword evaluation. Table 2 shows the de-tailed information of each data set.

In D#1, the average length of the 3783 opinion sentences is 21words. All these sentences are labeled manually with the topicwords. We develop a tool for volunteers to do the labeling withthe instruction as follows:

Suppose you are the writer of the sentence, given the sentimentword, please figure out in the sentence what is the topic word thatthe sentiment word related to. For example, in the sentence ”The z3was awful and z4 is not much better”, the corresponding topic wordfor ”awful” should be ”z3” and that for ”better” should be ”z4”.

We collect the articles which contain the 3783 opinion sen-tences and the final collection D#2 consists of 647 articles. Foradvertising keyword labeling, we divide the 647 articles into 10groups, and then request 10 annotators to label the keywords inarticles whose corresponding ads they consider appropriate(”appropriate” means not only relevant but also not annoying).These keywords are then taken as the correct advertising keywordsfor the articles.

6188 G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

6.2. Experiments on topic word extraction

Topic word extraction is the key to the extraction of advertisingkeywords, therefore we conduct separate experiments on it. Webegin with the evaluation strategy and criteria.

6.2.1. Evaluation strategy and criteriaTo the best of our knowledge, there is no previous work on sen-

timent based topic word extraction in the research area of senti-ment analysis. Therefore, in our experiments, we compare ourapproach with a naive distance based approach that extracts thenearest noun of the sentiment word as the topic word. We usethe accuracy as the evaluation criteria. The accuracy is defined tobe the percentage of the sentences whose automatically extractedtopic words (either by our approach or the distance based ap-proach) are the same as the user pre-labeled ones.

6.2.2. Results and discussionsTable 3 shows the results of the 3783 opinion sentences in D#1

by using our rule based approach, the distance based approach andtwo random approaches: Random-All and Random-Noun. The Ran-dom-All approach selects any one word from the sentence ran-domly as the topic word. Thus its corresponding accuracy valueis calculated as the inverse of the average sentence length (21words as shown before). Similarly, the Random-Noun approach isto select any noun in the sentence as the topic word and thus itsaccuracy value is the inverse of the average number of wordsparsed as N contained in the sentence (The number is 10.25. Thisnumber is a little bit high because Minipar parses all pronounsand numbers in addition to actual nouns as N and meanwhile someparsing errors also exist.). We take the -Noun case into account asboth rule and distance based approaches impose the nounconstraint.

As can be seen, both our proposed approach and the distancebased approach significantly outperform random approaches. Par-ticularly, our approach obtains 55% accuracy, while the distancebased approach obtains 42% accuracy. There is almost 31% perfor-mance improvement. This shows that our approach is effective inextracting the topic word. Moreover, this result reveals the factthat the relationship between sentiment word and topic word

Table 3Performance results of our rule based approach compared with the distance basedapproach and random approaches. The accuracies of random approaches arecalculated as the inverse of average sentence length (for Random-All) or the inverseof average number of contained words parsed as N (for Random-Noun). TheCorrectSent column is the number of sentences whose topic words are extractedcorrectly.

Approaches CorrectSent Accuracy

Rule 2063 0.55Distance 1581 0.42Random-Noun – 0.10Random-All – 0.05

Table 4Some examples for different approaches. The Sentiment column shows the sentiment wordby Minipar in the sentence. The Labeled column shows manually labeled topic words. Theapproaches, respectively.

Sentences Sentim

So the next thing is, does anyone know where I could get a cheap dealersmanual for the car?

Cheap

The coil tower on these vehicles are notorious for corrosion. NotorioBut in the end, a nice turbo setup would cost approx. Cost

can be better characterized by syntactic structure rather thandistance.

Table 4 gives some examples of our rule based approach com-pared with other approaches.

6.3. Experiments on advertising keyword extraction

In this section, we first describe our evaluation criteria andstrategy, and then show the results and discussions.

6.3.1. Evaluation criteria and strategyIn order to evaluate the performance of different advertising

keyword extraction methods, we consider the quality of advertis-ing keywords from two aspects: the commercial value (CV) andtraditional measurements (precision, recall and F1-measure)which are calculated based on the manually labeled results. Thecommercial value of extracted keywords in an article is definedas the percentage of extracted keywords which are of real valuein some particular online advertising environment, like Google’sAdWords and Yahoo’s Sponsored Search. It measures the qualityof keywords from the perspective of advertisers while traditionalmeasurements emphasize the consumers’ interests.

For comparison, we implement another keyword extraction ap-proach based on the famous keyword weighting schema TFIDF,which is regarded as the most useful feature for advertising key-word extraction (Yih et al., 2006). It would be important to notethat, however, our proposed approach can be easily incorporatedinto previous approaches, since most of them only consider topicalrelevance while our approach considers consumers’ attitudes. Ouraim in the experiments is to show the effectiveness of our ap-proach instead of showing superior performance over sophisti-cated previous approaches. Therefore, a comparison to a typicalfeature in advertising keyword extraction is enough. For our rulebased approach, the advertising keywords of an article are com-posed of those of the opinion sentences in the article. All these key-words are not weighted. For the TFIDF based approach, we removethe stop words and select the top 10 keywords with the highestTFIDF values as the advertising keywords. The average number ofthe labeled keywords in an article is 2.76 and that of those ex-tracted by our rule based approach is 2.26.

One important issue in our evaluation strategy is about the rec-ognition of the commercial value of a word. We argue that a wordis commercially valuable if and only if there is at least one adver-tiser that bids on it. Particularly, in order to quantitatively evaluateit, we submit a keyword to some search engine and check if there isany ad shown in the result page. If there is ad shown, this keywordis considered commercially valuable. In current experiments, wemake use of Google’s search engine and its AdWords service.Fig. 7 gives two search examples (‘‘shoes” and ‘‘black”) on Google.From the results, we can see that there are ads in the search resultpage of ‘‘shoes” but no ads for ‘‘black”. So the keyword ‘‘shoes” isconsidered commercially valuable while ‘‘black” is not. This con-clusion is also consistent to common sense.

contained in the sentence. The Nouns column shows the words which are parsed as NRule and Distance columns show the extracted topic words by rule and distance based

ent Nouns Labeled Rule Distance

Thing, anyone, where, manual,dealers, car

Manual Manual Dealers

us Tower, coil, vehicles, corrosion Tower Tower VehiclesEnd, setup, turbo, approx Setup Setup Approx

Fig. 7. Google search results for words ‘‘shoes” and ‘‘black”. As can be seen, there are ads in the result page of ‘‘shoes” but no ads for ‘‘black”. This means word ‘‘shoes” hashigher commercial value than ‘‘black”.

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191 6189

6.3.2. Results and discussionsIn our experiments, we report CV, precision, recall and F1-mea-

sure of the top 10 selected keywords as the overall results for theTFIDF based approach. Considering that the average number ofadvertising keywords manually labeled and extracted by rulebased approach is close to three, we also evaluate the performanceof TFIDF based approach for the top three selected keywords. Table5 lists all the results of different approaches.

From the table we can see that our approach outperforms theTFIDF based approach (both TFIDF-Average and TFIDF-Top3) inall the four measurements, except the precision in TFIDF-Top3.The improvement of CV (17.3% over TFIDF-Average and 8.6% overTFIDF-Top3) means that the advertising keywords extracted by

our approach are more valuable from the perspective of advertisersand thus ensures these keywords would be related with ads in-stead of being abandoned because of being meaningless foradvertisers.

The superior performance of our approach in traditional mea-surements precision, recall and F1-measure shows that our ap-proach can extract more advertising keywords with highaccuracy than other approaches. Note that, the F1-measure canbe essentially taken as the targeted-ness of the keywords fromthe view of consumers as it is a combination of precision and recall.Usually the more targeted the advertising keywords are, the higherrevenue the advertisers will obtain. The reason is that with theimprovement in the targeted-ness of keywords, ad relevance

Table 5Results of CV, precision, recall and F1-measure for our rule based approach and theTFIDF based approach. For TFIDF, we consider two different variations TFIDF-Averageand TFIDF-Top3 which take the average value of top 10 and top 3 keywords,respectively.

Approaches CV Precision Recall F1

Rule 0.265 0.101 0.375 0.159TFIDF-Average 0.226 0.092 0.211 0.128TFIDF-Top3 0.244 0.104 0.145 0.121

6190 G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

increases, therefore consumers will get less disturbed by ads andthus will remain well or even better impression on advertisingWeb sites. Finally, the market share would not be ill affected alot by ads and advertisers and Web sites will be likely to get moreprofits. Therefore, F1-measure is more important than the othertwo measurements. In the exception of the results, although ourapproach is a little inferior in precision compared with TFIDF-Top3, we gain 31.4% improvement in the F1-measure, whichmeans our approach still outperforms TFIDF-Top3.

Note that all the values in our measurements are relatively low.However, in online advertising-like applications, an improvementof X% in ad matching can lead to an improvement of X% in theend result unlike many other tasks where the effect of performanceenhancement on the end result is not linear (Mishne & Rijke, 2006).Therefore, regarding the magnitude of online advertising market, atiny improvement would lead to a considerable increase in the fi-nal revenue for advertisers.

6.4. Experiments on ad selection

The above experiments have examined the performance of ourapproach for advertising keyword extraction, which is a key steptowards ad placement. We now conduct experiments on selectingappropriate ads. Since we do not have the corpus of real world ads,we represent our recommendations as descriptions such as ”autosthose perform excellent in safety”. As described before, there are twokinds of ads that would be promoted to consumers as they maycomplain on a certain brand or feature. For simplicity, we focusonly on features in our current experiments.

We compare DASA with the naive strategy that extracts adver-tising keywords in the same way but does not consider the atti-

Fig. 8. Comparison results of our approach DASA and TE-ALL which does notconsider the attitudes of consumers.

tudes of consumers, i.e. take all the topic words as theadvertising keywords (TE-ALL). Volunteers are requested to makethe judgment on the recommended ads (the ad descriptions in-deed). In our experiments, we construct a prototype system inwhich volunteers provide texts commenting on some products(only on features of autos in current work) and then get two listsof ads returned by DASA and TE-ALL. Volunteers are unaware ofwhich list is the result of which approach. They are then asked todetermine which one they think is better to meet their needs withless annoying.

Fig. 8 shows the comparison between DASA and TE-ALL. The Y-axis is the number of times that volunteers regard the results of anapproach are better than those of the other one. An obvious obser-vation from the figure is that our approach (DASA) outperforms TE-ALL. For 35 times, volunteers judge that the results of our approachare better than those of TE-ALL, while the contrast is 22 times. Theresults show that it is reasonable to take consumers’ attitudes intoaccount.

7. Conclusions

Targeted advertising is of great importance for Internet compa-nies to gain revenue from both advertisers and consumers. Previousapproaches focus only on the topical relevance while the consum-ers’ attitudes are ignored. These approaches fail to meet the actualneeds of consumers especially when they may have negative atti-tudes towards the mentioned topics. In this paper, we take the atti-tudes of consumers into account and propose a novel advertisingstrategy DASA to promote ads according to what they are unsatis-fied with. We use pre-set rule based sentiment analysis techniquesto tackle topic word extraction and consumers’ attitude identifica-tion in advertising keyword extraction. We also design a prototypesystem for users to provide product information for the sake of adselection. For evaluation, we conduct separate experiments to mea-sure the quality of topic words, advertising keywords and placedads. Specifically, we propose to use commercial value to measurethe quality of advertising keywords from the perspective of adver-tisers. Experiments show that our strategy outperforms the TFIDFbased approach in advertising keyword extraction and the naive ap-proach which does not consider the attitudes of consumers in adselection. The results indicate that it is reasonable to take the atti-tudes of consumers into consideration.

References

Bhargava, H. K., & Feng, J. (2002). Paid placement strategies for internet searchengine. In Proc. 2002 Int. WWW Conf. (WWW’02) (pp. 117–123).

Broder, A., Fontoura, M., Josifovski, V., & Riedel, L. (2007). A semantic approach tocontextual advertising. In Proc. 2007 Int. Conf. on research and development ininformation retrieval (SIGIR’07), 2007 (pp. 559–566).

Chakrabarti, D., Agarwal, D., & Josifovski, V. (2008). Contextual advertising bycombining relevance with click feedback. In Proc. 2008 Int. WWW Conf.(WWW’08) (pp. 417–426).

Ding, X., & Liu, B. (2007). The utility of linguistic rules in opinion mining. In Proc.2007 Int. Conf. on research and development in information retrieval (SIGIR’07) (pp.811–812).

Esuli, A., & Sebastiani, F. (2005). Determining the semantic orientation of termsthrough gloss classification. In Proc. 14th ACM Int. Conf. on information andknowledge management (CIKM’05) (pp. 617–624).

Feng, J., Bhargava, H. K., & Pennock, D. (2003). Comparison of allocation rules forpaid placement advertising in search engines. In Proc. 5th Int. Conf. on electroniccommerce (ICEC’03) (pp. 294–299).

Hatzivassiloglou, V., & McKeown, K. R. (1997). Predicting the semantic orientationof adjectives. In Proc. 1997 ACL (ACL’04).

Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. In Proc. 2004Int. Conf. knowledge discovery and data mining (KDD’04) (pp. 168–177).

Lacerda, A., Cristo, M., Goncalves, M., Fan, W., Ziviani, N., & Ribeiro-Neto, B. (2006).Learning to advertise. In Proc. 2006 Int. Conf. on research and development ininformation retrieval (SIGIR’06) (pp. 549–556).

Lu, Y., & Zhai, C. (2008). Opinion integration through semi-supervised topicmodeling. In Proc. 2008 int. WWW Conf. (WWW’08) (pp. 121–130).

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191 6191

Mccoy, S., Everard, A., Polak, P., & Galletta, D. (2007). The effects of onlineadvertising. Communications of the ACM, 50(3), 84–88.

Mei, Q., Cai, D., Zhang, D., & Zhai, C. (2008). Topic modeling with networkregularization. In Proc. 2008 Int. WWW Conf. (WWW’08) (pp. 101–110).

Mei, Q., & Ling, X. (2007). Topic sentiment mixture: Modeling facets and opinions inweblogs. In Proc. 2007 Int. WWW Conf. (WWW’07) (pp. 161–175).

Mishne, G., & Rijke, M. (2006). Language model mixtures for contextual adplacement in personal blogs. In Proc. 5th Int. Conf. NLP (FinTAL).

Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis usingsubjectivity summarization based on minimum cuts. In Proc. 42nd Ann. meetingof the association for computational linguistics (ACL’04) (pp. 271–278).

Ribeiro-Neto, B., Cristo, M., Golgher, P., & Moura, E. (2005). Impedance coupling incontent-targeted advertising. In Proc. 2005 Int. Conf. on research and developmentin information retrieval (SIGIR’05) (pp. 496–503).

Tesniere, L. (1959). Elements de syntaxe structurale. Paris: Librairie C. Klincksieck.Turney, P. (2002). Thumbs up or thumbs down? Semantic orientation applied to

unsupervised classification of reviews. In Proc. 40th Ann. meeting of theassociation for computational linguistics (ACL’02) (pp. 417–424).

Turney, P. D., & Littman, M. L. (2003). Measuring praise and criticism: Inference ofsemantic orientation from association. ACM Transactions on Information Systems,21(4), 315–346.

Wang, C., Zhang, P., Choi, R., & D’Eredita, M. (2002). Understanding consumersattitude toward advertising. In Proc. 8th Americas Conf. on information system(pp. 1143–1148).

Wu X., & Bolivar, A. (2008). Keyword extraction for contextual advertisement. InProc. 2008 Int. WWW Conf. (WWW’08) (pp. 1195–1196).

Yih, W., Goodman, J., & Carvalho, V. R. (2006). Finding advertising keywords on webpages. In Proc. 2006 Int. WWW Conf. (WWW’06) (pp. 213–222).