Proceedings of the 2010 Named Entities Workshop, ACL 2010, pages 116–125,Uppsala, Sweden, 16 July 2010. c©2010 Association for Computational Linguistics

Think Globally, Apply Locally: Using Distributional Characteristics forHindi Named Entity Identification

Shalini Gupta Pushpak BhattacharyyaDepartment of Computer Science and Engineering

IIT BombayMumbai, India.

{shalini, pb}@cse.iitb.ac.in

Abstract

In this paper, we present a novel ap-proach for Hindi Named Entity Identifica-tion (NEI) in a large corpus. The key ideais to harness the global distributional char-acteristics of the words in the corpus. Weshow that combining the global distribu-tional characteristics along with the localcontext information improves the NEI per-formance over statistical baseline systemsthat employ only local context. The im-provement is very significant (about 10%)in scenarios where the test and train cor-pus belong to different genres. We alsopropose a novel measure for NEI basedon term informativeness and show that itis competitive with the best measure andbetter than other well known informationmeasures.

1 Introduction

NER is the task of identifying and classifyingwords in a document into predefined classes likeperson, location, organization, etc. It has many ap-plications in Natural Language Processing (NLP)NER can be divided into two sub-tasks, NamedEntity Identification (NEI) and Named EntityClassification (NEC). In this paper, we focus onthe first step, i.e., Named Entity Identification.NEI is useful in applications where a list of NamedEntities (NEs) is required. Machine Translationneeds identification of named entities, so that theycan be transliterated.

For Indian languages, it is tough to identifynamed entities because of the lack of capitaliza-tion. Many approaches based on MEMM (Saha etal., 2008b), CRFs (Li and McCallum, 2003) andhybrid models have been tried for Hindi NamedEntity Recognition. These approaches use onlythe local context for tagging the text. Many ap-

plications need entity identification in large cor-pora. When such a large corpus is to be tagged,one can use the global distributional characteris-tics of the words to identify the named entities.The state-of-the-art methods do not take advantageof these characteristics. Also, the performanceof these systems degrades when the training andtest corpus are from different domain or differentgenre. We present here our approach-CombinedLocal and Global Information for Named EntityIdentification (CLGIN) which combines the globalcharacteristics with the local context for HindiNamed Entity Identification. The approach com-prises of two steps: (i) Named Entity Identifica-tion using Global Information (NGI) which usesthe global distributional characteristics along withthe language cues to identify NEs and (ii) Com-bining the tagging from step 1 with the MEMMbased statistical system. We consider the MEMMbased statistical system (S-MEMM) as the Base-line. Results show that the CLGIN approach out-performs the baseline S-MEMM system by a mar-gin of about 10% when the training and test corpusbelong to different genre and by a margin of about2% when both, training and test corpus are similar.NGI also outperforms the baseline, in the formercase, when training and test corpus are from dif-ferent genre. Our contributions in this paper are:

• Developing an approach of harnessing theglobal characteristics of the corpus for HindiNamed Entity Identification using informa-tion measures, distributional similarity, lex-icon, term co-occurrence and language cues

• Demonstrating that combining the globalcharacteristics with the local contexts im-proves the accuracy; and with a very signif-icant amount when the train and test corpusare not from same domain or similar genre

• Demonstrating that the system using only the

116

global characteristics is also quite compara-ble with the existing systems and performsbetter than them, when train and test corpusare unrelated

• Introducing a new scoring function, whichis quite competitive with the best measureand better than other well known informationmeasures

Approach Description

S-MEMM(Baseline)

MEMM based statistical system withoutinserting global information

NGI Uses global distributional characteristicsalong with language information for NEIdentification

CLGIN Combines the global characteristics de-rived using NGI with S-MEMM

Table 1: Summary of Approaches

2 Related Work

There is a plethora of work on NER for En-glish ranging from supervised approaches likeHMMs(Bikel et al., 1999), Maximum Entropy(Borthwick, 1999) (Borthwick et al., 1998), CRF(Lafferty et al., 2001) and SVMs to unsupervised(Alfonseca and Manandhar, 2002), (Volker, 2005)and semi-supervised approaches (Li and Mccal-lum, 2005). However, these approaches do notperform well for Indian languages mainly due tolack of capitalization and unavailability of goodgazetteer lists. The best F Score reported for HindiNER using these approaches on a standard cor-pus (IJCNLP) is 65.13% ((Saha et al., 2008a)).Higher accuracies have been reported (81%) (Sahaet al., 2008b), albeit, on a non-standard corpus us-ing rules and comprehensive gazetteers.

Current state-of-the-art systems (Li and McCal-lum, 2003) (Saha et al., 2008b) use various lan-guage independent and language specific features,like, context word information, POS tags, suffixand prefix information, gazetteer lists, commonpreceding and following words, etc. The perfor-mance of these systems is significantly hamperedwhen the test corpus is not similar to the trainingcorpus. Few studies (Guo et al., 2009), (Poibeauand Kosseim, 2001) have been performed towardsgenre/domain adaptation. But this still remains anopen area. Moreover, no work has been done to-wards this for Indian languages.

Select words based on

Information Measure

Applying Pruning

Heuristics

Corpus

NEIG Tagged DataSet

Applying Augmenting

Heuristics

Threshold (Set using Development Set

Step 1

Tagging using Global Distribution

(NEIG)

Trained ModelStatistical

System (MEMM)

Step 2

MEMM Based Statistical

System(S-MEMM)

Final Tagged DataSet

Added as a feature

Features (Context Words, POS Tags, Suffix Info, Gazetteers,

Lexicon, etc.)

DataSet to be Tagged

Figure 1: Block diagram of CLGIN Approach

One shortcoming of current approaches is thatthey do not leverage on global distributional char-acteristics of words (e.g., Information Content,Term Co-occurrence statistics, etc.) when a largecorpus needs NEI. Rennie and Jaakkola (2005)introduced a new information measure and usedit for NE detection. They used this approachonly on uncapitalized and ungrammatical Englishtext, like blogs where spellings and POS tags arenot correct. Some semi-supervised approaches(Collins and Singer, 1999), (Riloff and Jones,1999), (Paşca, 2007) have also used large availablecorpora to generate context patterns for named en-tities or for generating gazetteer lists and entityexpansion using seed entities. Klementiev andRoth (2006) use cooccurrence of sets of termswithin documents to boost the certainty (in across-lingual setting) that the terms in questionwere really transliterations of each other.

In this paper, we contend that using such globaldistributional characteristics improves the perfor-mance of Hindi NEI when applied to a large cor-pus. Further, we show that the performance ofsuch systems which use global distribution charac-teristics is better than current state-of-the-art sys-tems when the training and test corpus are not sim-ilar (different domain/genre) thereby being moresuitable for domain adaptation.

3 MEMM based Statistical System(S-MEMM)

We implemented the Maximum Entropy MarkovModel based system(Saha et al., 2008b) for NEIdentification. We use this system as our Base-line and compare our approaches NGI and CLGINwith this baseline. We used various language de-pendent and independent features. An important

117

Input Text: िलब ने िससार नद क मछिलय का समूल नाश कर दया| बाँकड़ ने मुःकराकर कहा क कल वह मछली पकड़ने जायेगा ह |

English Translation: Lib destroyed all the fishes of Sisaar River. Bankad smilingly said, that he would surely go for fishing tomorrow.

Transliteration: Lib ne Sisaar Nadi ki machliyon ka samool naash kar diya. Bankad ne muskurakar kaha ki kal vah machli pakadne jayega hi 

िलब, िससार, नद , ि ँ

Word  (Transliteration,             Info Value िलब

Extract    Nouns

E Hi hमछिलय , नाश, बाँकड़, कल, मछली Lib, Sisaar, Nadi, 

Translation)िलब (Lib, Lib) 2.6718िससार (Sisaar, Sisaar) 0.9982

( d )

िससारमछिलयबाँकड़Calculate 

Extract High Info Value Terms  (Ab

Lib, Sisaar, River, fishes,

machliyon, naash, Bankad, kal, machli

नद (Nadi, River) 0.2839मछिलय (machliyon, fishes) 0.4622नाश (naash, destruction) 0.1097ँ ( k d k d)

ड़मछली

Apply Term

Information Content

(Above Threshold)

Lib, Sisaar, River, fishes, destruction, Bankad, tomorrow, fish

बाँकड़ (Bankad, Bankad) 1.3175कल (kal, tomorrow) 0.2288मछली (machli, fish) 0.6148

Apply Term Excluding Heuristics (Lexicon Suffix

Apply Augmenting H i ti

(Lexicon, Suffix, Dist. Sim.)

िससार नदHeuristics (Term Co‐occurrence)

िलब, िससार, बाँकड़

Input Text: िलब ने िससार नद क मछिलय का समूल नाश कर दया| बाँकड़ ने मुःकराकर कहा क कल वह मछली पकड़ने जायेगा ह |Output:

English Translation: Lib destroyed all the fishes of Sisaar River. Bankad smilingly said, that he would surely go for fishing tomorrow.

Transliteration: Lib ne Sisaar Nadi ki machliyon ka samool naash kar diya. Bankad ne muskurakar kaha ki kal vah machli pakadne jayega hi

Figure 2: An Example explaining the NGI approach

modification was the use of lexicon along with tra-ditionally used gazetteers. Gazetteers just improvethe recall whereas including the lexicon improvesthe precision. The state-of-art Hindi NER sys-tems do not use lexicon of general words but wefound that using lexicons significantly improvesthe performance. Unlike English, NEs in Hindi arenot capitalized and hence it becomes important toknow, if a word is a common word or not.

Features used in S-MEMM were:

• Context Words: Preceding and succeeding twowords of the current word

• Word suffix and prefix: Fixed length (size: 2)suffix information was used. Besides, suffixlist of common location suffixes was created

• First word and last word information• Previous NE Tag information• Digit information• Gazetteer Lists: Person and Location names,

Frequent words after and before person, orga-nization and location names, list of commoninitials, stopwords, etc.

• POS Tag Information• Lexicons: If the stemmed word was present in

the lexicon, this feature was true.

4 Our Approach-CLGIN

In this section, we describe our approach, CLGINin detail. It combines the global information fromthe corpus with the local context. Figure 1 gives

the block diagram of the system while tagging acorpus and Figure 2 explains the approach usingan example. This approach involves two steps.Step 1 of CLGIN is NGI which creates a listof probable NEs (both uni-word and multi-word)from the given corpus and uses it to tag the wholecorpus. Sections 4.1 and 4.2 explain this step indetail. Later, in step 2, it combines the taggingobtained from step 1, as a feature in the MEMMbased statistical system. Output thus obtainedfrom the MEMM system is the final output of theCLGIN approach. The creation of list in step 1,involves the following steps

• A list of all words which appeared as a noun atleast once in the the corpus is extracted.

• List is ordered on the basis of the informationcontent derived using the whole corpus. Wordsabove the threshold (set during training usingthe development set) are selected as NEs.

• Heuristics are applied for pruning and aug-menting the list.

• Multi-word NEs derived using term co-occurrence statistics along with language char-acteristics are added to the NE list.

The above process generates a list of NEs (uni-word and multi-word). In the second step, we pro-vide this tagging to the S-MEMM along with otherset of features described in Section 3

During training, the cutoff threshold is set forselecting NEs (in bullet 2) above. Also the taggingobtained from the step 1 is added as a feature to

118

S-MEMM and a model is trained during the train-ing phase. The following sections describe this ap-proach in detail.

4.1 Information Measures/Scoring FunctionsVarious measures have been introduced for de-termining the information content of the words.These include, IDF (Inverse Document Fre-quency) (Jones, 1972) , Residual IDF (Church andGale, 1995), xI - measure (Bookstein and Swan-son, 1974), Gain (Papineni, 2001), etc. We intro-duced our own information measure, RF (Ratio ofFrequencies).

4.1.1 RF (Ratio of Frequencies)NEs are highly relevant words in a document(Clifton et al., 2002) and are expected to have highinformation content (Rennie and Jaakkola, 2005).It has been found that words that appear frequentlyin a set of documents and not so frequently in therest of the documents are important with respect tothat set of documents where they are frequent.

We expected the NEs to be concentrated in fewdocuments. We defined a new criteria which mea-sures the ratio of the total number of times theword appears in the corpus to the number of doc-uments containing a word.RF (w) = cf(w)df(w)where cf(w) is the total frequency of a word inthe whole corpus and df(w) is the document fre-quency. This measure is different from the TF-IDFmeasure in terms of the term frequency. TF-IDFconsiders the frequency of the word in the docu-ment. RF considers it over the whole corpus.

We use the scoring function (information mea-sure) to score all the words. During training, wefix a threshold using the development set. Dur-ing testing, we pick words above the threshold asNEs. We then apply heuristics to augment this listas well as to exclude terms from the generated list.

4.2 Heuristics for Pruning and AugmentingNE List

Distributional Similarity: The underlying ideaof Distributional Similarity is that a word is char-acterized by the company it keeps (Firth, 1957).Two words are said to be distributionally similarif they appear in similar contexts. From the previ-ous step (Sect. 4.1), we get a list of words havinghigh score. Say, top t, words were selected. Inthis step, we take t more words and then clustertogether these words. The purpose at this phase is

primarily to remove the false positives and to in-troduce more words which are expected to be NEs.For each distinct word, w in the corpus, we cre-ate a vector of the size of the number of distinctwords in the corpus. Each term in the vector rep-resents the frequency with which it appears in thecontext (context window: size 3) of word, w. Itwas observed that the NEs were clustered in someclusters and general words in other clusters. Wetag a cluster as a NE cluster if most of the wordsin the cluster are good words. We define a wordas good if it has high information content. If thesum of the ranks of 50% of the top ranked word islow, we tag the cluster as NE and add the wordsin that set as NEs. Also, if most of the words inthe cluster have higher rank i.e. lower informationcontent, we remove it from the NE set.

This heuristic is used for both augmenting thelist as well to exclude terms from the list.Lexicon: We used this as a list for excludingterms. Terms present in the lexicon have a highchance of not being NEs. When used alone, thelexicon is not very effective (explained in Sec-tion 5.2). But, when used with other approaches,it helps in improving the precision of the sys-tem significantly. State-of-art Hindi NER systemsuse lists of gazetteers for Person names, locationnames, organization names, etc. (Sangal et al.,2008), but lexicon of general words has not beenused. Unlike English, for Indian languages, it isimportant to know, if a word is a general wordor not. Lexicons as opposed to gazetteers aregeneric and can be applied to any domain. Un-like gazetteers, the words would be quite commonand would appear in any text irrespective of thedomain.

Suffixes: NEs in Hindi are open class words andappear as free morphemes. Unlike nouns, NEs,usually do not take any suffixes (attached to them).However, there are few exceptions like, lAl Ekl�k� bAhr (laal kile ke baahar, (outside Red Fort))or when NEs are used as common nouns, d�f kogA\EDyo\ kF j!rt h{ (desh ko gandhiyon ki za-roorat hai, The country needs Gandhis.) etc. Weremove words appearing with common suffixeslike e\ (ein), ao\ (on), y�\g� (yenge), etc. from theNE list.

Term Co-occurrence: We use the term co-occurrence statistics to detect multi-word NEs. Aword may be a NE in some context but not in an-other. E.g. mhA(mA (mahatma “saint”) when ap-

119

pearing with gA\DF (Gandhi “Gandhi”) is a NE,but may not be, otherwise. To identify such multi-words NEs, we use this heuristic. Such words canbe identified using Term Co-occurrence. We usethe given set of documents to find all word pairs.We then calculate Pointwise Mutual Information(PMI) (Church and Hanks, 1990) for each of theseword pairs and order the pairs in descending orderof their PMI values. Most of the word pairs belongto the following categories:

• Adjective Noun combination (Adjectives fol-lowed by noun): This was the most frequentcombination. E.g. BFnF g\D (bheeni gandh“sweet smell”)

• Noun Verb combination: Edl DwknA (dildhadakna, “heart beating”)

• Adverb verb combination: EKlEKlAkrh\snA (khilkhilakar hansna, “merrily laugh”)

• Cardinal/Ordinal Noun Combination: TowFd�r (thodi der, “some time”)

• Named Entities• Hindi Idioms: uSl� sFDA (ullu seedha)• Noun Noun Combination: HyAtF aEjt (khy-

ati arjit, “earn fame”)

• Hindi Multiwords: jof Krof (josh kharosh)We need to extract NEs from these word pairs. Thefirst four combinations can be easily excluded be-cause of the presence of a verb, cardinals and ad-jectives. Sometimes both words in the NEs appearas nouns. So, we cannot reject the Noun Nouncombination. We handle rest of the cases by look-ing at the neighbours (context) of the word pairs.

We noticed three important things here:

• Multiwords which are followed (alteast once)by m�\ (mein), s� (se), n� (ne), k� (ke), ko(ko) (Hindi Case Markers) are usually NEs.We did not include kF (ki) in the list be-cause many words in the noun-noun combi-nation are frequently followed by ki in thesense of EkyA/ krnA (kiya/karna, “do/did”)e.g. HyAtF aEjt kF (khyati arjit ki, “earnedfame”), prF"A uFZ kF (pariksha uttirandki, “cleared the exam”), etc.

• There were word pairs which were followedby a single word most of the time. E.g I-Vi\EXyA (East India, “East India”) was followedby k\pnF (Company, “Company”) in almost allthe cases. When Company appears alone, itmay not be a NE, but when it appears with East

Corpus No. of Tagged No. of No. of Source GenreDocuments Words NEs

Gyaan 1570 569K 21K Essay, Biography,Nidhi History and Story

Table 2: Corpus Statistics

India, it appears as a NE. Other examples ofsuch word pairs were: KA iNn� (Khan Ibnu,“Khan Ibnu”) followed by alFsm (Alisam,“Alisam”)

• There were word pairs which were followedby uncommon words were not common wordsbut were different words each time, it ap-peared. i.e. Most of the words following theword pair were not part of lexicon. gvnrjnrl (governor general, “Governor Gen-eral”) followed by [ dlhOsF, bhd� r, solbrF,m{VkA', lOX ((dalhousie, bahadur, solbari,metkaf, lord), “Dalhousie, Bahadur, Solbari,Metkaf, Lord”)] Such words are multi wordNEs.

4.3 Step 2: Combining NGI with S-MEMM

The tagging obtained as the result of the step 1(NGI), is given as input to the MEMM based sta-tistical system (S-MEMM). This feature is intro-duced as a binary feature OldTag=NE. If a word istagged as NE in the previous step, this feature isturned on, otherwise OldTag=O is turned on.

5 Experiments and Results

We have used Gyaan Nidhi Corpus for eval-uation which is a collection of various booksin Hindi. It contains about 75000 documents.The details of the corpus are given in Table2. Names of persons, locations, organizations,books, plays, etc. were tagged as NE and othergeneral words were tagged as O (others). Thetagged documents are publicly made available athttp://www.cfilt.iitb.ac.in/ner.tar.gz.

We use the following metrics for evaluation:Precision, Recall and F-Score. Precision is theratio of the number of words correctly tagged asNEs to the total number of words tagged as NEs.Recall is the ratio of the number of words cor-rectly tagged as NEs to the total number of NEspresent in the data set. F Score is defined as(F = 2 ∗ P ∗R/(P + R))

120

5.1 Comparison of Information MeasuresWe compare the performance of the variousterm informativeness measures for NEI which areResidual IDF1, IDF 2, Gain3 and x′ measure 4

and the measure defined in Section 4.1.1. Table3 shows the results averaged after five-fold crossvalidation. The graphs in the Figure 3 to Figure7 show the distribution of words (nouns) over therange of values of each information measure.

Scoring Function Prec. Recall F Score

Residual IDF 0.476 0.537 0.504IDF 0.321 0.488 0.387x-dash Measure 0.125 0.969 0.217RF (Our Measure) 0.624 0.396 0.484Gain 0.12 0.887 0.211

Table 3: Comparison of performance of variousinformation measures

The best results were obtained using ResidualIDF followed by Ratio of Frequencies (RF).

Method Prec Recall F ScoreS -MEMM (Baseline) 0.871 0.762 0.812Res. IDF 0.476 0.537 0.504Res. IDF + Dist Sim (DS) 0.588 0.522 0.553Res. IDF + Lexicon (Lex) 0.586 0.569 0.572Res. IDF + DS + Suffix 0.611 0.524 0.563Res. IDF + Lex + Suffix 0.752 0.576 0.65Res. IDF + Lex + Suffix + Term Cooccur (NGI) 0.757 0.62 0.68CLGIN 0.879 0.784 0.829

Table 4: Performance of various Approaches(Here, train and test are similar)

5.2 NGI and CLGIN Approaches (Trainingand Test Set from Similar Genre)

Table 4 compares the results of S-MEMM, NGIapproach and CLGIN. Besides, it also shows thestep wise improvement of NGI approach. Thefinal F-Score achieved using NGI approach was68%. The F-Score of the Baseline system im-plemented using the MaxEnt package1 from theOpenNLP community was 81.2%.

Using the lexicon alone gives an F-Score ofonly 11% (Precision: 5.97 Recall: 59.7 F-Score:10.8562). But, when used with Residual IDF, the

1Observed IDF - Expected IDF2IDF = -log df(w)

D3Gain = dw

D( dw

D− 1− log dw

D)

4x′(w) = df(w)− cf(w)1http://maxent.sourceforge.net/index.html

45

50 Residu

35

40

45

20

25

30

f Words

10

15

20

% o

0

5

Residual 

ual IDF

General Words

Named Entities

IDF Values

Figure 3: Distribution of Residual IDF values overthe nouns in the corpus

performance of the overall system improves sig-nificantly to about 57%. Note that, the use of lexi-con resulted in an increase in precision (0.5860)which was accompanied by improvement in re-call (0.5693) also. The cutoff thresholds in bothcases (Rows 2 and 4 of Table 4) were different.Suffix information improved the systems perfor-mance to 65%. As words were removed, morewords from the initial ordered list (ordered on thebasis of score/information content) were added.Hence, there was a small improvement in recall,too. Improvement by distributional similarity waseclipsed after the pruning by lexicon and suffix in-formation. But, in the absence of lexicon; distri-butional similarity and suffix information can beused as the pruning heuristics. Adding the multi-word NEs to the list as explained in the section 4.2using term co-occurrence statistics, improved theaccuracy significantly by 3%. Word pairs were ar-ranged in the decreasing order of their PMI valuesand a list was created. We found that 50% of theNE word pairs in the whole tagged corpus lied inthe top 1% of this word pairs list and about 70%of NE word pairs were covered in just top 2% ofthe list.

CLGIN which combines the global informa-tion obtained through NGI with the Baseline S-MEMM system gives an improvement of about2%. After including this feature, the F-Score in-creased to 82.8%.

5.3 Performance Comparison of Baseline,NGI and CLGIN (Training and Test Datafrom different genre)

In the above experiments, documents were ran-domly placed into different splits. Gyaan Nidhiis a collection of various books on several top-

121

35 Ga

25

30

15

20

of  W

ords

10

5

% 

0

5

ain

General Words

Named Entities

Gain

Figure 4: Distribution of Gain values over thenouns in the corpus

25 ID

20

10

15

of W

ords

5

10

% o

0

DF

General Words

N d E titiNamed Entities

IDF

Figure 5: Distribution of IDF values over thenouns in the corpus

60 Ratio Of Fr

40

50

30

40

of W

ords

10

20

% o

0

10

Ratio o

requencies

General Words

Named Entities

of Frequencies

Figure 6: Distribution of Ratio of Frequencies(RF)values over the nouns in the corpus

ics. Random picking resulted into the mixing ofthe documents, with each split containing docu-ments from all books. But, in this experiment,we divided documents into two groups such thatdocuments from few books (genre: Story and His-tory) were placed into one group and rest into an-other group (Genre: Biography and Essay). Table5 compares the NGI and CLGIN approaches with

25x' Me

20

10

15

of W

ords

5

10% 

0

x' M

easure

General Words

Named Entities

Measure

Figure 7: Distribution of xI measure values overthe nouns in the corpus

S-MEMM and shows that the CLGIN results aresignificantly better than the Baseline System,when the training and test sets belong to differentgenre. The results were obtained after 2-fold crossvalidation.

Method Prec. Recall F Score

S-MEMM 0.842 0.479 0.610NGI 0.744 0.609 0.67CLGIN 0.867 0.622 0.723

Table 5: Performance of various Approaches(Here, train and test are from different genre)

Similar improvements were seen when the setswere divided into (Story and Biography) and (Es-say and History) (The proportions of train and testsets in this division were uneven). The F Scoreof NGI system was 0.6576 and S-MEMM was0.4766. The F Score of the combined system(CLGIN) was 0.6524.

6 Discussion and Error Analysis

6.1 RF and other information measuresAs can be seen from the graphs in Figures 3 to 7,Residual IDF best separates the NEs from the gen-eral words. The measure introduced by us, Ratioof Frequencies is also a good measure, althoughnot as good as Residual IDF but performs betterthan other measures. The words having RF valuegreater than 2.5 can be picked up as NEs, giving ahigh recall and precision. It is evident that IDF isbetter than both, Gain and xI measure, as most ofthe general words have low IDF and NEs lie in thehigh IDF zone. But, the general words and NEsare not very clearly separated. As the number ofnouns is about 7-8 times the number of NEs, the

122

words having high IDF cannot be picked up. Thiswould result in a low precision, as a large num-ber of non-NEs would get mixed with the generalwords. Gain and xI measure do not demarcate theNEs from the general words clearly. We observedthat they are not good scoring functions for NEs.

Information Gain doesn’t consider the fre-quency of the terms within the document itself. Itonly takes into account the document frequencyfor each word. xI measure considers the fre-quency within document but it is highly biasedtowards high frequency words and hence doesn’tperform well. Hence, common words like smy(samay, “time”), Gr (ghar, “home”), etc. havehigher scores compared to NEs like BArt(bharat,“India”), klkA (kalkatta, “Calcutta”), etc. Ourmeasure on the other hand, overcomes this draw-back, by considering the ratio. We could havecombined the measures, instead of using only thebest measure “Residual IDF”, but the performanceof “Gain”, “IDF” and “x’-measure” was not good.Also, results of “RF” and “Residual IDF” werequite similar. Hence, we did not see any gain incombining the measures.

6.2 S-MEMM, NGI and CLGIN

The results in Section 5 show that adding theglobal information with the local context helps im-prove the tagging accuracy especially when thetrain and test data are from different genre. Sev-eral times, the local context is not sufficient todetermine the word as a NE. For example, whenthe NEs are not followed by post positions orcase markers, it becomes difficult for S-MEMM toidentify NEs, e.g., V{gor ek apvAd h{\, (tagore ekapvaad hain,“Tagore is an exception”) or when theNEs are separated by commas, e.g. s� k� mArF d,c� àFlAl.. (Sukumari Dutt, Chunnilal ... “Suku-mari Dutt, Chunnilal ..”). In such cases, becauseof the frequency statistics, the NGI approach isable to detect the words V{gor (Tagore, “Tagore”),d (Dutt, “Dutt”), etc. as NEs and frequently theCLGIN approach is able to detect such words asNEs.

The false positives in NEIG are words whichare not present in the lexicon (uncommon words,words absent due to spelling variations e.g.sA\p/sA p (sanp “snake”)) but have high informa-tiveness. Using the context words of these wordsis a possible way of eliminating these false pos-itives. Many of the organization names having

common words (m\Xl (mandal, “board”)) andperson names (like þkAf (prakash,“light”)) arepresent in the lexicon are not tagged by NEIG.Some errors were introduced because of the re-moval of morphed words. NEs like g� SbAno\, Vop�(Gulbano, Tope) were excluded.

Many of the errors using CLGIN are because ofthe presence of the words in the lexicon. This ef-fect also gets passed on to the neighbouring words.But, the precision of CLGIN is significantly highcompared to NGI because CLGIN uses context, aswell.

The statistical system (S-MEMM) provides thecontext and the global system(NGI) provides astrong indication that the word is a NE and theperformance of the combined approach(CLGIN)improves significantly.

7 Conclusion and Future Work

We presented an novel approach for Hindi NEIwhich combines the global distributional charac-teristics with local context. Results show that theproposed approach improves performance of NEIsignificantly, especially, when the train and testcorpus belong to different genres. We also pro-posed a new measure for NEI which is based onterm informativeness. The proposed measure per-forms quite competitively with the best known in-formation measure in literature.

Future direction of the work will be to studythe distributional characteristics of individual tagsand move towards classification of identified enti-ties. We also plan to extend the above approachto other Indian languages and other domains. Wealso expect further improvements in accuracy byreplacing the MEMM model by CRF. Currently,we use a tagged corpus as development set to tunethe cut-off threshold in NGI. To overcome this de-pendence and to make the approach unsupervised,a way out can be to find an approximation to theratio of the number of nouns which are NEs to thenumber of nouns and then use this to decide thecut-off threshold.

Acknowledgments

We would like to acknowledge the efforts of Mr.Prabhakar Pandey and Mr. Devendra Kairwan fortagging the data with NE tags.

123

ReferencesEnrique Alfonseca and Suresh Manandhar. 2002. An

Unsupervised Method For General Named EntityRecognition and Automated Concept Discovery. InProceedings of the 1 st International Conference onGeneral WordNet.

Daniel M. Bikel, Richard Schwartz, and Ralph M.Weischedel. 1999. An Algorithm that LearnsWhat’s In A Name.

A. Bookstein and D. R. Swanson. 1974. Probabilis-tic Models for Automatic Indexing. Journal of theAmerican Society for Information Science, 25:312–318.

Andrew Borthwick, John Sterling, Eugene Agichtein,and Ralph Grishman. 1998. Nyu: Description ofthe MENE Named Entity System as used in MUC-7. In In Proceedings of the Seventh Message Under-standing Conference (MUC-7.

Andrew Eliot Borthwick. 1999. A Maximum En-tropy Approach to Named Entity Recognition. Ph.D.thesis, New York, NY, USA. Adviser-Grishman,Ralph.

Kenneth Church and William Gale. 1995. InverseDocument Frequency (IDF): A Measure of Devi-ations from Poisson. In Third Workshop on VeryLarge Corpora, pages 121–130.

Kenneth Ward Church and Patrick Hanks. 1990. WordAssociation Norms, Mutual Information, and Lexi-cography.

Chris Clifton, Robert Cooley, and Jason Rennie. 2002.Topcat: Data mining for Topic Identification in aText Corpus.

Michael Collins and Yoram Singer. 1999. Unsuper-vised Models for Named Entity Classification. InIn Proceedings of the Joint SIGDAT Conference onEmpirical Methods in Natural Language Processingand Very Large Corpora, pages 100–110.

J.R. Firth. 1957. A Synopsis of Linguistic Theory1930-1955. In In Studies in Linguistic Analysis,pages 1–32.

Honglei Guo, Huijia Zhu, Zhili Guo, Xiaoxun Zhang,Xian Wu, and Zhong Su. 2009. Domain Adaptationwith Latent Semantic Association for Named EntityRecognition. In NAACL ’09, pages 281–289, Mor-ristown, NJ, USA. Association for ComputationalLinguistics.

Karen Sprck Jones. 1972. A Statistical Interpretationof Term Specificity and its Application in Retrieval.Journal of Documentation, 28:11–21.

Alexandre Klementiev and Dan Roth. 2006. NamedEntity Transliteration and Discovery from Multi-lingual Comparable Corpora. In Proceedings ofthe main conference on Human Language Technol-ogy Conference of the North American Chapter of

the Association of Computational Linguistics, pages82–88, Morristown, NJ, USA. Association for Com-putational Linguistics.

John D. Lafferty, Andrew McCallum, and FernandoC. N. Pereira. 2001. Conditional Random Fields:Probabilistic Models for Segmenting and LabelingSequence Data. In ICML ’01: Proceedings of theEighteenth International Conference on MachineLearning, pages 282–289, San Francisco, CA, USA.Morgan Kaufmann Publishers Inc.

Wei Li and Andrew McCallum. 2003. Rapid Devel-opment of Hindi Named Entity Recognition usingConditional Random Fields and Feature Induction.ACM Transactions on Asian Language InformationProcessing (TALIP), 2(3):290–294.

Wei Li and Andrew Mccallum. 2005. Semi-supervisedSequence Modeling with Syntactic Topic Models.In AAAI-05, The Twentieth National Conference onArtificial Intelligence.

Marius Paşca. 2007. Organizing and Searching theWorld Wide Web of facts – Step Two: Harnessingthe Wisdom of the Crowds. In WWW ’07: Proceed-ings of the 16th international conference on WorldWide Web, pages 101–110, New York, NY, USA.ACM.

Kishore Papineni. 2001. Why Inverse DocumentFrequency? In NAACL ’01: Second meeting ofthe North American Chapter of the Association forComputational Linguistics on Language technolo-gies 2001, pages 1–8, Morristown, NJ, USA. Asso-ciation for Computational Linguistics.

Thierry Poibeau and Leila Kosseim. 2001. ProperName Extraction from Non-Journalistic Texts. In InComputational Linguistics in the Netherlands, pages144–157.

Jason D. M. Rennie and Tommi Jaakkola. 2005. UsingTerm Informativeness for Named Entity Detection.In SIGIR ’05: Proceedings of the 28th annual inter-national ACM SIGIR conference on Research anddevelopment in information retrieval, pages 353–360, New York, NY, USA. ACM.

Ellen Riloff and Rosie Jones. 1999. Learning Dic-tionaries for Information Extraction by Multi-LevelBootstrapping. In AAAI ’99/IAAI ’99: Proceedingsof the sixteenth national conference on Artificial in-telligence and the eleventh Innovative applicationsof artificial intelligence conference innovative ap-plications of artificial intelligence, pages 474–479,Menlo Park, CA, USA. American Association forArtificial Intelligence.

Sujan Kumar Saha, Sanjay Chatterji, Sandipan Danda-pat, Sudeshna Sarkar, and Pabitra Mitra. 2008a. AHybrid Named Entity Recognition System for Southand South East Asian Languages. In Proceedings ofthe IJCNLP-08 Workshop on Named Entity Recog-nition for South and South East Asian Languages,

124

pages 17–24, Hyderabad, India, January. Asian Fed-eration of Natural Language Processing.

Sujan Kumar Saha, Sudeshna Sarkar, and Pabitra Mi-tra. 2008b. A Hybrid Feature Set Based MaximumEntropy Hindi Named Entity Recognition. In Pro-ceedings of the Third International Joint Conferenceon Natural Language Processing, Kharagpur, India.

Rajeev Sangal, Dipti Sharma, and Anil Singh, editors.2008. Proceedings of the IJCNLP-08 Workshop onNamed Entity Recognition for South and South EastAsian Languages. Asian Federation of Natural Lan-guage Processing, Hyderabad, India, January.

Johanna Volker. 2005. Towards Large-Scale, Open-Domain and Ontology-Based Named Entity Classi-fication. In Proceedings of the International Confer-ence on Recent Advances in Natural Language Pro-cessing (RANLP’05, pages 166–172. INCOMA Ltd.

125