Finding homogenious word sets Towards a dissertation in NLP
description
Transcript of Finding homogenious word sets Towards a dissertation in NLP
Finding homogenious word setsTowards a dissertation in NLP
Chris Biemann
NLP Department, University of Leipzig
Universitetet i Oslo, 12/10/2005
2
Outline
• Preliminaries: Co-occurrences
• Unsupervized methods- Language Seperation- POS tagging
• Weakly Supervized Methods- gazetteer building for NER- semantic lexicon extension- extension of lexical-semantic word nets
3
Statistical Co-occurrences• occurrence of two or more words within a well-defined
unit of information (sentence, nearest neighbors, document, window ...)
• Significant Co-occurrences reflect relations between words
• Significance Measure (log-likelihood):- k is the number of sentences containing a and b together- ab is (number of sentences with a)*(number of sentences with b)- n is total number of sentences in corpus
( , ) log log !
with number of sentences,
.
sig A B x k x k
n
abx
n
4
Unsupervized methods• Unsupervized means: no training data, there is nothing like a „training
set“• This means: the discovery and usage of any structure in language must
be entirely algorithmical• Unsupervized means knowledge-free: No prior knowledge allowed.• Famous unsupervized method: clustering.
Advantages:• language-independent• no need to build manual ressources (cheap)• Robust
Disadvantages:• Labeling problem• Unaware of errors• Often not traceable• difficult to interpret / evaluate
5
Unsupervized Language DiscriminationSupervized Language Identification: • needs training • Operates on letter n-grams or common words as features• Works almost error-free for texts from 500 letters on
Drawbacks:• Does not work for previously unknown languages• Danger of misclassifying instead of reporting „unknown“
Example: http://odur.let.rug.nl/~vannoord/TextCat/Demo • “xx xxx x xxx …” classified as Nepali• “öö ö öö ööö …” classified as Persian
Unsupervized Language Discrimination: Task: Given a mixed-language corpus, split it into the different languages.
Biemann, C., Teresniak, S. (2005): Disentangling from Babylonian Confusion - Unsupervized Language Identification, Proceedings of CICLing-2005, Computational Linguistics and Intelligent Text Processing, Mexico City, Mexico and Springer LNCS 3406, pp. 762-773
6
Co-occurrence Graphs • The entirety of all
significant co-occurrences is a co-occurrence graph G(V,E) withV: Vertices = WordsE: Edges (v1, v2, s) with v1, v2 words, s significance value.
• Co-occurrence graph is- weighted- undirected
• Small-world-property
7
Chinese Whispers - Motivation
• (small-world) graphs consist of regions with a high clustering coefficient and hubs that connect those regions
• The nodes in cluster regions should be assigned the same label per region
• Every node gets a label and whispers it to its neighbouring nodes. A node changes to a label if most of its neighbours whisper this label – or it invents a new one
• Under assumption of semantic closeness when being strongly connected there should emerge motivated clusters
8
Chinese Whispers AlgorithmAssign different labels to every node in the graph;
For iteration i from 1 to total_iterations {
mutation_rate= 1/(i^2);
For each word w in the graph {
new_label of w = highest ranked label in neighbourhood of w;
with probability mutation_rate: new_label of w = new class label;
}
labels = new_labels;
}
• graph clustering algorithm• linear time in the number of nodes • random mutation can be omitted but showed better results
for small graphs
AL1
DL2
EL3
BL4
CL3
58
63
deg=1deg=2
deg=3deg=5
deg=4
9
Chinese Whispers on 7 Languages
10
Chinese Whispers on 7 languages
11
Assigning languages to sentences
• Use word-based language identification tool• Largest clusters form word lists for different languages• A sentence is assigned a cluster label if
- it contains at least 2 words from the cluster and - not more words from another cluster
Questions for Evaluation:• up to what number of languages is that possible ?• How much can the corpus be biased ?
12
Evaluation
Mix of seven languages, equal number of sentences:• Languages used: Dutch, Estonian, English, French, German, Icelandic and
Italian • At least 100 sentences per language are necessary for consistent clustersTwo languages with strong bias:• At least 500 sentences out of 100‘000 needed to find the smaller language• Tested on English in Estonian, Dutch in German, French in Italian
13
Common mistakes
• Unclassified: - mostly enumerations of sport teams - very short sentences, e.g. headlines- legal act ciphers in estonian case, e.g. 10.12.96 jõust.01.01.97 - RT I 1996 , 89 , 1590
• Misclassified: mixed-language-sentences, likeFrench: Frönsku orðin "cinéma vérité" þýða "kvikmyndasannleikur“
English: Die Beatles mit "All you need is love".
14
Induction of POS InformationGiven:
Unstructured monolingual text corpus
Goal: Induction of POS Tags for many (all) words.Result is a list of words with the corresponding tag. Application on text (the actual POS tagging) is another task.
Motivation: • POS information is a processing step in a variety of NLP applications
such as parsing, IE, indexing• POS taggers need a considerable amount of hand-tagged training data
which is expensive and only available for major languages• Even for major languages, POS taggers are suited for well-formed
texts and do not cope well with domain-dependent issues as being found e.g. in eMail or spoken corpora
15
Literature Overview[Schütze 93, Schütze 95, Clark 00, Freitag 04] show a similar
architecture on high level, but differ in details.
Steps to achieve word classes:
1. Calculation of global contexts using a window of 1-2 words to left and right and the most frequent 150-250 words as features
2. Clustering of these contexts gives word classes
Paper Finch & Chater 92
Schütze 93
Schütze 95
Clark 00 Freitag 04
Context 4 x 150 4 x 5000, by SVD: 15
2 x 250by SVD: 50
2 x classes 2 x 5000
Similarity Mutual Inf. and 4 x 500by SVD: 50
Cosine KL divergence
Mutual Inf.
Clustering 1-Nearest Neighbour (?) Buckshot
500 classesBuckshot, 200 classes
Iterative,77,100,150 c‘s
Co-clustering
200 classes
16
Method Description• Contexts: the most frequent N (100, 200, 300) words are used for 4 x
N context vectors for the most frequent 10‘000 words in the corpus• Cosine similarity between all pairs of the 10‘000 top words is
calculated• Transformation to a graph: Draw an edge with weight
1/ (1-cos(x,y)) between x and y, if cos(x,y) is above some threshold • Chinese Whispers (CW) on graph results in word class clusters
Differences to prev. methods:• CW Clustering does not need number of classes as input• No dimensionality reduction techniques as SVD• Explicit threshold for similarity
17
Toy Example (1)
Corpus fragments:
... _KOM_ sagte der Sprecher bei der Sitzung _ESENT_
... _KOM_ rief der Vorsitzende in der Sitzung _ESENT_
... _KOM_ warf in die Tasche aus der Ecke _ESENT_
Features: der(1), die(2), bei(3), in(4), _ESENT_(5), _KOM_(6)
Word 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6
sagte 1 1
rief 1 1
warf 1 1 1
Sprecher 1 1 1
Vorsitzende 1 1 1
Tasche 1 1 1
Sitzung 1 1 2 2
Ecke 1 1
Position: -2 -1 +1 +2
18
Toy Example (2)Cos(x,y) sagte rief warf Sprecher Vorsitzende Tasche Sitzung Ecke
sagte 1
rief 1 1
warf 0.4082 0.4082 1
Sprecher 0 0 0 1
Vorsitzende 0 0 0.3333 0.6666 1
Tasche 0 0 0 0.4082 0.3333 1
Sitzung 0 0 0 0.3333 0.3333 0.1666 1
Ecke 0 0 0 0.4082 0.4082 0 0.6666 1
Here, CW cuts graph in 2 partitions: nouns and verbs.
1000
15
15
30
30
17
17
171717
151512
19
Norwegian – Labels
20
corpus size and features: CP vs. coverage
cluster purity vs. coverage for different corpus sizes and features
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
0,5 0,55 0,6 0,65 0,7 0,75 0,8 0,85 0,9 0,95 1
cluster purity
cove
rage
de010K_100
de010K_200
de010K_300
de050K_100
de050K_200
de050K_300
de100K_100
de100K_200
de100K_300
de1M_200
de10M_200
de10_it0_85
21
Example: time words in Norwegian
22
Cluster sizes and clusters per word class
• When optimizing CP, words of the same word class tend to end up in several clusters, especially for open word classes
• Open word classes are the most interesting word classes for further processing steps like IE, relation learning..
• Cluster sizes are Zipf-distributed, there are always many small clusters
• Hierarchical CW could be used to lower the number of clusters while staying in POS distinctions
23
Outlook: Constructing a POS tagger
• Using word clusters to initialize a POS tagger
• Evaluation based on types instead of tokens
Open questions:• Context window backoff model for unknown words• Leave out or take in unclustered high frequency words (as
singletons) ?• Can the many classes per POS be unified using tagger
behaviour?
24
Weakly Supervized Methods
Weakly supervized means:• Very little training data and prior knowledge• Learning from labeled and unlabeled data• bootstrapping methods
Advantages:• Very little input: still cheap• No labeling problem• Easier to evaluate
Disadvantages:• Subject to error propagation• Stopping criterion difficult to define
25
Bootstrapping of lexical items
For learning by bootstrapping, two things are needed: A start set of some known items with classes and a rule set that states, how more information can be obtained using known items.
Generic bootstrapping algorithm: Knowledge=0New=Start_setWhile New>0
Knowledge+=NewNew=0New=find new items using Knowledge and Rule_set
known items
new items
Phase of exhaustionPhase of growth
iteration
# items
26
Benefits and Bothers of Bootstrapping
Pro:• Only small start sets (seeds) are needed, those can be
rapidly prepared• Process needs no further supervision (weakly supervized
learning)
Cons:• Danger of Error Propagation• When to stop is unclear
27
Patterns for word classes and their relations
Examples for word classes in text:• Islands: „On the island of Cuba ...“, „carribbean island of
Trinidad“• Companies: „the ACME Ltd. Incorporated“• Verbs of utterance: „she said: <something>“• Person names: John W. Smith, Ellen Meyer
Observation: • Words belonging to the same class can be interchanged
without hurting the relation• Sometimes no trigger words
28
Problem definition
Be Ri: A1 ... An n-ary relations over word sets A1..An.
Given:
• Some elements of sets A1..An
• Large corpus
Needed:
• Sets A1..An
• (a1..an) Ri
Necessary: rules for classification
29
Pattern Matching Rules
• Annotate Text with known items and flat features (tagging is nice, but Tagsets of 4 tags will do for English)" ... said Jonas Berger , who .. "
... LC UC LN PM LC ..
• Use rules likeUC* LN -> FN FN UC* -> LNto classify "Jonas" as first name
• Rules of this kind are weak hypotheses because they sometimes misclassify, e.g. in
“As Berger turned over, ...““... tickets at Agency Berger, Munich."
Rules alone are not sufficient.
30
Pendulum-Algorithm: Bootstrapping with verification
Initialize Knowledge, Rules, New_items
While New_items>0:
Last_new_items=New_items New_items=0
for all Last_new_items i
fetch text containing i from corpus
find candidates in text by using Knowledge and Rules
verify candidate k:
fetch text containing k
rate k on basis of text
New_items+=candidates with high ratings Knowledge+=New_items
Search step
Verification step
Quasthoff, U.; Biemann, Chr.; Wolff, Chr. : Named entity learning and verification: EM in large corpora. In: Proceedings of CoNLL-2002 , The Sixth Workshop on Computational Language Learning, 31 August and 1 September 2002 in association with Coling 2002 in Taipei, Taiwan
Biemann, Chr.; Böhm, K.; Quasthoff; U.; Wolff, Chr.: Automatic Discovery and Aggregation of Compound Names for the use in Knowledge Representations. Proc: I-KNOW ’03, International Conference on Knowledge Management, Graz and Journal of Universal Computer Science (JUCS), Volume 9, Number 6, Pp. 530-541, Juni 2003
31
Explanations on the Pendulum
• The same rules are used for both search and verification of candidates
• Previously known and previously learned items are used for both search and verification of candidates
• A word is tonly taken into knowledge, if it occurs– multiple times and– at high rate
in the corpus with its classification.
32
Example: island names and island specifiers
33
Results – German Person NamesStart Set and prior knowledge:
9 first names, 10 last names, 15 rules, 12 reg-exps for titles
Corpus: Projekt Deutscher Wortschatz, 36 Mio. Sentences
Found: 42000 items, of which74% LNPrec>99%, 15% FN Prec>80% 11% TIT Prec>99%
Items per iteration step
step
New items Total items
34
Extending a semantic lexicon using co-occurrences and HaGenLex
Size for nouns: about 13 000.
50 semantic classes for nouns are constructed from allowed combinations of:
• 16 semantic features (binary), e.g. HUMAN+, ARTIFICIAL- • 17 ontologic sorts, e.g. concrete, abstract-situation...
sort (hierarchy)
semantic featuressemantic classes
WORD SEMANTIC CLASSAggressivität nonment-dyn-abs-situationAgonie nonment-stat-abs-situationAgrarprodukt nat-discreteÄgypter human-objectAhn human-objectAhndung nonment-dyn-abs-situationÄhnlichkeit relationAirbag nonax-mov-art-discreteAirbus mov-nonanimate-con-potagAirport art-con-geogrAjatollah human-objectAkademiker human-objectAkademisierung nonment-dyn-abs-situation... ...
35
Underlying Assumptions
• Harris 1968: Distributional Hypothesissemantic similarity is a function over global contexts of words. The more similar the contexts, the more similar the words
• Projected on nouns and adjectives: nouns of similar semantic classes are modified through similar adjectives
36
Neighbouring Co-occurrences and Profiles
• Neighbouring co-occurrence: a pair of words that occur next to each other more often than to be expected under assumption of statistical independence.
• The neighbouring co-occurrence relation between adjectives as left neighbours and nouns as right neighbours approximates typical head-modifier structures
• The set of adjectives that co-occur significantly often to the left of a noun is called ist adjective profile (analogous definition of noun profile for adjectives)
• For experiments, I used the most recent German corpus of Projekt Deutscher Wortschatz, 500 million tokens
37
Example: neighbouring profiles
amount: 160‘000 nouns, 23‘400 adjectives
word adjective / noun profile
Buch neu, erschienen, erst, neuest, jüngst, gut, geschrieben, letzt, zweit, vorliegend, gleichnamig herausgegeben, nächst, dick, veröffentlicht, ...
Käse gerieben, überbacken, kleinkariert, fett, französisch, fettarm, löchrig, holländisch, handgemacht, grün, würzig, selbstgemacht, produziert, schimmelig,
Camembert gebacken, fettarm, reif
überbacken Schweinesteak, Aubergine, Blumenkohl, Käse
erlegt Tier, Wild, Reh, Stück, Beute, Großwild, Wildkatzen, Büffel, Rehbock, Beutetier, Wal, Hirsch, Hase, Grizzly, Wildschwein, Thier, Eber, Bär, Mücke,
ganz Leben, Bündel, Stück, Volk, Wesen, Vermögen, Herz, Heer, Arsenal, Dorf, Land, Können, Berufsleben, Paket, Kapitel, Stadtviertel, Rudel, Jahrzehnt, ...
Word transl. adjektive / noun profile translations
book new, published, first, newest, most recent, recently, good, written, last, second, onhand, eponymous, next, thick, ...
cheese grated, baked over, small minded, fat, French, low-fat, holey, Dutch, hand-made, green, spicey, self-made, produced, moldy
camembert baken, low-fat, ripe
baked over steak, aubergine, cauliflower, cheese
brought down animal, game, deer, piece, prey, big game, wild cat, buffalo, roebuck, prey animal, whale, hart, bunny, grizzly, wild pig, boar, bear, ...
whole life, bundle, piece, population, kind, fortune, heart, army, anrsenal, village, country, ability, career, packet, chapter, quater, pack, decade ...
38
Mechanism of Inheritance
Algorithm:Initialize adjective and noun profiles;Initialize the start set;As long as new nouns get classified {
calculate class probabilities for each adjective;for all yet unclassified nouns n {
Multiply class probabilities per class of modifying adjectives; Assign the class with highest probabilities to n;
} }
Which class is assigned to N4 in the next step?
Class probabilities per adjective:• count number of classes• normalize on total number of class wrt. noun classes• normalize to 1
39
Experimental DataDistribution of semantic classes (total: 6045)
nonment-dyn-abs-situationhuman-objectprot-theor-concept
nonoper-attributeax-mov-art-discretenonment-stat-abs-situationanimal-object
nonmov-art-discretement-stat-abs-situationnonax-mov-art-discretetem-abstractum
mov-nonanimate-con-potagart-con-geograbs-infoart-substance
nat-discretenat-substanceprot-discretenat-con-geogr
prot-substancemov-art-discretemeas-unitoper-attribute
institutionment-dyn-abs-situationplant-objectmov-nat-discretecon-info
con-geogrcon-objectanimate-objectprot-method
dyn-abs-situationobjectnonmov-nonanimate-con-potagabs-geogr
stat-abs-situationmodalityrelationcon-potag
prot-con-objectnonmov-nat-discretenoninstit-abs-potagthc-relation
nonanimate-con-potagabs-situationabs-potag
• 5133 nouns comply to minAdj=5, that means maximal recall=84.9%• In all experiments, 10-fold-cross validation was used
40
Results: Global Classification
• Classification was carried out directly on 50 semantic classes• Different measuring points correspond to parameters minAdj in
{5,10,15,20}, maxClass in {2, 5, 50}• Results too poor for lexicon extension
Precision/Recall for global classifier
00,10,20,30,40,50,60,70,80,9
1
0 0,2 0,4 0,6 0,8 1
Precision
Re
ca
ll
41
Combining Single ClassifiersArchitecture: binary classifiers for single features, then
combinding the outcome. Parameter: minAdj=5, maxClass=2
ANIMAL +/-ANIMATE +/-ARTIF +/-AXIAL +/-... (16 features)
... (17 sorts)
ab +/-abs +/-ad +/-as +/-
Selection:compatible
semantic classes that are minimal
w.r.t hierarchy and unambiguous.
result classor
reject
42
Results: Single Semantic Features
• for bias > 0.05 good to excellent precision• total precision: 93.8% (86.8% for feature +)• total recall: 70.7% (69.2% for feature +)
43
Results: Ontologic Sorts
• for bias > 0.10 good to excellent precision• total precision: 94.1% (89.5% for sort +)• total recall: 73.6% (69.6% for sort +)
44
Results: Comb. Semantic Classes
• no connection between amount of class and results visible• total precision: 82.3%• total recall: 32.8% • number of newly classified nouns: 8500 (minAdj=2: ~ 13‘000)
45
Typical mistakesPflanze (plant) animal-object instead of plant-objectzart, fleischfressend, fressend, verändert, genmanipuliert, transgen, exotisch, selten, giftig, stinkend,
wachsend...
Nachwuchs (offspring) human-object instead of animal-objectwissenschaftlich, qualifiziert, akademisch, eigen, talentiert, weiblich, hoffnungsvoll, geeignet, begabt,
journalistisch...
Café (café) art-con-geogr instead of nonmov-art-discrete (cf. Restaurant)Wiener, klein, türkisch, kurdisch, romanisch, cyber, philosophisch, besucht, traditionsreich, schnieke,
gutbesucht, ...
Neger (negro) animal-object instead of human-objectweiß, dreckig, gefangen, faul, alt, schwarz, nackt, lieb, gut, brav
but:
Skinhead (skinhead) human-object (ok){16,17,18,19,20,21,22,23,30}ährig, gleichaltrig, zusammengeprügelt, rechtsradikal, brutal
In most cases the wrong class is semantically close. Evaluation metrics did not account for that.
Biemann, C., Osswald, R. (2005): Automatic Extension of Feature-based Semantic Lexicons via Contextual Attributes, Proceedings of 29th annual meeting of Gfkl, Magdeburg 2005
46
Extending CoreNet – Korean WordNetCoreNet Characteristics• Rather large groups of words per concept as opposed to fine-grained WordNet structure• Same concept hierarchy is used for all word classes
Size of KAIST Korean corpus: • 38 Million tokens, • 2.3 Million sentences, • 3.8 Million types
Word class Lemmas Senses
NOUN 28,823 56,523
VERB 1,757 4,717
ADJECTIVE 804 1,392
47
Pendulum-Algorithm on co-occurrences
LastLearned=StartSet;
Knowledge=StartSet;
NewLearned=0;
while (LastLearned>0) {
for all i in LastLearned {
Candidates=getCooccurrences(i);
for all c in Candidates {
VerifySet=getCooccurrences(c);
if |VerifySet Knowledge| >threshhold {
NewLearned+=c;
Knowledge+=c;
}
}
}
LastLearned=NewLearned;
NewLearned=0;
}
Search step
Verification step
48
Sample step
Seed:
Search with yields (amongst others):
Verifiy:
49
Evaluation
• Selection of concepts performed by a non-Korean speaker
• Evaluation performed manually, only new words counted
• Heuristics for avoiding result set infection- iteratively lower threshold for verification from 8 downto 3 until the result set is too large- take lowest threshold for result set with reasonable size (not exceeding start set)
• Typical run needed 3-7 iterations to converge
Biemann, C., Shin, S.-I., Choi, K.-S. (2004): "Semiautomatic Externsion of CoreNet using a Bootstrapping Mechanism on Corpus-based Co-occurrences", Proceedings of the 20th International Conference on Computational Linguistics (COLING04) Genf, Switzerland
50
Results
Not enough for automatic extension, but a good source for candidates
CoreNet ID Name of Concept Size # new # ok precision
50 human good/bad 119 36 5 13.89%111 human relation 274 3 2 66.67%113 partner / co-worker 123 23 8 34.78%114 partner / member 71 5 3 60.00%181 human ability 213 7 2 28.57%430 store 128 12 11 91.67%471 land, area 260 10 2 20.00%548 insect, bug 75 43 6 13.95%552 part of animal 736 10 6 60.00%553 head 139 7 4 57.14%577 forehead 72 4 2 50.00%590 legs and arms 86 7 3 42.86%672 plant (vegetation) 461 30 15 50.00%817 cloths 246
343934
2311887
52.94%37.67% Sum:
51
Problems... ...and possible solutions
• „Coverage is low“- increase corpus size for relevant domains- make use of other features, e.g. patterns
• „Precision is not satisfactionary“- obtain multiple concepts simultaneously- meta-level bootstrapping- make use of other features, e.g. POS tags for word class information
This work gives a baseline of what is reachable without employing language-dependent features
52
From Text to Ontologies
Text Text TextText
sort by language
lang. 1 ...lang. 2 lang. n
assign word classes
text with POS labels
Determine patterns
and extract word pairs
assign semantic properties for words
typed relations and instances
53
Questions?
THANK YOU!
54
55
56
57
Abstract
Methods are introduced that find sets of words that have something in common in some way by corpus analysis. Having the objective of vastly automizing the task and putting the knowledge in algorithms instead of training sets, two kinds of methods can be distinguished: completely unsupervized methods (clustering) and weakly supervized methods (bootstrapping).
Two unsupervized variants for standard preprocessing steps will be discussed, namely language identification and part-of-speech tagging. In both, a novel, efficient graph clustering algorithm is employed.
After a general introduction to bootstrapping, which needs only a minimal training set, three bootstrapping experiments will be described: Gazetteer construction for Named Entity Recognition, extension of a semantic lexicon and expansion of a lexical-semantic word net.
Follow-ups on the latter two can give rise to automatic ontology creation and extension.