Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
gather annotation of types, events, relations, coref
Lora Aroyo and Chris Welty
From Crowd Knowledge to Machine Knowledge
Text
Text
1Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Position
The human disagreement & vagueness are part of the
event & relations semantics
Flickr: elkabong2Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
PositionArtificially restricting humans does not help machines to learn.
Machines will learn from diversity
Flickr: elkabong3Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Disagreement Framework
• ontology: disagreements on the basic status of events themselves as referents of linguistic utterances, e.g. are people events or do events exist at all.
• granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc.
• interpretation: disagreements that result from (non-granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September.
4Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Disagreement Framework
• ontology: disagreements on the basic status of events themselves as referents of linguistic utterances, e.g. are people events or do events exist at all.
• granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc.
• interpretation: disagreements that result from (non-granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September.
4Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Approach Principles1. tolerate, capture & exploit disagreement
2. understand the disagreement by creating a space of possibilities (frequencies & similarities)3. score the machine output based on where it falls in this space
4. adaptable to new annotation tasks
Flickr: auroille5Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo Lora Aroyo
Event Extractioncrowdsourcing ground truth data
6Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
7Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
7Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Event Participants Disagreement
Prime minister Benjamin
Netanyahu
Benjamin Netanyahu
Israeli Prime minister
Cabinet
Benjamin Netanyahu’s
Cabinet
Israeli Cabinet
his Cabinet
Israeli Government
{TOLD}
50%
35%
15%
10%
15%
5%
45%
15%
8Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Temporal Disagreement
Spring 1998
March 1, 1998
March 1998
SundayPrime minister
Benjamin Netanyahu
Benjamin Netanyahu
Israeli Prime minister
{TOLD}
50%
35%
15%
25%
15%
50%
5%
9Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Spatial Disagreement
Lebanon
IsraelSouthern Lebanon
Israel's Northern Frontier
Middle East
{WILLING TO WITHDRAW}
35%
45%
10%
30%
65%
10Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
it seems to refer to an inference or communicated feeling more than specific event.
does not refer to an event
a group of people did something specific at a specific point in 6me.
refers to an event
the actors in ques6on (top Israeli officials) performed an ac6on during a specified 6me (Sunday).
refers to an event
it refers to what the israelis did on sunday, a specific 6me.
Top Israeli officials SENT strong new SIGNALS Sunday that Israel wants to withdraw from southern Lebanon, ...
11Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Because it is describing a historical issue concerning the resolu6on of 1978
refers to an event
That 1978 resolu6on calls for Israel's uncondi6onal WITHDRAWAL from the self-‐declared security zone it occupies in south Lebanon, ...
it is not a par6cular movement that has or is going on but a request that the country of Israel remove their forces from the zone they occupy.
does not refer to an event
the sentence is speaking of a demand for a withdrawal that had not yet occurred.
does not refer to an event
12Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Relation Extractioncrowdsourcing ground truth data
Lora Aroyo
13Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
6 experiments• 2 professional
annotators per sentence @20sentences
• 10 CFworkers per sentence @20sentences
• 20 CFworkers per sentence @20sentences
• 30 CFworkers per sentence @20sentences
• 30 CFworkers per sentence @10sentences + relations definitions
• 10 CFworkers per sentence @20sentence explanation validation
14Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
The Task
15Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
The Steps: Example Sentence (1)
16Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
The Steps: Example Sentence (2)
17Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food and Drug Administration (FDA) for the treatment of SCHIZOPHRENIA and bipolar disorder.
?OLANZAPINEIs SCHIZOPHRENIA related to
treated_by treats may_treat
?RESPIRATORY ALKALOSIS
Is HYPERVENTILATION related to
cause_of cause symptom_ofmay_causediagnosed_by
RESPIRATORY ALKALOSIS is a medical condition in which increased respiration (HYPERVENTILATION) elevates the blood pH (a condition generally called alkalosis).
18Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
He was the first physician to identify the relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY.
?HEMOPHILIC ARTHROPATHY
Is HEMOPHILIA related to
treated_by may_causeother
cause_of cause
has_manifestation
symptom_of
It just says there is a relation between the two but gives no specifics about what the relation is
There is a relationship between the two disorders but the sentence does not indicate what that relationship is.
identify the relationship between
relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY
19Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Distributions Compared
0"
20"
40"
60"
80"
100"
120"
140"
sTB" sT" sMT" sPB" sP" sMP" sDB" sD" sMD" sCO" sC" sMC" sLO" sHM" sDF" sSS" sOTH" sNO"
10"workers" 20"workers" 30"workers"
20Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Sentence-Relation DistributionProfessional Annotators
21Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Sentence-Relation Distribution10w x 20s
22Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Sentence-Relation Distribution10w x 20s
caused_bycaused_by
caused_by
is differentiate
d from
is_related
is_type_of,
has_type,
x
is_a
includes
is_related to
maybe related to
22Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Sentence-Relation Distribution20w x 20s
caused_bycaused_by
caused_by
is differentiate
d from
is_related
is_type_of,
has_type,
x
is_a
includes
is_related to
maybe related to
23Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Sentence-Relation Distribution30w x 20s
caused_bycaused_by
caused_by
is differentiate
d from
is_related
is_type_of,
has_type,
x
is_a
includes
is_related to
maybe related to
24Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Sentence-Relation Distribution30w x 10s
with relation explanations
25Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
The Task
26Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
The Steps: Example Sentence
27Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
The Dark Side of Crowdsourcing Disagreement
• disagreement is beautiful, except when it results from spamming• crowdsourcing has to account for people that want to get paid for
not doing any work• spammers generate disagreement for the wrong reasons• most spam detection requires gold standard
28Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Phase I:A. Collect event annotations +
motivations
Extraction of Putative Events
input: putative events
Phase I:C. Filtering spam
event annotations
input: output of A
Phase II:A. Collect event types
+ motivations
input: list of events
Phase II:B. Filtering spam
event types
input: output of A
Manual selection of Gold Questions
input: output of A
Manual selection of
Gold Questions
input: output of A
Phase III:A. Collect event
modalities + motivations
input: list of events
Phase III:B. Filtering spam event modalities
input: output of A Manual
selection of Gold Questions
input: output of A
Phase IV:A. Collect event
role fillers + motivations
Phase IV:B. Filtering spam event role fillers
input: output of A Manual
selection of Gold Questions
input: output of A
input: list of events
• a new way of measuring ground truth
• a new set of semantic features for learning in event extraction
29Wednesday, October 17, 12
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Questions?
@laroyohttp://lora-aroyo.org
30Wednesday, October 17, 12
Top Related