Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic...
Transcript of Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic...
![Page 1: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/1.jpg)
Question-Answer Driven Semantic Role Labeling
Using Natural Language to Annotate Natural Language
1
Luheng He, Mike Lewis, Luke Zettlemoyer
EMNLP 2015
University of Washington
![Page 2: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/2.jpg)
Semantic Role Labeling (SRL)
2
who did what to whom, when and where?
![Page 3: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/3.jpg)
Semantic Role Labeling (SRL)
2
Predicate Argument
Role
They increased the rent drastically this year PatentAgent
Manner
Time
![Page 4: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/4.jpg)
Semantic Role Labeling (SRL)
2
Predicate Argument
Role
They increased the rent drastically this year PatentAgent
Manner
Time
• Defining a set of roles can be difficult • Existing formulations have used different sets
![Page 5: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/5.jpg)
Existing SRL Formulations and Their Frame Inventories
3
Frame: Change_position_on_a_scale This frame consists of words that indicate the change of an Item's position on a scale (the Attribute) from a starting point (Initial_value) to an end point (Final_value). The direction (Path) …Lexical Units:…, reach.v, rise.n, rise.v, rocket.v, shift.n, …
Roleset Id: rise.01 , go up
Arg1-: Logical subject, patient, thing rising Arg2-EXT: EXT, amount risenArg3-DIR: start pointArg4-LOC: end pointArgm-LOC: medium
FrameNet1000+ semantic frames,
10,000+ frame elements (roles)
PropBank10,000+ frame files
with predicate-specific roles
Unified Verb Index, University of Colorado http://verbs.colorado.edu/verb-index/ PropBank Annotation Guidelines, Bonial et al., 2010 FrameNet II: Extended theory and practice, Ruppenhofer et al., 2006 FrameNet: https://framenet.icsi.berkeley.edu/
![Page 6: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/6.jpg)
4
• Introduce a new SRL formulation with no frame or role inventory
• Use question-answer pairs to model verbal predicate-argument relations
• Annotated over 3,000 sentences in weeks with non-expert, part-time annotators
• Showed that this data is high-quality and learnable
This Talk: QA-SRL
![Page 7: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/7.jpg)
Our Annotation Scheme
5
They increased the rent this year .
Given sentence and a verb:
![Page 8: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/8.jpg)
Our Annotation Scheme
5
Who increased something ?
They increased the rent this year .
Given sentence and a verb:
Step 1: Ask a question about the verb:
![Page 9: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/9.jpg)
Our Annotation Scheme
5
Who increased something ? They
They increased the rent this year .
Given sentence and a verb:
Step 1: Ask a question about the verb:
Step 2: Answer with words in the sentence:
![Page 10: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/10.jpg)
Our Annotation Scheme
5
Who increased something ? They
They increased the rent this year .
Given sentence and a verb:
Step 1: Ask a question about the verb:
Step 2: Answer with words in the sentence:
Step 3: Repeat, write as many QA pairs as possible …
![Page 11: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/11.jpg)
Our Annotation Scheme
5
Who increased something ? They
What is increased ? the rent
When is something increased ? this year
They increased the rent this year .
Given sentence and a verb:
Step 1: Ask a question about the verb:
Step 2: Answer with words in the sentence:
Step 3: Repeat, write as many QA pairs as possible …
![Page 12: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/12.jpg)
6
ARG1 ARG4
ARG3
ARG2The rent rose 10% from $3000 to $3300
??????
???
amount risen
start point
end point
• Depends on pre-defined frame inventory
• Annotators need to: 1) Identify the Frameset 2) Find arguments in the sentence 3) Assign labels accordingly
• If frame doesn’t exist, create new
The Proposition Bank: An Annotated Corpus of Semantic Roles, Palmer et al., 2005 http://verbs.colorado.edu/propbank/framesets-english/rise-v.html
Frameset: rise.01 , go up
Arg1-: Logical subject, patient, thing rising Arg2-EXT: EXT, amount risen Arg3-DIR: start point Arg4-LOC: end point Argm-LOC: medium
Previous Method: Annotation with Frames
![Page 13: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/13.jpg)
Wh-Question Answer
the rentWhat rose ?
10%
$3000
$3300
How much did something rise ?
What did something rise from ?
What did something rise to ?
ARG1 ARG4
ARG3
ARG2The rent rose 10% from $3000 to $3300
??????
???
amount risen
start point
end point
Our Method: Q/A Pairs for Semantic Relations
![Page 14: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/14.jpg)
8
Predicate Argument (Verbal) Predicate
Answer
RoleQuestion
Question-Answer Driven SRL (QA-SRL)
SRL
Large Role InventoryNo Role Inventory!
Comparing to Existing SRL Formulations
![Page 15: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/15.jpg)
9
• Easily explained • No pre-defined roles, few syntactic assumption • Can capture implicit arguments • Generalizable across domains
Advantages
![Page 16: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/16.jpg)
9
• Easily explained • No pre-defined roles, few syntactic assumption • Can capture implicit arguments • Generalizable across domains
Advantages
Limitations • Only modeling verbs (for now) • Not annotating verb senses directly • Can have multiple equivalent questions
![Page 17: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/17.jpg)
9
• Easily explained • No pre-defined roles, few syntactic assumption • Can capture implicit arguments • Generalizable across domains
Advantages
Limitations • Only modeling verbs (for now) • Not annotating verb senses directly • Can have multiple equivalent questions
Challenges • What questions to ask? • Quality - Can we get good Q/A pairs? • Coverage - Can we get all the Q/A pairs?
![Page 18: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/18.jpg)
Outline
10
Learning Tasks and Baselines
Data Collection and Analysis
Motivation and Intuition
Future Work and Conclusion
• Annotation Task Design • Dataset Statistics • Quality Analysis
• Semantic Role Labeling • Our Method: QA-SRL
![Page 19: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/19.jpg)
Question-Answer Driven SRL
11
Given sentence s, target verb v
Annotate all possible question-answer pairs <q,a>
![Page 20: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/20.jpg)
Question-Answer Driven SRL
11
Given sentence s, target verb v
Annotate all possible question-answer pairs <q,a>
• Question q should start with a wh-word and contain the target verb v
• Answer a should be a phrase from the sentence s. Multiple correct answers are allowed.
![Page 21: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/21.jpg)
Writing Questions
12
q 2 WH⇥AUX⇥ SBJ⇥TRG⇥OBJ1⇥PP⇥OBJ2
![Page 22: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/22.jpg)
Writing Questions
12
q 2 WH⇥AUX⇥ SBJ⇥TRG⇥OBJ1⇥PP⇥OBJ2
WH: Who, What, When, Where, Why, How, How much
![Page 23: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/23.jpg)
Writing Questions
12
q 2 WH⇥AUX⇥ SBJ⇥TRG⇥OBJ1⇥PP⇥OBJ2
WH: Who, What, When, Where, Why, How, How much
AUX: Auxiliary verbs, including negations. i.e. is, might, wo n’t
![Page 24: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/24.jpg)
Writing Questions
12
q 2 WH⇥AUX⇥ SBJ⇥TRG⇥OBJ1⇥PP⇥OBJ2
WH: Who, What, When, Where, Why, How, How much
AUX: Auxiliary verbs, including negations. i.e. is, might, wo n’t
SBJ, OBJ1, OBJ2: someone, something, do something, etc.
![Page 25: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/25.jpg)
Writing Questions
12
q 2 WH⇥AUX⇥ SBJ⇥TRG⇥OBJ1⇥PP⇥OBJ2
WH: Who, What, When, Where, Why, How, How much
AUX: Auxiliary verbs, including negations. i.e. is, might, wo n’t
SBJ, OBJ1, OBJ2: someone, something, do something, etc.
TRG: Target verb, including inflected forms.
![Page 26: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/26.jpg)
Writing Questions
12
q 2 WH⇥AUX⇥ SBJ⇥TRG⇥OBJ1⇥PP⇥OBJ2
WH: Who, What, When, Where, Why, How, How much
AUX: Auxiliary verbs, including negations. i.e. is, might, wo n’t
SBJ, OBJ1, OBJ2: someone, something, do something, etc.
TRG: Target verb, including inflected forms.
PP: Preposition. i.e. to, for, from, about, etc.
![Page 27: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/27.jpg)
WH* AUX SBJ TRG* OBJ1 PP OBJ2
Who built something
What had someone said
When was someone expected to do something
Where might something rise from
Writing Questions
13
![Page 28: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/28.jpg)
14
Annotation Interface
![Page 29: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/29.jpg)
14
Annotation Interface
![Page 30: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/30.jpg)
Dataset Statistics
15
0
2750
5500
8250
11000
Sentences Verbs QA Pairs
newswire (PropBank) Wikipedia
8,109
3,336
1,241
![Page 31: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/31.jpg)
Dataset Statistics
15
0
2750
5500
8250
11000
Sentences Verbs QA Pairs
newswire (PropBank) Wikipedia
10,798
4,440
1,959
8,109
3,336
1,241
![Page 32: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/32.jpg)
Cost and Speed
16
0
0.4
0.8
1.2
1.6
Cost per Verb Cost per Sentence
newswire Wikipedia
$1.57
$0.58
• Part-time freelancers from upwork.com (hourly rate: $10) • ~2h screening process for native English proficiency
![Page 33: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/33.jpg)
Cost and Speed
16
0
0.4
0.8
1.2
1.6
Cost per Verb Cost per Sentence
newswire Wikipedia
$1.01
$0.45
$1.57
$0.58
• Part-time freelancers from upwork.com (hourly rate: $10) • ~2h screening process for native English proficiency
![Page 34: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/34.jpg)
Cost and Speed
16
0
0.4
0.8
1.2
1.6
Cost per Verb Cost per Sentence
newswire Wikipedia
$1.01
$0.45
$1.57
$0.58
2.254.5
6.759
Time per Sentence
6min
9min
• Part-time freelancers from upwork.com (hourly rate: $10) • ~2h screening process for native English proficiency
![Page 35: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/35.jpg)
Sample Annotation
17
QA-SRL PropBank (CoNLL-2009)
Who would play something ? the soft-spoken clarinetist / he ARG0: he
What would be played ? her favorite tune from the record ARG1: tune
When would someone play something? his mother ’s birthday /
Sentence: Clad in his trademark black velvet suit , the soft-spoken clarinetist announced that . . . and that it was his mother ’s birthday , so he was going to play her favorite tune from the record .
![Page 36: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/36.jpg)
Sample Annotation
17
QA-SRL PropBank (CoNLL-2009)
Who would play something ? the soft-spoken clarinetist / he ARG0: he
What would be played ? her favorite tune from the record ARG1: tune
When would someone play something? his mother ’s birthday /
Sentence: Clad in his trademark black velvet suit , the soft-spoken clarinetist announced that . . . and that it was his mother ’s birthday , so he was going to play her favorite tune from the record .
match
precision loss
match
![Page 37: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/37.jpg)
18
0
22.5
45
67.5
90
All Roles Core Roles Adjuncts
Precision Recall
86.381.4
Core Roles: A0-A5 Adjuncts: ADV, CAU,DIR, EXT, LOC, MNR, PNC, PRD, TMP
Agreement with PropBank: Results
![Page 38: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/38.jpg)
18
0
22.5
45
67.5
90
All Roles Core Roles Adjuncts
Precision Recall
89.886.3 85.981.4
Core Roles: A0-A5 Adjuncts: ADV, CAU,DIR, EXT, LOC, MNR, PNC, PRD, TMP
Agreement with PropBank: Results
![Page 39: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/39.jpg)
18
0
22.5
45
67.5
90
All Roles Core Roles Adjuncts
Precision Recall
63.6
89.886.3
59.9
85.981.4
Core Roles: A0-A5 Adjuncts: ADV, CAU,DIR, EXT, LOC, MNR, PNC, PRD, TMP
Agreement with PropBank: Results
![Page 40: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/40.jpg)
Inter-Annotator Agreement
19
• QA Equivalence: Same wh-word + Overlapping answers • Agreed QA Pairs: Proposed by at least 2 of the 5 annotators
![Page 41: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/41.jpg)
Inter-Annotator Agreement
19
• QA Equivalence: Same wh-word + Overlapping answers • Agreed QA Pairs: Proposed by at least 2 of the 5 annotators• Agreed QA pairs by five annotators: 2.6-2.8 QA/verb• One annotator can recover: 2.2-2.3 QA/verb (80%)
![Page 42: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/42.jpg)
Wh-words vs. PropBank Roles
20
Who What When Where Why How HowMuch
ARG0 1575 414 3 5 17 28 2ARG1 285 2481 4 25 20 23 95ARG2 85 364 2 49 17 51 74ARG3 11 62 7 8 4 16 31ARG4 2 30 5 11 2 4 30ARG5 0 0 0 1 0 2 0
AM-ADV 5 44 9 2 25 27 6AM-CAU 0 3 1 0 23 1 0AM-DIR 0 6 1 13 0 4 0AM-EXT 0 4 0 0 0 5 5AM-LOC 1 35 10 89 0 13 11AM-MNR 5 47 2 8 4 108 14AM-PNC 2 21 0 1 39 7 2AM-PRD 1 1 0 0 0 1 0AM-TMP 2 51 341 2 11 20 10
![Page 43: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/43.jpg)
Outline
21
Learning Tasks and Baselines
Data Collection and Analysis
Motivation and Intuition
Future Work and Conclusion
• Question Generation • Answer Identification
• Annotation Task Design • Dataset Statistics • Quality Analysis
• Semantic Role Labeling • Our Method: QA-SRL
![Page 44: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/44.jpg)
Question Generation
22
Given sentence s and target verb v, predict a set of questions that are grammatical and answerable.
Task
In the future, automate part of the annotation process, further reduce cost and speed up annotation.
Motivation
![Page 45: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/45.jpg)
Question Generation: Basic Idea
23
They increased the rent this year .
• Pick a role in the sentence • Predict the right pronoun. • Fill in the rest of the question.
s =
![Page 46: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/46.jpg)
Question Generation: Basic Idea
23
They increased the rent this year .
Who increased something ?✓role not presentWhy was something increased ?✗
• Pick a role in the sentence • Predict the right pronoun. • Fill in the rest of the question.
s =
![Page 47: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/47.jpg)
Question Generation: Basic Idea
23
They increased the rent this year .
Who increased something ?✓role not presentWhy was something increased ?✗
wrong pronounWhat increased someone ?✗
• Pick a role in the sentence • Predict the right pronoun. • Fill in the rest of the question.
s =
![Page 48: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/48.jpg)
Question Generation: Basic Idea
23
They increased the rent this year .
Who increased something ?✓role not presentWhy was something increased ?✗
wrong pronounWhat increased someone ?✗
• Pick a role in the sentence • Predict the right pronoun. • Fill in the rest of the question.
wrong templateWhen increased someone something ?✗
s =
![Page 49: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/49.jpg)
Question Generation: 2-Step Method
24
Step 1: Role/Pronoun Prediction as Multi-label Learning
L ={role:pronoun val | role 2 R}
R ={R0,R1,R2,R2[pp], wh,wh[pp]}wh 2{Where,When,Why,How,HowMuch}
*Details can be found in paper
![Page 50: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/50.jpg)
Question Generation: 2-Step Method
24
Step 1: Role/Pronoun Prediction as Multi-label Learning
Step 2: Template-based Generation with Abstract Questions
L ={role:pronoun val | role 2 R}
R ={R0,R1,R2,R2[pp], wh,wh[pp]}wh 2{Where,When,Why,How,HowMuch}
Who increased something ?WH SBJ Voice OBJ1 OBJ2R0 / active R1 /
R0:someone R1:something
*Details can be found in paper
![Page 51: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/51.jpg)
Question Generation: Results
25
0%
22.5%
45%
67.5%
90%
prec@1 prec@3 prec@5 prec@1 prec@3 prec@5
Grammatical Answerable
newswire Wikipedia
![Page 52: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/52.jpg)
Question Generation: Results
25
0%
22.5%
45%
67.5%
90%
prec@1 prec@3 prec@5 prec@1 prec@3 prec@5
Grammatical Answerable
82%86%90%
77.2%78.7%84%
newswire Wikipedia
![Page 53: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/53.jpg)
Question Generation: Results
25
0%
22.5%
45%
67.5%
90%
prec@1 prec@3 prec@5 prec@1 prec@3 prec@5
Grammatical Answerable
40%
53.3%
72%
38.4%
51.3%
66%
82%86%90%
77.2%78.7%84%
newswire Wikipedia
![Page 54: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/54.jpg)
Question Generation: Results
25
0%
22.5%
45%
67.5%
90%
prec@1 prec@3 prec@5 prec@1 prec@3 prec@5
Grammatical Answerable
40%
53.3%
72%
38.4%
51.3%
66%
82%86%90%
77.2%78.7%84%
newswire Wikipedia• In question prediction: 2 Question/verb answerable • In annotated data: 2.6-2.8 QA/verb
![Page 55: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/55.jpg)
Answer Identification
26
Given sentence s, target verb v, and question q, predict a word in the sentence that answers the question q.
Task
In the future, build an end-to-end SRL system trained by QA-SRL data. (Analogy to SRL - questions:roles, answers:arguments).
Motivation
![Page 56: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/56.jpg)
Answer Identification: Basic Idea
27
Who would increase something ?
The leasing office said they would increase the rent .
Arcs from k-best dependency trees
s =
v = q =increase
Annotated answer spans. Space: 2|s|
![Page 57: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/57.jpg)
Answer Identification: Basic Idea
27
Who would increase something ?
The leasing office said they would increase the rent .
Arcs from k-best dependency trees
s =
v = q =increase
Annotated answer spans. Space: 2|s|
Training samples: hs, v, q, o�cei, hs, v, q, theyi Space: |s|
![Page 58: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/58.jpg)
Answer Identification: Basic Idea
27
Who would increase something ?
The leasing office said they would increase the rent .
Arcs from k-best dependency trees
Ex. of correct predictions: “office”, “leasing”, “they”
Ex. of wrong predictions: “rent”✓✗
s =
v = q =increase
Annotated answer spans. Space: 2|s|
Training samples: hs, v, q, o�cei, hs, v, q, theyi Space: |s|
![Page 59: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/59.jpg)
Answer Identification: Results
28
0
22.5
45
67.5
90
newswire Wikipedia
Random Classifier
26.926.3
![Page 60: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/60.jpg)
Answer Identification: Results
28
0
22.5
45
67.5
90
newswire Wikipedia
Random Classifier
82.378.7
26.926.3
![Page 61: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/61.jpg)
Outline
29
Learning Tasks and Baselines
Data Collection and Analysis
Motivation and Intuition
Future Work and Conclusion• Generalization • Question Suggestion • Training a Joint Parser
• Question Generation • Answer Identification
• Annotation Task Design • Dataset Statistics • Quality Analysis
• Semantic Role Labeling • Our Method: QA-SRL
![Page 62: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/62.jpg)
Future Work: Generalization
30
• Generalize to non-verbal predicates:
• Generalize to other languages:
他们 今年 涨了 房租 。
Q: 房租 什么 时候 涨了 ?they this year increased the rent
rent when increasedA: 今年
this year
S: The rent increase came as a shock to us .
Q: Who was shocked ? A: us
![Page 63: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/63.jpg)
Future Work: Automatic Question Suggestion
31
Automatic Question Generator
Annotated QA Pairs
“I can ’t believe they increased the rent by so much .”Given new sentence and verb:
training
![Page 64: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/64.jpg)
Future Work: Automatic Question Suggestion
31
Automatic Question Generator
Annotated QA Pairs
“I can ’t believe they increased the rent by so much .”Given new sentence and verb:
training Who increased something ? What increased ? How did something increase ? When did someone increase ?
suggest
![Page 65: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/65.jpg)
Future Work: Automatic Question Suggestion
31
Automatic Question Generator
Annotated QA Pairs
“I can ’t believe they increased the rent by so much .”Given new sentence and verb:
training
Human Annotators
write answers
Who increased something ? What increased ? How did something increase ? When did someone increase ?
suggest
![Page 66: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/66.jpg)
Future Work: Automatic Question Suggestion
31
Automatic Question Generator
Annotated QA Pairs
“I can ’t believe they increased the rent by so much .”Given new sentence and verb:
training
Who increased something? - they What increased? - the rent How did something increase? - by so muchWhen did someone increase? - unanswerable
Human Annotators
write answers
Who increased something ? What increased ? How did something increase ? When did someone increase ?
suggest
![Page 67: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/67.jpg)
Future Work: Training a Joint Parser
32
Joint A* CCG Parsing and Semantic Role Labeling, Lewis et al., EMNLP-2015. (Presentation: Sunday 6B)
• Use question-answer pairs to train a joint parser, to improve on both syntax and semantics
• Combine with other SRL data, i.e. PropBank, FrameNet
![Page 68: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/68.jpg)
Contributions
33
• Introduced question-answer driven semantic role labeling (QA-SRL).
• High quality QA annotation with a lightweight template-based scheme.
• Two new QA-SRL learning baselines: question generation and answer identification.
• Releasing data and annotation tool - https://dada.cs.washington.edu/qasrl/
![Page 69: Question-Answer Driven Semantic Role Labelingluheng.github.io/files/qasrl_slides.pdf · Semantic Role Labeling (SRL) 2 Predicate Argument Role They increased the rent drastically](https://reader034.fdocuments.us/reader034/viewer/2022051810/6016e8171ef0a730b6017be0/html5/thumbnails/69.jpg)
34
Thank You! Questions?
QA-SRL Project Page: https://dada.cs.washington.edu/qasrl/
Predicate Answer
Wh-QuestionNo Frame Inventory!