CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been...
Transcript of CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been...
![Page 1: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/1.jpg)
CoQA: A Conversational Question Answering Challenge
1
Danqi Chen* Christopher D. ManningSiva Reddy* =
![Page 2: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/2.jpg)
2
Question Answering
Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.
Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…
![Page 3: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/3.jpg)
2
Question Answering
Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.
Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…
Q: Who had malaria? A: Didier Drogba
![Page 4: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/4.jpg)
2
Question Answering
Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.
Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…
Q: Who had malaria? A: Didier Drogba
Q: Where is Didier from? A: Ivory Coast
![Page 5: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/5.jpg)
3
………………………………..…………
![Page 6: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/6.jpg)
4
Conversational Question Answering
Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.
Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…
![Page 7: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/7.jpg)
4
Conversational Question Answering
Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.
Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…
Q: When?
![Page 8: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/8.jpg)
4
Conversational Question Answering
Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.
Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…
Q: When?
![Page 9: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/9.jpg)
4
Conversational Question Answering
Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.
Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…
Q: Against who?
Q: When?
![Page 10: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/10.jpg)
4
Conversational Question Answering
Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.
Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…
Q: Against who?
Q: When?
![Page 11: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/11.jpg)
4
Conversational Question Answering
Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.
Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…
Q: Is he going to play with malaria?
Q: Against who?
Q: When?
![Page 12: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/12.jpg)
4
Conversational Question Answering
Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.
Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…
Q: Is he going to play with malaria?
Q: Against who?
Q: When?
A: Yes
A: Fulham
A: Wednesday
![Page 13: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/13.jpg)
5
• 127,000 questions and answers in English
• 8,000 conversations (15 turns on average)
• Collected from 7 genres: Wikipedia, News, Literature, Children's stories, English exams, Reddit, Science articles
![Page 14: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/14.jpg)
6
CoQA: Three Main Goals
![Page 15: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/15.jpg)
7
CoQA goal #1: conversational questions
The Virginia governor’s race is shaping up to be a foregone conclusion. Democrat Terry McAuliffe, the longtime political fixer and moneyman, hasn’t trailed in a poll since May…
Q: What are the candidates running for? A: Governor
![Page 16: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/16.jpg)
7
CoQA goal #1: conversational questions
The Virginia governor’s race is shaping up to be a foregone conclusion. Democrat Terry McAuliffe, the longtime political fixer and moneyman, hasn’t trailed in a poll since May…
Q: What are the candidates running for? A: GovernorQ: Where?
![Page 17: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/17.jpg)
7
CoQA goal #1: conversational questions
The Virginia governor’s race is shaping up to be a foregone conclusion. Democrat Terry McAuliffe, the longtime political fixer and moneyman, hasn’t trailed in a poll since May…
Q: What are the candidates running for? A: GovernorA: VirginiaQ: Where?
![Page 18: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/18.jpg)
7
CoQA goal #1: conversational questions
The Virginia governor’s race is shaping up to be a foregone conclusion. Democrat Terry McAuliffe, the longtime political fixer and moneyman, hasn’t trailed in a poll since May…
Q: What are the candidates running for? A: GovernorA: Virginia
Q: Who is the democratic candidate? A: Terry McAuliffe
Q: Who is his opponent?
Q: Where?
![Page 19: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/19.jpg)
8
CoQA goal #2: free-form answers
Jessica is turning 80. Her granddaughter Annie is coming over in the afternoon and Jessica is very excited to see her. Her daughter Melanie and Melanie’s husband Josh were coming as well.
Q: How many are visiting?
Q: Who?
A: Three
A: Annie, Melanie and Josh
![Page 20: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/20.jpg)
9
CoQA goal #2: free-form answers
Span overlap found 67%
No span overlap found 33%
![Page 21: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/21.jpg)
9
CoQA goal #2: free-form answers
Span overlap found 67%
No span overlap found 33%
Yes/No 79%
One word edit 8% Q: what is the cost to end users? R: The service is free to users A: It is freeMultiple edits 6%
Counting 5%
Other 2%
![Page 22: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/22.jpg)
10
CoQA goal #3: diverse genre
Train != Test
![Page 23: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/23.jpg)
11
Conversational vs. isolated questions
SQuADCoQA
![Page 24: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/24.jpg)
12
CoQA vs. QuAC
[Choi et al. 2018]
Wikipedia, extractive answers
7 genres, free-form answers with
extractive rationale
![Page 25: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/25.jpg)
12
CoQA vs. QuAC
[Choi et al. 2018]
questioner and answerer sees the passage
only answerer sees the passage
Wikipedia, extractive answers
7 genres, free-form answers with
extractive rationale
![Page 26: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/26.jpg)
12
CoQA vs. QuAC
[Choi et al. 2018]
questioner and answerer sees the passage
only answerer sees the passage
answer length: avg = 2.7 answer length: avg = 15.1
Wikipedia, extractive answers
7 genres, free-form answers with
extractive rationale
![Page 27: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/27.jpg)
12
CoQA vs. QuAC
[Choi et al. 2018]
questioner and answerer sees the passage
only answerer sees the passage
answer length: avg = 2.7 answer length: avg = 15.1
human performance: 88.8 F1 human performance: 74.7 F1
Wikipedia, extractive answers
7 genres, free-form answers with
extractive rationale
![Page 28: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/28.jpg)
13
Other datasets
[Saeidi et al. 2018]
[Elgohary et al. 2018]
![Page 29: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/29.jpg)
14
Linguistic phenomena
70%
30%
IsolatedCoreference
![Page 30: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/30.jpg)
15
Linguistic phenomena
27%
43%
30%
Lexical MatchParaphrasingPragmatics
Did he keep the puppy?
R: He walked the puppy home A: Yes
![Page 31: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/31.jpg)
Generate free text as answers
Highlight spans as answers
16
Learning models
![Page 32: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/32.jpg)
Generate free text as an answer
Context Vector
<start>
Atte
ntio
n
Dis
tribu
tion
...
Enco
der
Hid
den
Stat
es
Decoder
Hidden States
The Virginia governor ’s race … <q> What … <q> Where
Passage
weighted sum
Answer Generation
Dialog history and current question
…
…
Vocabulary D
istributiona zoo
Pvocab
[See et al. 2017]
![Page 33: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/33.jpg)
18
What are the candidates … Where ?
…
Dialog history and current question
q
Highlight a text span as an answer
[Chen et al. 2017]
![Page 34: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/34.jpg)
18
What are the candidates … Where ?
…
Dialog history and current question
…
Passage
...
The Virginia governor ’s race … …. hasn’t trailed since May
P
q
Highlight a text span as an answer
[Chen et al. 2017]
![Page 35: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/35.jpg)
18
What are the candidates … Where ?
…
Dialog history and current question
…
Passage
...
The Virginia governor ’s race … …. hasn’t trailed since May
P
q
Highlight a text span as an answer
p(answer) = maxi,j
p(starti) p(endj)
…
[Chen et al. 2017]
![Page 36: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/36.jpg)
19
Highlight first and then generate
Question: Did she plan to have any visitors?
Generate: Yes
Highlight: Her granddaughter Annie was coming over
(discriminator + generator)
![Page 37: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/37.jpg)
20
Evaluation
average F1 score of word overlap
Question: What did James do? Predicted: Took the puppy home Gold: Took him home
![Page 38: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/38.jpg)
21
Results
Generate Highlight Highlight + Generate
Human
avg
F1 S
core
![Page 39: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/39.jpg)
21
Results
Generate Highlight Highlight + Generate
Human
avg
F1 S
core
44.1
![Page 40: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/40.jpg)
21
Results
Generate Highlight Highlight + Generate
Human
avg
F1 S
core
52.644.1
![Page 41: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/41.jpg)
21
Results
Generate Highlight Highlight + Generate
Human
avg
F1 S
core 65.1
52.644.1
![Page 42: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/42.jpg)
21
Results
Generate Highlight Highlight + Generate
Human
avg
F1 S
core
88.8
65.152.6
44.1
![Page 43: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/43.jpg)
22
Results: analysis
Question: Does this help or hurt their memory of the event?Only Generator: help
![Page 44: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/44.jpg)
22
Results: analysis
Question: Does this help or hurt their memory of the event?Only Generator: help
Question: Does this help or hurt their memory of the event?Only Highlighter: Answer is not a span
![Page 45: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/45.jpg)
22
Results: analysis
Question: Does this help or hurt their memory of the event?Only Generator: help
Question: Does this help or hurt their memory of the event?
Generator: hurt
Highlighter: this obsession may prevent 1their brains from remembering
Question: Does this help or hurt their memory of the event?Only Highlighter: Answer is not a span
![Page 46: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/46.jpg)
23
Importance of conversation history
10
87
0 1 2 all
Models Humans
History =
![Page 47: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/47.jpg)
23
Importance of conversation history
10
87
0 1 2 all
Models Humans
19.9
61.5
History =
![Page 48: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/48.jpg)
23
Importance of conversation history
10
87
0 1 2 all
Models Humans
79.8
19.9
66.261.5
History =
![Page 49: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/49.jpg)
23
Importance of conversation history
10
87
0 1 2 all
Models Humans
85.379.8
19.9
6666.261.5
History =
![Page 50: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/50.jpg)
23
Importance of conversation history
10
87
0 1 2 all
Models Humans
86.485.379.8
19.9
64.36666.261.5
History =
![Page 51: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/51.jpg)
24
Amazing progress since launch
BERT
![Page 52: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/52.jpg)
24
Amazing progress since launch
BERT
![Page 53: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/53.jpg)
• Do models actually resolve coreferences, or do they exploit shortcuts?
• Unsupervised and low-resource scenario
• Transfer CoQA to other dialogue problems
25
What’s next?
GPT-2
![Page 54: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/54.jpg)
• CoQA combines challenges from both dialogue and reading comprehension
• Grounded evaluation for dialogue models
• Contains several reasoning phenomena
• An avenue for transfer learning in dialogue
26
Conclusions
https://stanfordnlp.github.io/coqa/
![Page 55: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/55.jpg)
27
Acknowledgements
![Page 56: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/56.jpg)
28
CoQA goal #2: free-form answers
We asked the answerer to first highlight a piece of text (as the rationale) and then edit the text to generate a natural answer
![Page 57: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/57.jpg)
28
CoQA goal #2: free-form answers
We asked the answerer to first highlight a piece of text (as the rationale) and then edit the text to generate a natural answer
Rationale: Jessica is turning 80Question: How old would she be?
Answer: 80
![Page 58: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/58.jpg)
28
CoQA goal #2: free-form answers
We asked the answerer to first highlight a piece of text (as the rationale) and then edit the text to generate a natural answer
Rationale: Jessica is turning 80Question: How old would she be?
Answer: 80
CoQA NarrativeQA
Human agreement 88.8 59.0
![Page 59: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/59.jpg)
why? why what?
decide that's not a good question
29
Data collection setup #8
![Page 60: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/60.jpg)
30
Data collection setup #10
did they go? So we went over and the girls had a wonderful time
Did Tina catch butterfly, yes or no please? She did everything she could to try and catch that pretty butterfly but it kept flying way to high for her to catch
"Yes or no?" "No she did not"
![Page 61: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/61.jpg)
• Answerer: abstractive and extractive
• Questioner: as you type feedback
• In-window chat interface for discussion
• Pay no matter what!
• Be nice
31
Data collection setup
![Page 62: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/62.jpg)
32
![Page 63: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/63.jpg)
33
![Page 64: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/64.jpg)
34
![Page 65: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/65.jpg)
• Stay away from boring stories
• Not more than one entity
• No coreferences
• No multiple events
35
Stories
![Page 66: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/66.jpg)
A gramophone record (phonograph record in American English) or vinyl record, commonly known as a "record", is an analogue sound storage medium in the form of a flat polyvinyl chloride (previously shellac) disc with an inscribed, modulated spiral groove. The groove usually starts near the periphery and ends near the center of the disc.…
36
Boring story from SQuAD
![Page 67: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/67.jpg)
Everton were founder members of the Premier League in 1992, but struggled to find the right manager. Howard Kendall had returned in 1990 but could not repeat his previous success, while his successor, Mike Walker, was statistically the least successful Everton manager to date…..
37
Good story from SQuAD
![Page 68: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/68.jpg)
38
Answer dependent
![Page 69: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/69.jpg)
38
Answer dependent
“I think the flow of this one turned out really awkward. I got a lot of irrelevant/nonsense questions.” — turker
![Page 70: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/70.jpg)
39
Disagreements
![Page 71: CoQA: A Conversational Question · 2019-06-05 · Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up](https://reader034.fdocuments.us/reader034/viewer/2022042400/5f0ecaa67e708231d440f51d/html5/thumbnails/71.jpg)
• Pragmatics/common-sense
40
Linguistic Phenomena