Learning to reason by reading text and answering questions · Learning to reason by reading text...
Transcript of Learning to reason by reading text and answering questions · Learning to reason by reading text...
![Page 1: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/1.jpg)
Learningtoreasonbyreadingtextandansweringquestions
MinjoonSeoNaturalLanguageProcessingGroup
UniversityofWashingtonMay26,2017
@Kakao Brain
![Page 2: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/2.jpg)
Whatisreasoning?
![Page 3: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/3.jpg)
SimpleQuestionAnsweringModel
Whatis“Hello”inFrench? Bonjour.
![Page 4: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/4.jpg)
Examples
• Mostneuralmachinetranslationsystems(Choetal.,2014;Bahdanau etal., 2014)• Needveryhighhiddenstatesize(~1000)• Noneedtoquerythedatabase(context)à veryfast
• Mostdependency,constituencyparser(Chenetal.,2014;Kleinetal.,2003)• Sentimentclassification(Socher etal.,2013)
• Classifyingwhetherasentenceispositiveornegative• Mostneuralimageclassificationsystems
• Thequestionisalways“Whatisintheimage?”
• Mostclassificationsystems
![Page 5: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/5.jpg)
SimpleQuestionAnsweringModel
Whatis“Hello”inFrench? Bonjour.
Problem:parametricmodelhasfinitecapacity.
“Youcan’tevenfitasentenceintoasinglevector”-DanRoth
![Page 6: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/6.jpg)
QAModelwithContext
English French
Hello Bonjour
Thankyou Merci
Whatis“Hello”inFrench? Bonjour.
Context(KnowledgeBase)
![Page 7: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/7.jpg)
Examples
• WikiQA(Yangetal.,2015)• QASent(Wangetal.,2007)• WebQuestions (Berant etal.,2013)• WikiAnswer (Wikia)• Free917(Cai andYates,2013)
• Manydeeplearningmodelswithexternalmemory (e.g.MemoryNetworks)
![Page 8: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/8.jpg)
QAModelwithContext
Eats IsA
(Amphibian, insect) (Frog, amphibian)
(insect,flower) (Fly,insect)
Whatdoesafrogeat? Fly
Context(KnowledgeBase)
Somethingismissing…
![Page 9: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/9.jpg)
QAModelwithReasoningCapability
Eats IsA
(Amphibian, insect) (Frog, amphibian)
(insect,flower) (Fly,insect)
Whatdoesafrogeat? Fly
Context(KnowledgeBase)
FirstOrderLogicIsA(A, B)^IsA(C,D)^Eats(B,D)à Eats(A,C)
![Page 10: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/10.jpg)
Examples
• Semanticparsing• GeoQuery (Krishnamurthyetal.,2013;Artzi etal.,2015)
• Sciencequestions• AristoChallenge(Clarketal.,2015)• ProcessBank (Berant etal.,2014)
• Machinecomprehension• MCTest (Richardsonetal.,2013)
![Page 11: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/11.jpg)
“Vague”linebetweennon-reasoningQAandreasoningQA• Non-reasoning:• Therequiredinformationisexplicitinthecontext• Themodeloftenneedstohandlelexical/syntacticvariations
• Reasoning:• Therequiredinformationmaynot beexplicitinthecontext• Needtocombinemultiplefactstoderivetheanswer
• Thereisnoclearlinebetweenthetwo!
![Page 12: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/12.jpg)
Ifourobjectiveisto“answer”difficultquestions…• Wecantrytomakethemachinemorecapableofreasoning(bettermodel)
• Wecantrytomakemoreinformationexplicitinthecontext(moredata)
OR
![Page 13: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/13.jpg)
QAModelwithReasoningCapability
Eats IsA
(Amphibian, insect) (Frog, amphibian)
(insect,flower) (Fly,insect)
Whatdoesafrogeat? Fly
Context(KnowledgeBase)
FirstOrderLogicIsA(A, B)^IsA(C,D)^Eats(B,D)à Eats(A,C)
Whomakesthis?Tellmeit’s notme…
![Page 14: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/14.jpg)
ReasoningQAModelwithUnstructuredData
Whatdoesafrogeat? Fly
Frogisanexampleofamphibian.Fliesareoneofthemostcommoninsectsaroundus.Insectsaregoodsourcesofproteinforamphibians.…
Contextinnaturallanguage
![Page 15: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/15.jpg)
Iaminterestedin…
• Naturallanguageunderstanding• Naturallanguagehasdiversesurfaceforms(lexically,syntactically)
• Learningtoreadtextandreasonbyquestionanswering(dialog)• Textisunstructureddata• Derivingnewknowledgefromexistingknowledge
• End-to-endtraining• Minimizinghumanefforts
![Page 16: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/16.jpg)
![Page 17: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/17.jpg)
Reasoningcapability
NLUcapability End-to-end
![Page 18: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/18.jpg)
AAAI2014EMNLP2015
ECCV2016CVPR2017
ICLR2017ACL2017
ICLR2017
![Page 19: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/19.jpg)
Reasoningcapability
NLUcapability End-to-end
GeometryQA
![Page 20: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/20.jpg)
GeometryQA
In the diagram at the right, circle O has a radius of 5, and CE = 2. Diameter AC is perpendicular to chord BD. What is the length of BD?
a) 2 b) 4 c) 6d) 8 e) 10
EB D
A
O
5
2
C
![Page 21: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/21.jpg)
GeometryQAModel
WhatisthelengthofBD? 8
In the diagram at the right, circle O has a radius of 5, and CE = 2. Diameter AC is perpendicular to chord BD.
FirstOrderLogic
Localcontext Globalcontext
![Page 22: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/22.jpg)
Method
• Learntomapquestiontologicalform• Learntomaplocalcontexttologicalform• Textà logicalform• Diagramà logicalform
• Globalcontextisalreadyformal!• Manually defined• “IfAB=BC,then<CAB=<ACB”
• Solveronalllogicalforms• Wecreatedareasonablenumericalsolver
![Page 23: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/23.jpg)
Mappingquestion/texttologicalform
IntriangleABC,lineDEisparallelwithlineAC,DBequals4,ADis8,andDEis5.FindAC.(a)9(b)10(c)12.5(d)15(e)17
B
D E
A C
IsTriangle(ABC) ∧ Parallel(AC, DE) ∧
Equals(LengthOf(DB), 4) ∧ Equals(LengthOf(AD), 8) ∧ Equals(LengthOf(DE), 5) ∧ Find(LengthOf(AC))
TextInput
Logicalform
Difficulttodirectlymaptexttoalonglogicalform!
![Page 24: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/24.jpg)
Mappingquestion/texttologicalformIntriangleABC,lineDEisparallelwithlineAC,DBequals4,ADis8,andDEis5.FindAC.(a)9(b)10(c)12.5(d)15(e)17
B
D E
A C
IsTriangle(ABC)Parallel(AC, DE)Parallel(AC, DB)Equals(LengthOf(DB), 4)Equals(LengthOf(AD), 8)Equals(LengthOf(DE), 5)Equals(4, LengthOf(AD))…
Over-generatedliterals0.960.910.740.970.940.940.31…
Textscores1.000.990.02n/an/an/an/a…
Diagramscores
Selectedsubset
TextInput
Logicalform
Ourmethod
IsTriangle(ABC) ∧ Parallel(AC, DE) ∧
Equals(LengthOf(DB), 4) ∧ Equals(LengthOf(AD), 8) ∧ Equals(LengthOf(DE), 5) ∧ Find(LengthOf(AC))
![Page 25: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/25.jpg)
Numericalsolver
Literal EquationEquals(LengthOf(AB),d) (Ax-Bx)2+(Ay-By)2-d2 =0Parallel(AB,CD) (Ax-Bx)(Cy-Dy)-(Ay-By)(Cx-Dx)=0PointLiesOnLine(B,AC) (Ax-Bx)(By-Cy)-(Ay-By)(Bx-Cx)=0Perpendicular(AB,CD) (Ax-Bx)(Cx-Dx)+(Ay-By)(Cy-Dy)=0
• Findthesolutiontotheequationsystem• Useoff-the-shelfnumericalminimizers(WalesandDoye,1997;Kraft,1988)
• Numericalsolvercanchoosenot toanswerquestion
• Translateliteralstonumericequations
![Page 26: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/26.jpg)
Dataset• Trainingquestions(67questions,121sentences)• Seoetal.,2014• Highschoolgeometryquestions
• Testquestions (119questions,215sentences)• Wecollectedthem• SAT(UScollegeentranceexam)geometryquestions
• Wemanuallyannotatedthetextparseofallquestions
![Page 27: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/27.jpg)
Results(EMNLP2015)
0
10
20
30
40
50
60
Textonly Diagramonly
Rule-based GeoS Studentaverage
SATScore(%
)
***0.25penaltyforincorrectanswer
![Page 28: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/28.jpg)
Demo(geometry.allenai.org/demo)
![Page 29: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/29.jpg)
Limitations
• Datasetissmall• Requiredlevelofreasoningisveryhigh• Alotofmanualefforts(annotations,ruledefinitions,etc.)• End-to-endsystemissimplyhopeless
• Collectmoredata?• Changetask?• Curriculumlearning?(Domorehopeful tasksfirst?)
![Page 30: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/30.jpg)
Reasoningcapability
NLUcapability End-to-end
DiagramQA
![Page 31: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/31.jpg)
DiagramQA
Q:Theprocessofwaterbeingheatedbysunandbecominggasiscalled
A:Evaporation
![Page 32: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/32.jpg)
IsDQAsubsetofVQA?
• Diagramsandrealimagesareverydifferent• Diagramcomponentsaresimplerthanrealimages• Diagramcontainsalotofinformationinasingleimage• Diagramsarefew(whereasrealimagesarealmostinfinitelymany)
![Page 33: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/33.jpg)
Problem
Whatcomesbeforesecondfeed? 8
Difficulttolatentlylearnrelationships
![Page 34: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/34.jpg)
Strategy
Whatdoesafrogeat? Fly
DiagramGraph
![Page 35: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/35.jpg)
DiagramParsing
![Page 36: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/36.jpg)
QuestionAnswering
![Page 37: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/37.jpg)
Attentionvisualization
![Page 38: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/38.jpg)
Results(ECCV2016)
Method Trainingdata Accuracy
Random(expected) - 25.00
LSTM+CNN VQA 29.06
LSTM+CNN AI2D 32.90
Ours AI2D 38.47
![Page 39: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/39.jpg)
Limitations
• Youcan’treallycallthisreasoning…• Rathermatchtingalgorithm• Nocomplexinferenceinvolved
• Youneedalotofpriorknowledgetoanswersomequestions!• E.g.“Flyisaninsect”,“Frogisanamphibian”
![Page 40: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/40.jpg)
TextbookQAtextbookqa.org (CVPR2017)
![Page 41: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/41.jpg)
Reasoningcapability
NLUcapability End-to-end
MachineComprehension
![Page 42: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/42.jpg)
QuestionAnsweringTask(StanfordQuestionAnsweringDataset,2016)
Q:WhichNFLteamrepresentedtheAFCatSuperBowl50?
A:DenverBroncos
![Page 43: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/43.jpg)
WhyNeuralAttention?
Q:WhichNFLteamrepresentedtheAFCatSuperBowl50?
Allowsadeeplearningarchitecturetofocusonthemostrelevantphraseofthecontexttothequery
inadifferentiablemanner.
![Page 44: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/44.jpg)
OurModel:Bi-directionalAttentionFlow(BiDAF)
Attention
Modeling
MLP+softmax
𝑖$ = 0 𝑖' = 1
BarakObamaisthepresidentoftheU.S. WholeadstheUnitedStates?
Attention
![Page 45: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/45.jpg)
(Bidirectional)AttentionFlow
Modeling Layer
Output Layer
Attention Flow Layer
Phrase Embed Layer
Word Embed Layer
x1 x2 x3 xT q1 qJ
LSTM
LSTM
LSTM
LSTM
Start End
h1 h2 hT
u1
u2
uJ
Softm
ax
h1 h2 hT
u1
u2
uJ
Max
Softmax
Context2Query
Query2Context
h1 h2 hT u1 uJ
LSTM + SoftmaxDense + Softmax
Context Query
Query2Context and Context2QueryAttention
WordEmbedding
GLOVE Char-CNN
Character Embed Layer
CharacterEmbedding
g1 g2 gT
m1 m2 mT
![Page 46: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/46.jpg)
Char/WordEmbeddingLayers
Modeling Layer
Output Layer
Attention Flow Layer
Phrase Embed Layer
Word Embed Layer
x1 x2 x3 xT q1 qJ
LSTM
LSTM
LSTM
LSTM
Start End
h1 h2 hT
u1
u2
uJ
Softm
ax
h1 h2 hT
u1
u2
uJ
Max
Softmax
Context2Query
Query2Context
h1 h2 hT u1 uJ
LSTM + SoftmaxDense + Softmax
Context Query
Query2Context and Context2QueryAttention
WordEmbedding
GLOVE Char-CNN
Character Embed Layer
CharacterEmbedding
g1 g2 gT
m1 m2 mT
![Page 47: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/47.jpg)
CharacterandWordEmbedding
• Wordembeddingisfragileagainstunseenwords• Charembeddingcan’teasilylearnsemanticsofwords• Useboth!
• CharembeddingasproposedbyKim(2015)
Seattle
SeattleCNN
+MaxPooling
concat
Embeddingvector
![Page 48: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/48.jpg)
PhraseEmbeddingLayer
Modeling Layer
Output Layer
Attention Flow Layer
Phrase Embed Layer
Word Embed Layer
x1 x2 x3 xT q1 qJ
LSTM
LSTM
LSTM
LSTM
Start End
h1 h2 hT
u1
u2
uJ
Softm
ax
h1 h2 hT
u1
u2
uJ
Max
Softmax
Context2Query
Query2Context
h1 h2 hT u1 uJ
LSTM + SoftmaxDense + Softmax
Context Query
Query2Context and Context2QueryAttention
WordEmbedding
GLOVE Char-CNN
Character Embed Layer
CharacterEmbedding
g1 g2 gT
m1 m2 mT
![Page 49: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/49.jpg)
PhraseEmbeddingLayer• Inputs:thechar/wordembeddingofqueryandcontextwords• Outputs:wordrepresentationsawareoftheirneighbors(phrase-awarewords)
• ApplybidirectionalRNN(LSTM)forbothqueryandcontext
Modeling Layer
Output Layer
Attention Flow Layer
Phrase Embed Layer
Word Embed Layer
x1 x2 x3 xT q1 qJ
LSTM
LSTM
LSTM
LSTM
Start End
h1 h2 hT
u1
u2
uJ
Softm
ax
h1 h2 hT
u1
u2
uJ
Max
Softmax
Context2Query
Query2Context
h1 h2 hT u1 uJ
LSTM + SoftmaxDense + Softmax
Context Query
Query2Context and Context2QueryAttention
WordEmbedding
GLOVE Char-CNN
Character Embed Layer
CharacterEmbedding
g1 g2 gT
m1 m2 mT
Context Query
![Page 50: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/50.jpg)
AttentionLayer
Modeling Layer
Output Layer
Attention Flow Layer
Phrase Embed Layer
Word Embed Layer
x1 x2 x3 xT q1 qJ
LSTM
LSTM
LSTM
LSTM
Start End
h1 h2 hT
u1
u2
uJ
Softm
ax
h1 h2 hT
u1
u2
uJ
Max
Softmax
Context2Query
Query2Context
h1 h2 hT u1 uJ
LSTM + SoftmaxDense + Softmax
Context Query
Query2Context and Context2QueryAttention
WordEmbedding
GLOVE Char-CNN
Character Embed Layer
CharacterEmbedding
g1 g2 gT
m1 m2 mT
![Page 51: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/51.jpg)
AttentionLayer
• Inputs:phrase-awarecontextandquerywords• Outputs:query-awarerepresentationsofcontextwords
• Context-to-queryattention:Foreach(phrase-aware)contextword,choosethemostrelevantwordfromthe(phrase-aware)querywords• Query-to-contextattention:Choosethecontextwordthatismostrelevanttoanyofquerywords.
Modeling Layer
Output Layer
Attention Flow Layer
Phrase Embed Layer
Word Embed Layer
x1 x2 x3 xT q1 qJ
LSTM
LSTM
LSTM
LSTM
Start End
h1 h2 hT
u1
u2
uJ
Softm
ax
h1 h2 hT
u1
u2
uJ
Max
Softmax
Context2Query
Query2Context
h1 h2 hT u1 uJ
LSTM + SoftmaxDense + Softmax
Context Query
Query2Context and Context2QueryAttention
WordEmbedding
GLOVE Char-CNN
Character Embed Layer
CharacterEmbedding
g1 g2 gT
m1 m2 mT
![Page 52: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/52.jpg)
Context-to-QueryAttention(C2Q)
Q:WholeadstheUnitedStates?
C:BarakObamaisthepresidentoftheUSA.
Foreachcontextword,findthemostrelevantqueryword.
![Page 53: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/53.jpg)
Query-to-ContextAttention(Q2C)
WhileSeattle’sweatherisveryniceinsummer,itsweatherisveryrainyinwinter,makingitoneofthemostgloomycitiesintheU.S.LAis…
Q:Whichcityisgloomyinwinter?
![Page 54: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/54.jpg)
ModelingLayer
Modeling Layer
Output Layer
Attention Flow Layer
Phrase Embed Layer
Word Embed Layer
x1 x2 x3 xT q1 qJ
LSTM
LSTM
LSTM
LSTM
Start End
h1 h2 hT
u1
u2
uJ
Softm
ax
h1 h2 hT
u1
u2
uJ
Max
Softmax
Context2Query
Query2Context
h1 h2 hT u1 uJ
LSTM + SoftmaxDense + Softmax
Context Query
Query2Context and Context2QueryAttention
WordEmbedding
GLOVE Char-CNN
Character Embed Layer
CharacterEmbedding
g1 g2 gT
m1 m2 mT
![Page 55: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/55.jpg)
ModelingLayer
• Attentionlayer:modelinginteractionsbetweenqueryandcontext• Modelinglayer:modelinginteractionswithin(query-aware)contextwordsviaRNN(LSTM)
• Divisionoflabor:letattentionandmodelinglayerssolelyfocusontheirowntasks• Weexperimentallyshowthatthisleadstoabetterresultthanintermixingattentionandmodeling
![Page 56: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/56.jpg)
OutputLayer
Modeling Layer
Output Layer
Attention Flow Layer
Phrase Embed Layer
Word Embed Layer
x1 x2 x3 xT q1 qJ
LSTM
LSTM
LSTM
LSTM
Start End
h1 h2 hT
u1
u2
uJ
Softm
ax
h1 h2 hT
u1
u2
uJ
Max
Softmax
Context2Query
Query2Context
h1 h2 hT u1 uJ
LSTM + SoftmaxDense + Softmax
Context Query
Query2Context and Context2QueryAttention
WordEmbedding
GLOVE Char-CNN
Character Embed Layer
CharacterEmbedding
g1 g2 gT
m1 m2 mT
![Page 57: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/57.jpg)
Training
• Minimizesthenegativelogprobabilitiesofthetruestartindexandthetrueendindex
𝑦*+ Trueendindexofexamplei
𝑦*, Truestartindexofexamplei
𝐩+ Probabilitydistributionofstopindex
𝐩, Probabilitydistributionofstartindex
![Page 58: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/58.jpg)
Previouswork
• Usingneuralattentionasacontroller(Xiong etal.,2016)• UsingneuralattentionwithinRNN(Wang&Jiang,2016)• Mostoftheseattentionsareuni-directional
• BiDAF (ourmodel)• usesneuralattentionasalayer,• Isseparatedfrommodelingpart(RNN),• Isbidirectional
![Page 59: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/59.jpg)
VGG-16
Modeling Layer
Output Layer
Attention Flow Layer
Phrase Embed Layer
Word Embed Layer
x1 x2 x3 xT q1 qJ
LSTM
LSTM
LSTM
LSTM
Start End
h1 h2 hT
u1
u2
uJ
Softm
ax
h1 h2 hT
u1
u2
uJ
Max
Softmax
Context2Query
Query2Context
h1 h2 hT u1 uJ
LSTM + SoftmaxDense + Softmax
Context Query
Query2Context and Context2QueryAttention
WordEmbedding
GLOVE Char-CNN
Character Embed Layer
CharacterEmbedding
g1 g2 gT
m1 m2 mT
BiDAF (ours)
ImageClassifierandBiDAF
![Page 60: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/60.jpg)
StanfordQuestionAnsweringDataset(SQuAD)(Rajpurkar etal.,2016)
• MostpopulararticlesfromWikipedia• QuestionsandanswersfromTurkers• 90ktrain,10kdev,?test(hidden)• Answermustlieinthecontext• Twometrics:ExactMatch(EM)andF1
![Page 61: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/61.jpg)
SQuAD Results(http://stanford-qa.com)asofDec2
(ICLR2017)
![Page 62: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/62.jpg)
Now..
![Page 63: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/63.jpg)
50
55
60
65
70
75
80
NoCharEmbedding NoWordEmbedding NoC2QAttention NoQ2CAttention DynamicAttention FullModel
EM F1
Ablationsondevdata
![Page 64: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/64.jpg)
InteractiveDemo
http://allenai.github.io/bi-att-flow/demo
![Page 65: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/65.jpg)
AttentionVisualizations
There%are%13 natural%reserves%in%Warsaw%–among%others%,%Bielany Forest%,%KabatyWoods%,%Czerniaków Lake%.%About%15%kilometres (%9%miles%)%from%Warsaw%,%the%Vistula%river%'s%environment%changes%strikingly%and%features%a%perfectly%preserved%ecosystem%,%with%a%habitat%of%animals%that%includes%the%otter%,%beaver%and%hundreds%of%bird%species%.%There%are%also%several%lakes%in%Warsaw%– mainly%the%oxbow%lakes%,%like%Czerniaków Lake%,%the%lakes%in%the%Łazienkior%Wilanów Parks%,%Kamionek Lake%.%There%are%lot%of%small%lakes%in%the%parks%,%but%only%a%few%are%permanent%– the%majority%are%emptied%before%winter%to%clean%them%of%plants%and%sediments%.
Howmany
naturalreserves
arethere
inWarsaw
?
[]hundreds, few, among, 15, several, only, 13, 9natural, ofreservesare, are, are, are, are, includes[][]Warsaw, Warsaw, Warsawinter species
Where
did
Super
Bowl
50
take
place
?
Super%Bowl%50%was%an%American%football%game%to%determine%the%champion%of%the%National%Football%League%(%NFL%)%for%the%2015%season%.%The%American%Football%Conference%(%AFC%)%champion% Denver%Broncos%defeated%the%National%Football%Conference%(%NFC%)%champion%Carolina%Panthers%24–10%to%earn%their%third%Super%Bowl%title%.%The%game%was%played%on%February%7%,%2016%,%at%Levi%'s%Stadium%in%the%San%Francisco%Bay%Area%at%Santa%Clara%,%California .%As%this%was%the%50th%Super%Bowl%,%the%league%emphasized%the%"%golden%anniversary%"%with%various%goldZthemed%initiatives%,%as%well%as%temporarily%suspending%the%tradition%of%naming%each%Super%Bowl%game%with%Roman%numerals%(%under%which%the%game%would%have%been%known%as%"%Super%Bowl%L%"%)%,%so%that%the%logo%could%prominently%feature%the%Arabic%numerals%50%.
at, the, at, Stadium, Levi, in, Santa, Ana
[]
Super, Super, Super, Super, Super
Bowl, Bowl, Bowl, Bowl, Bowl
50
initiatives
![Page 66: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/66.jpg)
EmbeddingVisualizationatWordvsPhraseLayers
January
September
August
July
May
may
effect and may result in
the state may not aid
of these may be more
Opening in May 1852 at
debut on May 5 ,
from 28 January to 25
but by September had been
![Page 67: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/67.jpg)
Howdoesitcomparewithfeature-basedmodels?
![Page 68: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/68.jpg)
CNN/DailyMail ClozeTest(Hermannetal.,2015)
• ClozeTest(PredictingMissingwords)• ArticlesfromCNN/DailyMail• Human-writtensummaries• Missingwordsarealwaysentities• CNN– 300karticle-querypairs• DailyMail – 1Marticle-querypairs
![Page 69: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/69.jpg)
CNN/DailyMail ClozeTestResults
![Page 70: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/70.jpg)
TransferLearning(ACL2017)
![Page 71: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/71.jpg)
SomelimitationsofSQuAD
![Page 72: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/72.jpg)
Reasoningcapability
NLUcapability End-to-end
bAbIQA&Dialog
![Page 73: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/73.jpg)
ReasoningQuestionAnswering
![Page 74: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/74.jpg)
DialogSystem
U:CanyoubookatableinRomeinItalianCuisine
S:Howmanypeopleinyourparty?
U:Forfourpeopleplease.
S:Whatpricerangeareyoulookingfor?
![Page 75: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/75.jpg)
DialogtaskvsQA
• DialogsystemcanbeconsideredasQAsystem:• Lastuser’sutteranceisthequery• Allpreviousconversationsarecontexttothequery• Thesystem’snextresponseistheanswertothequery
• Posesafewuniquechallenges• Dialogsystemrequirestrackingstates• Dialogsystemneedstolookatmultiplesentencesintheconversation• Buildingend-to-enddialogsystemismorechallenging
![Page 76: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/76.jpg)
Ourapproach:Query-Reduction
<START>Sandragottheapplethere.Sandradroppedtheapple.Danieltooktheapplethere.Sandrawenttothehallway.Danieljourneyedtothegarden.
Q:Whereistheapple?
Reducedquery:
Whereistheapple?WhereisSandra?WhereisSandra?WhereisDaniel?WhereisDaniel?WhereisDaniel?à garden
A:garden
![Page 77: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/77.jpg)
Query-ReductionNetworks• Reducethequeryintoaneasier-to-answerqueryoverthesequenceofstate-changingtriggers(sentences),invectorspace
Sandragottheapplethere.
!"
!"
#""
#"$
%""
%"$
Where isSandra?
Sandradroppedtheapple
!$
!$
#$"
#$$
%""
%$$
Danieltooktheapplethere.
!&
!&
#&"
#&$
%""
%&$
Where isDaniel?
Sandrawenttothehallway.
!'
!'
#'"
#'$
%""
%'$
Where isDaniel?
Danieljourneyedtothegarden.
!(
!(
#("
#($
%""
%($ → *+
Where isDaniel?
Whereistheapple?
#
garden
Where isSandra?
∅ ∅ ∅ ∅
![Page 78: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/78.jpg)
QRNCell
𝛼 𝜌
1 − ×
× +
𝐱𝑡 𝐪𝑡
𝐡𝑡−1 𝐡𝑡
𝐳𝑡 𝐡𝑡
sentence query
reducedquery(hiddenstate)
updategatecandidatereducedquery
updatefunc reductionfunc
![Page 79: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/79.jpg)
CharacteristicsofQRN
• Updategatecanbeconsideredaslocalattention• QRNchoosestoconsider/ignoreeachcandidatereducedquery• Thedecisionismadelocally(asopposedtoglobalsoftmax attention)
• SubclassofRecurrentNeuralNetwork(RNN)• Twoinputs,hiddenstate,gatingmechanism• Abletohandlesequentialdependency(attentioncannot)
• Simplerrecurrentupdateenablesparallelization overtime• Candidatehiddenstate(reducedquery)iscomputedfrominputsonly• Hiddenstatecanbeexplicitlycomputedasafunctionofinputs
![Page 80: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/80.jpg)
Parallelizationcomputedfrominputsonly,socanbetriviallyparallelized
Canbeexplicitlyexpressedasthegeometricsumofpreviouscandidatehiddenstates
![Page 81: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/81.jpg)
Parallelization
![Page 82: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/82.jpg)
CharacteristicsofQRN
• Updategatecanbeconsideredaslocalattention• SubclassofRecurrentNeuralNetwork(RNN)• Simplerrecurrentupdateenablesparallelization overtime
QRNsitsbetweenneuralattentionmechanismandrecurrentneuralnetworks,takingtheadvantageofbothparadigms.
![Page 83: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/83.jpg)
bAbI QADataset
• 20 differenttasks• 1kstory-questionpairsforeachtask(10kalsoavailable)• Syntheticallygenerated• Manyquestionsrequirelookingatmultiplesentences• Forend-to-endsystemsupervisedbyanswersonly
![Page 84: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/84.jpg)
What’sdifferentfromSQuAD?
• Synthetic• Morethanlexical/syntacticunderstanding• Differentkindsofinferences• induction,deduction,counting,pathfinding,etc.
• Reasoningovermultiplesentences• InterestingtestbedtowardsdevelopingcomplexQAsystem(anddialogsystem)
![Page 85: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/85.jpg)
bAbI QAResults(1k)(ICLR2017)
0
10
20
30
40
50
60
LSTM DMN+ MemN2N GMemN2N QRN(Ours)
AvgError(%)
AvgError(%)
![Page 86: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/86.jpg)
bAbI QAResults(10k)
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
MemN2N DNC GMemN2N DMN+ QRN(Ours)
AvgError(%)
AvgError(%)
![Page 87: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/87.jpg)
DialogDatasets
• bAbI DialogDataset• Synthetic• 5differenttasks• 1kdialogsforeachtask
• DSTC2*Dataset• Realdataset• EvaluationmetricisdifferentfromoriginalDSTC2:responsegenerationinsteadof“state-tracking”• Eachdialogis800+utterances• 2407possibleresponses
![Page 88: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/88.jpg)
bAbI DialogResults(OOV)
0
5
10
15
20
25
30
35
MemN2N GMemN2N QRN(Ours)
AvgError(%)
AvgError(%)
![Page 89: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/89.jpg)
DSTC2*DialogResults
0
10
20
30
40
50
60
70
MemN2N GMemN2N QRN(Ours)
AvgError(%)
AvgError(%)
![Page 90: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/90.jpg)
bAbI QAVisualization
𝑧/ = Localattention(updategate)atlayerl
![Page 91: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/91.jpg)
DSTC2(Dialog)Visualization
𝑧/ = Localattention(updategate)atlayerl
![Page 92: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/92.jpg)
So…
![Page 93: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/93.jpg)
Reasoningcapability
NLUcapability End-to-end
Isthispossible?
![Page 94: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/94.jpg)
Reasoningcapability
NLUcapability End-to-end
Orthis?
![Page 95: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/95.jpg)
So… Whatshouldwedo?
• Disclaimer:completelysubjective!
• Logic(reasoning)isdiscrete• Modelinglogicwithdifferentiablemodelishard• Relaxation:eitherhardtooptimizeorconvergetobadoptimum(lowgeneralizationerror)• Estimation:Low-biasorlow-variancemethodsareproposed(Williams,1992;Jangetal.,2017),butimprovementsarenotsubstantial.• Bigdata:howmuchdoweneed?Exponentiallymany?• Perhapsnewparadigmisneeded…
![Page 96: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/96.jpg)
“Ifyougotabilliondollarstospendonahugeresearchproject,whatwouldyouliketodo?”
“I'dusethebilliondollarstobuildaNASA-sizeprogramfocusingonnaturallanguageprocessing(NLP),inallofitsglory(semantics,pragmatics,etc).”
MichaelJordanProfessorofComputerScienceUCBerkeley
![Page 97: Learning to reason by reading text and answering questions · Learning to reason by reading text and answering questions Minjoon Seo Natural Language Processing Group University of](https://reader033.fdocuments.us/reader033/viewer/2022042220/5ec6a2870dbd4d54f536c972/html5/thumbnails/97.jpg)
TowardsArtificialGeneralIntelligence…
Naturallanguageisthebesttooltodescribeandcommunicate“thoughts”
Askingandansweringquestionsisaneffectivewaytodevelopdeeper“thoughts”