Algorithms that learn to think on their...
Transcript of Algorithms that learn to think on their...
![Page 1: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/1.jpg)
Hal Daumé III | University of Maryland | [email protected] | @haldaume3
Algorithmsthat learn to
think on their feet(now, with amazing bonus prize!)
![Page 2: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/2.jpg)
Algorithms that think on their feet2 Hal Daumé III ([email protected])
What is NLP?
Fundamental goal: deep understanding of text Not just string processing or keyword matching
End systems that we want to build Simple: Spelling correction, text categorization, etc. Complex: Speech recognition, machine translation, information
extraction, dialog interfaces, question answering Unknown: human-level comprehension (more than just NLP?)
![Page 3: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/3.jpg)
Algorithms that think on their feet3 Hal Daumé III ([email protected])
Why is language hard?
Ambiguity abounds (some headlines) Iraqi Head Seeks Arms Teacher Strikes Idle Kids Kids Make Nutritious Snacks Stolen Painting Found by Tree Local HS Dropouts Cut in Half Enraged Cow Injures Farmer with Ax Hospitals are Sued by 7 Foot Doctors Ban on Nude Dancing on Governor's Desk Scientists study whales from space
Why are these funny? What does ambiguity imply about the role of learning?
![Page 4: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/4.jpg)
Algorithms that think on their feet4 Hal Daumé III ([email protected])
Despite ambiguity, language is predictable
The brain uses this information!
Can we use predictabilityto make decisions beforeall of the input is observed?
I like my coffee with cream and ___________
YES!!!
This is crummy weather for San ___________
asparagus
ta Claus
![Page 5: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/5.jpg)
Algorithms that think on their feet5 Hal Daumé III ([email protected])
Simultaneous Simultaneous Machine Machine
Interpretation Interpretation
Ich bin mit dem Zug nach Ulm gefahren I am with the train to Ulm traveled I (. . . . . . waiting. . . . . . ) traveled by train to Ulm
QuizbowlQuizbowl (Incremental(Incremental QuestionQuestion Answering)Answering)
Outline
double-e with a
fridge
He HeAlvin Grissom II
![Page 6: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/6.jpg)
Algorithms that think on their feet6 Hal Daumé III ([email protected])
Simultaneous (machine) interpretation Dozens of defendants Judges from four nations
(three languages) Status quo: speak, then
translate After Nuremberg,
simultaneous translations became the norm
Long wait bad →conversation
NuremburgTrials
![Page 7: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/7.jpg)
Algorithms that think on their feet7 Hal Daumé III ([email protected])
Why simultaneous interpretation is hard Human languages have vastly different word orders
About half are OV, the other half are VO This comes with a lot more baggage than just verb-final
man-TOP store-LOC go-PAST
the man went to the store
food-OBJ buy-DESIRE man-TOP store-LOC go-PAST
the man who wanted to buy food went to the store
Running (German/English) Example:
Ich bin mit dem Zug nach Ulm gefahren I am with the train to Ulm traveled I (. . . . . . waiting. . . . . . ) traveled by train to Ulm
![Page 8: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/8.jpg)
Algorithms that think on their feet8 Hal Daumé III ([email protected])
We have a set of actions (predict / translate) Wait Predict clause-verb Predict next word Commit (“speak”)
In a changing environment (state) The words we've seen so far Our models' internal predictions
With a well defined notion of “optimal action” at training time
Model for interpretation decisions
![Page 9: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/9.jpg)
Algorithms that think on their feet9 Hal Daumé III ([email protected])
Example of interpretation trajectory
Ich bin mit dem Zug nach Ulm gefahren I am with the train to Ulm traveled I (. . . . . . waiting. . . . . . ) traveled by train to Ulm
![Page 10: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/10.jpg)
Algorithms that think on their feet10 Hal Daumé III ([email protected])
DAgger: Dataset Aggregation
ππ**
Collect trajectories from expert π* Dataset D0 = ( s, π*(s) ) | s ~ π*
Train π1 on D0
Collect new trajectories from π1
But let the expert steer! Dataset D1 = ( s, π*(s) ) | s ~ π1
Train π2 on D0 ∪ D1
In general: Dn = ( s, π*(s) ) | s ~ πn
Train πn on ∪i<n Di
ππ11
ππ22
If N = T log T,
L(πn) < T N + O(1)
for some n
(Ro
ss+G
ord
on
+Ba
gn
ell,
AIS
tats
201
1)
![Page 11: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/11.jpg)
Algorithms that think on their feet11 Hal Daumé III ([email protected])
Evaluating performance and baselines
(Gris
som
II e
t a
l., E
MN
LP 2
014)
![Page 12: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/12.jpg)
Algorithms that think on their feet12 Hal Daumé III ([email protected])
Evaluating performance and baselines
(Gris
som
II e
t a
l., E
MN
LP 2
014)
![Page 13: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/13.jpg)
Algorithms that think on their feet13 Hal Daumé III ([email protected])
Evaluating performance and baselines
(Gris
som
II e
t a
l., E
MN
LP 2
014)
![Page 14: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/14.jpg)
Algorithms that think on their feet14 Hal Daumé III ([email protected])
Evaluating performance and baselines
(Gris
som
II e
t a
l., E
MN
LP 2
014)
![Page 15: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/15.jpg)
Algorithms that think on their feet15 Hal Daumé III ([email protected])
Evaluating performance and baselines
(Gris
som
II e
t a
l., E
MN
LP 2
014)
![Page 16: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/16.jpg)
Algorithms that think on their feet16 Hal Daumé III ([email protected])
Evaluating performance and baselines
(Gris
som
II e
t a
l., E
MN
LP 2
014)
![Page 17: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/17.jpg)
Algorithms that think on their feet17 Hal Daumé III ([email protected])
Evaluating performance and baselines
(Gris
som
II e
t a
l., E
MN
LP 2
014)
![Page 18: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/18.jpg)
Algorithms that think on their feet18 Hal Daumé III ([email protected])
Training the policy
(Gris
som
II e
t a
l., E
MN
LP 2
014)
Actions: Commit translate(revealed words)
Predict (verb/next) translate(revealed + predicted)
Wait get_next_words()
Delayed feedback: latency BLEU
Features: Output & confidence of predictors Internal translation / language model scores Previous decisions made by policy
![Page 19: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/19.jpg)
Algorithms that think on their feet19 Hal Daumé III ([email protected])
Evaluating performance
0 0.2 0.4 0.6 0.8 1 1.2 1.4
(Gris
som
II e
t a
l., E
MN
LP 2
014)
![Page 20: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/20.jpg)
Algorithms that think on their feet20 Hal Daumé III ([email protected])
Simultaneous Simultaneous Machine Machine
Interpretation Interpretation
Ich bin mit dem Zug nach Ulm gefahren I am with the train to Ulm traveled I (. . . . . . waiting. . . . . . ) traveled by train to Ulm
QuizbowlQuizbowl (Incremental(Incremental QuestionQuestion Answering)Answering) double-e with a
fridge
Outline
double-e with a
fridge
He HeMohit Iyyer
![Page 21: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/21.jpg)
Algorithms that think on their feet21 Hal Daumé III ([email protected])
Humans doing incremental prediction Game called “quiz bowl” Two teams play each other
Moderator reads a question When a team knows the
answer, they buzz in If right, they get points;
otherwise, rest of the question is read to the other team
Hundreds of teams in the US alone
Example . . .
(Iyye
r et
al.,
AC
L 20
14)
![Page 22: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/22.jpg)
Algorithms that think on their feet22 Hal Daumé III ([email protected])
Quizbowl exampleWith Leo Szilard, he invented a doubly-eponymous
(Iyye
r et
al.,
AC
L 20
14)
![Page 23: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/23.jpg)
Algorithms that think on their feet23 Hal Daumé III ([email protected])
Quizbowl exampleWith Leo Szilard, he invented a doubly-eponymous refrigerator with no moving parts. He did not take interaction with neighbors into account when formulating his theory
(Iyye
r et
al.,
AC
L 20
14)
![Page 24: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/24.jpg)
Algorithms that think on their feet24 Hal Daumé III ([email protected])
Quizbowl exampleWith Leo Szilard, he invented a doubly-eponymous refrigerator with no moving parts. He did not take interaction with neighbors into account when formulating his theory of heat capacity, so
(Iyye
r et
al.,
AC
L 20
14)
![Page 25: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/25.jpg)
Algorithms that think on their feet25 Hal Daumé III ([email protected])
Quizbowl exampleWith Leo Szilard, he invented a doubly-eponymous refrigerator with no moving parts. He did not take interaction with neighbors into account when formulating his theory of heat capacity, so Debye adjusted the theory for low temperatures. His summation convention automatically sums repeated indices in tensor products. His name is attached to the A and B coefficients
(Iyye
r et
al.,
AC
L 20
14)
![Page 26: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/26.jpg)
Algorithms that think on their feet26 Hal Daumé III ([email protected])
Quizbowl exampleWith Leo Szilard, he invented a doubly-eponymous refrigerator with no moving parts. He did not take interaction with neighbors into account when formulating his theory of heat capacity, so Debye adjusted the theory for low temperatures. His summation convention automatically sums repeated indices in tensor products. His name is attached to the A and B coefficients for spontaneous and stimulated emission, the subject of one of his multiple groundbreaking 1905 papers. He further developed the model of statistics sent to him by
(Iyye
r et
al.,
AC
L 20
14)
![Page 27: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/27.jpg)
Algorithms that think on their feet27 Hal Daumé III ([email protected])
Quizbowl exampleWith Leo Szilard, he invented a doubly-eponymous refrigerator with no moving parts. He did not take interaction with neighbors into account when formulating his theory of heat capacity, so Debye adjusted the theory for low temperatures. His summation convention automatically sums repeated indices in tensor products. His name is attached to the A and B coefficients for spontaneous and stimulated emission, the subject of one of his multiple groundbreaking 1905 papers. He further developed the model of statistics sent to him by Bose to describe particles with integer spin. For 10 points, who is this German physicist best known for formulating
(Iyye
r et
al.,
AC
L 20
14)
![Page 28: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/28.jpg)
Algorithms that think on their feet28 Hal Daumé III ([email protected])
Quizbowl exampleWith Leo Szilard, he invented a doubly-eponymous refrigerator with no moving parts. He did not take interaction with neighbors into account when formulating his theory of heat capacity, so Debye adjusted the theory for low temperatures. His summation convention automatically sums repeated indices in tensor products. His name is attached to the A and B coefficients for spontaneous and stimulated emission, the subject of one of his multiple groundbreaking 1905 papers. He further developed the model of statistics sent to him by Bose to describe particles with integer spin. For 10 points, who is this German physicist best known for formulating the special and general theories of relativity?
(Iyye
r et
al.,
AC
L 20
14)
![Page 29: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/29.jpg)
Algorithms that think on their feet29 Hal Daumé III ([email protected])
Solving incrementally Action: buzz now or wait
Content Model is constantly generating guesses Oracle provides examples where it is correct The Policy generalizes to test data Features represent our state
(Iyye
r et
al.,
AC
L 20
14)
![Page 30: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/30.jpg)
Algorithms that think on their feet30 Hal Daumé III ([email protected])
Evaluation methodology Mechanical Turk to
collect human data 7000 questions were
answered in the first day
Over 43000 questions were answered in the space of two weeks
Total of 461 unique users
Leaderboard to encourage users
Big problem:
“this man shot at Aaron Burr”is very different from
“Aaron Burr shot at this man”
(Iyye
r et
al.,
AC
L 20
14)
![Page 31: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/31.jpg)
Algorithms that think on their feet31 Hal Daumé III ([email protected])
Challenge: modeling compositionality
invented
double-e with a
he fridge
no moving
parts
(Iyye
r et
al.,
AC
L 20
14)
![Page 32: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/32.jpg)
Algorithms that think on their feet32 Hal Daumé III ([email protected])
Challenge: modeling compositionality
invented
double-e with a
he fridge
no moving
parts
(Iyye
r et
al.,
AC
L 20
14)
![Page 33: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/33.jpg)
Algorithms that think on their feet33 Hal Daumé III ([email protected])
Challenge: modeling compositionality
invented
double-e with a
he fridge
no moving
parts
(Iyye
r et
al.,
AC
L 20
14)
![Page 34: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/34.jpg)
Algorithms that think on their feet34 Hal Daumé III ([email protected])
Challenge: modeling compositionality
invented
double-e with a
he fridge
no moving
parts
e(fridge) = f( wfridge + W [ e(a) e(double-e) e(with) ])
(Iyye
r et
al.,
AC
L 20
14)
![Page 35: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/35.jpg)
Algorithms that think on their feet35 Hal Daumé III ([email protected])
Results on question-answering task
Full
2nd Sent
1st Sent
25 35 45 55 65 75 85
History QuestionsBOW (QB)
RNN (QB)BOW (wiki)
Combined
(Iyye
r et
al.,
AC
L 20
14)
![Page 36: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/36.jpg)
Algorithms that think on their feet36 Hal Daumé III ([email protected])
Results on question-answering task
Full
2nd Sent
1st Sent
25 35 45 55 65 75 85
Literature QuestionsBOW (QB)
RNN (QB)BOW (wiki)
Combined
(Iyye
r et
al.,
AC
L 20
14)
![Page 38: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/38.jpg)
Algorithms that think on their feet38 Hal Daumé III ([email protected])
But the true test... RESULTS!
SUPER thanks to Ken Jenningsfor being a great sport!
![Page 39: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/39.jpg)
Algorithms that think on their feet39 Hal Daumé III ([email protected])
Simultaneous Simultaneous Machine Machine
Interpretation Interpretation
Ich bin mit dem Zug nach Ulm gefahren I am with the train to Ulm traveled I (. . . . . . waiting. . . . . . ) traveled by train to Ulm
QuizbowlQuizbowl (Incremental(Incremental QuestionQuestion Answering)Answering) double-e with a
fridge
Outline
BONUSPRIZE!
A BETTER WAY TO DO THE LEARNING!
double-e with a
fridge
AlekhAgarwal
Kai-WeiChang
AkshayKrishnamurthy
JohnLangford
![Page 40: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/40.jpg)
Algorithms that think on their feet40 Hal Daumé III ([email protected])
Structured learning with partial feedback
Loss of a single structured label can be observed Labels are never observed
![Page 41: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/41.jpg)
Algorithms that think on their feet41 Hal Daumé III ([email protected])
Solution strategy Use randomization to estimate losses Apply “standard” learning-to-search to losses
![Page 42: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/42.jpg)
Algorithms that think on their feet42 Hal Daumé III ([email protected])
Learning to search Convert structured prediction into a search problem
search space and actions
Define structured features over each state
Construct a reference policy (Ref) Ref usually defined using true label
Learn a policy that imitates Ref Implement with a cost-sensitive classifier
![Page 43: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/43.jpg)
Algorithms that think on their feet43 Hal Daumé III ([email protected])
Structured contextual bandit challenge True label is not available => Hard to define good Ref
Existing L2S algorithms give:
Can use status quo system as Ref, But competing with this Ref is not useful!
Main goal: Learning to search with: A suboptimal reference => improve on Ref Partial feedback
R (π)≤R(πref
)+o(1)
![Page 44: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/44.jpg)
Algorithms that think on their feet44 Hal Daumé III ([email protected])
Learning to search “schematic”
Desiridata: Compete with Ref (global opt if Ref is optimal and realizable) Local optimality
![Page 45: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/45.jpg)
Algorithms that think on their feet45 Hal Daumé III ([email protected])
Effect of Roll-in,out policies
Theorem: LOLS minimizes a combination ofregret to Ref and regret to its own one-step deviations
Theorem: Can take Ω(2T) steps to reach local optimality
![Page 46: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/46.jpg)
Algorithms that think on their feet46 Hal Daumé III ([email protected])
Does it work in practice? Experiments on Dependency Parsing
LOLS always good, even with Ref is bad
![Page 47: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/47.jpg)
Algorithms that think on their feet47 Hal Daumé III ([email protected])
Learning with partial feedback Loss of a single structured label can be observed Reference policy is not optimal
Apply an ϵ-greedy strategy Regret to ref and one-step deviations still bounded
![Page 48: Algorithms that learn to think on their feetusers.umiacs.umd.edu/~hal/talks/15-10-ucsc-feetthinking.pdf · 2015. 12. 18. · 7 Hal Daumé III (me@hal3.name) Algorithms that think](https://reader033.fdocuments.us/reader033/viewer/2022051408/5ff6d5f75b667c17ba0a5641/html5/thumbnails/48.jpg)
Reasoning with incomplete information is useful for speed and modeling
Imitation learning can help us build such systems Even when you can't construct a perfect
oracle & have incomplete information! Wide range of new, interesting problems to
work on! Improve upon human interpreters? Compete against specific opponents? Distance supervision via structured bandits
Thanks! Questions?
He HeJordan B-G Alvin Grissom II Mohit IyyerAlekh Agarwal K-W Chang Akshay K. John Langford
I am on thejob market!
umiacs.umd.edu/~hhe
ICPR '10 EMNLP'13 ICC'15CVPR '11 EMNLP'14 Fusion'15EMNLP'12 NIPS '14 EMNLP'15NIPS '12 SLT '14 + more