Grounding Language with Points and Paths in Continuous Spaces
-
Upload
gabriel-albert -
Category
Documents
-
view
62 -
download
2
description
Transcript of Grounding Language with Points and Paths in Continuous Spaces
![Page 1: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/1.jpg)
Berkeley
N L P
Grounding Language with Points and Paths in Continuous Spaces
Jacob Andreas and Dan KleinUC Berkeley
Berkeley
N L P
![Page 2: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/2.jpg)
Formal grounding
On June 26th, Facebook stock cost $65 per share
quote {date: 2014-06-26,stock: FB,price: $65
}
![Page 3: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/3.jpg)
Perceptual grounding
On June 26th, Facebook stock reboundedafter a bruising swoon
?
![Page 4: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/4.jpg)
Perceptual grounding
On June 26th, Facebook stock reboundedafter a bruising swoon
![Page 5: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/5.jpg)
Perceptual grounding
On June 26th, Facebook stock reboundedafter a bruising swoon
A after B A, B A before B B, A rebounded { sgn(slope) = +1 }
bruising { sgn(slope) = -1, abs(slope) = +2.3 }
![Page 6: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/6.jpg)
Continuous spaces everywhere
On June 26th, Facebook stock reboundedafter a bruising swoon
A deep red sunset
Keep a little to the left of the post
Beat the eggs gently, until they form stiff peaks
![Page 7: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/7.jpg)
Three tasks
Color Time series Navigation
![Page 8: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/8.jpg)
Predicting colors
bluepastel bluedark pastel blue
H
V S
![Page 9: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/9.jpg)
Regression model
bluepastel bluedark pastel blue
H
V S
![Page 10: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/10.jpg)
Regression model
dark pastel blue
0
0
-40
0
-37
-25
216
80
90
H
S
V
216
43
75
+ + =
![Page 11: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/11.jpg)
Regression model
dark pastel blue
![Page 12: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/12.jpg)
dark pastel blue{dark, pastel, blue}
Regression model
H 216S 43V 75
![Page 13: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/13.jpg)
Experiment setup
![Page 14: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/14.jpg)
Sample predictions
electric green pale blue dark brown
pale green indigo
![Page 15: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/15.jpg)
Prediction error
Hue Sat Val Mean0
0.1
0.2
0.3
0.4
BaselineLast wordFull model
![Page 16: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/16.jpg)
A guessing game
pale blue
![Page 17: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/17.jpg)
A guessing game
Prediction accuracy0
0.2
0.4
0.6
0.8
1
0.50
0.78 0.810.86
BaselineLast wordFull modelHuman
![Page 18: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/18.jpg)
Predicting time series
stocks rebounded after a bruising swoon
1 2
2 1
![Page 19: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/19.jpg)
Predicting time series
stocks rebounded after a bruising swoon
![Page 20: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/20.jpg)
Predicting time series
stocks rebounded after a bruising swoon{stocks, rebounded} {after, a, bruising, swoon}
2 1
sgn(slope): -1abs(slope): 3.1curvature: 0.5
sgn(slope): 1abs(slope): 2.7curvature: -0.1
![Page 21: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/21.jpg)
Learning & inference
• Need parameters for linear prediction model & log-linear alignment model: easy with EM
• For small number of path segments, possible to sum exactly over latent alignments
• Otherwise, approximation of your choice
![Page 22: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/22.jpg)
Experiment setup
Market rallies to new highs
![Page 23: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/23.jpg)
Sample predictions
Reference
Predicted
U.S. stocks end lower as economic worries persist[U.S. stocks end lower]2 [as economic worries persist]1
![Page 24: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/24.jpg)
A guessing game
Prediction accuracy0
0.2
0.4
0.6
0.8
0.5
0.59 0.61
0.72
BaselineNo alignmentFull modelHuman
![Page 25: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/25.jpg)
Peeking at parameters
sgn(slope) abs(slope)
rise
swoon
sharply
0.27
-0.57
-0.22
-0.78
0
0.28
![Page 26: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/26.jpg)
Following instructions…
and then we're going to turn north again
and immediat-- well a distance below that turning point there's a fenced meadow
but you should be avoiding that by quite a distance
okay so we've turned and we're going up north again
continue straight up north
and then we're going to turn to the west on a curvature right sort of
…
![Page 27: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/27.jpg)
Navigation results
Precision Recall F-measure0
0.1
0.2
0.3
0.4
0.5
0.6
BranavanVogelThis work
![Page 28: Grounding Language with Points and Paths in Continuous Spaces](https://reader035.fdocuments.us/reader035/viewer/2022062304/56813320550346895d99efde/html5/thumbnails/28.jpg)
Conclusions
• New model for predicting grounded representations of meaning in arbitrary real-valued spaces
• Beats strong baselines on a diverse range of tasks
• Code and data available online athttp://cs.berkeley.edu/~jda