On the Winograd Schema Challenge - Towards a New Test for Thinking Machines
Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry...
-
Upload
ethelbert-hawkins -
Category
Documents
-
view
218 -
download
3
Transcript of Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry...
![Page 1: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/1.jpg)
Michael BernsteinComputer Science
MIT
Jeff ShragerSymbolic SystemsStanford University
Terry WinogradComputer ScienceStanford University
TaskposéExploring Fluid Boundaries in an Associative Window Visualization
![Page 2: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/2.jpg)
http://blog.strawberryice.org.uk
Artifacts of information work
[Hutchings et al. 2004, Czerwinski et al. 2004, Gonzales & Mark 2004]
![Page 3: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/3.jpg)
http://flickr.com/photos/judxapp
Artifacts of information work
[Hutchings et al. 2004, Czerwinski et al. 2004, Gonzales & Mark 2004]
![Page 4: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/4.jpg)
Rooms [Henderson & Card 1986]
![Page 5: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/5.jpg)
+Task Gallery [Robertson et al. 2000]
![Page 6: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/6.jpg)
GroupBar [Smith et al. 2003]
![Page 7: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/7.jpg)
WindowScape [Tashman 2006]
![Page 8: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/8.jpg)
TaskTracer [Dragunov et al. 2004]
![Page 9: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/9.jpg)
Classification
?
assigning each window to a relevant task
Fire
fox
Mic
roso
ft
Exc
el
Fire
fox
iTu
nes
Fire
fox
AIM
Ou
tlook
Microsoft Word
![Page 10: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/10.jpg)
Classification Can Be The Wrong Model
When asked which task would be correct:“Users are often not 100% sure themselves or may provide different answers in different contexts. Users are often able to tell the system what it is not, but not what it is.”
[Stumpf et al. 2005], emphasis added
vs.
![Page 11: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/11.jpg)
“Buying a Birthday Gift”
![Page 12: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/12.jpg)
Classification
?
assigning each window to a relevant task
Fire
fox
Mic
roso
ft
Exc
el
Fire
fox
iTu
nes
Fire
fox
AIM
Ou
tlook
Microsoft Word
![Page 13: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/13.jpg)
Association
Fire
fox
Mic
roso
ft
Exc
el
Fire
fox
iTu
nes
Fire
fox
AIM
Ou
tlook
Microsoft Word
a continuous measure of two windows’ relatedness
?
![Page 14: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/14.jpg)
![Page 15: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/15.jpg)
![Page 16: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/16.jpg)
![Page 17: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/17.jpg)
related windows move near each other
![Page 18: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/18.jpg)
windows may belong to multiple groupings
![Page 19: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/19.jpg)
large thumbnails anchor more important windows
![Page 20: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/20.jpg)
laid out via a spring-embedded graph
![Page 21: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/21.jpg)
Values to Calculate
1. Window ImportanceWindowRank algorithm
2. Pairwise Window AssociationWindowRank-weighted switch ratios
![Page 22: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/22.jpg)
WindowRank Algorithm
AX
XY
NN YXSwitches
AXSwitchesXWindowRankAWindowRank
),(
),()()(1
…proportional to the number of switches X made to the window of interest.
For each other Window X,inherit X’s WindowRank…
WindowRank = 100
25% of switches
75% of switches
WindowRank += 25
WindowRank += 75
Measure of window importancePageRank algorithm run on a window switch graph
![Page 23: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/23.jpg)
Window Association AlgorithmSimple proof-of-concept association algorithmWeights window switch ratios by WindowRank
),(),(),(
)(
)(
),(
),(),(
},{
ABtioWeightedRaBAtioWeightedRaBAnAssociatio
XWindowRank
AWindowRank
XASwitches
BASwitchesBAtioWeightedRa
BAXX
Window A’s vote is the ratio of switches it made to B…
…proportional to its WindowRank when compared to B
![Page 24: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/24.jpg)
Field Study
10 undergraduate students were asked to use Taskposé one hour a day for one week on their main computer
Actual median usage was 40.8 hours, using Taskposé to switch windows 156 times
![Page 25: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/25.jpg)
Lessons Learned
General support for an association-based window switch visualization
Window importance tracking (6.0 / 7) and relationship tracking (5.5 / 7) are useful
Importance tracking is accurate (5.5 / 7)
but relationship tracking needs improvement (4.0 / 7)
![Page 26: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/26.jpg)
Future Work
Improve association algorithms via machine learning techniques such as distance metric learning
Design a one-dimensional visualization
Directly compare a classificatory visualization to an associative visualization
![Page 27: Michael Bernstein Computer Science MIT Jeff Shrager Symbolic Systems Stanford University Terry Winograd Computer Science Stanford University Taskposé Exploring.](https://reader033.fdocuments.us/reader033/viewer/2022052701/56649e255503460f94b1384e/html5/thumbnails/27.jpg)
TaskposéExploring Fluid Boundaries in an Associative Window Visualization
Special thanks to Todd Davies, Scott Klemmer, the SymbolicSystems Program and the Stanford HCI Group
Michael [email protected]