Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident...
Transcript of Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident...
![Page 1: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/1.jpg)
![Page 2: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/2.jpg)
![Page 3: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/3.jpg)
Computation & memory
Connectivity & data
Learning & reasoning prowess
Exciting Times
Opportunities & directions
![Page 4: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/4.jpg)
(I. Beinlich, et al)
CATECHOLAMINE
ANESTHESIA INSUFFICIENT
HYPOVOLEMIA
CARDIAC OUTPUT
STROKE VOLUME
HR BP HR EKG HR SAT
LV FAILURE
PCWP
HEART RATE
ERROR LOW OUTPUT
ERROR CAUTER
TPR
BLOOD PRESSURE
ANAPHYLAXIS
SAO2
ARTERIAL CO2
DISCONNECTION KINKED TUBE INTUBATION
PRESSURE
VENT TUBE VENT LUNG
FIO2
EXPIRED CO2
PA SAT
SHUNT
MINUTE VENTILATION
PAP
PULMONARY EMBOLUS
VENT MACHINE
MV SETTING
HISTORY LV FAILURE
CVP
LVED VOLUME
VENT ALV
Advances in Capturing Expertise
![Page 5: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/5.jpg)
*
![Page 6: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/6.jpg)
Best actions under uncertainty
Data Prediction Decisions
Case library
Decision Model
Decisions
Predictive Model
Predictions
Data
![Page 7: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/7.jpg)
Ambient, “in-stream” data resources
Exciting Directions
Example: Lac Kivu earthquake, Congo
Rwandan call densities: 6 days, 140 cell towers, 10.5m calls
with A. Kapoor, N. Eagle
![Page 8: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/8.jpg)
⋯ ⋯
⋯ ⋯
ℎ1 ℎ2 ℎ𝑗 ℎ𝐽 1
𝑣1 𝑣2 𝑣𝑖 𝑣𝐼 1
Exciting Directions
⋯ ⋯ ℎ1 ℎ2 ℎ𝑗 ℎ𝐽 1
𝑣1 𝑣2 𝑣𝑖 𝑣𝐼 1
⋯ ⋯ 𝑙1 𝑙2 𝑙𝑗 𝑙𝐽
𝑣1 𝑣2 𝑣𝑖 𝑣𝐼 1
Causality
Active learning
Lifelong learning
Deep learning
![Page 9: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/9.jpg)
Learning & Inference in the World
Four efforts
• Transportation
• Healthcare
• Citizen science
• Integrative AI
![Page 10: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/10.jpg)
Transportation
Heterogeneous data sources
User models
![Page 11: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/11.jpg)
Event store
Multiple views on traffic
Operator ID: Nick
Heading: INCIDENT
Message: INCIDENT
INFORMATION
Cleared 1637: I-405 SB
JS I-90 ACC BLK RL CCTV
1623 – WSP, FIR ON SCENE
Incident reports Weather
Major events
Fusion of Heterogeneous Evidence
Day & time
Road properties
& topology
with J. Apacible, P. Koch, J. Krumm, P. Newson, R. Sarin, S. Srinivasan, M. Subramani,
![Page 12: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/12.jpg)
Max likely time jam will last
System’s confidence
Traffic prediction service Core predictions
Predicting Future Flows
• System-wide status & dynamics
• Incident reports
• Major events
• Weather
• Day & time
• Road properties & topology
• Holiday status
• Event store
• Learning
• Inference
![Page 13: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/13.jpg)
E1 E2 E3
H1 H2
E4
World model
E2 E3
H1 H2
E4
User model
Beyond Domain Focus: Models of User
Extend system with model of user’s knowledge
![Page 14: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/14.jpg)
Models of Surprise
Learn what surprises people
…now and in the future
Expectations
Real
World
Outcome
Database of surprising events
Data store
Future traffic
Pro
bability
! ! • System-wide status
& dynamics
• Incident reports
• Sporting events
• Weather
• Time and day
• Topology
• Road properties
• Holiday status
Events at t
Human
Forecaster
Major events
Weather
Time and day
Holiday status
Events at t-T
Machine
learning
![Page 15: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/15.jpg)
Traffic prediction service Core predictions
Integrating Surprise Forecasting
• System-wide status & dynamics
• Incident reports
• Major events
• Weather
• Day & time
• Road properties & topology
• Holiday status
• Event store
• Learning
• Inference
Surprise forecasting models
![Page 16: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/16.jpg)
GPS library
~1,000,000 km
~100,000 trips
Extend Predictions to Unsensed Roads
![Page 17: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/17.jpg)
Clearflow maps.bing.com * Windows phone
72 cities across North America
Flows assigned to ~60 million streets every few minutes
t
t t
![Page 18: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/18.jpg)
Clearflow
maps.bing.com * Windows phone
![Page 19: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/19.jpg)
Healthcare
High-stakes challenges
Working across cultures
Coupling prediction & decision
with M. Bayati, M. Braverman, R. Caruana, J. Gatewood, P. Koch, M. Smith, J. Wiens
![Page 20: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/20.jpg)
~20% within 30 days
~35% in 90 days
Estimated cost to Medicare in 2004:
$17.4 billion
Costly Challenge
![Page 21: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/21.jpg)
Learning from a Case Library
• Large hospital in Wash DC
• All visits during the years 2001 to 2009 (e.g., ~300,000 ED visits)
• Admissions, discharge, transfer (ADT)
• Chief complaint in free text
• Age, gender, demographics
• Diagnosis codes (ICD-9)
• Lab results and studies
• Medications
• Vital signs
• Procedures
• Admitting and attending MD codes
• Fees and billing
~25,000 variables considered in dataset
![Page 22: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/22.jpg)
Graphical Model for Readmission
![Page 23: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/23.jpg)
Performance of Classifier
False positive rate
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Tru
e p
os
itiv
e r
ate
Train
Test
![Page 24: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/24.jpg)
Identifying Discriminatory Evidence
Weight Feature description Frequency 0.68398 Dx0->2 = Excessive vomiting in pregnancy 0.31%
0.61306 Dx3->2 = Personal history of malignant neoplasm 0.28%
0.58281 Dx0->2 = Heart failure 0.30%
0.56708 Dx0->1 = Nephritis, nephrotic syndrome, and nephrosis 0.09%
0.56649 Dx3->2 = Heart failure 0.28%
0.54663 Complaint sentence contains ''suicidal'' 0.17%
0.48415 Dx1->2 = Disorders of function of stomach 0.07%
0.47257 Dx5->0 = Diseases Of The Genitourinary System 0.15%
0.46136 Dx0->2 = Chronic airway obstruction, not elsewhere classified 0.10%
0.44555 Dx4->2 = Depressive disorder, not elsewhere classified 0.10%
0.44257 Stayed 14 hours in the ER 0.10%
0.43890 Dx0->1 = Other psychoses 0.32%
0.43513 Dx0->0 = Diseases Of The Blood And Blood-Forming Organs 0.46%
0.42582 Complaint sentence contains ''dialysis'' 0.19%
0.41888 Dx0->2 = Depressive disorder, not elsewhere classified 0.27%
0.41302 Dx1->1 = Nephritis, nephrotic syndrome, and nephrosis 0.29%
0.38506 Complaint sentence contains ''fluid'' 0.10%
0.37474 69 < Age 9.22%
![Page 25: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/25.jpg)
Multiple Predictions
• ED discharge Inpatient within 72 hours
• Inpatient discharge Inpatient within 30 days
• CHF discharge CHF inpatient within 30 days
• Death within 30 days
• Inpatient infection within 48hrs, 72hrs, stay
C.Difficile, MRSA, VRE
Team
M. Bayati, M. Braverman, E. Horvitz, P. Koch, P. Oka, J. Wiens , N. Donegan, L. Pic-Aluas, G. Ruiz, M. Smith
![Page 26: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/26.jpg)
New Kinds of Models: Predict Surprises
Predict readmission surprises:
“The patient you’re discharging will likely return within 3
days with a 10 diagnosis that is not currently on the chart.”
![Page 27: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/27.jpg)
Translation: Research Open World
![Page 28: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/28.jpg)
Engineering for Tractability and Availability
![Page 29: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/29.jpg)
Predictive platform goes live…
![Page 30: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/30.jpg)
Data Predictions Decisions
Will this patient
bounce back?
Invest in aggressive
outpatient follow up?
? Predictions
Data
Decisions
? F-
F+
p(B|E)
![Page 31: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/31.jpg)
Expected Value of Fielding System for Population
Data Recommended action Learning / Prediction
Inferences E1
.
.
.
.
En
Train
Test
$ D readmission
?
![Page 32: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/32.jpg)
Congestive Heart Failure: Train: 2004-2007 / Test: 2008 E
ffic
acy (
% r
ed
uctio
n)
Cost of intervention ($)
70% - 80%
60% - 70%
50% - 60%
40% - 50%
30% - 40%
20% - 30%
10% - 20%
0% - 10%
Expected Value of Fielding System for Population
![Page 33: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/33.jpg)
Citizen Science
Human + machine intelligence
Multiple roles of machine intelligence
![Page 34: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/34.jpg)
Zooniverse: Classification & discovery in astronomy
Sloan Digital Sky Survey:
~106 galaxies, ~120k quasars, ~225k stars
Citizen Science
Galaxy Zoo
View & classify galaxies online
886k galaxies, 34m votes, 100k participants
with E. Kamar, S. Hacker, P. Koch, C. Lintott, A. Smith
![Page 35: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/35.jpg)
Classification & Discovery in Astronomy
![Page 36: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/36.jpg)
Classification & Discovery in Astronomy
![Page 37: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/37.jpg)
Classification & Discovery in Astronomy
![Page 38: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/38.jpg)
Classification & Discovery in Astronomy
![Page 39: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/39.jpg)
Sloan Digital Sky Survey: Image Analysis
453 features
![Page 40: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/40.jpg)
Fuse human & machine perceptual effort
Mesh Human & Machine Intelligence
Machine learning, prediction, action
Machine
perception
Human
perception
Optimize task routing & stopping
![Page 41: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/41.jpg)
CrowdSynth & Zcion
Machine learning, prediction, action
Machine
perception
Human
perception
![Page 42: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/42.jpg)
~450 features
Machine learning, prediction, action
Machine
perception Human
perception
E. Kamar, S. Hacker, P. Koch, A. Smith, C. Lintott, H.
Ideal fusion, routing, stopping
CrowdSynth & Zcion
![Page 43: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/43.jpg)
label
𝑓𝑖𝑚𝑎𝑔𝑖𝑛𝑔
𝑓𝑤𝑜𝑟𝑘𝑒𝑟𝑠 𝑓𝑣𝑜𝑡𝑒𝑠
Predict Next Votes & Ground Truth
Answer Next vote
![Page 44: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/44.jpg)
Predict Next Contributions & Ground Truth
All votes
Crowdsynth
![Page 45: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/45.jpg)
Learn about Abilities & Engagement
Computer Vision
Activity
Correct answer
Experience
![Page 46: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/46.jpg)
Learn about Abilities & Engagement
Computer Vision
Correct answer
Experience
Activity
![Page 47: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/47.jpg)
Dreams of Richer Machine Intelligence
with D. Bohus, P. Choudhury, R. Hughes, E. Kamar, P. Koch, S. Rosenthal, N. Saw, A. Thompson, W. Wang
Intelligence via composition
Principles of situated sensing & action
![Page 48: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/48.jpg)
• Leveraging tapestry of components
• Understanding synergies & dependencies
• Whole more than sum?
Integrative AI: Intelligence via Composition
Learning Inference
NLP
Vision
Speech recognition Planning
Motion control
Speech generation
Localization
Whole >> i part i ?
![Page 49: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/49.jpg)
Situated Interaction Project
![Page 50: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/50.jpg)
![Page 51: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/51.jpg)
Models of Multiparty Collaboration
shuttle
![Page 52: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/52.jpg)
Models of Multiparty Collaboration
What can I do
for you?
Shuttle?
![Page 53: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/53.jpg)
Models of Multiparty Collaboration
Are the two of
you together?
Shuttle?
![Page 54: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/54.jpg)
system
user
active interaction
suspended interaction
other interaction
1
1
t1 t2
1
t3 t4
1
t5
2
1
2
t6 t7 t8
1
2
t9
1
2 3
t10 t11
1
t12
1
t13 t14
Active Suspended
Engage({1},i1)
Maintain({1},i1)
Engage({2},i1)
Engage({1,2},i1)
Disengage({1,2},i1)
Engage({3},i2) Maintain({3},i2) Disengage({3},i2)
Engage({1,2},i1)
Maintain({1},i1)
Disengage({1},i1)
Active
Active
Active
Track conversational dynamics
Time-critical turn-taking decisions
Contributions and Turns in the Open World
![Page 55: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/55.jpg)
wide-angle camera
4-element microphone array
touch screen
card reader
speakers
Speech
Synthesis
Output
Management
Avatar
Synthesis
Behavioral control
Dialog and interaction planning
Tracker Speech
Recognition
Conversational
Scene
Analysis
Machine learning about interaction
Models of user frustration, task time
Receptionist
Composing and Exercising a Platform
![Page 56: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/56.jpg)
Composing and Exercising a Platform
![Page 57: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/57.jpg)
![Page 58: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/58.jpg)
Multiple Experiments & Refinements
![Page 59: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/59.jpg)
P: arrow indicates
direction of
attention
P: P is an addressee
P: P has floor
P: P is the target of
the floor release
P: P is speaking
P: P is releasing the
floor
P: P is trying to take
the floor (performs
TAKE action)
indicates system’s
gaze direction
Multiple Experiments & Refinements
![Page 60: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/60.jpg)
Studies in the Wild
![Page 61: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/61.jpg)
Current Focus: The Assistant
![Page 62: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/62.jpg)
Presence & Availability Predictor
Awareness & Coordination
Multiparty Engagement & Dialog
Multiple components
Perception, learning, reasoning
The Assistant
![Page 63: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/63.jpg)
Coordinate: Presence & Availability Forecasting
![Page 64: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/64.jpg)
BusyBody: Attention & Interruption
![Page 65: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/65.jpg)
![Page 66: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/66.jpg)
New Methods and Principles
𝑡2
A
B
C
D
0.75
0.25
0.75
0.25
0.75
0.25
0.9
0.7
0.5
Learning & inference from real-time streams
Information value in streaming settings
Time-critical tradeoffs
![Page 67: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/67.jpg)
![Page 68: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/68.jpg)
Clarity, preferences, and handles
Decision-theoretic mediation
Differential privacy
Protected sensing & personalization
Privacy, Data, and Machine Learning
…and optimism Urgency
![Page 69: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/69.jpg)
Documents
Web activity
GPS, wifi
. . .
Results from web search engine
Personalized
ranker
Content & activities
store
Example: PSearch
With J. Teevan and S. Dumais
![Page 70: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/70.jpg)
Time Images & videos
Appts & events
Desktop & search activity
Whiteboard capture
Locations
Example: Lifebrowser
![Page 71: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/71.jpg)
Applications of sensing, learning, and reasoning still in infancy
Studies and themes
Unprecedented value to people and society
Principles Applications Principles …
Summary
![Page 72: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/72.jpg)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it
should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
![Page 73: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/73.jpg)
![Page 74: Computation & memory Learning & reasoning prowess · • System-wide status & dynamics • Incident reports • Sporting eventsMajor events • Weather • Time and day • Topology](https://reader031.fdocuments.us/reader031/viewer/2022031509/5caadc0188c993e3548c6d37/html5/thumbnails/74.jpg)