SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.
-
date post
22-Dec-2015 -
Category
Documents
-
view
219 -
download
1
Transcript of SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.
![Page 1: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/1.jpg)
SAMI: Situational Awareness from Multi-
modal Input
Naveen Ashish
![Page 2: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/2.jpg)
Talk Organization Why are we at RESCUE interested ? Situational Awareness (SA)
– Introduction System architecture Research challenges Expected outcomes and artifacts Extraction system demonstration
![Page 3: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/3.jpg)
Team
Naveen AshishSharad MehrotraNalini VenkatasubramanianUtz WestermannDmitry KalashnikovStella ChenVibhav GogatePriya GovindarajanRam HariharanJohn HutchinsonYiming MaDawit SeidJay LickfettChris DavisionQuent Cassen
Bhaskar RaoMohan TrivediRajesh HegdeSangho ParkShankar Shivappa
Ron EguchiMike Mio
Jacob Green
![Page 4: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/4.jpg)
Information from Various Sources
People/Victims at disaster
Emergency responders
News, video, audio footage
GIS, satellite imagery, maps
Pushing “Human-as-sensor”
![Page 5: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/5.jpg)
More Data ≠ More Information
SA
Where are the fire personnel ?Have all medical supplies reached ?What areas should we start evacuating first ?
![Page 6: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/6.jpg)
Situational Awareness Wide variety of fields
– Beginning in mid-80s, accelerating thru 90s– Fighter aircraft, ATM, Power plants,
Manufacturing Definitions
– "the perception of elements in the environment along with a comprehension of their meaning and along with a projection of their status in the near future"
– "the combining of new information with existing knowledge in working memory and the development of a composite picture of the situation along with projections of future status and subsequent decisions as to appropriate courses of action to take"
Situational awareness and decision making Areas
– Cognitive science– Information processing– Human factors
Knowing what is going on
![Page 7: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/7.jpg)
Abstraction of Information
Multimodal Input: Text, Audio, Video
Events
Awareness
![Page 8: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/8.jpg)
First-cut Architecture
EVENT BASE
Qu
ery
ing
an
d
A
na
lysi
s
Graph View
VISUALIZATION and USER INTERFACES
SpatialIndexingPDF Histogram
KNOWLEDGE: ONTOLOGIES
Text
Audio
Video
Internet
RAW DATA
EVENT EXTRACTION
REFINEMENTDisambiguationLocation
Centered around EVENTS as fundamental abstractions
![Page 9: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/9.jpg)
Research Areas
Event ModelingEvent Extraction
Disambiguation
Location Uncertainty
Graph Analysis
GIS Querying
![Page 10: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/10.jpg)
Event Modeling What is an event ? Event Representation
RELIABILITY
PEOPLE EVACUATION
LOCATION
TIME
REPORT
TYPE
NAME LOCATION
AGENCY
FROM TO
OPERATION
NUMBER
![Page 11: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/11.jpg)
Domain Knowledge
Captured as Ontologies
ROAD EVACUATION
EVACUATION
AIREVACUATION
IS-A
IS-A
THAILAND
SOUTHERN REGION
…….
PHUKETPHUKET,CHANGWAT
![Page 12: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/12.jpg)
Event Extraction Long history of information extraction
– IR (MUC efforts)– Web data extraction
DARPA ACE– Entities, Relations, Events– Events in 2004
Event extraction accuracy is still low SA Domain
– Stream of information– Duplicated, ambiguous– Reliability– Conversations
Modalities– Text
![Page 13: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/13.jpg)
Semantics Driven Approach
Semantics Driven Challenges
– Framework– Ontologies
What semantics required for event extraction ? Application
With NLP, ML techniques Performance
– SA specific Duplicates, reconciliation, temporal,
conversations …..
![Page 14: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/14.jpg)
Disambiguation
![Page 15: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/15.jpg)
Disambiguation
![Page 16: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/16.jpg)
– point-location
in terms of landmarks
uncertain, not (x,y)
– reasoning on such data support all types of queries
Uncertainty is a Challenge
Report 1: “... a massive accident involving a hazmat truck on I5-N between Sand Canyon and Alton Pkwy ...”
Report 2: “... a strange chemical smell on Rt133 between I405
and Irvine Blvd ...”
Report 1 R
epor
t 2
![Page 17: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/17.jpg)
Implications of Uncertainty in Text How to model uncertainty?
– probabilistic model– P(location | report)
e.g. report says “near building A”
Queries– cannot be answered exactly...
use probabilistic queries all events: P(location R | report)
> 0– SA requirements
triaging capabilities fast response
– top-k – threshold: P(location R | report) >
-RQ, k-RQ, k -RQ
How to map text to probabilities?– use spatial ontologies
AB
R
![Page 18: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/18.jpg)
Graph Analysis
GAAL Inherent spatio-
temporal properties Graphs are powerful
for querying and analysis
![Page 19: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/19.jpg)
Current FGDC Search
GIS Search
![Page 20: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/20.jpg)
Progressive Refinement of Data
GIS Search
![Page 21: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/21.jpg)
Deliverables, Outcomes, Artifacts “Vertical” thrusts
– Event extraction system (TEXT)– Disambiguation system– GIS search system
Overall system demonstration ? “By-products”
– Ontologies Computer science research areas
DatabasesSemantic-Web
Information Retrieval
Intelligent Agents (AI)
![Page 22: SAMI: Situational Awareness from Multi-modal Input Naveen Ashish.](https://reader035.fdocuments.us/reader035/viewer/2022081513/56649d805503460f94a6498a/html5/thumbnails/22.jpg)
Thank you !
http://sami.ics.uci.edu