Recognising Situations in context aware systemsusing Dempster-Shafer Theory
Dr. Susan McKeeverNov 4th 2013
Context Aware systems – e.g. Smart home
• Sensors in a smart home• Situation tracking – what is the user doing? What
activity are they undertaking?• E.g Monitoring elderly
Context Aware systems• Pervasive /ubiquitious /ambient systems – embedded
in the environment s• E.g. intelligent homes, location tracking system
• They understand their own “context”.• Context-awareness is the ability to track the state of
the environment in order to identify situations
• Situations are human understandable representations of the environment, derived from sensor data
Research focus: e.g .Gator Tech Smart home
Van Kasteren sensored smart home
14 digital sensors For a month:
7 Situations:Preparing breakfast, dinner, drink, leave house, use toilet, take shower, go to bed
Abstracting sensor data to situations
Location sensor reading(X,Y,Z, ID239, 12:30:04)
Sensor 1, 2, 3
Abstracted Context
Situations
John located in Kitchen @ time 12:30
John is ‘preparing meal’
Is abstracted to
Is evidence of
Sensor 1, 2, 3Sensor 1, 2, 3
Application e.g. elderly alert system
Sensor dataSituation
Recognition
Situation(s) occurring at time, t
12:53 preparing breafast
(12:53, 0)(2.15,5.04,3.16, 12:34)
Situation Recognition
Knowledge• Expert? Past data?
• Situation recognition is a critical, continuous, dynamic process – often required in real time.
• The recognition process is difficult and uncertain – no single approach suitable for all
Situation Recognition - ScenarioScenario“The person is in the kitchen. It is morning time. They carry out a series of tasks, such as taking cereal out of the groceries cupboard, using the kettle, opening the fridge, and using the toaster”
Human Observer: “preparing breakfast”
Why?
•Individual tasks may not confirm that breakfast is in progress, but together, indicate the ’preparing breakfast’ situation.
•Morning time
•Informative sensors e.g. toaster
Recognising situations – Automated
Sensor overlap - Kettle and fridge: ’preparing drink?
Different people “prepare breakfast” in different ways.. Individual efinitions?
Gaps of seconds or minutes occuring with no sensor activity – classify?
Sensors can breakdown and have error rate – toaster sensor doesn’t fire?
As more tasks are done, system is more certain of ‘preparing breakfast situation’ – Temporal aspect
The person does not prepare breakfast in the same way every day.
The tasks are not necessarily performed in any particular order.
Co-occurring situations? (’on telephone’); Cannot o-occur (’user asleep’)? -Valid combinations of situations.
A second occupant now enters the kitchen – how to distinguish?
Recognising situations – Some approaches
• Machine learning techniques, inc.• Bayesian networks• Decision trees• Hidden Markhov modelsreliant on training data
• Specification based approaches, inc.• Logic approaches• Fuzzy logic• Temporal logic
Problems to be solved (not exhaustive)How to recognise situations in pervasive
environments, allowing for particular challenges:
1. Uncertainty (sensor data, situation definitions, context fuzziness)
2. Difficulties in obtaining training data
My solution: Use and enhance evidence theory (Dempster Shafer theory)
Why Dempster Shafer theoryDevised in 1970s
Mathematical theory for combining separate pieces of information (evidence) to calculate the belief in an event.
Applied in military applications, cartography, image processing, expert systems, risk management, robotics and medical diagnosis
Key features:(1) its ability to specifically quantify and preserve
uncertainty (2) its facility for assigning evidence to combinations
Various researchers applying in pervasive systems
Approach
• Apply Dempster Shafer (evidence) theory to situation recognition• Create a network structure to propagate
evidence from sensors
• Extend the theory to allow for:• New operations needed support evidence
processing of situation• Temporal features of situation• Rich (static and dynamic) sensor quality
Dempster Shafer theory: ExampleTwo sensors are used to detect user location in an office.
The locations of interest are: (1) Cafe, (2) the user’s desk, (3) the meeting room and (4)
‘lobby’ in the building.
Meeting roomCafé User’s desk Lobby
Sensor 1 Sensor 2
Any uncertainty is assigned to ‘ignorance’ hypthesis 𝞱– {desk ^ cafe ^ meetingRoom ^ lobby}
Frame of Discernment‘hypotheses’
(allows combinations)
Each sensors assigns belief as a ‘mass function’ which totals per sensor to 1
Evidence sources
Dempster Shafer theory: Example
Sensor 1Detects the user’s location in the cafe. The sensor is 70% reliable, so its belief is assigned across the frame as {cafe 0:7; 0:3 𝞱)Sensor 2
The second sensor has conflicting evidence, assigning{meetingRoom 0:2, desk ^cafe^lobby 0:6, 0:2 𝞱}To combine evidence source:Use dempster combination rule
mass functions
Dempster Shafer theory: Combination rule
M12 (A) is the combination of two evidence sources or mass functions for a hypotheses A.
Denominator is a normalisation factor 1-K where K = conflicting evidence
Evidence sources must sum to 1:
Dempster Shafer theory: example
Conflict (K ) = 0.14 ;
All evidence is normalised by 1-K giving:
Café 0.65; meeting 0.07; desk/café/lobby 0.21, uncertainty 0.07
Sensor 1
Sensor 2Combined evidence
Dempster Shafer theory: problems
Zadeh’s paradox
Conflicting sensor: Appear to agree completely if any agreement – not intuitive
Dempster Shafer theory: problems
Single sensor dominance
A single sensor can overrule a majority of agreeing sensors if it disagrees:
e.G .if 5 sensors determine a user location in a house, a single “categorical” (certain) sensor that assigns all its belief to a contradictory option will negate the evidence from the remaining 4.
Sensor 1 Sensor 2 Sensor 3 Sensor5Sensor 4
Kitchen 0.7
Kitchen 0.6
Kitchen 0.8
Kitchen 0.9
Sitting room
1
Dempster Shafer theory: gapsNo support for evidence spread over time.Assumes evidence is all co-occuring but in reality evidencemay be spread over time.
e.g. detecting “prepare dinner” situation detected by sensors on cupboards and fridges.
GroceriesCupboardAccessed
FridgeAccessed
Freezer Accessed
PansCupboardAccessed
PlatesCupboardAccessed
Prepare Dinner Timeline
40 minutes
Dempster Shafer theory: gapsOnly deals with fusing evidence: no “theory” for propogating evidence across other rules in order to recognise situations Limited to just combining n “sources”: Need a set of additional mathemtical operations for propogating evidence
Sensor 1, 2, 3
Abstracted Context
Situations
Sensor 1, 2, 3Sensor 1,
2, 3Sensor 1,
2, 3
Abstracted Context
Situations
Sensor 1, 2, 3Sensor 1,
2, 3Sensor 1,
2, 3
Abstracted Context
Situations
Sensor 1, 2, 3Sensor 1,
2, 3Location sensor reading(X,Y,Z, ID239, 12:30:04)
John located in Kitchen @ time 12:30
John is ‘preparing meal’
Is abstracted to
Is evidence of
sensor Sensor
ContextValue
situation SituationSituation
Sensor
ContextValue
ContextValue
ContextValue
ContextValue
ContextValue Context
Value
Certainty0.n
Certainty0.n
Certainty0.n
Sensor Level
Abstracted Context
Situations
sensor sensor sensor
situation situation
Dempster Shafer theory: gapsOnly deals with fusing evidence: no “theory” for propogating evidence across other rules in order to recognise situations (and a way to capture all this knowledge)
Recognising situations – Using Dempster Shafer theory
• Want an approach that reduces or eliminates reliance on training data. OK (provided we can define mass functions to say what sensor readings mean)
• That allows for “uncertainty” OK • That allows temporal information to be included To be added• That allows sensors belief to be propogated (distributed) up into
situation hierachies based on “knowledge” rules To be added• That addresses the issue of Zadeh’s paradox and dominant
sensors To be added• Ultimately: Develop a full decision making architecture for real
time situation recognition (overleaf) To be added
Needed to extend Dempster Shafer theory
Knowlege
Sensor Readings
Belief Distribution
DecisionStage
RecognisedSituations
Valid situationcombinations
At time t
Applicati-ons
Develop a full decision making architecture for real time situation recognition using extended DS theory
Extended DS theory
Prep Breakfast 0.3,Take a shower 0.6
Knowledge: an interconnected hierarchy of sensor and situations
sensor Sensor
ContextValue
situation SituationSituation
Sensor
ContextValue
ContextValue
ContextValue
ContextValue
ContextValue Context
Value
Certainty0.n
Certainty0.n
Certainty0.n
Sensor Level
Abstracted Context
Situations
sensor sensor sensor
situation situation
PlatesUsed
CupUsed
FridgeUsed
GroceriesUsed
MicrowaveUsed
Pans Used
FreezerUsed
GetDrink
PrepareBreakfast
PrepareDinner
<2> <15> <62>
0.80.2
0.8 0.8
0.4
0.8
Morning
PlatesCupboard
Cup Fridge GroceriesCupboard
Microwave Pans CupboadrFreezer TimeMoning
Nighttime
VanKasteren e.g. 3 of the situations
First : Define a notation for knowledge capture : denoting sensor evidence /context/ situations –
Situation DAG
sensor Sensor
Situation Situation
Situation
ContextValue
Certainty0.n
Certainty0.n
Certainty0.n
Discount0.n
< 5> > 10 >
ContextValue
ContextValue
ContextValue
ContextValue
ContextValue
Belief distribution
Situations
Sensors
Context Values
Belief distribution
First : Define a notation for denoting sensor evidence /context/ situations – Situation DAG i.e to capture the knowledge of what sensors indicate what situation
is a type of
is evidence of
< duration> Duration of situation, evidence not in sequenceDuration of situation, evidence in sequence
>duration > Sensor, context value or situation
Discount 0.n Discount factor applied to a sensor: 0< n <1
Certainty 0.n Certainty applied to an inference rule: 0 < n < 1
Second: Create evidence propogation rules to distribute/propogate belief up to situation level
sensor Sensor
ContextValue
situation SituationSituation
Sensor
ContextValue
ContextValue
ContextValue
ContextValue
ContextValue Context
Value
Certainty0.n
Certainty0.n
Certainty0.n
Sensor Level
Abstracted Context
Situations
sensor sensor sensor
situation situation
TranslateSensor readings into beliefs here ..
Up to situation certainties here
Second: Create evidence propogation rules to distribute/propogate belief up to situation level
sensor Sensor
ContextValue
situation SituationSituation
Sensor
ContextValue
ContextValue
ContextValue
ContextValue
ContextValue Context
Value
Certainty0.n
Certainty0.n
Certainty0.n
Sensor Level
Abstracted Context
Situations
sensor sensor sensor
situation situation
Is a type of:
e.g. Situation X is occuring if either Situation Y OR Z is occuringOccupant is “resting” if they are “watching TV” or “in bed”
Second: Create evidence propogation rules to distribute/propogate belief up to situation level:
Examples
Distributing combined belief across single situations
Second: Create evidence propogation rules to distribute/propogate belief up to situation level:
Examples: Sensor QualitySome sensors are inherently lower quality as an evidence source
e.g. Calendar sensor is indicative of real calendar owner’s location 70% of the time – Discount (d) evidence from the sensor
Third: Include temporal evidence:
GroceriesCupboardAccessed
GroceryCupboardaccessed
Freezer Accessed
PlatesCupboardAccessed
FridgeAccessed
Prepare Dinner Timeline40 minutes
Different Sensors fire intermittently – no single sensor sufficient for situation recognition
(1) Use absolute time as evidence(2) Find a way to combine transitory evidence
GroceriesCupboardAccessed
FridgeAccessed
Freezer Accessed
PansCupboardAccessed
PlatesCupboardAccessed
Prepare Dinner: Time Extended Evidence Time
FridgeExtended
FridgeExtended
FridgeExtended
FridgeExtended
FridgeExtended
GroceriesCupboardExtended
GroceriesCupboardExtended
GroceriesCupboardExtended
GroceriesCupboardExtended
PlatesCupboardExtended
PlatesCupboardExtended
PlatesCupboardExtended
Freezer Extended
Freezer ExtendedPansCupboardExtended
Prepare DinnerStarts
Prepare BreakfastEndsSituation
Duration
Third: extend evidence for duration of situation
Fusing time extended evidence:
Adjust Dempster Shafer fusion rules to allow for time extension of evidence
Two transitory extended mass functions for hypothesis h with duration t dur, a t time t +t rem
Fourth: Allow for Zadeh’s and Single sensor dominance
Use an alternative combination rule (Murphy’s) which averages out the evidence BEFORE fusing
Use a simpler averaging rule to fuse evidenceLacks convergenceRemoves Zadeh’s problem
Two options:
Fifth: Combine all this and apply to real world data for situation recogntion
Knowlege
Sensor Readings
Belief Distribution
Decision
StageRecognisedSituations
Valid situation
combinations
At time t
Applicati-ons
Extended DS theory
Prep Breakfast 0.3,Take a shower 0.6
Test our approach using annotated datasets of sensor readings
ExperimentsData set (1) “Van Kasteren”Heavily used by other researchers - compare results on situation recognition 7 situation annotated, 14 sensors
Data set (2)“CASL”Office data set: 3 situations annotated, •Location sensors, •Calendar sensor, •Keyboard sensor
Question Data set1 How accuracy is our DS
approach for situation recognition?
Both
2 Do DS temporal extensions improve situation recognition?
Van Kasteren
3 Do DS quality extensions improve situation recognition?
CASL
Evaluation
Various sub questions also addressed: comparison with published results, comparison of DS fusion rules, impact of quality on situation transitions, quality parameter sensitivity, static versus dynamic quality
Evaluation1. 2 annotated published real world datasets –
VanKasteren (Smart home) and CASL (office-based)
2. Situation DAGs created for both datasets
3. Situation recognition accuracy measured using f-measure of timesliced data sets;
4. Recognition accuracy using temporal and quality extensions evaluated
5. J45 Decision Tree and Naive Bayes used for comparison , and published results ; Cross validation used.
leave house
use toilet take shower
go to bed prepare breakfast
prepare dinner
get drink0.00
0.20
0.40
0.60
0.80
1.00
No time Absolute time Time Extended
Use of DS theory with temporal extensions for situation recognition
F-Measure for each situation using DS theory – (1) no time, (2) absolute time, (3) time extended (VanKasteren dataset )
Temporal DS theory compared to two other approches: Naïve Bayes, J48 decision tree.
leave house
use toilet take shower
go to bed prepare breakfast
prepare dinner
get drink0
0.2
0.4
0.6
0.8
1
No time EDN Temporal EDN Naïve Bayes J48
Situations
Our approach compared to the three available published results
Same experimental measures
* Excludes timeslices with no sensors firing which are harder to infer – ‘inactive’ Timeslices harder to infer
*
Use of DS theory with temporal extensions • Use of temporal extensions significantly
improves situation accuracy (over baseline DS theory alone)
• Performs better than J45, Naive Bayes (particularly with limited training data). This improvement narrows when more training data used (LODO)
• Achieves 69% class accuracy in comparison to VanKasteren (49.2%) and Ye*(88.3%)
Use of DS theory with quality extensions
0.00
0.20
0.40
0.60
0.80
1.00
No Quality With Quality
F-Measure for each situation using DS theory – with and without quality
• Use of quality parameters significantly improves situation recognition accuracy (over baseline)
• Performance close to Naive Bayes (4%) and J48 (2%) -
• Each individual sensor’s quality contributes to improvement
• Sensitivity analysis of quality parameters indicates the relative quality of sensors may be important
• Time based dynamic quality parameters impact situation transitions – application dependant
Use of DS theory with quality extensions
Our DS theory is a viable approach to situation recognition:• Not reliant on training data• Incorporates domain knowledge• Caters for uncertainty• Encoding temporal and quality knowledge improves
performance over basic DS approachBUT
• Knowledge must be available• Different fusion rules appropriate in different scenarios –
requires expert “evidence theory” knowledge• Environment changes – no feedback loop for drift• Potentially high computation effort can be reduced
Conclusions
Contributions1. A situation recognition approach based on DS
theory 2. Selection of existing and creation of new
evidential operations and algorithms to create evidence decision networks
3. Temporal and quality extensions to DS theory 4. Diagramming technique to capture structure of
evidence for an environment (Situation DAG)5. A thorough application, evaluation and analysis
of the extended DS theory approach6. An analysis of alternative fusion rules
Related Publications• Journal
1. Journal of Pervasive and Mobile Computing2. JAISE Volume 2, Number 2 2010
• International Conferences1. EuroSSC Smart Sensing UK 20092. ICITST Pervasive Services Italy 2008
• International (Peer viewed) Workshops1. Pervasive 2010, Helsinki, Finland2. CHI 2009 Boston, US3. QualConn 2009, Stuttgart, Germany4. Pervasive 2009, Sydney, Australia,
Questions?
ExperimentsEstablish situation DAG for each dataset
SystemDevelopers
-Users-Application
experts
Sensors
ContextValues
Situations
Top Related