Context-Awareness based on Lifelog

Context-Awareness based on Lifelog

Sangkeun Lee

Intelligent Database Systems Lab.

Seoul National University

Copyright 2009 by CEBT

Introduction

The whole story begins with my survey ‘A Survey of Context-Aware Systems’

Context & Context-awareness

General Process in Context-Aware Systems

History of Context-aware Systems

– Generations of Context-aware Systems

– Trend Analysis

Let’s briefly revisit the survey first

2


Context & Context-aware Systems

Context

Any information that can be used to characterize the situation of entities relevant to the interaction between a user and an application (Dey and Abowd ,2001)

Context-aware Systems

The systems that use context to provide relevant information and/or services to the user, where relevancy depends on the user’s task (Dey and Abowd ,2001)

3


General Process in Context-Aware Systems

Generally, processing context-awareness can be summarized into following four steps

Sense

– Getting values from context sources (e.g. sensors)

– Context-sources doesn’t have to be hardware devices

– (e.g. virtual sensors, other applications)

Abstraction

– Context Aggregation Temperatures of previous 10 minutes -> Very Cold

– Context Interpretation GPS signal -> City name

(Recognize)

– Understanding current context & trigger the most suitable action

Action

– Better user’s experience

4


Trend Analysis

5


Insights from the Survey

What makes context-Awareness so difficult?

Realizing useful Application/Domain Specific context-aware systems is not a big problem

– Using simple, non-flexible context models

– In a restricted & known scenarios (e.g. museum, …)

– Not a ‘true’ context-awareness for a ubiquitous environment

However, realizing a flexible and general context-aware system is a big problem

– If you use ontological context model,

then you get flexibility and expressiveness of representing context information,

but recognizing (understanding) the represented context data causes computational costs, less scalability – not practical!

6


Approach to achieve feasibility and flexibility at the same time?

As I mentioned earlier, context-aware systems are

The systems that use context to provide relevant information and/or services to the user, where relevancy depends on the user’s task (Dey and Abowd ,2001)

Recommender system

– Recommender systems or recommendation engines form or work from a specific type of information filtering system technique that attempts to present information items that are likely to be of interest to the user. (Wikipedia)

– Similar ! but there are many working systems They use user’s ratings, logs, previous histories, ….

By getting hints from recommender systems that are practically used in many domains, we present a practical way of processing context-awareness by separating ‘Recognize’ step into two steps ‘Collect’ & ‘Match’

Existing approach : Sense - Abstraction – Recognize – Action (Analytic)

Practical Approach : Collect - Sense – Abstraction - Match – Action (Empiric)

7


Profile

Query

Context

t1

t2

t3

t4

t5

t2

t4

t6

Informa-tion

t7

t8

t9

Service

t10

t11

t12

Sense

Practical Context-awareness –an Empiric Approach

Match

Collect


Practical Context-awareness –an Empiric Approach

Is it reasonable to do this? Yes!

– Context history can be used to establish trends and predict future context values. [Anind K. Dey and Gregory D. Abowd, 2001]

– “Context history is generally believed to be useful, but it is rarely used” [Guanling Chen and David Kotz, 2001]

– “Context histories may be used to establish trends and predict future context values.” [Matthias Baldauf, 2007]

Many systems keeps recording context history (although they do not utilize them)

– Context Toolkit, CoBrA, CASS, SOCAM and CORTEX save sensed context data persistently in a database.

9


Collect & Match

My research consists of two parts

Collect

– Sense and store context histories

– LifeLogOn: Log on to Your Lifelog Ontology! (ISWC ‘09)

– Entity-Event Lifelog Ontology Model (EELOM)for Lifelog Ontology Schema Definition (APWEB ‘10)

– LifeLogOn: A Practical Lifelog System for Building and Exploiting Lifelog Ontology (UMC ‘10)

Match

– Find the most suitable information/services based on current context & context histories (logs) using available techniques such as Collaborative filtering, Context-aware collaborative filtering, contents based matching,

Simple If-then rules, many other heuristics

– A flexible framework for context-aware matching is needed

10


Collect – LifeLogOn: A Lifelog System

Integrate currently available logs from different devices and create semantic relationships among logs

Transforms relational log data and metadata into instance-level Ontology and stores in knowledge base

Music Listening LogsMovie Watching Logs

GPS LogsPhone Call LogsE-mail History

Schedules

Music Listening Logs

Movie Watching Logs

GPS LogsPhone Call Logs

E-mail History Schedules

* Sematnic Relationships

LifeLogOn


Collect – ‘Entity-Event Lifelog Ontol-ogy Model’

EELOM consists of domain, entity and events A domain is a group of entities and events, and it has a domain name for representing it.

An entity is a thing which has a separate existence and can be uniquely identified (e.g. a person, a location, a device, a timestamp) An entity is represented as a set of attributes. Entities can be shared among domains.

An event is an activity or interaction that can be uniquely identified. It is represented as a set of en-tities, and each entity has its role for the event.

Why not RDF? OWL?

- It does not allow complex relationships (e.g. hierarchy of entities),- but it is still flexible and expressive

enough to cover logs of many domains

- Easy to construct index structure by considering everything as entities & events


Collect – LifeLogOn Overview

Log ID E-mail Time Date Event

453 2412 1:11PM 09/03/21 Played

454 2151 12:06AM 09/04/22 Played

455 2151 12:10AM 09/04/22 Stopped

Title Genre Artist BPM Tags

453 2412 1:11PM 09/03/21 Played

454 2151 12:06AM 09/04/22 Played

455 2151 12:10AM 09/04/22 Stopped

Music Domain

Music

Atmosphere

Album Genre

Artist

BPMTitle

Play

User

Time DateOther

Context

MusicLocation

Stop

User

Time DateOther

Context

MusicLocation


453 2412 1:11PM 09/03/21 Played

454 2151 12:06AM 09/04/22 Played

455 2151 12:10AM 09/04/22 StoppedTitle Genre Artist BPM Tags

453 2412 1:11PM 09/03/21 Played

454 2151 12:06AM 09/04/22 Played

455 2151 12:10AM 09/04/22 Stopped


453 2412 1:11PM 09/03/21 Played

454 2151 12:06AM 09/04/22 Played

455 2151 12:10AM 09/04/22 Stopped

Last.fm Music Log

iTunes Music Log

AMG Metadata

CDDB Metadata

GPS Log from Cell phone

Temperature Log from Forecast.com

Music@1Groove

Hyori’s back KpopHyori

9010 minute

Play@1

Location@1Korea

Kangwon

12:31:42Chunchon

User@1

Male

Hiphop Seoul,KoreaSinging

27Matt

Time@130

1202

Play@2Time@212

901

Location@2Korea

Seoul

55:24:66Seoul

Music@2Bright

Yellow Submarine OldpopBeatles

110 Yesterday

take@1Photo@1SonyCD2

640KB

Home.jpg640x480

LifeLogOn Instance-Level Ontology GeneratorLifeLogOn Instance-Level Ontology Generator

Log–Ontology Schema Mapping

OntologySchema

OntologyGenerator

OntologyInstances

RelationalLogData

Ontology Schema Definition Tool

Ontology Schema Definition Tool

Visualization & Browsing ToolVisualization & Browsing Tool

Log to Ontology Mapper

Log to Ontology Mapper


What can we use LifeLogOn for?

You can create your own lifelog ontology without understanding any ontology languages

You can search something not sure about Find a song you listened at your birthday

party and but only know the filename of the photos at the party

Find photos that you took when you are talking with your friend on the phone, saying "It's so beautiful here!"

Find any events and entities those you know their context information such as time, location,...

... even more

14


We have the context history collec-tion…

15

Many issues remain- Building a lifelog ontology using real data set- Implementing Loggers (maybe on iPhone, …)- Log Ontology Abstraction/Summarization- Better Visualization- Entity Matching, …


Profile

Query

Context

t1

t2

t3

t4

t5

t2

t4

t6

Informa-tion

t7

t8

t9

Service

t10

t11

t12

Sense

Let’s Get Back to Context-awareness

Match

Collect


Paper Reading

Context-Aware Collaborative Filtering System: Predicting the User’s Preference in the Ubiquitous Computing Environment - Annie Chen, 2005

predicts a user’s preference in different context situations based on past experiences

시간 상황 정보를 고려한 협업 필터링을 이용한 음악 추천 – 2009, 이동주

Context-Aware Recommendation by Aggregating User Context – 2009, 신동민

Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions – 2005, Gediminas Adomavicius

And so on, …

17


Problem Definition

Recommender System

Estimating ratings for the items that have not been seen by a user

User (explicitly/implicitly - logs) rates Item

Input: User, Ratings Output: Item

Context-aware Recommender System (similar but different!)

Predicting a user’s preference in different context situations based on past experiences

User (explicitly/implicitly) rates Item + each rating has Context Info

Input: User, Current Context Output: Item

18


Problem Definition

Context-awareness based on Lifelog Ontology

In our Lifelog Ontology, there are no classification of users, items, contexts…

All the things in the ontology are Events & Entities

19


Problem Definition

Assume that the system knows the current context (situation)

Context-aware applications are such as Find out the most suitable songs to the current user

Find out the most suitable person to be invited to the current location

Find out the most similar songs to the song that currently user’s listening to

Find out the most related anything to the current situation

…

Relatedness can be defined in various ways

We can conclude that a context-aware system based on lifelogs is a system that allows dynamically & flexibly composing recommendation based on lifelogs

20


Context-Awareness Based on Lifelogs

21

Our Lifelog model

Lifelog Ontology LO is a set of events EV

A context history is represented by a event EV that are composed of a set of entities EN

– e.g. {Sangkeun Lee, Yesterday, 2009/12/24, Summer, Seoul, SNU}

A current context C is represented by a set of entities EN

– e.g. {2009/12/25, Hyo-ri Lee, Winter, Hungry, Happy, Busan, …}


Simple Flexible Match Algorithm (on-going work)

Collaborative Filtering (e.g. Music recommendation)

22

User

Similar User

Based on logs

SongsUser’s rat-

ing

Rating prediction based on similar user’s rait-ings



Generalized Model

Raiting is one possible relatedness between users & songs

User similarity is one possible relatedness between users

Entity – Entity Relatedness Can be calculated by similarity of events related to each entity

Is it only suitable for same type of entities? Not sure.

23

Entity

Event

Event

Entity

……

User

Song

Song

User

… …

Song

Location

Time

…


Simple Flexible Match Algorithm 1 (on-going work)

Collaborative Context-aware Filtering

24

Entity

Similar Entity

EventsEntity relatedness in current context

Predict relatedness based on similar entity’s related-ness

C

E1

E2

E3E4

E5

E6

C

E1

E2

E3E4

E5

E6

C

E1

E2

E3E4

E5

E6

Entity relatedness in current context


Simple Flexible Match Algorithm 2 (on-going work)

Context-Aware Filtering

Find out the most related entities to current context

Context Similarity Can be defined as an aggregation of entity relatedness

Entity Rank The most related entity to current context

25

Current Context can be represented in a set of entities

(Assume that all the entities are the categorical values)

C

E1

E2

E3E4

E5

E6

CH1

E1

E2

E3E4

E5

E6

CH1

E1

E2

E3E4

E5

E6

CH1

E1

E2

E3E4

E5

E6

CH1

E1

E2

E3E4

E5

E6

CH1

E1

E2

E3E4

E5

E6

CH1

E1

E2

E3E4

E5

E6

Event Rank

User Time Location Tempera-ture

…

1 Matt Summer Chunchon Hot …

2 Kang Summer Chunchon Hot …

3 Matt Winter Chunchon Hot …

4 Steve Fall Seoul Hot …

5 … … … … …

Event ranking based on similarity to current context

Entity Rank can be calculated by weighted sum



Context-Aware Filtering

Flexible?– Context Similarity, Entity Relatedness can be defined in various

ways by controlling Weight parameters of context similarity, weight parameters of entity relatedness

– You can decide what to recommend (Any entity can be recommended)

26


Simple Match is Not the Only Way – Graph Tra-verse

Assume that Current Context is

{Matt Lee, DSC_2128, 2009/07/23, 018-2144-8842}

Result Entity: matt183/018-2144-8842/

Entity: SNU. Main Gate/37:27:14.60N:126:57:12.33E/37:27:14.60N:126:57:12.33E/

Entity: GS24-TB/

Entity: Green House/37:27:27.29N:126:57:62.52E/37:27:27.29N:126:57:62.52E/

Entity: Jay Yeon/Female/[email protected]/101-1106 Suwon apt. Suwon Si, Republic Of Korea/Deejay09/010-4255-5541/

Entity: 2009/01/04/

Entity: Lazy Rhapsody/The Duke: Creole Rhapsody (1931-1932) (disc 1)/Duke Ellington & His Orchestra/196440/

Entity: SNU. Rear Gate/37:27:13.79N:126:57:18.23E/37:27:13.79N:126:57:18.23E/

Entity: 12:35:45/

Entity: Warren Gong/[email protected]/138-3 Naksung apt. BongChun dong, Kwanak-gu, Seoul Si, Republic Of Korea/gongfit/019-258-8163/

Entity: SNU Library/37:27:72.11N:126:57:11.32E/37:27:72.11N:126:57:11.32E/

27


Simple Match is Not the Only Way – Simple If-then Rules

Simple if then rules

sometimes much powerful and useful then other recommendation algorithm

Many context-aware actions can be defined as simple if-then rules

– Alert me when ‘Sangkeun Lee’ is in ‘IDS Lab’

– Turn on light when ‘Sangkeun Lee’ arrives at home

– ….

CAUTION : should not require complex reasoning or inference

28


Conclusion & Discussion

The Ideal Context-awareness

Find out the most related anything to the current situation

Considering everything useful

Ignoring information noise

…

29

Context-Awareness based on Lifelog

Documents

Transcript of Context-Awareness based on Lifelog