COMP327Mobile ComputingSession: 2011-2012

Lecture Set 11 - Context Aware Systems

1

In this Lecture Set

• Context Aware Systems

• Context Dependent Mobile Scenarios

• Spatial Awareness

• Augmented Reality

• Services and Service Discovery

• Jini, UPnP, DLNA

2

Context Aware (CA) Systems

• Systems that are aware of their own situation in the physical, virtual, and user environment.• e.g. a “travel guide” that only provides information on buildings near to the user’s

location, or a “memories” application that identifies and focusses on familiar faces when photos are taken

• Typically adapt their operation or goal based on contextual cues from the environment or the user’s actions

• Implicit behaviour, rather than explicitly directed by the user

• Lessen the (cognitive) load on the user

• Requires ability to not only sense environment, but to determine what is relevant to the system’s task(s)

• May require additional processing to convert raw sensor data into relevant information

• May also need a representation of the relevant knowledge model used to comprehend contextual cues and direct behaviour

3

Classifying CA systems• Various ways of classifying CA software:

• Proactive Triggering

• e.g. performing some interaction based on environmental perceptions

• Streamlining Interaction

• Reducing irrelevant information - e.g. travel guide for current location

• Memories for past events

• Contextual retrieval, e.g. based on spatial or feature-based cues

• Reminders for future contexts

• e.g. tagging details regarding current context for future access

• Optimising patterns of behaviours

• e.g. changing interface based on situation (from screen to text-to-voice)

• Sharing Experiences

• Social networking based on shared contexts

4

Classifying CA systems

• Passive context-aware systems

• New context is presented to the user, to inform them of change.

• User can then explicitly determine if the use of an application should change

• Examples include

• Changing information regarding environment (weather meters)

• Context-based tagging (e.g. by camera)

• Active context-aware system

• Behaviour of the applications change automatically

• Examples include

• Task filtering (e.g. information filtering, based on current wireless network speed)

• Context-base task activation (e.g. routing)

• Content adaptation (e.g. for people with disability)

5

Context Creation

• New contexts can be created based on sensor data (captured by the device or nearby sensors)

• Lower-level raw contexts may need to be processed into higher level contexts

• Raw data may need to be scaled or transformed

• e.g. electrical signal on a temperature gauge should be converted into a Celsius value...

• ...or absolute geo-location position should be converted into an address or identification of a building

• Often this abstraction is more useful

• e.g. “this photo was taken at my parents home last christmas”

6

Challenges in Context Awareness

• There are many challenges in modelling contexts

• Environmental cues may be inaccurate or erroneous

• The user contexts may be incorrectly determined or predicted, or just ambiguous

• Lack of alignment with cues and the internal representation of contexts

• The use of contexts may reduce user privacy

• Awareness of context shifts or changes in application may overload or distract the user

7

Examples of Context Awareness for Mobile Devices• The Active Badge Location System

• Call Routing for Mobile Users

• Initially in 1989, published by Want et. al. 1992

• Users were tracked around a building using active badges and embedded sensors in the environment

• Calls could then be routed to users through the nearest telephone extension

• Electric Elves

• Agent-based Mobile Personal Assistant

• Deployed for 6 months in 2000 (Tambe et al, 2005)

• Agents would act on behalf of their users coordinating routine office tasks

• Exploited an awareness of factors such as user’s location and calendar to make decisions (e.g. nominating speakers for seminars, or delaying meetings)

8

Spacial Awareness• Often considered one of the main drivers for mobile services

• Often referred to as Location-Aware Systems

• However, often the notions of orientation and of nearby objects (as polygons) are also significant

• Services can be related spatially

• e.g. ordered in terms of proximity on a list

• located on a localised map

• Peer to Peer based services can also relate the proximity of users to each other, rather than to any fixed geographical point

• Services may also be discovered locally, and then use a local PAN (rather than internet-based) communication

9

Examples of Spatially Aware Applications

• Navigation

• Where is the nearest coffee shop? Give me directions...

• Context Change

• My route is congested - is there a better route?

• Query Location Context

• Tell me about the building in front of me

• Personal Emergency

• I’m having a heart attack! My car has broken down!

• Location/Time based offers

• Download Starbucks tune of the week. Special offer on restaurant X

10

Examples of Spatially Aware Applications

• Asset Tracking:

• Enterprise:

• Why does it always take so long to deliver to customer X?

• Public:

• How long before the next bus will arrive at this bus-stop?

• Personal:

• Where has Smudge the cat gone? My car has been stolen - can it be tracked?

• Location Tagging

• Where were those photos taken?

• Location and time synchronisation

• Used can share and synchronise their context (location, activity, mood) to arrange ad-hoc meetings

11

Location Acquisition• Various approaches based on:

• triangulation

• based on angulation (bearing) given two or more fixed points

• lateration

• determines distance from fixed points - e.g. determining Time of Arrival

• proximity

• recognising RFID tags or being within a WIFI cell

• scene analysis

• image recognition of iconic buildings, or text recognition of street names

12

Triangulation• The process of measuring the location of a point by measuring the angles to it from

known points.

• Uses law of sines (trigonometry)

• If A and B are fixed points, and the location of O is known if the following are known:

Distance AB

Angles at A and B

• Given that:

sin A = Y/a

sin B = Y/b

cos A = X/a

• Can calculate:

Y = a * Sin A which is also equal to Y = b * Sin B

X = a * Cos A

X = AB - b * Cos B

A B

O

a bY?

X?

13

Global Positioning System

• Positioning based on satellites in medium earth orbit

• Between 24-32 satellites, launched between 1989 and 1994

• Used to determine latitude, longitude, altitude (and time)

• Each satellite transmits:

• the time the signal was sent

• precise orbital information of that satellite

• general information of all the GPS satellite status (the almanac)

• Trilateration is used to combine the distances to determine location

• Due to possible error propagation, positioning is improved by using additional satellite signals

14

Assisted GPS

• A network carrier dependent GPS system

• Improves startup performance of GPS within mobile phones by using additional information sent by the carrier

• Orbital data and/or almanac for GPS satellites, to help lock onto signals more rapidly

• Atomic time information

• Accurate surveyed coordinates of the cell tower location can be used to determine ionosperic conditions, and correct for errors

15

Mobile Phone Localisation• Determines the location of a phone based on power levels of signals

and antenna patterns

• Doesn’t need active calls to work

• Can achieve a precision of 50 meters in urban areas with high density of cells

• Lower precision in rural areas, where cell towers are few and far between!!!

• Three approaches

• Network Based

• Relies on the Provider’s network to (non-intrusively) identify the location of the handset.

• Cell identification (i.e. knowing which cell the phone is in) gives a course-grained precision.

• GSM uses trilateration to determine a better location precision.

• Used by legislative frameworks such as E911

• Handset Based

• Relies on applications running on the phone itself to determine position using various methods

• Including GPS, cell identification, signal strengths of different towers, etc.

• These are then processed, and sent to the receiver

• Hybrid Systems

• Combines both network and handset-determined positioning to get the best accuracy

• May vary the mechanism used depending on local conditions and available signals

16

iPhone Location Services

• iPhone makes use of a combination of GPS, GSM and Wi-Fi positioning methods

• GPS is used whenever possible

• Can be inaccurate indoors or due to urban canyons (e.g. in New York)

• Exploits Skyhook’s WiFi Positioning System (WPS)

• This relies on a large database of known WiFi access points

• See www.skyhookwireless.com for more details

• The MAC address and Received Signal Strength is used from each detected Wifi Point to generate an approximate location through trilateration

• Good accuracy, but dependent on high density of Wifi access points

• Prone to error if Access Points are moved (e.g. consumers change addresses!!!)

• Uses GSM localisation based on cell towers

17

Core Location Framework

• A framework for providing details about:

• Location, in terms of

• latitude, longitude, altitude,

• horizontal accuracy (latitude/longitude radius)

• vertical accuracy (altitude)

• Heading, in terms of

• heading (degrees from magnetic or true north)

• heading accuracy

• Regions

• Uses a three tiered approach

• The more accurate the approach, the more power it consumes

• GPS

• Wifi

• Cell Network

18

Core Location Manager• This is the object responsible for managing locations, and

generating notifications of location change

• Uses a delegate protocol to inform changes:

• Responding to Location Events

• – locationManager:didUpdateToLocation:fromLocation:

• – locationManager:didFailWithError:

• Responding to Heading Events

• – locationManager:didUpdateHeading:

• – locationManagerShouldDisplayHeadingCalibration:

- (void) locationManager:(CLLocationManager *)manager didUpdateToLocation:(CLLocation *)newLocation fromLocation:(CLLocation *)oldLocation { NSLog(@"LocationView new location: latitude %+.6f, longitude %+.6f\n", newLocation.coordinate.latitude, newLocation.coordinate.longitude);}

19

Core Location Manager

• The Manager is configures to determine the type of events generated:

• desiredAccuracy - how accurate should the location be?

• Affects what technology is used

• Values go from 3km to nearest 10m, to best, to navigation

• distanceFilter - how often should the events be generated?

locationManager = [[CLLocationManager alloc] init]; [locationManager setDesiredAccuracy:kCLLocationAccuracyNearestTenMeters]; [locationManager setDistanceFilter:10]; [locationManager setDelegate:self];

20

Proximity Analysis

• Makes use of short-range transceivers at fixed positions

• Determines when tags such as RFIDs are within range

• Typically determine relative distance, as opposed to absolute positions

• Typically used for asset tracking

• Can trigger location events for tourist guides

• e.g. in museums or galleries

21

Scene Analysis

• Relies on the abstraction of image markers from terrain scenes

• These are then cross-correlated from a database of known markers

• E.g. church spires, canals, railway lines etc

• Good for orientation, given a rough location

• Can be used for absolute positioning, but requires significant image information on unique buildings!

22

AugmentedReality

• Combining live direct (or indirect) view of a physical real-world environment with virtual CGI

• Typically in real-time using the semantics of environmental elements

• Identify digital “cues” (e.g. 2D barcode) and overlay with a digital image

• Video camera identifies orientation of the cue, and replaces it with the image, transforming it in 3 dimensions

• Can be used to enhance the environment

• Provide details about the environment, by “projecting” labels or directions onto the street or building being viewed

• Can be used for entertainment

• Project games into the real-world, by identifying cues to determine location or features

23

AugmentedReality

• To develop augmented reality applications, several technological challenges need to be overcome:

• Live video is required to project the graphics onto

• i.e. the app needs to display in real time what the user is looking at, and add the “augmented images” onto that video

• Positioning information required, to determine relative location of other artefacts that may be annotated

• Applications that annotate locations or give directions (e.g. Yelp) first need to determine the user’s location (using GPS or other location mechanism), device orientation (using accelerometer) and compass direction (using magnetometer) to determine what the user is looking at

• Then the objects that are being looked at need to be presented over the video feed, in the correct position (e.g. building annotations or directions)

• Image analysis is required to identify cues to overlay graphics

• Applications such as Word Lens identify text, and replace this with other text corresponding to a translation (e.g. English-Spanish)

• Other Apps identify patterns (and their orientation) to determine what should be displayed, and in what orientation it should be displayed

24

AugmentedReality

• Using Augmented Reality apps in real world (on the go) scenarios can be hazardous!

• Conventional apps require some attention from reality/environment

• Users on the go might focus some time on the app, but will also retain awareness of their environment

• Augmented reality apps provide some awareness of reality/environment

• Users can be lured into relying on the app to provide that awareness of their environment

• However, this can restrict full awareness, or become fixated on details, and thus act in a dangerous, or unexpected way

• e.g. loss of peripheral vision etc

25

Service Discovery• Historically, devices have required specific interfaces or

drivers to connect to other devices

• e.g. offering printing or visualisation services

• With the increasing number and diversity of device types, managing individual services is no longer feasible

• Discovery architectures have emerged to support the dynamic (run-time) discovery and use of services offered by other devices

• Particularly important with standardised bus services (e.g. firewire or USB), or wireless (PAN) communication

• e.g. Bluetooth

26

Capabilities supported by discoverable devices• Ability to:

• make other devices aware of its own existence and presence

• advertise the services it offers, and to provide additional information when necessary

• dynamically search and use other services

• Requires zero administration

• Interaction with other devices in the environment fulfils some goal

27

Context and Service Discovery

• Services (and/or devices) may be contextual, and usable within those contexts

• Location or availability of services, bound to a physical device

• e.g. printers, projectors, scanners etc

• Services that provide location information to a mobile device

• e.g. tourist information services, support for augmented reality

• Many existing services rely on determining location, then searching for services relevant to that location

• Alternatively, devices could locate localised services

• This is already done for communication services such as WiFi

28

Universal Plugand Play

• UPnP originally emerged from Microsoft's Plug and Play mechanism

• Uses established standards such as TCP/IP, UDP and HTTP for networking, communication and broadcast

• XML and SOAP are used for higher level comms.

• Devices can:

• dynamically join a network and obtain an IP address

• announce themselves and convey their capabilities upon request

• learn about the presence and capabilities of other devices

29

Digital LivingNetwork Alliance

• DLNA is a consumer electronics standard to support discovery and sharing of content across a home network

• Various types of devices

• Digital media servers (DMS) - e.g. home PC, Network Attached Storage (NAS)

• Serve content to other devices over the network

• Digital media players (DMP) - e.g. TV, stereo, projector

• Discover content from DMS and provide playback and control

• Digital media controlers (DMC) - e.g. mobile device, digital cameras

• Control media sent to a DMR

• Digital media renderers (DMR) - e.g. video displays, remote speakers

• Render content that is controled by a DMC

• Upload/Download capabilities also exist for mobile devices

30

Jini• Service Discovery system based on Java

• Assumes a number of distributed devices running a Java virtual machine

• RMI (Remote Method Invocation) ius used to call objects which reside on other devices.

• Discovery and Join mechanism (similar to UPnP) allow services to advertise themselves

• Lookup services act as service repositories (similar to control points)

• Devices advertise services for a limited time; to remain they need to renew their lease

• Ensures that the repository does not become stale

31

Exercises...

• Describe why context can be relevant for mobile applications, and detail two different scenarios where mobile applications could be used within a tourist environment.

• Discuss the privacy and ethical issues of location determination. Is it acceptable for a provider to be able to determine the location and context of a customer when an incident occurs?

• Why is there a need for service discovery by mobile devices? Discuss how devices could discover other services within a home environment, and speculate how a mobile device might be used to control the output of a rendering device

32

To Recap...

• In this lecture set, we covered:

• The advantages and usage of context

• How context can be represented or constructed

• The challenges of developing context aware systems

• Spatial Awareness

• How it can be used

• How location can be determined

• Other spatial modalities (e.g. compass direction)

• Service Discovery

• UPnP, DLNA and Jini

33

Further Reading

• Ubiquitous Computing: Smart Devices, Environments and InteractionsStefan Poslad (Wiley, 2009)• Chapter 7

• Pervasive ComputingUwe Hansmann, Lothar Merk, Martin S. Nicklous and Thomas Stober (Springer, 2003)• Chapter 15

• Wikipedia !!!

34