Post on 27-Jan-2016
description
ISTD 2003
Interactive Systems Technical Design
Lecture #4
Context-aware interaction with mobile devices
ISTD 2003
Context• Lenat et al.: ”There are a dozen mostly-independent
dimensions along which context vary (Absolute time, Type of time, Absolute place, Type of place, Culture, Sophistication/Security, Granularity, Epistemology, Argument-preference, Topic, Justification and Let’s) Conversely, each region of that 12-dimensional space implicitly defines a context”
• Abowd et al.: ”Any information that can be used to characterize the situation of an entity, where an entity can be a person, place, or physical or computation object”
• Schmidt et al.:” Interrelated conditions where something exists or occurs”
• Simplified: ”Description of the situation”, ”Snapshot of the world”
ISTD 2003
Context Space Dimensions (Lenat et al.)
1. Absolute time : a certain time interval in which events occur
2. Type of time: a non-absolute type of time period, e.g. ”after breakfast”
3. Absolute place: a particular location, where events occur, e.g. ”Oulu”
4. Type of place: a non-absolute type of place, e.g. ”in bed”
5. Culture: linguistic, religious, ethnic, age-group, wealth etc. of typical actors
6. Sophistication/Security: who already knows this, who could learn it, etc.
7. Granularity: phenomena and details which are (not) ignored
8. Epistemology (Modality/Disposition): who wants/believes this context to
be true?
9. Argument-Preference: local rules for resolving pro-con argument disputes
10. Topic/Usage: aspects and applications
11. Justification: are things in this context generally proven, observed, on faith, …
12. Let’s: local bindings of variables etc. that are true in that context
ISTD 2003
Context Categories (Dey)
• Categorization helps to uncover the relevant pieces of context wrt. a given application
• Who’s, Where’s, When’s and What’s of entities• Incoming context is used to encode some
action in the application• Most common contexts
• Location• Time• Identity• Activity
ISTD 2003
Context-Awareness
• Abowd et al.: ”Context-awareness or context aware computing can be defined as the use of context to provide task-relevant information and/or services to a user”
• Pascoe et al.: ”The ability of a device or program to sense, react or adapt to its environment of use”
• Simplified: ”Use of context”
ISTD 2003
Context-Awareness Categories (Dey)
• Attempt to describe how context is used
• Categories• Presentation of information and services to a
user• Automatic execution of a service• Tagging of context information for later retrieval
• Show primary types of features to consider wrt.a particular context-aware application
ISTD 2003
Context (Engineer’s View :)
• Time (absolute, relative, …)• Location (absolute, relative, symbolic, …)• People (profile, presence status, location, …)• Physical environment (objects, weather, …)• Virtual environment (data, …)• Operation/activity (task, mission, objectives,
…)• Device context
• Note!• Absolute vs relative context information• Physical vs symbolic/logical context information
ISTD 2003
Mobile Device Context
• Environment• Illumination, temperature, humidity, noisiness, …• People, machines, networks, services, …
• Location & position• Coordinates, places, indoors, outdoors, …• Near/far wrt. user, tilted, orientation, …
• Activity• Touched, held, moving, shaking, looked at, talked to,
power on/off …
• Note! A mixture of absolute/physical and relative/symbolic/logical context information
ISTD 2003
Context-Awareness and Mobile Devices
• Goal: Improved interaction• Adaptive services and interfaces• Automated context recognition preferred over
manua
• Adaptive interfaces• Modality (how the user uses the device)• Visual appearance• Functions (different functionalities in different
contexts)
• Presentation of context-relevant information• Decrease information overload• Selective context sharing
ISTD 2003
Context-Awareness and Mobile Devices (cont.)
• Challenges for multidiciplinary research• Data (context) acquisition & hardware issues• Context recognition• Distributed computing & protocols to handle
mobility• Human Computer Interaction issues for creating
usable applications• Knowledge engineering issues for representing
and manipulating context information and utilizing background information
ISTD 2003
Context-Awareness for Improved Interaction
Operation sequence Example (proximity sensor)
1. Context recognition IR emitter & receiver• Sensors 1. IR emitter emits light• External information 2. IR receiver measures the amount of
reflection,which depends on the surface type, size of
objectand distance
2. Context representation Proximity representation(a) Direct (sensor) values (a) Distance to the user in cm(b) Inferred states (b) Proximity state (Close, InRange, OutOfRange)
3. Context utilization Adapt UI according to distance(loudspeaker volume, font size, …)
ISTD 2003
Case Study
”Sensitive mobile phones”Microsoft Research, USA
Hinckley K, Pierce J, Sinclair M & Horvitz E (2000) Sensing techniques for mobile interaction. Proc. UIST'00, 91-100. (Best Paper Award)
Hinckley K & Horvitz (2001) Toward more sensitive mobile phones. Proc. UIST’01, 191-192.
http://research.microsoft.com/users/kenh/
ISTD 2003
Overview
• Context of interaction is recognized with simple sensors• Particularly, information on how the device is
being held
• Context is used to improve interaction techniques• Holding the device in certain ways indicates
what action is being performed (or a small set of possible / likely actions)
• Complete automation of particular functions• Speedups and improvements of particular
functions
ISTD 2003
Mobile Device: Casio Cassiopeia E105Augmented with Sensors
ISTD 2003
Sensor #1: Proximity Range Sensor
• IR emitter (light-emitting diode)• IR light modulated at 40Khz
• IR receiver (tuned to sense 40Khz only)• Measures the amount of reflection
• Different readings for different surfaces and sizes of object• Pretty consistent, since we are mostly looking at
the same thing (skin)
• Estimates the distance to an object• Range 5 – 40 cm
ISTD 2003
Sensor #2: Touch Sensors
• Two separate sensors (“plates”)• “Holding” sensor on back and sides• “Bezel” sensor at the left edge of screen
• Sensor ”plates”• Two regions: ”active” area and ”ground”• Conductive paint
• Sensor “plates” detect capacitance changes• Capacity changes when the user’s hand touches
the “plate“ vs. not touching
ISTD 2003
Sensor #3: Tilt Sensor
• Two-axis linear accelerometer (in plane of device)• Gravity provides constant acceleration (1g)• Direction of acceleration vector x,y tilt• Direction of gravity vector can’t be recognized
• Screen up vs. screen down can’t be recognized• Could be solved with a gravity-activated switch
• Can also detect patterns of movement
• Realized with a single chip (ADXL05)• Cheap and relatively easy
ISTD 2003
Sensor Architecture
• PIC micro-controller (Microchip 16C73A)• Constantly reads sensors (~400 samples / sec)• Reports values to PDA via serial port
• Context server (“Broker”) application on PDA• Receives and processes values• Makes information available to other
applications via API• Polled values• Subscribed events
ISTD 2003
Processed Values From Sensors
• Things taken directly from processed sensor data without recognition / inference
• Tilt sensor• TiltAngleLR, TiltAngleFB• HzLR, MagnitudeLR, HzFB, MagnitudeFB
• Dominant frequency and magnitude of movements (separate x,y measures)
• Obtained via FFT over short window
ISTD 2003
Processed Values From Sensors (cont.)
• Proximity sensor• Proximity
• Estimate in cm
• ProximityState• Close (< ~7cm)• InRange (~7-25cm)
• OutOfRange• Duration in
that state
ISTD 2003
Contexts Inferred From Touch Sensors
• Holding (duration)• From back / side touch sensors• Is user holding the device?• If so how long he has been holding it?
• TouchingBezel (duration)• Similar for bezel touch sensor• Not considered to be touching it until
duration>0.2 sec• Thresholds like this is used in a number of places
ISTD 2003
Contexts Inferred From Tilt Sensor
• LookingAt (duration)• Small range of angles appropriate for typical viewing
• Moving (duration)• Any movement and how long since last still period
• Walking (duration)• Detected by repetitive motion in 1.4-3Hz range
• Shaking• Device is being shaken vigorously
ISTD 2003
Contexts Inferred From Tilt Sensor (cont.)
Angle
Y (FB) Tilt
X (LR) Tilt
Walking
Holding at side
Looking at display
Operation sequence: ”entering an elevator, looking at display, holding the device down at side, walking to a meeting”
ISTD 2003
Context-Aware Interaction Techniques
• Automatic power on
• Voice memo recording
• Portrait / Landscape display mode selection
• Tilt Scrolling• LCD contrast compensation
• Display changes for viewing while walking
ISTD 2003
Automatic Power On
• PIC processor is on even when PDA is off
• PDA is powered up when you pick it up to use it• Holding in orientation for use• Specifically
• Holding + LookingAt • In portrait orientation (but not Flat) • For 0.5 seconds
Extremely natural interaction• Pick it up and its ready to use• Avoids most false positives
ISTD 2003
Voice Memo Recording
• Interaction: Pick it up like a cell phone and talk into it”• Holding + Proximity + proper orientation (tilt)
• Audio feedback (critical)• Click when pickup gesture recognized
• Beep to start recording
• Double beep at stop• Stop via loss of preconditions
Again, very natural interaction: pick it up as necessary to do a particular thing and it does it
ISTD 2003
Display Mode Selection
• DisplayOrientation event / data• Flat, Portrait, PortraitUpsideDown,
LandscapeLeft, LandscapeRight• Refresh event when changed
• Dead bands importantto remove jitter• Must cross all the way
through zone to count
• Flat is important • Don’t change orientation
• Interacts/interfereswith scrolling
• Input controls remapped tocurrent orientation
ISTD 2003
Tilt Scrolling
• Must explicitly touch bezel touch area to enable• Not quite as nice, but still pretty easy
• Then tilt up, down, left, right to scroll• Rate controlled (exponential)• Dead band of 4 degrees to prevent jitter
• Application menu and Start bar hidden during scrolling
• LCD contrast compensation• Contrast adjusted according to the viewing angle
ISTD 2003
Tilt Scrolling (cont.)
• Interference: Tilt scrolling interferes with display orientation selection• Hence explicit clutch• “I’m scrolling now” event for explicit disabling
across applications• BUT… what about when you let go
• Can’t tilt back prior to releasing clutch • Not scrolling anymore once released
• Handled by basically disallowing display change in direction of scroll tilt immediately after a scroll
ISTD 2003
Future Work: Display Changesfor Viewing while Walking
• If you hold the device in reading orientation (Holding + LookingAt) and walk (Walking), it will increase font size, etc.• Walking detected by repetitive motion in 1.4–3
Hz range
ISTD 2003
Lessons and Issues
• Sensor fusion• Typically need multiple evidence for a situation
in order to avoid false positives• E.g., “looking at” via both angle and touching instead
of just angle
• Need recognition error tolerant designs• False positive or negative is not a catastrophe • In general recognition errors can easily ruin a
good interaction• Recovery costs can easily destroy benefits
ISTD 2003
Lessons and Issues (cont.)
• Interaction/interference between techniques• Handled by explicit disabling
• Event saying “We’re using that now” • Used to disable other effect• Somewhat ad hoc
• General issue that needs work• Probably needs some sort of general conflict resolution
mechanism• But not clear what that is, or how to do it
ISTD 2003
Lessons and Issues (cont.)
• Big take away point• Knowing context of operation (e.g. how held)
allows space of possible current interactions to be constrained
• Less explicit actions required to invoke an action or provide a input
• E.g., typically avoid button presses or other mode entry
• Can result in simpler and more natural interaction• Best case: Picking it up to do something causes the
something to “just happen”
ISTD 2003
Video
Video