© 2003, Carla Schlatter Ellis Display Management: Sensing User Intention and Context (HOTOS 2003)

© 2003, Carla Schlatter Ellis

Display Management: Sensing User Intention and

Context (HOTOS 2003)

2© 2003, Carla Schlatter Ellis

Outline

• Motivation and Research Objective• FaceOff Architecture and Prototype• Evaluation

– Best Case Feasibility Study– Responsiveness Study

• Future Work• Related Work

– Dark Windows


Motivation

• Current energy management techniques tied to process execution

• Can we use low power sensors to match I/O behavior more directly to user behavior and reduce system energy consumption?

Sensing User Intention and Context

for Energy Management


Case Study: FaceOff

• Displays:– Typically responsible for large power drain– Power State can be controlled by software– State transition strategies naïve

A display is only necessary if someone is looking at it.


Image Capture

Face Detector

Main ControlLoop Face=on

No Face=off


Prototype

• IBM ThinkPad T21 running RedHat Linux– Base Power Consumption = 9.6 Watts

– Max CPU = 8.5 Watts over Base

– Display = 7.6 Watts

• Logitech QuickCam Web Cam– Power Consumption = 1.5 Watts

• Software components:– Image capture, face detection, display power state

control


Face Detection

• Skin detection used for prototype

• Real time proprietary methods exist


Outline




– Dark Windows


Best Case Feasibility Study

• What is the potential for energy savings?– Assume perfect accuracy – Best case user behavior – start it and leave.

• Tradeoff of energy costs:– CPU/Camera vs. Display

• Effect on System Performance– Network file transfer (113 MB)– CPU intensive process (Linux kernel compile)– MP3 Song (no display necessary)


File Transfer Traces


Kernel Compile Traces


Energy and Time Comparisons

Energy (J) Default With FaceOff % SavingsCompile 12506.85 11023.07 11.86Transfer 6795.42 4791.19 29.49

Time (s) Default With FaceOff % OverheadCompile 575 603.5 4.96Transfer 348.6 351.3 0.77


MP3 Application

• Playing an MP3– Display not necessary– Song completes before default timeout turns off

display

• Energy comparison– 3,403 J with FaceOff vs. 4,714 J with Default– 28% energy savings

• No noticeable effect on playback


Responsiveness Study

• Use full prototype including skin detection

• Establish baseline timing

• Examine Responsiveness– varying system load– varying polling rate


Responsiveness Timing

Face arrives (or departs)

pollinglatency

detectionlatency

Image acquired detection completedisplay signaled

Total responsiveness latency


Baseline Detection Latency

• Measured over a period of one hour with no programs other than background processes running

• Latency increased over time– Started at ~110ms– Increased to ~160ms– Why?

• Appears to be an effect of Linux scheduler reducing priority of long running jobs


Detection Latency over Time


Detection Latency Under Load

Workload Average(99% Confidence)

Maximum Minimum

Network Transfer

175±7ms 305ms 116ms

Kernel Compile

230±5ms 669ms 51ms

MP3 Song 154±3ms 229ms 84ms


Outline




– Dark Windows


Varying Polling Rate

• Reduce overhead by reducing polling rate– Increases responsiveness latency

• Adaptive polling rate– Eliminate polling in presence of UI events– Begin polling as duration without UI events

increases and face is detected– Reduce polling when no face present

• Similar problem with latency increase upon return


Optimization with Motion Sensor

• Combine adaptive polling & motion sensing

• Meet responsiveness requirements with minimal FaceOff system overhead

• Eliminate image polling when no motion

• Switch display state on immediately when motion detected and restart image polling


Implementation

• Prototype using X10 ActiveHome Wireless Motion Sensor and Receiver– Receiver connects to serial port– Reading port blocks until sensor triggers– Takes up to 10 seconds to recharge

• Promising addition to FaceOff system


More Roles for Sensors

• Touch Sensor– Detect picking up of a PDA

• Light, Sound sensors– Adjust display brightness (Compaq iPAQ)– Adjust speaker volume

• 802.11 Signal Strength sensor– Determine possibility of offloading

computation


Enhanced Sensors

• “Active Camera”– Perform some or all of the face detection

• Color filtering– Preprocessing skin color segmentation

• Low Power processor for external sensor control, computation


Discussion: Other ideas for using sensors to save

energy?


Future Work

• Continue work on optimizing responsiveness• Comprehensive user study

– Survey of usability– Characterization of usage patterns

• End-to-end experiment

• Implementation with available very low power camera/motion sensor and prototype for small device (handheld)


Conclusions

• Context information offers promising method of energy management

• FaceOff illustrates feasibility of approach

• Available very low power sensors as well as optimization techniques would improve upon the FaceOff energy savings


Outline




– Dark Windows


Related Work• Display Power Management

– Industry Specifications• APM, ACPI, DPMS

– Zoned Backlighting– Energy-Adaptive Display System Design

• Attentive/Perceptual UIs– Smart Kiosk System: Gesture analysis– CAMSHIFT: Game control– IBM PupilCam: Head gesture recognition


ACPIAdvanced Configuration and Power

InitiativeBrought to you by Intel, Microsoft, and Toshiba

and designed to enable OS Directed Power Management (OSPM).

• Goal is to be able to move power management into software for more sophisticated policies

• Abstract OS-HW interface• Replaces APM interface


What ACPI Offers• Standardization industry-wide

(Vendors to support ACPI in products instead of building their own power mgt)

• System and device power states• Thermal model

– Thermal zones, indicators, cooling methods

• BIOS interfaces– Motherboard configuration tables – Interpreted control methods

• Plug-and-play• Complexity moved into OS


What ACPI Offers• System

– Mechanisms for putting computer as a whole in sleep/wake states

• Devices– ACPI tables describe motherboard devices

• Power states• Controls for managing states

• Processor – Detecting idle state and swapping to low power

• Batteries– Querying and controlling battery behavior


Power States

G: global states apply to entire system and are visible to user

D: states of individual devices

S: sleeping states within the G1 state

C: CPU states G2-S5Soft off

LegacyG0-S0

working

G3mech off

G1Sleep

S1S2 S3

S4

Dxmodem x DxHDD xDxcdrom x Cxcpu

wakeup


ACPI Internal Structure

AC PI T ab lesAC PI B IO SAC PI R eg isters

K ernel

D eviceD river

ACPIRegisterInterface

ACPI T ableInterface

ACPI BIO SInterface

Platform H ardw are

Existingindustrystandardregister

interfaces to:CM O S, PIC,

PIT s, ...

AC PI D river/AM L Interpreter

ApplicationsO S

D ependentA pplication

A PIs

O S Specifictechnolog ies,

in terfaces, and code.

O SIndependenttechnologies,

interfaces,code, andhardw are.

BIO S

O SPM System Code

- Hardw are/Platform- Provided by ACPI CA

- ACPI Spec Covers this area.- O S specific technology


Transmeta Crusoe ACPI Power States


OSDM: OnNow

Applications

OS

HW

OnNow WIN32 ext

ACPI Spec

SetSystemPowerState– initiate sleep state, query apps(?)

SetThreadExecutionState– specifies level of support needed

(e.g. display required)

WM_POWERBROADCAST– a message notifying of power state

changes to which applications can respond

SetWaitableTimer– ensure PC is awake at scheduled

time

RequestDeviceWakeupRequestWakeupLatency - to specify latency requirements

GetSystemPowerStatus and GetDevicePowerState


Outline




– Dark Windows


Energy-Adaptive Display System Designs for Future

Mobile EnvironmentsS. Iyer, L. Luo, R. Mayo, P.

Ranganathan, MOBISYS 2003


Opportunity• New technology:

OLEDs – Organic Light Emitting Diodes– Energy consumption on a per-pixel basis

by determining each pixel’s brightness and color

– Energy consumption of different regions of screen to be changed independently

– No separate backlight– In development by Kodak, Sanyo, Sony,…


Dark Windows

• Software support – modifications to windowing system to ensure energy is spent mostly on window-of-focus (capturing user’s area of visual interest)

• Non-active screen area is changed (dimmed or re-colored for energy optimization)


Justification• Usage study – what are the user’s needs and

how well do they match display characteristics?• Characterization of display usage in Microsoft

Windows by 17 “typical” users• Application logger – recorded, for up to 14

days, when user was active.– Window of focus (the one accepting keyboard input)

– its size, location, title– Size of total screen area used (all non-minimized

windows


Screen Usage Results• On average

only 59% of area used by window-of-focus,additional 17% bybackground windows

• High variability among users

• Large fraction of smaller windows have very low content (notifications, alerts) – don’t need full color characteristics of display to convey it.


Dark Windows Design• Prototyped on X Windows under Linux• Used VNC – Virtual Network Computing

Server – provides a virtual representation of display – virtual framebuffer where pixels can be manipulated between application and display

• Track window-of-focus, apply modifications to pixels outside of it


Modifications

• Half Dimmed – areas outside window-of-focus are dimmed to 50% brightness

• Fully Dimmed – areas outside are turned off• Gray Scale – areas outside are changed to

gray by setting rgb to average value.• Green Scale – areas outside are changed to

green which is lowest power color for OLEDs. Dims to 67%


Evaluation• Energy benefits measured by generating synthetic

trace from usage study and playing trace on prototype.

• 15 inch OLED displays were not available so they used a software power model to calculate power consumption– Controller power set to 0.5W– Driver power – 1W– 1024x768 pixels individually consume -- red power 4.3W,

green power 2.2W, blue power 4.3W.

• User experience – users want to see background but willing to use dark windows


Results based on default Teal Background*

* Benefits depend on original Choiceof backgroundand windowcolors.


Conclusions

• While benefits depend on usage scenario, significant energy savings can be achieved with these optimizations

• Further opportunities for application specific adaptivity– More meaningful notions of area of focus can be

defined (e.g. most recent email message, most recently changed region of screen)

– Better match to (low) content – e.g., notifications could be audible signal instead of popup window

© 2003, Carla Schlatter Ellis Display Management: Sensing User Intention and Context (HOTOS 2003)

Documents

Transcript of © 2003, Carla Schlatter Ellis Display Management: Sensing User Intention and Context (HOTOS 2003)