Pervasive Displays - ETH ZPervasive displays as part of pervasive / ubiquitous computing ... using...

Pervasive Displays

Reto Achermann [email protected]

March 12, 2013 1 Distributed Systems Seminar

PART I:

PERVASIVE DISPLAYS


htt

p:/

/ww

w.1

815.c

h/im

ages/4

419_1.jp

g

htt

p:/

/ww

w.g

oogle

.com

/sync/im

ages/s

ync-c

onta

cts

.png

htt

p:/

/ww

w.g

oogle

.com

/gla

ss/s

tart

/what-

it-d

oes/

htt

p:/

/g-e

cx.im

ages-a

mazon.c

om

/im

ages/G

/01/e

lectr

onic

s/d

eta

il-p

age/B

004T

T0

1JS

-clo

udl.jp

g

Pervasive Displays

Just “screens everywhere” ?


htt

p:/

/ww

w.t

imessquare

nyc.o

rg/im

age.a

spx?id

=1838&

wid

th=

137

0&

heig

ht=

87

0

Not really…

Pervasive Displays

Pervasive “spreading widely throughout an area or a group of people”

Display “electronic device for visual presentation”

“printing the arrangement and choice of type in a style intended to attract attention”

“a collection of objects for public viewing”

- Oxford English Dictionary


Pervasive / Ubiquitous Computing

Pervasive displays as part of pervasive / ubiquitous

computing

Principles of Ubiquitous Computing “The purpose of a computer is to help you do something else.”

“The best computer is a quiet, invisible servant.”

“The more you can do by intuition the smarter you are; the computer should extend your unconscious.”

“Technology should create calm.”


htt

p:/

/en.w

ikip

edia

.org

/wik

i/F

ile:M

ark

_w

eis

er.

jpg

- Mark Weiser

(1952-1999)

“Today’s public display systems are largely closed and

isolated, with tightly controlled screen access.”

Analogy:

Old mobile phones – current public display systems

- Nigel Davies et al.


In the early days…


htt

p:/

/4.b

p.b

logspot.

com

/_U

tUF

vF

_vR

lo/S

FA

lhV

1cw

XI/

AA

AA

AA

AA

AU

U/G

5G

pdA

rwqtg

/s1600/1

904+

coke+

ad.jp

g

htt

p:/

/sto

re.u

shis

tory

.org

/im

ages/m

ediu

m/w

ant-

dalton.jp

g

Public Information Advertisement

… and today


htt

p:/

/ww

w.1

815.c

h/im

ages/4

419_1.jp

g

htt

p:/

/blo

g.h

slu

.ch/o

uto

fhom

edis

pla

ys/w

p-c

onte

nt/

blo

gs.d

ir/1

83/f

iles/p

oin

t-of-

transport

atio

n/a

pg_

ep

an

el_

zuerich_

2.jp

g

Information / News Display

Electronic Advertising

Information presented to the

user interleaved with ads

Content is very repetitive:

User gets bored

No interaction or possibility to

change content

Showed content may be

moving / flickering to attract

the attendance

… and today


htt

p:/

/uplo

ad.w

ikim

edia

.org

/wik

ipedia

/com

mons/t

hum

b/7

/7b

htt

p:/

/ww

w.h

ers

che.a

t/uplo

ads/p

lakat_

ap

pe

nzelle

r_kalte_fu

sse_gr1

0_zug

eschn_korr

.jpg

Poster Advertising

Augmentation

Interaction needs application

(tag reader) or sending SMS

Augmentation is sometimes

inaccurate / impossible without

location service

Maps just show “Here are you”

indicator (hard to find)

Display Types

Static Displays: Cannot change the content

shown

Provide no direct feedback

Cheap (big surface)

Require no energy

Posters, Maps, Objects

Dynamic Displays: Can change content shown

dynamically

Can provide direct feedback

Relatively expensive

Require energy and actively driven input

LCD Screens, Projectors


htt

p:/

/ww

w.1

815.c

h/im

ages/4

419_1.jp

g

htt

p:/

/ww

w.h

ers

che.a

t/uplo

ads/p

lakat_

ap

pe

nzelle

r_kalte_fu

sse_gr1

0_zug

eschn_korr

.jp

g

Both may be location / environment aware

Are today’s displays pervasive?

“Yes” in the sense of Oxford dictionary

No in the sense of ubiquitous computing People have adapted to ignore the displays

Fast moving pictures distract calmness

Almost no interaction / non-intuitive interaction

Interaction / augmented reality is “plug-and-play” instead of “arrive-and-operate”

Displays provide no / too less help in doing something


htt

p:/

/ww

w.b

eobachte

r.ch/t

ypo

3te

mp/p

ics/S

BB

-Auto

mate

n_d5c70a1bcd.jp

g

htt

p:/

/1.b

p.b

logspot.

com

/

htt

p:/

/ww

w.t

imessquare

nyc.o

rg/im

age.a

spx?id

=1838&

wid

th=

137

0&

heig

ht=

87

0

Observation: Smartphones are everywhere

“The first truly pervasively available interaction devices”

- Robert Hardy

12

htt

p:/

/ww

w.s

prin

gw

ise.c

om

/im

g/u

plo

ads/2

012/0

9/d

andeacon.jp

g

Observation: Smartphones are everywhere

Equipped with various sensors, radio interfaces, acoustic &

haptic feedback and touch displays

Problem: Interaction with displays using smart phones

today is inconvenient, not always working and clumsy

Idea: use of technology provided by the smartphone in a

smart way for interaction


The vision in the movies


htt

p:/

/jo

rdste

r4000.b

logspot.

ch/2

011/0

9/la

yo

ut-

desig

n-r

esearc

h.h

tml

htt

p:/

/ww

w.b

itre

bels

.com

/wp

-conte

nt/

uplo

ads/2

010/1

2/K

inect-

Min

ority

-Report

-UI-

2.jp

g

Minority Report,

DreamWorks Pictures

Avatar

20th Century Fox

Visions

Interactive Street Map

Intuitively select an element on the map

Get related information of selected object

Query / Filter:

Get list of all restaurants

in a specific area

Show directions to object


htt

p:/

/ww

w.p

re.e

thz.c

h/a

ddre

ss/

Visions

Interactive Mensa Menu

Show additional information: Menu ratings

Alternative Menus at other location

Filter according to preferences

Example: Diet help Caloric values directly added to diary

Do not show menu which do not fit

Example: Allergic / Health Hide menus containing certain ingredients

Show warnings


htt

p:/

/a5.m

zsta

tic.c

om

/us/r

1000/1

03/P

urp

le/e

f/b1/2

6/m

zl.jlm

dxofp

.320x480

-75.jp

g

Visions

Passive Interaction with Displays

No need for taking the phone out of the pocket

Walk-by / be present as interaction event

Display spontaneously react to your presence and shows

content you like (pictures, videos) Privacy?

Example: Screen at CAB Foyer You like to see images of D-INFK

events

Your phone tells the screen wirelessly

Screen displays selection of photos


Visions

Personalized Content

Content is chosen according to personal preferences Switch paintings of e-gallery

Show news you’re interested in / weather of your location

Show tailored advertisement

Showing related information to query

Several social issues

Example: Team Support A group of soccer fans meet

The displays show the colors of the favorite team


ww

w.g

oogle

.com

htt

p:/

/i.t

ele

gra

ph.c

o.u

k/m

ultim

edia

/arc

hiv

e/0

1655/f

lags-s

treet1

_1655777i.jp

gc

Visions

Augmented Displays

A tablet’s display augment the view of the current display Different angle / layer

Annotations

Show additional information of the elements shown

Example: Doctors Different experts analyze a patient

Cardiologist

Sinologist

Surgeon

All have different interests concerning the patient’s health state


htt

p:/

/jo

rdste

r4000.b

logspot.

ch/2

011/0

9/la

yo

ut-

desig

n-r

esearc

h.h

tml

Visions

Houses as Displays


Using the façade of a house as a game display

Borin

g e

t al 2011

htt

p:/

/tro

yh.u

s/p

hoto

s/C

ity%

20S

kylin

es%

20&

%20D

ow

nto

wn/1

37

04%

20D

alla

s%

20skylin

e%

20at%

20nig

ht.

jpg

Visions

Projected Displays


Visions

Video:

http://www.youtube.com/watch?feature=player_embedded

&v=df1NO7MoAUY

http://www.youtube.com/watch?feature=player_embedded&v=df1NO7MoAUY





PART II

INTERACTION WITH DISPLAYS


htt

p:/

/farm

4.s

taticflic

kr.

com

/3259/3

221222512_723

ed

0e

08

3_

z.jp

g?zz=

1

htt

p:/

/b.v

imeocdn.c

om

/ts/3

56/3

89/3

56389029_64

0.jp

g

1. Physical Buttons

Select elements by pressing buttons

Buttons can be marked to distinguish them blindly

Drawbacks: Not always clear what different buttons do

Unclear element highlighting

No button adaption to different content

Button interface is not extensible


htt

p:/

/pre

vie

w.t

urb

osquid

.com

/Pre

vie

w/2

011/0

9/

Physical Buttons

2. Touch Screens

Interact by touching the element directly on the screen

More intuitive than physical buttons, adaptable interface

Drawbacks: Indistinguishable buttons (no blind navigation)

Dirty displays

Not well suited for very large screens

Not working with Displays behind security glass

Displays far away

Wet fingers / gloves


htt

ps:/

/activecapta

in.c

om

/art

icle

s/m

obile

Phones/iP

hone/iP

hone_K

eyboard

.jp

g

Touch Screens

3. Bluetooth Device Names for Interaction

Idea: send service requests to displays wirelessly

Issues tackled: SMS: untrusted number (premium service), need to know display ID

App: download necessary (inconvenient)

Touching may not be possible

Observation: many users have device supporting

Bluetooth and have set a custom USB device names

Approach: Use of Bluetooth device discovery and

Bluetooth device names to send requests


Bluetooth

[Nigel Davis et al]

Bluetooth Device Names as Commands

No need for additional software

Users set the device name to a special command string, to

send a request command to the system

Identifier followed by service name

Examples:


Bluetooth

ec <service_name> <params>

ec youtube: eth ec map: CAB H52

Scanning Area

Principle


htt

p:/

/en.w

ikip

edia

.org

/wik

i/F

ile:B

lueto

oth

Logo.s

vg

Display

Smart

Phone Smart

Phone

Smart

Phone

Smart

Phone

Display Machine

Problem:

reading the device ID takes time

discovery happens too early / late

Bluetooth

Serving the Requests

Each request (recognized command) is put into a queue

Each request is served for a maximum specified time

If a user leaves the display area, the requests is marked as

served

Social issues arise

ec youtube: puke


Bluetooth

4. Visual Markers

Using special patterns for interaction

Identify the display / element by

special marker (QR Code / edges)

Touch like interaction at distance


[Ballagas et al. - Point & Shoot (2005)]

htt

p:/

/ww

w.id

ea.o

rg/b

log/w

p-c

onte

nt/

uplo

ads/2

01

1/0

9/q

r-code

-01.jp

g

Visual Markers

5. Mobile Interaction with NFC enabled displays

Near field Communication Wireless communication technology, Point-to-Point

Small Range: < 0.2m

Frequency: 13.56 MHz

Bandwidth: 424 MHz

Set-up time: <0.1s

Low-Power, tag is unpowered

Applications Payments (Credit Cards, …)

Keys

Data Exchange (Business Cards)


htt

p:/

/askbobra

nkin

.com

/google

-walle

t-nfc

.jp

g

htt

p:/

/en.w

ikip

edia

.org

/wik

i/F

ile:N

FC

-N-M

ark

-Logo.p

ng

NFC

NFC enabled displays

A mesh of NFC tags attached to the

rear, no need for visual markers

Interaction by “touching” i.e. holding

the device close

Mobile device reads content of the

NFC tag:

Object ID to look up on internet

Self contained information (no lookup)

Mobile device may augment the

display by showing information


Robert Hardy, Enrico Rukzio, Paul Holleis, Matthias Wagner

Mobile interaction with static and dynamic NFC-based displays

NFC

Resolution Problem of NFC Enabled Displays

Resolution Problem: Mesh grid of NFC tags is coarse.

One tag covers many elements

Dynamic Solution: “Zoom in”: Show pop up

Static Solution: Show list on mobile phone

Generic Approach: Filtering Enable filter to reduce the

result set


htt

p:/

/ww

w.p

re.e

thz.c

h/a

ddre

ss/

NFC Tag Area

Filters Zoom In

NFC

6. Accelerometers: PhoneTouch

Goal: Distinguish multiple users interacting with a touch

screen at the same time

Touchscreen registers location of touch event

Phone registers movement

using accelerometer

Tabletop server matches touch

event with phone movement

to identify user


Accelerometers

PhoneTouch: A Technique for Direct Phone Interaction on Surfaces Dominik Schmidt, Fadi Chehimi, Enrico Rukzio, Hans Gellersen

Computing Department, Lancaster University, Lancaster, UK

7. Touch Projector: Touch Screen from Distance

Observation: Not all display support touch input

Displays may be out of arm’s reach

Elements cannot be moved between displays easily

Idea: interaction through live video


Sebastian Boring, Dominikus Baur, Andreas Butz, Sean Gustafson, Patrick Baudisch

TouchProjector: Mobile interaction through video

Touch Projector

Touch Projector Usage

All touch events are routed through

a server (the environment manager)

Basic Usage: User points at display and touches element

User moves element within / between screens

User releases touch and element is placed on new location


Touch Projector

Touch Projector Implementation Issues

Screen too small on mobile device: Automatic zoom in

when display is recognized

Device needs to be pointed on screen: Freeze image on

mobile device for stability and fine tuning

Display identification based on computer vision may result

in incorrect identification


Touch Projector

8. Augmented Displays

Problem:

One screen – different people and different interests

Idea: Augment main display with different views / layers


htt

p:/

/qvecto

rs.n

et/

dow

nlo

ads/im

ages/f

ullp

revie

w/v

ecto

r-B

usin

ess-P

resenta

tio

n_fu

ll.jp

g

Augmentation

The Magic Lens Metaphor

Show alternate view of the data

Show annotations / information on the object


Augmentation

7. Gesture Recognition using Kinect

Improved hygiene: no need to touch


htt

p:/

/ww

w.d

igitaltre

nds.c

om

/com

putin

g/k

inect/

Gesture

SOCIAL ACCEPTANCE ISSUES

Security

Content

Privacy


htt

p:/

/gig

aom

2.f

iles.w

ord

pre

ss.c

om

/2012/0

6/c

om

score

-facebook.p

ng

Security: SMS / QR Codes

Sending an SMS can end in a premium service

QR codes may refer to phishing sites

QR codes / SMS numbers may be forged


htt

p:/

/mobhappy.c

om

/blo

g1/w

p-c

onte

nt/

uplo

ads/2

006/0

3//

sw

eetr

elie

f.jp

g

htt

p:/

/1.b

p.b

logspot.

com

/htt

p:/

/ww

w.t

-onlin

e.d

e/h

andy/s

mart

phone/id

_6

13

01

37

6/f

als

che

-qr-

codes-

expert

en-w

arn

en

-vor-

sm

art

phone

-phis

hin

g.h

tml

Social

Privacy Concerns: Unveiling Personal Data

Personalized display content unveil personal data e.g. advertisement of products recently bought

e.g. support for political party / sports team

User may need to fill in his preferences

Scanning QR code / sending SMS

unveils phone ID

Technology may be used to

track the user where he goes


htt

p:/

/medaid

pharm

acy.c

om

/wp

-conte

nt/

uplo

ads/2

011/0

5/t

op

-lo

go_sm

all3

.png

htt

p:/

/ww

w.m

aste

rfile

.com

/sto

ck-p

hoto

gra

phy/im

age/4

00

-0505966

4/s

peaker-

or-

film

-directo

r-and-

scre

en

---c

art

oon

-sty

le

Social

Effort Needed to Use the Displays

Must be kept as low as possible

No plug-and-play but arrive-and-operate:

the use has to be intuitive

Effort needed is percept as cost


htt

p:/

/help

desk.n

ex-t

ech.c

om

/prin

t.php?id

=657

htt

p:/

/tutz

net.

com

/2966

-fix

-google

-pla

y-s

tore

-err

or-

403-a

ndro

id/

Social

Content

User decides what to display May be inappropriate

(e.g. offensive views)

May distract other people (e.g. music styles)

The content providers must be trusted Not all display owners want all contents on their display

Concept of “Trusted Store” like an app store

Content war between users


htt

p:/

/ww

w.2

0m

in.c

h/s

chw

eiz

/new

s/s

tory

/1818593

5

Sunrise Demonstration Incident

Social

Summary of Today

Today’s displays are not really pervasive (not helpful)

Intuitive interaction with displays needed to turn them into

helping assistants

There are many technical & social issues to solve

Most of the basic

technology is already

available

45

htt

p:/

/ww

w.g

oogle

.com

/gla

ss/s

tart

/

THANKS

Questions ?


htt

p:/

/podcasts

.in

sitew

ebsitedesig

n.c

om

/in

ters

ect-

vid

eo-w

all-

sequence

-fro

m-c

huck-H

D-s

tills

-04.jp

g

References

Nigel Davies, Marc Langheinrich, Rui Jose, Albrecht Schmidt

Open Display Networks: A Communications Medium for the 21st Century

IEEE Computer, Volume 45, Issue 5, Pages 58-64, 2012.

Florian Alt et al.

Designing Shared Public Display Networks: Implications from Today's Paper-Based Notice Areas

Proceedings of the 9th international conference on Pervasive computing (Pervasive'11), San Francisco, CA, USA, 2011.

Sebastian Boring, Dominikus Baur, Andreas Butz, Sean Gustafson, Patrick Baudisch

TouchProjector: Mobile interaction through video

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'10), Atlanta, USA, 2010.

Robert Hardy, Enrico Rukzio, Paul Holleis, Matthias Wagner

Mobile interaction with static and dynamic NFC-based displays

Proceedings of the 12th international conference on Human computer interaction with mobile devices and services

(MobileHCI'10), Pages 123-132, Lisboa, Portugal, 2010.

Claudio S. Pinhanez

The Everywhere Displays Projector

Proceedings of the International Conference on Ubiquitous Computing (UbiComp'01), Pages 215-331, Atlanta, USA, 2001.

Gabor Soros, Peter Rautek, Hartmut Seichter, Eduard Groller

Augmented visualization with natural feature tracking

Proceedings of the 10th ACM Conference on Mobile and Ubiquitous Multimedia (MUM'11), Beijing, China, 2011.

John Hardy, Jason Alexander

Toolkit Support for Interacitve Projected Displays, MUM’12

Dominik Schmidt, Fadi Chehimi, Enrico Rukzio, Hans Gellersen

PhoneTouch: A technique for Direct Phone Interaction on Surfaces

Sebastian Boring, Sven Gehring, Alexander Wiethoff, Magdalena Blöckner, Johannes Schöning, Andreas Butz

Multi-User Interaction on Media Facades Through Live Video on Mobile Devices

CHI’11

47

Pervasive Displays - ETH ZPervasive displays as part of pervasive / ubiquitous computing ... using...

Documents

Transcript of Pervasive Displays - ETH ZPervasive displays as part of pervasive / ubiquitous computing ... using...