Post on 21-Mar-2020
Pervasive Displays
Reto Achermann acreto@student.ethz.ch
March 12, 2013 1 Distributed Systems Seminar
PART I:
PERVASIVE DISPLAYS
March 12, 2013 2 Distributed Systems Seminar
htt
p:/
/ww
w.1
815.c
h/im
ages/4
419_1.jp
g
htt
p:/
/ww
w.g
oogle
.com
/sync/im
ages/s
ync-c
onta
cts
.png
htt
p:/
/ww
w.g
oogle
.com
/gla
ss/s
tart
/what-
it-d
oes/
htt
p:/
/g-e
cx.im
ages-a
mazon.c
om
/im
ages/G
/01/e
lectr
onic
s/d
eta
il-p
age/B
004T
T0
1JS
-clo
udl.jp
g
Pervasive Displays
Just “screens everywhere” ?
March 12, 2013 3 Distributed Systems Seminar
htt
p:/
/ww
w.t
imessquare
nyc.o
rg/im
age.a
spx?id
=1838&
wid
th=
137
0&
heig
ht=
87
0
Not really…
Pervasive Displays
Pervasive “spreading widely throughout an area or a group of people”
Display “electronic device for visual presentation”
“printing the arrangement and choice of type in a style intended to attract attention”
“a collection of objects for public viewing”
- Oxford English Dictionary
March 12, 2013 4 Distributed Systems Seminar
Pervasive / Ubiquitous Computing
Pervasive displays as part of pervasive / ubiquitous
computing
Principles of Ubiquitous Computing “The purpose of a computer is to help you do something else.”
“The best computer is a quiet, invisible servant.”
“The more you can do by intuition the smarter you are; the computer should extend your unconscious.”
“Technology should create calm.”
March 12, 2013 5 Distributed Systems Seminar
htt
p:/
/en.w
ikip
edia
.org
/wik
i/F
ile:M
ark
_w
eis
er.
jpg
- Mark Weiser
(1952-1999)
“Today’s public display systems are largely closed and
isolated, with tightly controlled screen access.”
Analogy:
Old mobile phones – current public display systems
- Nigel Davies et al.
March 12, 2013 6 Distributed Systems Seminar
In the early days…
March 12, 2013 7 Distributed Systems Seminar
htt
p:/
/4.b
p.b
logspot.
com
/_U
tUF
vF
_vR
lo/S
FA
lhV
1cw
XI/
AA
AA
AA
AA
AU
U/G
5G
pdA
rwqtg
/s1600/1
904+
coke+
ad.jp
g
htt
p:/
/sto
re.u
shis
tory
.org
/im
ages/m
ediu
m/w
ant-
dalton.jp
g
Public Information Advertisement
… and today
March 12, 2013 8 Distributed Systems Seminar
htt
p:/
/ww
w.1
815.c
h/im
ages/4
419_1.jp
g
htt
p:/
/blo
g.h
slu
.ch/o
uto
fhom
edis
pla
ys/w
p-c
onte
nt/
blo
gs.d
ir/1
83/f
iles/p
oin
t-of-
transport
atio
n/a
pg_
ep
an
el_
zuerich_
2.jp
g
Information / News Display
Electronic Advertising
Information presented to the
user interleaved with ads
Content is very repetitive:
User gets bored
No interaction or possibility to
change content
Showed content may be
moving / flickering to attract
the attendance
… and today
March 12, 2013 9 Distributed Systems Seminar
htt
p:/
/uplo
ad.w
ikim
edia
.org
/wik
ipedia
/com
mons/t
hum
b/7
/7b
htt
p:/
/ww
w.h
ers
che.a
t/uplo
ads/p
lakat_
ap
pe
nzelle
r_kalte_fu
sse_gr1
0_zug
eschn_korr
.jpg
Poster Advertising
Augmentation
Interaction needs application
(tag reader) or sending SMS
Augmentation is sometimes
inaccurate / impossible without
location service
Maps just show “Here are you”
indicator (hard to find)
Display Types
Static Displays: Cannot change the content
shown
Provide no direct feedback
Cheap (big surface)
Require no energy
Posters, Maps, Objects
Dynamic Displays: Can change content shown
dynamically
Can provide direct feedback
Relatively expensive
Require energy and actively driven input
LCD Screens, Projectors
March 12, 2013 10 Distributed Systems Seminar
htt
p:/
/ww
w.1
815.c
h/im
ages/4
419_1.jp
g
htt
p:/
/ww
w.h
ers
che.a
t/uplo
ads/p
lakat_
ap
pe
nzelle
r_kalte_fu
sse_gr1
0_zug
eschn_korr
.jp
g
Both may be location / environment aware
Are today’s displays pervasive?
“Yes” in the sense of Oxford dictionary
No in the sense of ubiquitous computing People have adapted to ignore the displays
Fast moving pictures distract calmness
Almost no interaction / non-intuitive interaction
Interaction / augmented reality is “plug-and-play” instead of “arrive-and-operate”
Displays provide no / too less help in doing something
March 12, 2013 11 Distributed Systems Seminar
htt
p:/
/ww
w.b
eobachte
r.ch/t
ypo
3te
mp/p
ics/S
BB
-Auto
mate
n_d5c70a1bcd.jp
g
htt
p:/
/1.b
p.b
logspot.
com
/
htt
p:/
/ww
w.t
imessquare
nyc.o
rg/im
age.a
spx?id
=1838&
wid
th=
137
0&
heig
ht=
87
0
Observation: Smartphones are everywhere
“The first truly pervasively available interaction devices”
- Robert Hardy
12
htt
p:/
/ww
w.s
prin
gw
ise.c
om
/im
g/u
plo
ads/2
012/0
9/d
andeacon.jp
g
Observation: Smartphones are everywhere
Equipped with various sensors, radio interfaces, acoustic &
haptic feedback and touch displays
Problem: Interaction with displays using smart phones
today is inconvenient, not always working and clumsy
Idea: use of technology provided by the smartphone in a
smart way for interaction
March 12, 2013 13 Distributed Systems Seminar
The vision in the movies
March 12, 2013 14 Distributed Systems Seminar
htt
p:/
/jo
rdste
r4000.b
logspot.
ch/2
011/0
9/la
yo
ut-
desig
n-r
esearc
h.h
tml
htt
p:/
/ww
w.b
itre
bels
.com
/wp
-conte
nt/
uplo
ads/2
010/1
2/K
inect-
Min
ority
-Report
-UI-
2.jp
g
Minority Report,
DreamWorks Pictures
Avatar
20th Century Fox
Visions
Interactive Street Map
Intuitively select an element on the map
Get related information of selected object
Query / Filter:
Get list of all restaurants
in a specific area
Show directions to object
March 12, 2013 15 Distributed Systems Seminar
htt
p:/
/ww
w.p
re.e
thz.c
h/a
ddre
ss/
Visions
Interactive Mensa Menu
Show additional information: Menu ratings
Alternative Menus at other location
Filter according to preferences
Example: Diet help Caloric values directly added to diary
Do not show menu which do not fit
Example: Allergic / Health Hide menus containing certain ingredients
Show warnings
March 12, 2013 16 Distributed Systems Seminar
htt
p:/
/a5.m
zsta
tic.c
om
/us/r
1000/1
03/P
urp
le/e
f/b1/2
6/m
zl.jlm
dxofp
.320x480
-75.jp
g
Visions
Passive Interaction with Displays
No need for taking the phone out of the pocket
Walk-by / be present as interaction event
Display spontaneously react to your presence and shows
content you like (pictures, videos) Privacy?
Example: Screen at CAB Foyer You like to see images of D-INFK
events
Your phone tells the screen wirelessly
Screen displays selection of photos
March 12, 2013 17 Distributed Systems Seminar
Visions
Personalized Content
Content is chosen according to personal preferences Switch paintings of e-gallery
Show news you’re interested in / weather of your location
Show tailored advertisement
Showing related information to query
Several social issues
Example: Team Support A group of soccer fans meet
The displays show the colors of the favorite team
March 12, 2013 18 Distributed Systems Seminar
ww
w.g
oogle
.com
htt
p:/
/i.t
ele
gra
ph.c
o.u
k/m
ultim
edia
/arc
hiv
e/0
1655/f
lags-s
treet1
_1655777i.jp
gc
Visions
Augmented Displays
A tablet’s display augment the view of the current display Different angle / layer
Annotations
Show additional information of the elements shown
Example: Doctors Different experts analyze a patient
Cardiologist
Sinologist
Surgeon
All have different interests concerning the patient’s health state
March 12, 2013 19 Distributed Systems Seminar
htt
p:/
/jo
rdste
r4000.b
logspot.
ch/2
011/0
9/la
yo
ut-
desig
n-r
esearc
h.h
tml
Visions
Houses as Displays
March 12, 2013 20 Distributed Systems Seminar
Using the façade of a house as a game display
Borin
g e
t al 2011
htt
p:/
/tro
yh.u
s/p
hoto
s/C
ity%
20S
kylin
es%
20&
%20D
ow
nto
wn/1
37
04%
20D
alla
s%
20skylin
e%
20at%
20nig
ht.
jpg
Visions
Projected Displays
March 12, 2013 21 Distributed Systems Seminar
Visions
Video:
http://www.youtube.com/watch?feature=player_embedded
&v=df1NO7MoAUY
PART II
INTERACTION WITH DISPLAYS
March 12, 2013 22 Distributed Systems Seminar
htt
p:/
/farm
4.s
taticflic
kr.
com
/3259/3
221222512_723
ed
0e
08
3_
z.jp
g?zz=
1
htt
p:/
/b.v
imeocdn.c
om
/ts/3
56/3
89/3
56389029_64
0.jp
g
1. Physical Buttons
Select elements by pressing buttons
Buttons can be marked to distinguish them blindly
Drawbacks: Not always clear what different buttons do
Unclear element highlighting
No button adaption to different content
Button interface is not extensible
March 12, 2013 23 Distributed Systems Seminar
htt
p:/
/pre
vie
w.t
urb
osquid
.com
/Pre
vie
w/2
011/0
9/
Physical Buttons
2. Touch Screens
Interact by touching the element directly on the screen
More intuitive than physical buttons, adaptable interface
Drawbacks: Indistinguishable buttons (no blind navigation)
Dirty displays
Not well suited for very large screens
Not working with Displays behind security glass
Displays far away
Wet fingers / gloves
March 12, 2013 24 Distributed Systems Seminar
htt
ps:/
/activecapta
in.c
om
/art
icle
s/m
obile
Phones/iP
hone/iP
hone_K
eyboard
.jp
g
Touch Screens
3. Bluetooth Device Names for Interaction
Idea: send service requests to displays wirelessly
Issues tackled: SMS: untrusted number (premium service), need to know display ID
App: download necessary (inconvenient)
Touching may not be possible
Observation: many users have device supporting
Bluetooth and have set a custom USB device names
Approach: Use of Bluetooth device discovery and
Bluetooth device names to send requests
March 12, 2013 25 Distributed Systems Seminar
Bluetooth
[Nigel Davis et al]
Bluetooth Device Names as Commands
No need for additional software
Users set the device name to a special command string, to
send a request command to the system
Identifier followed by service name
Examples:
March 12, 2013 26 Distributed Systems Seminar
Bluetooth
ec <service_name> <params>
ec youtube: eth ec map: CAB H52
Scanning Area
Principle
March 12, 2013 27 Distributed Systems Seminar
htt
p:/
/en.w
ikip
edia
.org
/wik
i/F
ile:B
lueto
oth
Logo.s
vg
Display
Smart
Phone Smart
Phone
Smart
Phone
Smart
Phone
Display Machine
Problem:
reading the device ID takes time
discovery happens too early / late
Bluetooth
Serving the Requests
Each request (recognized command) is put into a queue
Each request is served for a maximum specified time
If a user leaves the display area, the requests is marked as
served
Social issues arise
ec youtube: puke
March 12, 2013 28 Distributed Systems Seminar
Bluetooth
4. Visual Markers
Using special patterns for interaction
Identify the display / element by
special marker (QR Code / edges)
Touch like interaction at distance
March 12, 2013 29 Distributed Systems Seminar
[Ballagas et al. - Point & Shoot (2005)]
htt
p:/
/ww
w.id
ea.o
rg/b
log/w
p-c
onte
nt/
uplo
ads/2
01
1/0
9/q
r-code
-01.jp
g
Visual Markers
5. Mobile Interaction with NFC enabled displays
Near field Communication Wireless communication technology, Point-to-Point
Small Range: < 0.2m
Frequency: 13.56 MHz
Bandwidth: 424 MHz
Set-up time: <0.1s
Low-Power, tag is unpowered
Applications Payments (Credit Cards, …)
Keys
Data Exchange (Business Cards)
March 12, 2013 30 Distributed Systems Seminar
htt
p:/
/askbobra
nkin
.com
-walle
t-nfc
.jp
g
htt
p:/
/en.w
ikip
edia
.org
/wik
i/F
ile:N
FC
-N-M
ark
-Logo.p
ng
NFC
NFC enabled displays
A mesh of NFC tags attached to the
rear, no need for visual markers
Interaction by “touching” i.e. holding
the device close
Mobile device reads content of the
NFC tag:
Object ID to look up on internet
Self contained information (no lookup)
Mobile device may augment the
display by showing information
March 12, 2013 31 Distributed Systems Seminar
Robert Hardy, Enrico Rukzio, Paul Holleis, Matthias Wagner
Mobile interaction with static and dynamic NFC-based displays
NFC
Resolution Problem of NFC Enabled Displays
Resolution Problem: Mesh grid of NFC tags is coarse.
One tag covers many elements
Dynamic Solution: “Zoom in”: Show pop up
Static Solution: Show list on mobile phone
Generic Approach: Filtering Enable filter to reduce the
result set
March 12, 2013 32 Distributed Systems Seminar
htt
p:/
/ww
w.p
re.e
thz.c
h/a
ddre
ss/
NFC Tag Area
Filters Zoom In
NFC
6. Accelerometers: PhoneTouch
Goal: Distinguish multiple users interacting with a touch
screen at the same time
Touchscreen registers location of touch event
Phone registers movement
using accelerometer
Tabletop server matches touch
event with phone movement
to identify user
March 12, 2013 33 Distributed Systems Seminar
Accelerometers
PhoneTouch: A Technique for Direct Phone Interaction on Surfaces Dominik Schmidt, Fadi Chehimi, Enrico Rukzio, Hans Gellersen
Computing Department, Lancaster University, Lancaster, UK
7. Touch Projector: Touch Screen from Distance
Observation: Not all display support touch input
Displays may be out of arm’s reach
Elements cannot be moved between displays easily
Idea: interaction through live video
March 12, 2013 34 Distributed Systems Seminar
Sebastian Boring, Dominikus Baur, Andreas Butz, Sean Gustafson, Patrick Baudisch
TouchProjector: Mobile interaction through video
Touch Projector
Touch Projector Usage
All touch events are routed through
a server (the environment manager)
Basic Usage: User points at display and touches element
User moves element within / between screens
User releases touch and element is placed on new location
March 12, 2013 35 Distributed Systems Seminar
Touch Projector
Touch Projector Implementation Issues
Screen too small on mobile device: Automatic zoom in
when display is recognized
Device needs to be pointed on screen: Freeze image on
mobile device for stability and fine tuning
Display identification based on computer vision may result
in incorrect identification
March 12, 2013 36 Distributed Systems Seminar
Touch Projector
8. Augmented Displays
Problem:
One screen – different people and different interests
Idea: Augment main display with different views / layers
March 12, 2013 37 Distributed Systems Seminar
htt
p:/
/qvecto
rs.n
et/
dow
nlo
ads/im
ages/f
ullp
revie
w/v
ecto
r-B
usin
ess-P
resenta
tio
n_fu
ll.jp
g
Augmentation
The Magic Lens Metaphor
Show alternate view of the data
Show annotations / information on the object
March 12, 2013 38 Distributed Systems Seminar
Augmentation
7. Gesture Recognition using Kinect
Improved hygiene: no need to touch
March 12, 2013 39 Distributed Systems Seminar
htt
p:/
/ww
w.d
igitaltre
nds.c
om
/com
putin
g/k
inect/
Gesture
SOCIAL ACCEPTANCE ISSUES
Security
Content
Privacy
March 12, 2013 40 Distributed Systems Seminar
htt
p:/
/gig
aom
2.f
iles.w
ord
pre
ss.c
om
/2012/0
6/c
om
score
-facebook.p
ng
Security: SMS / QR Codes
Sending an SMS can end in a premium service
QR codes may refer to phishing sites
QR codes / SMS numbers may be forged
March 12, 2013 41 Distributed Systems Seminar
htt
p:/
/mobhappy.c
om
/blo
g1/w
p-c
onte
nt/
uplo
ads/2
006/0
3//
sw
eetr
elie
f.jp
g
htt
p:/
/1.b
p.b
logspot.
com
/htt
p:/
/ww
w.t
-onlin
e.d
e/h
andy/s
mart
phone/id
_6
13
01
37
6/f
als
che
-qr-
codes-
expert
en-w
arn
en
-vor-
sm
art
phone
-phis
hin
g.h
tml
Social
Privacy Concerns: Unveiling Personal Data
Personalized display content unveil personal data e.g. advertisement of products recently bought
e.g. support for political party / sports team
User may need to fill in his preferences
Scanning QR code / sending SMS
unveils phone ID
Technology may be used to
track the user where he goes
March 12, 2013 42 Distributed Systems Seminar
htt
p:/
/medaid
pharm
acy.c
om
/wp
-conte
nt/
uplo
ads/2
011/0
5/t
op
-lo
go_sm
all3
.png
htt
p:/
/ww
w.m
aste
rfile
.com
/sto
ck-p
hoto
gra
phy/im
age/4
00
-0505966
4/s
peaker-
or-
film
-directo
r-and-
scre
en
---c
art
oon
-sty
le
Social
Effort Needed to Use the Displays
Must be kept as low as possible
No plug-and-play but arrive-and-operate:
the use has to be intuitive
Effort needed is percept as cost
March 12, 2013 43 Distributed Systems Seminar
htt
p:/
/help
desk.n
ex-t
ech.c
om
/prin
t.php?id
=657
htt
p:/
/tutz
net.
com
/2966
-fix
-pla
y-s
tore
-err
or-
403-a
ndro
id/
Social
Content
User decides what to display May be inappropriate
(e.g. offensive views)
May distract other people (e.g. music styles)
The content providers must be trusted Not all display owners want all contents on their display
Concept of “Trusted Store” like an app store
Content war between users
March 12, 2013 44 Distributed Systems Seminar
htt
p:/
/ww
w.2
0m
in.c
h/s
chw
eiz
/new
s/s
tory
/1818593
5
Sunrise Demonstration Incident
Social
Summary of Today
Today’s displays are not really pervasive (not helpful)
Intuitive interaction with displays needed to turn them into
helping assistants
There are many technical & social issues to solve
Most of the basic
technology is already
available
45
htt
p:/
/ww
w.g
oogle
.com
/gla
ss/s
tart
/
THANKS
Questions ?
March 12, 2013 46 Distributed Systems Seminar
htt
p:/
/podcasts
.in
sitew
ebsitedesig
n.c
om
/in
ters
ect-
vid
eo-w
all-
sequence
-fro
m-c
huck-H
D-s
tills
-04.jp
g
References
Nigel Davies, Marc Langheinrich, Rui Jose, Albrecht Schmidt
Open Display Networks: A Communications Medium for the 21st Century
IEEE Computer, Volume 45, Issue 5, Pages 58-64, 2012.
Florian Alt et al.
Designing Shared Public Display Networks: Implications from Today's Paper-Based Notice Areas
Proceedings of the 9th international conference on Pervasive computing (Pervasive'11), San Francisco, CA, USA, 2011.
Sebastian Boring, Dominikus Baur, Andreas Butz, Sean Gustafson, Patrick Baudisch
TouchProjector: Mobile interaction through video
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'10), Atlanta, USA, 2010.
Robert Hardy, Enrico Rukzio, Paul Holleis, Matthias Wagner
Mobile interaction with static and dynamic NFC-based displays
Proceedings of the 12th international conference on Human computer interaction with mobile devices and services
(MobileHCI'10), Pages 123-132, Lisboa, Portugal, 2010.
Claudio S. Pinhanez
The Everywhere Displays Projector
Proceedings of the International Conference on Ubiquitous Computing (UbiComp'01), Pages 215-331, Atlanta, USA, 2001.
Gabor Soros, Peter Rautek, Hartmut Seichter, Eduard Groller
Augmented visualization with natural feature tracking
Proceedings of the 10th ACM Conference on Mobile and Ubiquitous Multimedia (MUM'11), Beijing, China, 2011.
John Hardy, Jason Alexander
Toolkit Support for Interacitve Projected Displays, MUM’12
Dominik Schmidt, Fadi Chehimi, Enrico Rukzio, Hans Gellersen
PhoneTouch: A technique for Direct Phone Interaction on Surfaces
Sebastian Boring, Sven Gehring, Alexander Wiethoff, Magdalena Blöckner, Johannes Schöning, Andreas Butz
Multi-User Interaction on Media Facades Through Live Video on Mobile Devices
CHI’11
47