Haptic Touch Screens for Mobile Devices: Feedback ...hui/mobile/papers/Muller-paper.pdf · Haptic...
-
Upload
truongkhue -
Category
Documents
-
view
214 -
download
0
Transcript of Haptic Touch Screens for Mobile Devices: Feedback ...hui/mobile/papers/Muller-paper.pdf · Haptic...
Haptic Touch Screens for Mobile Devices:
Feedback & Interaction
in
Haptic Communication and Interaction in Mobile Contexts
University of Tampere, FI
11/12/2008
Sebastian Müller
Table of ContentsIntroduction..........................................................................................................................................2Theory..................................................................................................................................................3
The Haptic Sense.............................................................................................................................3What Information to deliver.............................................................................................................5How to deliver..................................................................................................................................5
Practice.................................................................................................................................................6Summary..............................................................................................................................................9References.............................................................................................................................................9
IntroductionTouchscreens bring a great amount of directness into the communication between man and machine.
Buttons can be directly pressed where they are instead of moving a mouse and text can be written
where it is supposed to be, just to name two examples. Even though usage of a touch screen requires
training and its technology does by no means replace complete computer architectures, touchscreen
technology can be helpful in a variety of ways.
The same holds for haptic technologies in touch screens. Haptic actuators can present users a feeling
they might be familiar with or can relate to. Several types of users can be addressed: Computer
savvy users can easily integrate the new information into their working (or gaming) habits. Haptic
feedback is in many cases faster and in any case more direct than visual feedback [10, 11]. People
who live with and around technological devices benefit from the visualindependent and sometimes
even attentionindependent aspects of haptics. Computer novices may appreciate the physical
metaphor that is constructed when pressing a button actually feels like a button and an error is
sensible. There is a threshold to overcome when people are unfamiliar with devices, their usage and
functioning, but additional information channels can only help people to learn intuitively. Often,
these novices are older people that have to deal with other problems than the average technical
gadget audience. Visual impairment means a lot in this domain. There are hardly any technical
devices that do not require reading a screen or blinking LED error messages. Touchscreens make
most tactile interfaces such as keyboards redundant. Obviously, it makes little sense to use haptic
touchscreens if users cannot read any screen, but haptically enhanced GUI elements are only at a
very early stage of investigation. Well adapted feedback may draw level with visual interaction some
day.
One of the driving forces of touchscreens and onscreen keyboards are mobile devices and
applications. Devices that require to fit into the pocket do not allow a lot of space, and compared to
the many technical finesses modern handheld devices carry, a keyboard with enough keys that are
large enough to be used by any user takes excessive amounts of space. For many people, the
keyboard is the most important input device and cannot be replaced by a smooth surface of virtual
buttons. It has already been shown in experiments that tactile feedback can significantly improve
entry speed and error rates [10, 11] up to empirical values experienced with normal keyboards.
Mobile application requires not only thoughtful use of space, but also context sensitivity. As mobile
devices can be taken anywhere, they can be used in any situation and therefore not rely on singular
modes of interaction or feedback channels. Opposed to the static keyboard on every computer, on
screen keyboards can deliver more information than just the position and status of a key, and
different than the vibration alarm of common mobile phones, tactile actuators have a much broader
range of actions, flexible in frequency, amplitude and space. Haptic screens are not the answer to all
contextsensitive interaction questions, but they offer more reliable input than plain onscreen
keyboards on smaller space than physical keyboards. Furthermore, if the haptic communication is
limited to fingertip/penscreen interaction, designers do not need to worry about appropriateness of
the feedback which is often in question when it is applied to other body parts [12].
Apart from stepping into the breach where other interaction techniques fail, haptic feedback on
touchscreens offer a wide variety of new applications and interactions that were not possible before.
Such as GUI elements are haptically enhanced, so can basically anything. The haptic TBar [9] is a
new GUI element that was specifically designed for haptic touchscreens. When this technique steps
out of vibrotactile motors, spatially arranged haptic actuators fed by software could make anything
tactile.
TheoryHaptic feedback on touchscreens is a rather new field of study, and therefore many aspects have not
been explored. This chapter deals with the physical possibilities and especially limits of the human
touch sense as well as the theoretical nature of touchscreen interaction.
The Haptic Sense
The haptic sense has been explored for humancomputer interaction for a much longer time. It
became popular and wellknown as the vibrating alarm in cell phones. Compared to other senses
often used in humancomputer interaction such as vision and audition, it is considered more direct.
This advantage is often paid for with lack of precision. Embedded in a device, haptic actuators
usually carry their movement over to the whole device. However, the fingertips are one of the parts
of the human body with the highest amount of somatic sensors. Considering that the fingertips are
also the most precise tool you have and they are in use when working on a touchscreen, they are the
perfect channel for haptic information.
Under certain conditions, haptic information can be ingested up to five times faster than visual
information [1]. This is mostly due to the fact that haptic information go through less processing
stages than visual information. This indicates that haptics do not mean so much to us as visuals do.
However, haptic feedback is not intended to replace or simulate visual information. Instead, the two
– in fact any second sense can support and influence each other. Tikka and Laitinen [2] found that
“audio [...] biases the perception and increases the perceived strength of the haptic stimulus.”
Since the uprising of virtual keyboards, the haptic capacity of fingertips has been investigated
intensively. Just noticeable movements and displeasing sensations are important landmarks in haptic
feedback design. Barfield and Furness [] found that vibrotactile sensations 2 millimetres apart from
each other can be identified as two distinct sensations. Furthermore, a shift of position of a
vibrotactile event as small as 0.2mm can be noticed. It is important to remember that these results
were found under laboratory conditions. Touchscreens, however, play an important role in the design
of mobile devices (due to the lack of space and replacement of physical keyboards with virtual
keyboards) and consequently cannot be used as definite values as the context and environment in
which these devices are used is highly variable.
One aspect of haptic feedback that does not apply to
the fingertips is the privacy effect: Haptic feedback
that is applied to other parts of the body such as the
back, stomach or feet needs to be more finetuned
for three reasons. First, many parts of the body do
not have as many somatic sensors as the fingertips
have. Haptic events therefore need to be stronger or
spatially more differentiated. Second, most body
parts are not used as much as the fingers for input
(which is, from an evolutionary perspective, the
reason why they do not have as many receptors as
the fingers) and touch is easier considered “sensual”. Third, touchscreens usually display
Illustration 1: Homunculus and brain [3]
information on demand, meaning that it requires the user's visual attention. When touchscreen
devices are carried around unused, the screens are usually turned off, and so is the haptic feedback.
The devices with haptic feedback that are sold or will be sold in the near future only respond to the
user's input. Future touchscreens might display haptic information independent from user input, but
so far it is difficult to see if these fall under the privacy effect at all.
What Information to deliver
The haptic information space consists of several dimensions. Most of them are unused in common
humancomputer interaction but nevertheless highly useful when designing haptic icons. Haptic
icons are single informationcarrying units consisting of haptic features. These features are spread in
the feature space of the human somatic sense. The principles of haptic icon design become more
important the more information is to be delivered. Questions that arise are: What is the smallest
perceivable haptic event? How many different tactile events can be differentiated? What is the
qualitative perception of specific features?
Koskinen, Kaaresoja, and Laitinen [14] conducted an experiment to find “the most pleasant tactile
feedback ”. Their results reveal a lot about subjective perception, and most of their results can be
found in other publications as well: The participants of their study found the feedback from a piezo
actuator slightly more pleasant than from a vibration motor. Also, the most pleasant stimuli had the
same current (46mA), but differed mostly in maximum displacement. Furthermore, vibration motor
stimuli are perceived most pleasant between 13 and 19 milliseconds. Although these results cannot
be generalized to all touchscreen architectures, the authors consider them indicators for other
technologies. In any case, tactically enhanced touchscreens are perceived better than simple
touchscreens.
An interesting observation about the difference between actual and perceived feedback can be made
with the Apple Inc.'s Mighty Mouse1: When plugged in, a small speaker inside the mouse plays
clicking sounds when a button is pressed or the mouse wheel (“Scroll Ball”) is moved. If the energy
source is cut off, buttons and wheel remain silent.
Haptic feedback on touchscreens has already been tested on a number of applications. Kaaresoja
and others [4] have demonstrated the applicability of tactile feedback on text entry, text selection,
scrolling and draganddrop applications. They enhanced simple clicks (usually left mouse button
clicks) with a response from a piezoelectric actuator. Moreover, dragging the stylus over the screen,
as in text selection or draganddrop, created several different feedbacks. In other experiments,
rotation and zooming were enhanced with a tactile feeling. It also seems convenient to differentiate
1 http://www.apple.com/mightymouse/
between left and right as well as single and doubleclicks, when the system allows this. Clicking
usually refers to the activation of GUI elements like buttons. One of the most eagerly awaited
applications of haptic feedback is the tactile onscreen keyboard. In some devices that are already
sold comprehensively, the selection of a button is simply connected to one short haptic event. If
properly designed, this event actually “feels” like the push of a button. As we will see later (see
Chapter 4), there already exist much better techniques of how to imitate the behaviour of a physical
keyboard.
Other information that could be transferred through haptic feedback are the notification of events
that run in the background, time (as the system time, time spent on a task or remaining time in a
countdown) or the strength of the input, if the touchscreen is pressuresensitive.
How to deliver
On account of the novelty of the concept, most experiments concerning the perception of haptic
feedback are made with unique prototypes of standard touchscreen devices upgraded with external
actuators. Therefore no widely adopted rules about what exactly specific haptic events should “feel”
like exist. Generally, haptic pulses are defined by frequency and amplitude. The higher both, the
stronger the perception. Ternes and MacLean have shown that the perception of frequency is
strongly subjective and might require individual haptic design, whereas amplitude is the strongest
differentiating factor [5].
Additionally, haptic icons can differ in length, such as many cellphones differentiate between calls
and text messages, and space. Wherever more sophisticated haptic feedback is required, higherlevel
Illustration 2: Interaction techniques with a handheld device
features consisting of different lowlevel features, such as rhythm [5], emerge.
The design of haptic feedback not only depends on the physical limits of sensors and actuators.
Especially in mobile devices, the appropriateness of signals depends on the context of usage. The
following is a list of design issues that may arise from other design decisions.
● Touchscreen technologies are various. Some are pressuresensitive and reactive, some
require solid surfaces. Most importantly, some can be used with a stylus. In that case the
haptic effect needs to be transferred through the stylus. Other sensors, such as capacitive
sensors, require the use of skin, i.e. fingertips, to work.
● Device layout. Depending on the layout and size of the whole device, users might use one
hand to hold the device and the other hand for input. Other devices can be held with both
hands and input happens with the thumbs. The use of a stylus might indicate the need for
higher accuracy than the use of fingertips. In these cases, the user might want to place the
device on a flat surface like a table or rest the wrist on the device for stabilization.
Vibrations on hard surfaces can create a lot of noise inappropriate under certain conditions.
● Distractions. Subjective perception of haptic events may be different depending on the
surrounding. A vibration that seems appropriate under laboratory conditions might not be
perceived at all in a packed subway where the user is subject to the physical forces of the
vehicle and flooded with other sensory perceptions.
● Task. The task that shall be performed on the device can influence any stage of design.
Depending on how specialized a device is, haptic feedback may be specialized as well.
Haptic feedback at a construction site needs to be much stronger (if not inappropriate at all)
than in a lecture or music hall.
PracticeIn this chapter, some already tested techniques will be presented. The design issues in the previous
chapters were of theoretical nature, the following are more practically oriented issues.
In 2001, Fukumoto and Sugimura [13] published their work about attaching an “electric to vibration
transducer” to a PDA. They also explored the possibilities and difficulties of designing haptic
feedback for different situations, for example comparing a handheld device to a large public
information terminal.
In 2003, Poupyrev and Maruyama published a design and implementation of a tactile interface [6].
They embedded a TouchEngine [7] in the touchscreen of a Sony Clié PDA, where the actuators were
placed at the sides of the display. The design of the device allowed the authors to place the actuators
between the covering glass plate and the TFT display, therefore requiring less energy. Although the
displacement of the glass measured only 0.5 mm, a “very strong tactile sensation” was reported. The
authors describe the tactile feedback in their device as “localized”, in the sense that only the glass
under the user's finger or stylus moves. Transfer of the movement onto the rest of the device was
inhibited with silicon dumpers on each side of the actuators.
Piezoelectric actuators are made of piezoceramic layers. In the case of the TouchEngine, each of
these layers is only 0.28 μm thick and can be assembled in any size and quantity. The displacement
does not depend on the thickness of the actuator but on the applied voltage. If the actuators were
thicker, they would separate surface and display which results in the parallax problem. The parallax
problem describes that the arrangement of two or more objects can appear different from different
perspectives. The further away display and surface of a touchscreen are from each other, the more
the user has to pay attention to look at the display at a right angle. This phenomenen can often be
observed on ATM machines, as their screen is protected by a thick layer of glass, which makes the
displayed area further away from the hardware buttons on the sides of the screen. When the screen
then points to one of the buttons (to ask the user for a selection), the assignment of display area to
buttons can be distracted depending on the user's point of view.
Johnny C. Lee and others published a paper on tactile feedback for touchscreens via a stylus in 2004
[8]. This idea limits tactile feedback to touchscreens used with styli, but on the other hand solves
several problems that arise with actuators under displays and offers other functions that do not apply
otherwise. Lee's system consists of a solenoid at the top of the pen, a pressuresensitive tip, a power
Illustration 3: Haptic display for small touch screens, after Poupyrev & Maruyama [6]
source, a location discovery system and a communication link to a host PC. Generally, the pressure
sensitivity and location discovery could be provided by a touchscreen, but this way the pen can be
used with any surface. It is even possible to draw own objects on a piece of paper and then to assign
them an individual haptic feeling. Lee's pen is connected to the host PC via a cable. It is however
imaginable to use a bluetooth connection which gives the user more freedom. Another advantage of
this design is that it can be used by multiple users at the same time. In the event of pressure on the
tip, each pen communicates its position to the host PC independently. The host PC, in return, can
trigger possible actions that result from the user's action and can give feedback about where and
what the user did (for example pressing a button or selecting text).
The total cost of components in Lee's prototype was less than 10 US$.
So far, we have focused on the how to design a device or its behaviour for haptic feedback
interaction. However, the technology offers new information channels which require, or at least
offer, new software and applications. One example of new interaction techniques is the TBar [9]. It
is a new GUI element that takes advantage of simulated haptic feeling to guide the user on the
screen towards a target object. The target object is represented as a short bar, like the upper bar of a
capital “T”, of haptic feedback that gets stronger
towards the middle. On its side at the centre is a longer
bar with the same features that spreads along the
largest possible area of the screen. When users blindly
feel the screen, they will eventually meet with the
larger bar and can then follow this bar until they reach
the crossing of both bars. Then changes of tactile intensity let the users know if they veer away from
the target.
The TBar suffers from some teething troubles yet. As the authors report, the increase of intensity
Illustration 4: Haptic Pen, after Lee et al. [8]
Illustration 5: TBar. After Hall, Hoggan & Brewster [9]
towards the centre of each bar was not as well distinguished as they had hoped. This means that the
bars need to be expanded, which makes them less accurate and keeps them from being used
numerously on small displays. Also, the question arises if the long bar might not be misleading
when the target object lies on an edge, because the user might navigate towards the wrong side of
the screen.
SummaryToday, the triumphant success of touchscreens, especially in mobile devices, seems unavoidable.
Every element of an electronic device needs to justify its use of space. Using the screen as an input
device helps saving space, but also brings a new directness to the interaction, which may be
especially helpful for inexperienced users. Missing haptic feedback is the major disadvantage for the
screen as an input device. Most marketready devices give only visual feedback, which requires the
user's visual attention. But current research shows possibilities how to add haptic and combined
(visual, audible, haptic) feedback to touchscreen devices. Although the presented feedback methods
are far away from simulating truly physical input devices, experiments show that they considerably
improve input speed and accuracy. Also, we have shown that current research is not solely on haptic
actuators combined with touchscreens, but new GUI elements are being developed, pens are
augmented with haptic actuators and the human haptic input channel is explored widely.
Little can be said about what turn this development will take the day after tomorrow. The
aforementioned patents by big companies such as Apple and Nokia give a hint. We might see three
dimensional touchscreens that can change their surface. There is little evidence of how well haptic
touchscreens will be received in practice, but a more holistic user experience with integrated haptic,
audible and visual feedback seems to be the way to go.
References[1]Geldard, F.A. Some Neglected Possibilities of Communication. In Science 131 (1960), 15831588.
[2]Tikka, V. and Laitinen, P. Designing Haptic Feedback for Touch Display: Experimental Study of Perceived Intensity and Integration of Haptic and Audio. In Lecture Notes in Computer Science 4129. Springer (2006), 36.
[3]Homunculus and brain.http://www.yogameditation.com/var/corporate/storage/images/media/images/bindu/27/homunculus_and_brain_engelsk/549301norNO/homunculus_and_brain_engelsk_image_300_w.jpg. Accessed 06.11.2008
[4]Kaaresoja, T. and Brown, L.M. and Linjama, J. SnapCracklePop: Tactile Feedback for Mobile Touch Screens. In Proceedings of Eurohaptics 2006, 565566.
[5]Ternes, D. and MacLean, K.E. Designing Large Sets of Haptic Icons with Rhythm. In Lecture Notes in Computer Science 5024. Springer (2008), 199.
[6]Poupyrev, I. and Maruyama, S. Tactile Interfaces for Small Touch Screens. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology 2 (2003). 217220.
[7]Poupyrev, I. and Rekimoto, J. and Maruyama, S. TouchEngine: A Tactile Display for Handheld Devices. In Conference on Human Factors in Computing Systems 2002. ACM Press New York, NY, USA (2002). 644645.
[8]Lee, J.C. and Dietz, P.H. and Leigh, D. and Yerazunis, W.S. and Hudson, S.E. Haptic pen: A Tactile Feedback Stylus for Touch Screens. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology. ACM New York, NY, USA (2004). 291294.
[9]Hall, M. and Hoggan, E. and Brewster, S.A. TBars: Towards Tactile User Interfaces for Touchscreen Mobiles. In Proc Mobile HCI (2008).
[10]Hoggan, E. and Brewster, S.A. and Johnston, J. Investigating the Effectiveness of Tactile Feedback for Mobile Touchscreens. ACM New York, NY, USA (2008).
[11]Leung, R. and MacLean, K. and Bertelsen, M.B. and Saubhasik, M. Evaluation of Haptically Augmented Touchscreen GUI Elements under Cognitive Load. In Proceedings of the 9th
International Conference on Multimodal Interfaces. ACM New York, NY, USA (2007), 374381.
[12]Bailenson, J.N. and Yee, N. and Brave, S. and Merget, D. and Koslow, D. Virtual Interpersonal Touch: Expressing and Recognizing Emotions Through Haptic Devices. In Human Computer Interaction 22 (3). Lawrence Earlbaum (2007), 325353.
[13]Fukumoto, M. and Sugimura, T. Active Click: Tactile Feedback for Touch Panels. In Conference on Human Factors in Computing Systems. ACM New York, NY, USA (2001), 121122.
[14]Koskinen, E. and Kaaresoja, T. and Laitinen, P. Feelgood Touch: Finding the most Pleasant Tactile Feedback for a Mobile Touch Screen Button. In Proceedings of the 10th international conference on Multimodal interfaces. ACM New York, NY, USA (2008). 297304.