Communication via the Skin: The Challenge of

Tactile Displays

Lynette Jones Department of Mechanical Engineering,

Massachusetts Institute of TechnologyCambridge, MA

                     

Spectrum of Tactile DisplaysSensory substitution Human-computer interactions Navigation/orientation

Visual impairments

Hearing impairments

Vestibular (balance) impairments

TSAS (Rupert et al.)

MIT tactile display

CyberTouch

Data Glove EST

Tactile mouse

Tactile belt

DataGlove

Tactile Displays - CTA ADA FocusUtilize a relatively underused sensory channel to convey

information that is private and discreet• Assist in navigation or threat

location in the battlefield• Increase SA in virtual

environments used for training• Enhance the representation of

information in displays

Torso-based Tactile Displays

        

                   

0.1 1 10 100 1000-20

-10

0

10

20

30

40

Frequency (Hz)

Disp

lace

men

t (dB

re 1

um

pea

k)

Abdomen

Finger

Forearm

•Function as an alert•Orientation and direction information •Sequential activation of array – vector conveys “movement” in environment•Effective in environments with reduced visibility – enhances situation awareness

Vibrotactile Sensitivity

Development of Tactile Display• Actuator (tactor) selection and

characterization• Development of body-based system

(configuration of display, power, wireless communication)

• Perceptual studies – optimize design of the display in terms of human perceptual performance

• Develop a framework for creating a tactile vocabulary – tactons

• Field studies – measure the efficacy of display for navigation, identifying location of environmental events, and examine robustness of system (e.g. impact of body armor)

Characteristics of the Actuators EvaluatedCylindrical

MotorPancakeMotor

R1Rototactor

LengthDiameter (mm)

12.84.0

14 25.46.4

Mass (g) 0.87 1.6 3

Peak frequency (Hz)

110(at 4 V)

103(at 8.8 V)

200(at 10 V)

Peak accel.(ms-2) 492 254

Voltage (V) Rated: 3Range: 2-4

82.5-8.8 10

Rated maximum current (mA) 130 65 170

Current (mA)at 3.3 V 91 39 56

0.6 1.0 1.4 1.8 2.2 2.6 3.0 3.40

25

50

75

100

Freq

uenc

y (H

z)

Voltage (V)

Pancake Motor

0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9 2.10

25

50

75

100

125

150

175

200

Freq

uenc

y (H

z)

Voltage (V)

Cylindrical motor

Rototactor

Pancakemotor

Cylindricalmotor

(Jones, Lockyer & Piateski, 2006)

C2 tactor

Tactaid

Prototypes 2003-2007

Tactile Display - Final ElementsCore components - Pancake motors, Wireless Tactile Control UnitContact area - ~ 300 mm2 (encased in plastic)Input signal – 130 Hz at 3.3 V, sinusoidal waveformPower – 9 V battery or 7.2 V Li-ion rechargeable, 2200 mAhDisplay – vest, waist band, sleeve

Visual Basic GUI

Actuator Evaluation – Frequencies and Forces

0 1 2 3 4 5 6 7 8 9 10 11 12 13 1480

100

120

140

160

180

Motor number

Freq

uenc

y (H

z)

0 1 2 3 4 5 6 7 8 9 10 11 12 13 140

0.5

1

1.5

2

2.5

Motor number

Forc

e (N

)

Mechanical properties not affected by encasing motors (Jones & Held, 2008)

Actuator Evaluation – Tactor spacing and Intensity

y = 0.3765x + 11.46R2 = 0.979

0

10

20

30

40

50

60

70

80

90

0 50 100 150 200 250

Peak Frequency in Impedance Head (Hz)

Peak

Fre

quen

cy o

n Sk

insi

m (H

z)

Mechanical testing of skin

Skinsim with accelerometers(Jones & Held, 2008)

Transitions – MIT Tactile DisplayARL

Investigated the efficacy of tactile and multimodal alerts on decision making by Army Platoon Leaders (Krausman et al., 2005, 2007) Analyzed the effectiveness of tactile cues in target search and localization tasks and when controlling robotic swarms (Hass, 2009)Evaluated Soldiers’ abilities to interpret and respond to tactile cues while they navigated an Individual Movement Techniques (IMT) course (Redden et al., 2006) Measured the effects of tactile cues on target acquisition and workload of Commanders and Gunners and determined the detectability of vibrotactile cues while combat assault maneuvers were being performed (Krausman & White, 2006; White et al., in press).

The MIT tactile displays have also been incorporated into multi-modal platforms developed by the University of Michigan, ArtisTech in the CTA test bed, and Alion MA&D for a robotics control environment.

Questions addressed – MIT Research• Can tactile signals be used to provide spatial cues about

the environment that are accurately localized? • How does the location and configuration of the tactile display

influence the ability of the user to identify tactile patterns?• What is the maximum size of a tactile vocabulary that could

be used for communication? • Which characteristics of vibrotactile signals are optimal for

generating a tactile vocabulary? • Can a set of Army Hand and Arm Signals be translated into

tactile signals that are accurately identified when the user is involved in concurrent tasks?

Localization of Tactile Cues for Navigation and Orientation

Navigation– Way-finding– Location of events – real and

simulated environments– Control of robots

Waist BackExperiments10 subjects in each

experimentEach tactor activated 5

times (randomly)Subject indicate location

of tactor vibrated

Navigation – Tactile Belt – One-dimensional Display

1

2

3

4

8

6

5

7

Navel

Spine

Left Right

Identification of tactor locationEight locations – 98% correct

(Inter-tactor spacing: 80-100 mm)Twelve locations – 74% correct

(Spacing: 55-66 mm)(Jones & Ray, 2008)

Inter-tactor distance 80-100 mm

Localization – Two-dimensional Display

Identification of tactor location16 locations – 59% correct (40-82%)

Within 1 tactor location: 95%Inter-tactor spacing: 40 mm vertical

60 mm horizontal Darker the shading, the more accurate the localization

(Jones & Ray, 2008)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 160

102030405060708090

100

Tactor number

Perc

ent c

orre

ct

Results• Spatial localization becomes more difficult as the number of

tactors increases and the inter-tactor distance decreases• Two-dimensional 16-tactor array on the back is unable to

support precise spatial mapping, for example between tactile location and visual target –driving or to highlight on-screen information

• One-dimensional array is very effective for conveying directions

Questions addressed• Can tactile signals be used to provide spatial cues about the

environment that are accurately localized? • How does the location and configuration of the tactile

display influence the ability of the user to identify tactile patterns?

• What is the maximum size of a tactile vocabulary that could be used for communication?

• Which characteristics of vibrotactile signals are optimal for generating a tactile vocabulary?

• Can a set of Army Hand and Arm Signals be translated into tactile signals that are accurately identified when the user is involved in concurrent tasks?

Location and Configuration of Tactile Display• Tested vibrotactile pattern recognition on forearm and back• Fabricated 3x3 (arm) and 4x4 (torso) arrays both controlled

by Wireless Tactile Control Unit (WTCU)• Tactile patterns varied with respect to spatial cues (location),

amplitude (number of tactors simultaneously active) and spatio-temporal sequence.

Group mean percentage of correct responses – averaged across tactors – 89%

Actual Pattern A B C D E F G H

A 80% 0% 1% 8% 2% 1% 8% 0%B 1% 86% 6% 3% 0% 1% 3% 0%C 2% 0% 95% 0% 2% 1% 0% 0%D 0% 8% 0% 90% 2% 0% 0% 0%E 0% 0% 1% 1% 96% 2% 0% 0%F 1% 4% 0% 1% 5% 86% 2% 1%G 0% 5% 3% 3% 0% 3% 84% 2%H 0% 1% 0% 1% 0% 0% 1% 97%

Subject Response

A

C D

E F

G H

B

Up Down

Right Left

Left, right, left

Blink X-shape 3 times

1

2

3

1 1

2

2

3

2

1

1

2

2

2

1

1

2

3

1,3

3 2

3

3

1,3

2

3

1,3

2

2

3

3

1

2 3

1

1

2

32

3

1

1

1, 2, 3

22

1,3

2

1,3

1,3

1, 2, 3

Top, bottom, top

Blink center 3 times

Results

(Piateski & Jones, 2005)

Tactile Patterns

2

3

4

2 2

4

3 3

4

2

3

4

1 1 1 1

3

2

1

3 3

1

2 2

1

3

2

1

4 4 4 4

1

1

2

2 3

1

2 3

3

4

4

4

1 2 3 4

4

4

3

3 2

4

3 2

2

1

1

1

4 3 2 1

A

2, 4

2, 4

2, 4

2, 4

1, 3

1, 3

1, 3

1, 3

2, 4 2, 4 2, 42, 4

1, 3 1, 3 1, 31, 31,2, 3,4

1,2, 3,4

B C D

EF G H

Up Down Right Left

Left, right, left, right

Top, bottom, top, bottom

Blink corners 4 times

Blink single motor 4 times100%100%100%

100%100%99%

99%

97%

Back -Tactile Pattern Recognition

(Piateski & Jones, 2005)

1 2 3 43,4

NM2,4

1,3

4,8 3,7 2,6 1,51,3 2,41,3

2,4Top, bottom, top, bottom

Bottom, top, bottom,

top

Right, left, right, left

Each corner vibrates twice

Up

H

Down

ED1,2

5,6

7,8

1

RightLeft

Left, right, left, right

Single tactor vibrates four

times

1234

4321

A B FC G

I

I

1,3

2,4

4 3 2 1

K

Corners vibrate together four times

Middle two rows

Outer corners then inner twice

L

Diagonal vibrates four

times

O

Two corners vibrate in turn twice

Tactile Vocabulary – 15-20 Tactons?

A B C D E F G H I J K L M N O70

75

80

85

90

95

100

Tactile pattern

Perc

ent c

orre

ct

Mean: 96%

J

(Jones, Kunkel, & Torres, 2007)

Experiment 1A

3

3 3

A C D

E F G H

B

Up Down Right Left

Up and right Blink X-shape 3 times

2

3 3

1,3

1,3

1,3

1

1, 2, 3

1 1

1

Up and left Left, right, left

3

3

3

3

3

3

3 3 3 3 2

2

2

2

2

2

2 2 2

2

2

2

2 2

1 1 1

1 1 1

1

1

1 1

1

2

2

2

3 3

3

3 3

3

Experiment 1B

A B C D E F G H0

20

40

60

80

100

Tactile pattern

Cor

rect

res

pons

es (%

)

Mean correct response rate:62% in Expt 1A 85% in Expt 1B IT: 1.48 bits IT: 2.15 bitsConfusion matrix (Expt 1A): A misidentified as F, whereas F mis-identified as D – errors not symmetrical.Tactile patterns that “moved” across the arm more accurately perceived than those that “moved” along the arm

Tactile Pattern Recognition – Effect of Stimulus Set

(Jones, Kunkel, & Piateski, 2009)

A C D

E F G H

B

Up Down Right Left

Blink center 3 times Blink X-shape 3 times

2

3 3

1

1

1, 2, 3 1,3 1,3 1,3

2

Top, bottom, top Left, right, left

3

3

3

3

3

3

3 3 3 3

2

2

2

2

2

2

2

2 2 2

2 2 2

2 2

1 1 1 1

1 1 1

1

1 1

2

2

1,3

1,3

1,3

Summary of FindingsArm vs back – both provide effective substrates for

communicationArray dimensions – marked effect on spatial localizationAsymmetries in spatial processing on the skinNeed to evaluate patterns in the context of the “vocabulary” usedTactile vocabulary size – absolute identification vs communicationInterceptor Body Armor - no effect on performance

Direction and orientation

Saltation

Tap on shoulder

Navigation path

Field ExperimentsFive subjects participatedEight patterns with five repetitions

Familiarization with visual analog initially

Brief training period outdoors

Navigation using only tactile cues, without feedback100% accuracy for 7/8 patterns presentedSingle error on 8th patternDemonstrated that navigation is accurate using only tactile cues as directions

(Jones, Lockyer & Piateski, 2006)

Questions addressed• Can tactile signals be used to provide spatial cues about the

environment that are accurately localized? • How does the location and configuration of the tactile display

influence the ability of the user to identify tactile patterns?• What is the maximum size of a tactile vocabulary that could

be used for communication? • Which characteristics of vibrotactile signals are optimal

for generating a tactile vocabulary? • Can a set of Army Hand and Arm Signals be translated into

tactile signals that are accurately identified when the user is involved in concurrent tasks?

Tactons (tactile icons)Structured tactile messages that can be used to communicate

information.These tactons must be intuitive and salient.

1

1

2

2 3

1

2 3

3

4

4

4

1 2 3 4

4

4

3

3 2

4

3 2

2

1

1

1

4 3 2 1

2, 4

2, 4

2, 4

2, 4

1, 3

1, 3

1, 3

1, 3

2, 4 2, 4 2, 4 2, 4

1, 3 1, 3 1, 3 1, 3

1,2, 3,4

C D

E F

G

Turn right Turn left

Arm horizontal Arm vertical

Stop at next cone

1,2, 3,4

H

Hop

Move forward Turn around

2

3

4

2 2

4

3 3

4

2

3

4

1 1 1 1

3

2

1

3 3

1

2 2

1

3

2

1

4 4 4 4

A B

Navigation tactons

Communication tactons

Assemble/rally

1,23,4

5,6 7,8

Tactons for hand-based

communicationFrequency

Duration, repetition rate

Waveform complexity

(Jones & Sarter, 2008)

Frequency Intensity Waveform Duration Location

Range:0.4-1000 Hz.

Optimal sensitivity:

150-300 Hz1

Absolute thresholds

across body sites: 0.07-14 m at 200 Hz3

Relatively insensitive to waveforms: sinusoidal, triangular,

square wave5

Burst duration: 80-500 ms (typical)

Differential thresholds: 7-

50%5

Localization accuracy varies with body site7

Body site influences perceived frequency

Changes with increased

voltage to a single tactor

and with number of

tactors activated

Amplitude modulation of

sinusoids effective for

varying roughness of

signals6

Pulse repetition rate (create temporal patterns - rhythms)

Inter-tactor spacing and array

configuration important

Differential thresholds:

18-50%2

Differential thresholds: 5-

30%4

Number of pulses:

1-5 (typical)

Localization superior near

anatomical points of reference (elbow,

spine)7

Tacton building blocks: Relevant properties of each variable

(Jones, Kunkel, & Piateski, 2009)

Arm and Hand Signals for Ground ForcesIdentify a set of structured

tactile messages (tactons) that can be used to communicate information.

1,7

4 4 4 4 1 1 1 1

2 2 2 2

3 3 3 3

4 4 4 41,7

1,7

1,7

2,6

2,6

2,6

2,6

3,5

3,5

3,5

3,5 4

41

1

2

2 3

3

Take coverIncrease speed Danger area

4

4

4

4

1

1

1

1

Advance or Move Out

2

2

2

2

3

3

3

3

Halt

4

3

2

1

4

2

3

1 11

2 2

3 3

4 4 2, 4

2, 4

2, 4

2, 4

1, 3

1, 3

1, 3

1, 3

Attention

Each tactile hand signal was designed to keep some of the iconic information of the matching visual hand signal

A B C D E F G0

20

40

60

80

100

Tactile pattern

Cor

rect

res

pons

es (%

)

Mean (N=10) percentage of correct responses (35 trials per subject) when identifying the hand signal with both the illustration and schematic available

(black - 98% correct) and with only the illustration available (red – 75% correct).

Hand and Arm Signals – Tactile-visual mapping

(Jones, Kunkel, & Piateski, 2009)

Questions addressed• Can tactile signals be used to provide spatial cues about the

environment that are accurately localized? • How does the location and configuration of the tactile display

influence the ability of the user to identify tactile patterns?• What is the maximum size of a tactile vocabulary that could

be used for communication? • Which characteristics of vibrotactile signals are optimal for

generating a tactile vocabulary? • Can a set of Army Hand and Arm Signals be translated

into tactile signals that are accurately identified when the user is involved in concurrent tasks?

Field Experiments – Concurrent activities

1 2 3 4 5 6 7 80

20

40

60

80

100

Hand Signal Identification

Walking

Jogging

Cognitive task

Hand signal

Perc

ent c

orre

ct

(Jones, Kunkel, & Piateski, 2009)

Nuclear, biological and chemical attack

AssembleAttentionIncrease speed

Take cover

Advance to left

Danger area

Halt

91%91%93%

Conclusions Vibrotactile patterns easily perceived on torso with little

training and single stimulus exposure Demonstrated feasibility of using sites that are non-

intrusive and body movements are not impeded Shown that the ability to perceive tactile patterns is not

affected by concurrent physical and cognitive activities Directional patterns are intuitive and can readily be used

as navigational and instructional cues Two-dimensional arrays provide greater capabilities for

communication, but one-dimensional arrays are effective for simple commands

AcknowledgementsEdgar TorresAmy LamDavid HeldChrista MargossianKatherine Ray

Brett LockyerMealani NakamuraErin PiateskiJacquelyn Kunkel

Research was supported through the Advanced Decision Architectures Collaborative Technology Alliance sponsored by the U.S. Army Research Laboratory under Cooperative Agreement DAAD19-01-2-0009.

ReferencesJones, L.A., Kunkel, J. & Piateski, E. (2009). Vibrotactile pattern recognition on the

arm and back. Perception, 38, 52-68.Jones, L.A. & Held, D.A. (2008). Characterization of tactors used in vibrotactile

displays. Journal of Computing and Information Sciences in Engineering, 044501-1-044501-5.

Jones, L.A. & Ray, K. (2008). Localization and pattern recognition with tactile displays. Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 33-39.

Jones, L.A. & Sarter, N. (2008). Tactile displays: Guidance for their design and application. Human Factors, 50, 90-111.

Jones, L.A., Kunkel, J., & Torres, E. (2007). Tactile vocabulary for tactile displays. Proceedings of the Second Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 574-575.

Jones, L.A., Lockyer, B., & Piateski, E. (2006). Tactile display and vibrotactile pattern recognition on the torso. Advanced Robotics, 20, 1359-1374.

Piateski, E. & Jones, L.A. (2005). Vibrotactile pattern recognition on the arm and torso. Proceedings of the First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 90-95.