UCD15 Talk - Enrico Furfaro - Sonification of Virtual and Real Surface Tapping: Evaluation of...

Sonification of virtual and real surface tapping: evaluation of behaviour changes, surface perception and emotional indices Enrico Furfaro

UX Architect @ We Love Digital

24th October 2015

Ana Tajadura-Jiménez, Research Fellow, UCL

Nadia Berthouze, Professor in Affective Computing, UCL

Frédéric Bevilacqua, Head Researcher, IRCAM

UCD 2015 - Humanity in digital landscapes: Mind, Cognition and Psychology

24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 2

Dr. Nadia Berthouze, Professor in Affective Computing, UCL


Enrico Furfaro UX Architect

Dr. A. Tajadura-Jiménez, Research Fellow and Principal Investigator


Dr. A. Tajadura-Jiménez, Research Fellow and Principal Investigator

Dr. Nadia Berthouze, Professor in Affective Computing, UCL


Enrico Furfaro UX Architect

What does Sonification mean?


Three aspects

Auditory displays

• Systems that systematically transform data into sounds that can be manipulated by human users through an interface.

Sonification

• Sonification is this transformation: data into sounds to convey information.

Sonic interaction design

• It is the study and exploitation of sound as one of the principal channels conveying information.


Why is it important?


Why is it important?

• Interaction with objects is increasingly mediated through their digital representation.

• Interaction with object is multisensory.

• Touch, vision and hearing contributing and interacting with each other

To what extent do we make use of the information conveyed by auditory feedback? How do we measure it?


Google’s Project Soli

The research


Our aim

To explore how the sounds resulting from tapping on a

surface inform :

1. The applied strength when tapping.

2. The user’s ability to tap & changes in emotional

states.

3. The physical features of the surface material

(hardness).

And to test a multidimensional measurement approach

to evaluate user experience related to these three

dimensions.


Me tapping on a table

1. Tapping behaviour

Changes might occur because of the auditory action loop.

• The effect for which users try to adjust their actions

based on the auditory feedback they receive.

Example: the audio feedback in real time helped elite

rowers to adjust their strokes and improved their

performance. (Schaffert et al.’10).


Listening to the sound of the boat motion increases boat velocity (Schaffert et al.’10)

Audio

Feedback

Motor Behaviour

2. User’s ability to tap

Users’ ability to tap might be affected because of the

sense of body can be altered by sound feedback .

• Sense of body = the mental representation of body.

• Crucial for our interaction with our environment.

• Obtained and continuously updated through sensing

and acting.

Example - the perceived length of participants' arm can

be altered by presenting real-time sound feedback

suggesting an altered distance. (Tajadura-Jiménez et

al.’12).


Tajadura-Jiménez et al.’12

2. User’s emotional states

Body movement is a medium to express and to regulate

one’s own emotions.

• We expected changes in behaviour and emotions.

For example, the strokes in ‘Fruit Ninja’ were

successfully used to detect players’ emotional states

(frustrated, excited, relaxed, bored) during the

gameplay.


Gao et al.’12

3. Physical features of the material

Changing audio feedback can result in changes in an

object perceived material properties.

• Both in case of natural surfaces and virtual haptic

surfaces.

Example:

• Reduced high frequencies led to biased perception

of sand paper’s texture (smoother). (Guest, et al.

2002)


Sandpaper

3. Physical features of the material

Changing audio feedback can result in changes in an

object perceived material properties.

• Both in case of natural surfaces and virtual haptic

surfaces.

Examples:

• Attenuating high frequencies led to biased

perception of sand paper (smoother). (Guest, et al.

2002)

• higher frequencies or louder sound while biting

crisps led to biased perception of crispness and

freshness (Zampini & Spence, 2003)


Kermit the frog eating crisps

The prototype


tapping on “real” surface

tapping on “virtual” surface

The tapping action triggers, in real-time, the presentation of pre-recorded tapping sounds

Surfaces & Components

24/10/2015 Enrico Furfaro, WLD | Sonification of surface tapping | UCD 2015 16

Two modes: “real” and “virtual”.

The measures



Design & procedure

Virtual

Real

Six conditions for each participant

“Strong” sound feedback

“Medium” sound feedback

“Weak” sound feedback

Design 2 surface type x 3 sound level strength

Timeline for each condition

0 s 10 s

baseline1

70 s 80 s

baseline2 Audio-feedback phase time

Questionnaire

x

1. Measuring tapping behaviour

The accelerometer was used to measure the tapping behaviour.

• Acceleration = tap’s strength

• Interval between taps = frequency of taps


Galvanic skin response (GSR)


2. Measuring emotional experience and ability to tap

Self Assessment Manikin (SAM)

Valence

Arousal

Dominance


2. Measuring emotional experience and ability to tap

7-point Likert scales: physical strength, ability to tap, aggressiveness.

Now please, draw an horizontal line through the vertical scale to indicate how much mental effort you feel you had to invest to complete the tapping task.

Subjective Mental Effort Questionnaire (SMEQ)

To measure perceived surface hardness we used a 7-point Likert scale.


3. Measuring surface hardness

The results



Results

Real vs. Virtual surface

• Harder surface, larger perceived strength and ability to tap, less stressed & more in control.

Surface hardness

• Virtual surface perceived softer when ‘weak’ feedback.

Emotions & ability to tap

• Audio motor incongruences led to unpleasant arousing experiences.

• Real surface. More negative valence when weak sound feedback and less able to tap.

• Virtual surface. Weak sound led to higher GSR values.

Behaviour

• Real & virtual surface. introducing the sound feedback sped up movements and decreased acceleration.

• Real surface. Weak sounds led to higher maxima acceleration


Takeaways

When measuring and designing digital representation of objects, consider that audio feedback plays a fundamental role in complementing or substituting haptic feedback.

1. it informs about physical properties of objects.

2. it can be used to manipulate and/or lead motor behaviour.

3. if congruent to the action, it leads to a more pleasant user experience.

4. it might affect the sense of body which relates to perceived body characteristics and self-esteem

Example of measures:

• Self Assessment Manikin (SAM)

• 7-point Likert scales

• Subjective Mental Effort Questionnaire (SMEQ)

• Bandura’s self-efficacy test

• GSR

• Sensors (for example accelerometer and piezo) to capture data and for sonification of objects.

Implications for design

1. To complement when interacting with digitally represented objects.

Digital representation of objects characterized by a limited amount of haptic & visual feedback:

• when extreme precision in applied strength is critical e.g. touch-less surgery or dismantling bombs.

• Shopping on-line (tactile-sensory substitution) perceived material properties & emotion responses are important .

2. To guide motor behaviour.

Motor behaviour changes in a self-controlled way:

• Physical rehabilitation (VR/augmented reality).

• Games promoting health- or fun-movement (rehabilitation, education, entertainment).


Nicole Stenger wearing one of the first VR prototypes in the early 90ies

THANKS! QUESTIONS?


[email protected]

@fasteedio

Sonification of Surface tapping original paper

• Tajadura-Jimenez, A; Bianchi-Berthouze, N; Furfaro, E; Bevilacqua, F; (2015) Sonification of surface tapping changes behavior, surface perception, and emotion. IEEE Multimedia , 22 (1) 48 - 57.

• Furfaro, E., Berthouze, N., Bevilacqua, F., Tajadura-Jiménez, A. (2013) Sonification of surface tapping: Influences on behaviour, emotion and surface perception. Interactive Sonification Workshop (ISon 2013), Bielefeld, 9th-10th December, 2013.

Sonification and Sonic Interaction design

• T. Hermann, A. Hunt, and J.G. Neuhoff, The sonification handbook. Berlin:Logos Publishing House, 2011.

• Franinović, Karmen, and Stefania Serafin. Sonic interaction design. Mit Press, 2013.

Sonification of boat motion improves athletes’ movements

• N. Schaffert, K. Mattes, and A.O. Effenberg, “Listen to the boat motion: acoustic information for elite rowers”, in Proc. Interaction Sonification workshop (ISon), Stockholm, Sweden, 2010

Sound feedback affects arm’s length perception

• A. Tajadura-Jiménez, A. Väljamäe, I. Toshima, T. Kimura, M. Tsakiris and N. Kitagawa, “Action sounds recalibrate perceived tactile distance”. Current Biology, 22(13), 2012, pp. R516-R517.


References

Fruit Ninja to detect players’ emotional states

• Y. Gao, N. Bianchi-Berthouze, and H. Meng, “What does touch tell us about emotions in touchscreen-based gameplay?”, in ACM Transactions on Computer-Human Interaction 19(4), pp. 31, 2012.

Body movements and emotions

• N. Bianchi-Berthouze, “Understanding the role of body movement in player engagement”. Human Computer Interaction 28(1), 2012, pp. 42-75.

• A. Kleinsmith, R. De Silva, and N. Berthouze, “Grounding affective dimensions into postures description”, Lecture Notes in Computer Science, Berlin / Heidelberg: Springer, 3784, 2005, pp.263-270.

Sound and perception of objects’ characteristics

• Guest, S., Catmur, C., Lloyd, D. & Spence, C. (2002). Audiotactile interactions in roughness perception. Experimental Brain Research, 146(2), 161-171

• D. Merrill, and H. Raffle, “The sound of touch”, in Proc. ACM Conference on Human Factors in Computing Systems 2007, ACM Press, 2007, pp. 2807-2812.

• Zampini, M., & Spence, C. (2004). The role of auditory cues in modulating the perceived crispness and staleness of potato chips. Journal of sensory studies,19(5), 347-363


References

UCD15 Talk - Enrico Furfaro - Sonification of Virtual and Real Surface Tapping: Evaluation of...

Design

Transcript of UCD15 Talk - Enrico Furfaro - Sonification of Virtual and Real Surface Tapping: Evaluation of...