Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction
description
Transcript of Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction
29/09/2011
1
Rhythmic Blueprints
Design and Evaluation of Rhythmic Interaction
Cumhur Erkut and Antti Jylhä
Aalto University, School of Electrical Engineering
Department of Signal Processing and Acoustics
Tampere, Finland MindTrek 2011 29/09/2011
Overview
• Getting know each other • Sensitizing: interactive rhythms in various scales
– Musical rhythms – Social rhythms
• Elements of rhythm – Pulse, beat, meter, and tempo – Resonance, synchronization, entrainment
• Design and evaluation models, and how we use them • Conclusions: Directions and Guidelines • Remember: http://blogs.aalto.fi/rhythmicity/
29/09/2011 2
29/09/2011
2
Michael H. Thaut, in Rhythm, Music, and the Brain, pp. 16-17
If we return briefly to the importance of temporal regulation for all our higher cognitive and motor functions, we may have very good reason to believe that rhythm in music, the element of temporal order, has a unique and profound influence on our perceptual processes related to cognition, affect, and motor function. Rhythm may enhance our brain operations through providing structure and anticipation in time. Rhythm may be one of the central processors to optimize our gestalt formation in the basic process of learning and perception.
29/09/2011 3
Rhythmic interaction
• Time is of the essence • A natural human capability
– Walking, hammering, talking… – Anticipation, mutual coordination
• Examples – Personal Orchestra {Borchers:2004hx} – Virtual interactive humanoids {Nijholt:2008ty} – Percussion robot Haile {Weinberg:2006wl} – B-keeper {Robertson:2007tf} – Hand clap interface {Jylha:2009gq}
29/09/2011 4
29/09/2011
3
Demo: Sonically Augmented Table and Rhythmic Interaction {Pesonen:2010ur}
29/09/2011 5
Elements of Musical Rhythms
• Pulses: events in a “pulse train” with regular temporal spacing – The inter-pulse interval always the same – The basis of rhythm perception {Thaut:2005te}.
• Beats: audible pulse markings – Sequenced events – May deviate from exact pulse timings in slight shifts
29/09/2011 6
29/09/2011
4
Elements of Musical Rhythms
• Tempo – Metric of (musical) rhythmic “speed” (rate of pulses/beats) – Inversely proportional to inter-pulse interval – Often measured as beats per minute (BPM) – In music, never completely stable
• Meter – Defines the rhythmic structure in music (and vice versa) – Often expressed as beats per measure
• E.g., 3/4, 4/4 – Relates to accentuation
29/09/2011 7
Related concepts
• Vibration: Mechanical response of a body to an external stimuli
• Frequency: Vibrations per second • Resonance: Tendency to vibrate at a certain frequency • Entrainment: A process between multiple bodies to align their
rhythmic resonances • Synchronization: The quasi-stable state of entrainment
29/09/2011 9
29/09/2011
5
Time-sensitive crowded scenes in movies
29/09/2011 10
Laws are simple: use in media?
29/09/2011 11
29/09/2011
6
Sound synthesis, control, and hierarchical events
29/09/2011 12
ClaPD
29/09/2011 13
29/09/2011
7
29/09/2011 14
Tutorial in a Nutshell …
29/09/2011 15
29/09/2011
8
Hand clapping interface for sonic interactions (AM ’08 / CHI ’09)
29/09/2011 16
Applications
• Potential in – Games and entertainment – Sound design – New HCI schemes
• Three example cases 1. Hand-clap driven sampler 2. Controlling music tempo 3. Synchronizing a virtual audience
29/09/2011 17
29/09/2011
9
Hand clap detection
Tempo estimation
Clap type identification
Application User Sound input
Sonic feedback
System dataflow
29/09/2011 18
Music tempo control
• The user claps to control the tempo of music – BPM of the user’s clapping is
mapped to that of the music
29/09/2011 19
29/09/2011
10
Virtual audience
• The user claps to synchronize a virtual audience with her clapping
• The user can sync the audience with or without reference music
• Interaction is immediate – The user is part of the
clapping crowd
29/09/2011 20
Demo video
• http://www.youtube.com/watch?v=7HLYGkayAGA
29/09/2011 21
29/09/2011
11
Interface implementation with PD
• Clap detection: [bonk~] (Puckette98)!
• Tempo estimation: [rhythm_estimator] (Seppänen01)!
• Combination: [clap_tracker]!
29/09/2011 22
Evaluation
• Informal evaluation – 2 subjects tested the example applications – Interface was found easy to use – Both subjects found that the tempo of the virtual audience or the
music drove their clapping
• Some latency appears in the system – Mostly from buffering and computations – Not reported as disturbing by the subjects
29/09/2011 23
29/09/2011
12
Negotiation
• Perceived tempo affected subjects’ clapping – The user negotiates with the computer to set the tempo
• Mutual coordination, ”dual-drive” – Process analogue: musical ensemble
29/09/2011 24
Explorations of rhythmic interaction with dancers
{Erkut:2009wu} 29/09/2011 25
29/09/2011
13
Design Blueprints: our version
Based on Z. Obrenovic, J. Abascal, and D. Starcevic, “Universal accessibility as a multimodal design issue,” Communications of the ACM, vol. 50, no. 5, pp. 83–88, 2007.
\cite{Erkut:2011ta} 29/09/2011 28
Rhythmic musical interface
29/09/2011 29
29/09/2011
14
29/09/2011 30
iPalmas (AM ‘09)
• An interactive Flamenco rhythm tutor for hand clapping
\cite{Jylha:2009uc} 29/09/2011 31
29/09/2011
15
Flamenco
29/09/2011 32
Rhythm in Flamenco
• Compas = meter in Flamenco • Usually 12-beat cycles
– Accentuation on certain beats depending on style, e.g. bulerias: X - - X - - X - X - X -
• Lots of percussion – Footwork, hand clapping, instruments…
• Focal performer leads – Improvisation requires communication
29/09/2011 33
29/09/2011
16
Palmas
• Supporting the compas
• Palmero/-a • Two types
– Hard and soft • Follow the focal
performer
http://www.youtube.com/watch?v=Yo38h7Wdc88
29/09/2011 34
Palmas skill requirements
1. Sordas and fuertes 2. Steady and accurate basic accompaniment 3. Decorative clapping 4. Starting to clap after silence 5. Reacting to tempo changes 6. Reacting to rhythmical cues
29/09/2011 35
29/09/2011
17
System (1)
• Flamenco palmas synthesis and tutor – Teaches the skills – Provides synthetic examples and accompaniment
• User’s clapping as input • Auditory and visual feedback
29/09/2011 36
System (2)
User Clap analysis
GUI
Clap synthesis
Visual feedback
Virtual tutor
29/09/2011 37
29/09/2011
18
System (3)
29/09/2011 38
Synthetic palmas
• ClaPD hand clap synthesis engine (Peltola et al. 2007) – Enveloped noise burst – Band-pass filter defines clap type – Coupled oscillator model for rhythmic interaction
• Pre-defined palmas patterns set synthesis parameters • User and/or tutor set the tempo
29/09/2011 39
29/09/2011
19
Analysis of clapping
• Clap type – Classification by template matching ( [bonk~] )
• Accent detection – Loudness-based
• Tempo – Probabilistic estimate based on inter-onset intervals
• Tempo steadiness – Temporal variance
• Correct accentuation – Comparison of user’s and tutor’s accent patterns
29/09/2011 40
Feedback (1)
• Clapping sounds from the system – Synchrony, accents, tempo changes
• Numeric feedback and sliders – Performance metrics
• Visualization – Dancing circles – Transcription of the pattern
29/09/2011 41
29/09/2011
20
Feedback (2)
29/09/2011 42
Prototype
• Built on Pure Data (Pd) • Synthesis of virtual palmeros • Clap type training • Basic compas training
– Tutor speeds up when the user gets better -> Tempo changes trained as well
29/09/2011 43
29/09/2011
21
Interlude: evaluation
Human System
29/09/2011 44
Interlude: evaluation
Human System Interaction
29/09/2011 45
29/09/2011
22
What can/should be evaluated
• Human factors – Rhythmic capabilities
• Perception • Production
• System factors – I/O latency – Computational complexity
• Interaction – Fluency, naturalness – Etc.
• The whole triptych
29/09/2011 46
Methods of evaluation
• Qualitative – Characterization of the capabilities/properties/phenomena
• Understanding – Interviews, observations, etc.
• Quantitative – Measurable quantities
• For quantifying the qualitative attributes • Also can be used to derive qualitative results
29/09/2011 47
29/09/2011
23
Human factors
• Sensorimotor synchronization (SMS {Repp:2005tb}) – Synchronization between sensory stimulus and motor response – Often metronome-based evaluation or simple rhythmic tasks
• “tapping to the beat” – Also, movement-to-music evaluation
• How movement trajectories correlate with the rhythmic elements
• Rhythmic capabilities – Production: e.g., rhythmic stability, accentuation – Perception: modal acuity – Some people can be actually “rhythm-deaf” (Phillips-Silver et al. 2011)
29/09/2011 48
System factors
• I/O latency – Plays a role in rhythmic applications
• In principle a constraint, but can be overcome in cyclic interactions
• I/O modalities – Auditory, visual, haptic – Streaming (continous) vs. event-based (discrete)
• Computational properties – Complexity, required processing power, memory, …
29/09/2011 49
29/09/2011
24
Evaluating interaction
• Qualitative studies • User tests
– Often based on simple tasks
• In-performance evaluation • Hybrid methods
29/09/2011 50
However…
• … evaluation of individual components is valuable, but does not necessarily explain everything Wholesome evaluation needed
29/09/2011 51
29/09/2011
25
Case example: iPalmas evaluation (CMJ ’11)
• Subjective experiment AND objective description – Both qualitative and quantitative measures
\cite{Jylha2011:CMJ} 29/09/2011 52
iPalmas revisited
Audiorecording
Detection andanalysis
Audiofeedback
Visualfeedback
Metrics:- Tempo- Accent- Deviation- Time stamp
Text file log
Post-analysisTempo
Transcription,circles,numeric feedback
Hand claps
Synthetic hand claps,
User
Tutor
Audio Mostly 2011 Jylhä & Erkut 29/09/2011 53
29/09/2011
26
Subjective experiment
• To evaluate 1. Human factors 2. System 3. Interaction
• Rhythmic tasks with realistic rhythmic patterns – Training, testing
• Logging of measurable quantities • Observations throughout the experiment • Interviews, verbal comments • Post-experiment questionnaire
29/09/2011 54
Evaluation of the first iPalmas system
• Performed to evaluate rhythmic interaction and the system (Jylhä et al. 2011)
• 16 subjects • 4 patterns • 4 tutors (audio and audiovisual, adaptive and fixed tempo) • Training phase and test phase, recall phase
Audio Mostly 2011 Jylhä & Erkut 29/09/2011 55
29/09/2011
27
Evaluation of the first iPalmas system
Audio Mostly 2011 Jylhä & Erkut 29/09/2011 56
Evaluation of the first iPalmas system
• Auditory information (clapping) the key to learning and performing – The reverb disturbing to some
• Transcription helpful • Circles pretty but not very helpful • Numeric FB useful only to some
– Helped in “tuning in” to clapping the accents
• Temporal variation in clapping – Tutor stops subject speeds up
• Adaptive mode helps performance
Audio Mostly 2011 Jylhä & Erkut 29/09/2011 57
29/09/2011
28
Tempo speed-up and fluctuation
0 20 40 60
160
180
200
Time (sec)
BPM
Audio only tutor (steady)
Subject11Subject1
0 20 40 60
160
180
200
Time (sec)
BPM
Audio only tutor (adaptive)
0 20 40 60
160
180
200
Time (sec)
BPM
Audiovisual tutor (steady)
0 20 40 60
160
180
200
Time (sec)
BPM
Audiovisual tutor (adaptive)
\cite{Jylha2011:CMJ} 29/09/2011
58
Quantitative findings
• Average time to start clapping: 10 cycles (40 s) • Accentuation correctness
– Audio-only: 68.1 % – Audiovisual: 73.7 % – Fixed tempo: 67.6 % – Adaptive tempo: 74.2 %
• Indication: for accentuated beats, IOI slightly longer (344.7 ms) than for non-accentuated beats (343 ms) – More prominent with the adaptive tutor
\cite{Jylha2011:CMJ} 29/09/2011 59
29/09/2011
29
Evaluation Blueprints: Summarize Qualitative, quantitative, and metrics
\cite{Erkut:2011ta} 29/09/2011 60
Direction1: SID
http://en.wikipedia.org/wiki/Sonic_interaction_design 29/09/2011 61
29/09/2011
30
Direction 2: Interaction Gestalts and Attributes
{Lim:2009tw}
Interaction gestalt
User experience • User experience
qualities
Interactive artifact • Artifact
properties
29/09/2011 62
Directions 1+2 Combined: New basic sonic interaction design
http://www.room50.org/stefanodellemonache See, \cite{Rocchesso:2009wi}, and \cite{Franinovic:2009wl} 29/09/2011
63
29/09/2011
31
Other directions
• Social rhythms: H. Lefebvre, “Rhythmanalysis: Space, Time and Everyday Life,” Book, pp. 1–129, 2004 => CSCW => Social games
• Rhythms and emotions: Thaut’05, but also recent CHI papers, e.g. \cite{Epp:2011hz}
• Gamification! • What else?
29/09/2011 64
References
• C. Epp, M. Lippold, and R. L. Mandryk, “Identifying emotional states using keystroke dynamics,” in CHI’11, Vancouver, BC, Canada, 2011, pp. 715–724.
• C. Erkut, A. Jylhä, and R. Discioglu, “A structured design and evaluation model with application to rhythmic interaction displays,” in Proc. New Interfaces for Musical Expression (NIME), Oslo, Norway, 2011, pp. 477–480.
• C. Erkut, A. Jylhä, and I. Ekman, “Recent advances in exploring self-induced sonic interactions in the context of performing arts,” in Intl. Workshop on Haptic and Audio Interaction Design, 2009, pp. 1–2.
• K. Franinovic, “Toward Basic Interaction Design,” available online at http://tdd.elisava.net/coleccion/25/franinovic-en
• A. Jylhä and C. Erkut, “A hand clap interface for sonic interaction with the computer,” CHI-EA, Apr. 2009.
• A. Jylhä, C. Erkut, I. Ekman, and K. Tahiroglu, “iPalmas - An interactive flamenco rhythm machine,” Proc. Audio Mostly, pp. 1–2, May. 2009.
• A. Jylhä, I. Ekman, C. Erkut, and K. Tahiroglu, “Design and Evaluation of Rhythmic Interaction with an Interactive Tutoring System,” Computer Music Journal, vol. 35, no. 2, pp. 36–48. 2011.
29/09/2011 68
29/09/2011
32
References
• Y.-K. Lim, E. Stolterman, H. Jung, and J. Donaldson, “Interaction gestalt and the design of aesthetic interactions,” Proc.. Conf. Designing Pleasurable Products and Interfaces, pp. 239–254, 2007.
• M. Pesonen, Sonically Augmented Table and Rhythmic Interaction, Master’s thesis, Aalto Univerity, School of Electrical Engineering, 2010.
• B. Repp, “Sensorimotor synchronization: A review of the tapping literature,” Psychonomic Bulletin and Review, vol. 12, no. 6, pp. 969–992, 2005.
• A. Robertson and M. Plumbley, “B-Keeper: A Beat-Tracker for Live Performance,” Proc. NIME, pp. 234–237, 2007.
• D. Rocchesso, P. Polotti, and S. D. Delle Monache, “Designing Continuous Sonic Interaction,” Intl. J. Design, vol. 3, no. 3, pp. 13–25, May. 2009.
• M. Thaut, Rhythm, Music, and the Brain. New York, NY, USA: Routledge, 2005.
• G. Weinberg and S. Driscoll, “Robot-human interaction with an anthropomorphic percussionist,” CHI '06, Apr. 2006.
29/09/2011 69
Discussion
• (Edward T) Hall's studies (on cultural rhythms) and (Saul) Greenberg's work on social rhythms (awareness)
• Should the tutorial be called "interaction with rhythms" instead of rhythmic interaction?
• Synesthesia as a resource? • Behavioral change: negotiation
already used in game design (transfer of adaptation)
• Adaptive coaching • Language learning and
prononciation
29/09/2011 70