Human-Technology Choreographies: Body, Movement, &...

Volume 13, Number 1, May 2017

SPECIAL ISSUE HUMAN–TECHNOLOGY CHOREOGRAPHIES: BODY, MOVEMENT, AND SPACE IN EXPRESSIVE INTERACTIONS

Antti Pirhonen, Kai Tuuri, and Cumhur Erkut Guest Editors

Pertti Hurme Editor in Chief

Published by the Open Science Centre, University of Jyväskylä, Finland

Volume 13 Number 1, May 2017

DOI: http://dx.doi.org/10.17011/ht/urn.201705302564

Contents From the Editors External Assessors as “Reviewers” for Quality Assurance of Open Access Journals

Pertti Hurme and Barbara J. Crawford

pp. 1–5

Guest Editors’ Introduction Human–Technology Choreographies: Body, Movement, and Space in Expressive Interactions

Antti Pirhonen, Kai Tuuri, & Cumhur Erkut

pp. 6–9

Original Articles

Technology Choreography: Studying Interactions in Microsoft’s Future Visions Through Dance

Olli Poutanen, Salu Ylirisku, & Petri Hoppu

pp. 10–31

Body, Space, and Emotion: A Perceptual Study Donald Glowinski, Sélim Yahia Coll, Naëm Baron, Maëva Sanchez, Simon Schaerlaeken, & Didier Grandjean

pp. 32–57

Musical Instruments, Body Movement, Space, and Motion Data: Music as an Emergent Multimodal Choreography

Federico Visi, Esther Coorevits, Rodrigo Schramm, & Eduardo Reck Miranda

pp. 58–81

Bodily Interactions in Motion-Based Music Applications Marcella Mandanici, Antonio Rodà, & Sergio Canazza

pp. 82–108

Designing a Computer Model of Drumming: The Biomechanics of Percussive Performance

John R. Taylor

pp. 109–141

Book Review

Public Access ICT Across Cultures: Diversifying Participation

in the Network Society Francisco J. Proenza (Ed.) Reviewed by Sakari Taipale

pp. 142–144

ISSN: 1795-6889

humantechnology.jyu.fi

Submission: humantechnologypublishing.jyu.fi

Editor in Chief Pertti Hurme, University of Jyväskylä, Finland

Founding Editor in Chief Pertti Saariluoma, University of Jyväskylä, Finland

Managing Editor: Barbara J. Crawford Editorial Assistant: Rachel Ferlatte Kuisma

Board of Editors

Morana Alač University of California San Diego, USA

Michael Arnold University of Melbourne, Australia

Jeffrey Bardzell Indiana University Bloomington, USA

Dhrubes Biswas Indian Institute of Technology, India

Adele Botha CSIR Meraka & University of South Africa

José Cañas University of Granada, Spain

Ling Chen Hong Kong Baptist University

Monica Divitini Norwegian University of Science and Technology

Markku Häkkinen Education Testing Service, USA

Päivi Häkkinen University of Jyväskylä, Finland

Sung H. Han Pohang University of Science and Technology, South Korea

Jun He University of Michigan, Dearborn,USA

Jörn Hurtienne Julius-Maximilians University, Germany

R. Malatesha Joshi Texas A & M University, USA

Pilar Lacasa University of Alcalá, Spain

Ann Light University of Sussex, UK

Kalle Lyytinen Case Western Reserve University, USA

Jim McGuigan Loughborough University, UK

Jan Pawlowski Hochschule Ruhr West, Germany, & University of Jyväskylä, Finland

Raul Pertierra Ateneo de Manila University, the Philippines

Roger Säljö University of Gothenburg, Sweden

Stephen Viller University of Queensland, Australia

Marita Vos University of Jyväskylä, Finland

Chihiro Watanabe Tokyo Institute of Technology (emeritus)

Human Technology is an interdisciplinary, scholarly journal publishing innovative, peer-reviewed articles exploring the issues and challenges within human–technology interaction and the human role in all areas of ICT-infused societies. Human Technology, published by the Open Science Centre, University of Jyväskylä, is distributed without a charge online.

ISSN: 1795-6889

www.humantechnology.jyu.fi Volume 13(1), May 2017, 1–5

1

From the Editors

EXTERNAL ASSESSORS AS “REVIEWERS” FOR QUALITY ASSURANCE OF OPEN ACCESS JOURNALS

The interdisciplinary journal Human Technology has been a venue for scholarly research on the interactions between humans and technology since 2005. The editorial philosophy on publishing has been steeped in, from the start, the value of open access (OA) research. The leadership of the Agora Center at the University of Jyväskylä, Finland, as publisher, and founding editor Professor Pertti Saariluoma decided that the journal’s OA financing model would be institution-funded in conjunction with the funding of special issues through the journal and the university working as partners on research grants. As a result, no author has been charged an article processing fee (APC) to submit or be published in the online-only journal. For its first 12 years, the journal was published by the interdisciplinary research unit, the Agora Center; with this current issue, the publishing responsibility has been assumed by the Open Science Centre, also at the University of Jyväskylä.

From the start, the publisher and editors of Human technology have endeavored to produce a high-quality journal with impeccable ethical standards and a robust peer review process. We established an editorial board with experts from a diversity of fields that research human–technology interaction and continually are expanding the number of disciplines and research focuses represented by members of our board. In virtually every way, the editorial team has embraced the ethical standards for high quality publishing, in line with long-established and well-respected journals.

By some estimates (e.g., Ware & Mabe, 2015, p. 6), as many as 10,000 publishers around the globe are actively publishing academic journals. But questions have arisen from academics across the spectrum regarding the quality of the newer, particularly OA and online, journals (see, e.g., Butler, 2013; Sorokowski, Kulczycki, Sorokowska & Pisanski, 2017). One way to substantiate the quality of a journal is by means of quality assurance undertaken by external organizations, a © 2017 Pertti Hurme & Barbara J. Crawford, and the Open Science Centre, University of Jyväskylä DOI: http://dx.doi.org/10.17011/ht/urn.201705272514

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Pertti Hurme Editor in Chief

Department of Language and Communication Studies University of Jyväskylä

Finland

Barbara J. Crawford Managing Editor

Department of Language and Communication Studies University of Jyväskylä

Finland

Hurme & Crawford

2

process known as “whitelisting” (see, e.g., Berger & Cirasella, 2015; Butler, 2013; Gasparyan, Yessirkepov, Diyanova, & Kitas, 2015; Shen & Björk, 2015), that is, articulating standards regarding peer review, ethical behavior, adherence to scope, and other measures of excellence through which journal editors and publishers can attain accreditation. In many ways, such organizations serve the role of the “reviewer” of journals, assuring they have academic quality and relevance and abide by accepted scholarly publishing standards that contribute to advancing scientific fields, rather than threatening their integrity (Bartholomew, 2014; Clark & Smith, 2015)

External validation provides important recognition for primarily young, open access, and interdisciplinary journals. Human Technology has been indexed by the Directory of Open Access Journals1 (DOAJ)—a key player in whitelisting OA journals (Berger & Cirasella, 2015)—since 2005, receiving reaccreditation under the directory’s revised guidelines in 2016. In February 2017, our journal was accepted by Scopus,2 Elsevier’s bibliographic database, and will be indexed there in the coming months. In the process for being considered for, and then accepted by, these two well-known indexers, we at Human Technology had to supply to the indexers’ editors a wide variety of materials that supported our claim of quality, of reputable OA practices, and adherence to ethical standards. We are pleased that our consistent emphasis on publishing ethics, attention to quality in accepting papers, and filling an important niche in the vast academic publishing world has been assessed and accepted as equally valuable as those journals published by large, established publishing houses.

Obviously, young, independent OA journals cannot have their quality assured right away; indeed, in the early years, many face several challenges. For instance, predatory journals (see, e.g., Cook, 2017) and low-quality journals (Gasparyan et al., 2015) threaten the reputations of good, ethics-abiding journals in that the unscrupulous and poor scientific behaviors of these journals cast a long shadow over all journals, particularly OA journals that cover their publication costs through APCs (Shen & Björk, 2015). Good, young journals accept that, inevitably, it takes time and ongoing attention to reputable and ethical publishing in order to, eventually, earn the recognition of the academic community and accreditation by the quality assessors.

But, considering the reality that quality journals may not yet have been vetted by external assessors such as DOAJ, Scopus, or Web of Science, a separate but equally important question is, How can authors and researchers know to which journal to submit their work or which OA papers reflect sufficiently peer-reviewed quality for citation? The following points are based in part on the checklist of the Open Access Scholarly Publishers’ Association,3 and Butler (2013), Clark and Smith (2015), and Hansoti, Langendorf and Murphy (2016):

Examine the reputation of the journal and publisher: Do you or your colleagues know the journal? Is the journal associated with a noted organization, such as a scientific association,

university, or research institute? Are there any unpleasant rumors about the journal or publisher on the Internet?

Carry out an analysis of the information on the journal’s Web site: Can you easily identify and contact the publisher? Does the journal articulate (and then follow) basic ethical standards? What are the academic affiliations and credentials of the editors and editorial

board members?

Role of Assessors of Quality for OA Journals

3

Is the journal clear about the type of peer review process it uses? Are there outside experts reviewing papers or are all reviewers associated with the

journal (e.g., editorial board members)? What is the editorial process like? A reputable journal will need at least several

months to assess a paper’s quality, get it through review, reassess, and allow time for revisions.

Does the journal have a peer-review label or some other independent notation of quality assurance?

How does the journal assess whether any parts of a paper have been plagiarized, knowingly or unknowingly?

What fees (if any) will be charged? Can you find the publication fee easily identified on the Web site, or is it hidden many pages back with obscure navigation or written in a less-than-clear manner?

Consider the source and approach: Are the research articles published through the journal under evaluation

available through a reputable indexer or research database? Predatory publishers and journals rarely are accepted by these organizations, even if the articles they publish are readily available on the Internet.

Are you being solicited for submission and in what manner? Most reputable journals do not need to send emails with enticing lures on quick publication.

Does the offer seem too good to be true? Chances are, if it does, it is. Please note that while a lacking in any of the above areas does not necessarily mean a journal is predatory or low quality, the number and/or combination of issues should at least merit caution.

In addition to the assurances provided by external assessors, such as DOAJ, Scopus, and Web of Science, authors and researchers can use one of several organizations as sources aimed at identifying quality journals. For example, national lists of reputable journals often are produced (and/or ranked) in collaboration with public organizations and the academic community. Comprehensive lists have been produced by several Nordic countries. Human Technology is included in the Finnish Publication Forum4 and the Norwegian Register for Scientific Journals, Series, and Publishers.5 Currently, academics in the Nordic countries, in conjunction with their governmental officials and several academic communities, are discussing the potential of creating a joint Nordic List of quality journals and publishers. The organizations involved are collaborating as well with DOAJ in regard to open access journals.6 Finally, although the list of potential, possible, and probable predatory journals and publishers,7 compiled by University of Colorado librarian Jeffrey Beall (2012, 2016a, 2016b), is no longer available, many found the list useful and influential (see, e.g., Butler, 2013). However, it also was controversial (see, e.g., Berger & Cirasella, 2015; Butler, 2013; Gasparyan et al., 2015; Straumsheim, 2017). Some academics have called for a more systematic way, and by a wider swath of academic players (e.g., librarians, researchers, accrediting agencies, journal publishers, publisher and editor associations), to identify problematic journals and publishers as a companion to organizations recognizing quality (Berger & Cirasella, 2015; Gasparayan et al., 2015). As the editor in chief and managing editor of a reputable journal that for several years was not yet evaluated by the major indexers, we see value in a formal way of indicating that a

Hurme & Crawford

4

young journal is not disreputable as it develops toward official acknowledgment via the well-known external assessors.

The publisher and editors of Human Technology share the long-term goal of producing a journal that will be accepted as well by the Thomson Reuters Web of Science to supplement DOAJ and Scopus in recognizing our journal’s continued emphasis on quality and ethical publishing and in contributing to the interdisciplinary field of human–technology interaction. For this we need to maintain our long-standing reputation as a quality journal, good submissions, and the hard work of our reviewers and editors, supported by our distinguished Board of Editors.

An additional support toward the journal’s goals is provided by our newly established (2017) internal Publisher’s Board, a team of five experts from the University of Jyväskylä with various scientific backgrounds. The Publisher’s Board members are Päivi Fadjukoff (adjunct professor, Psychology), Marja Kankaanranta (research professor, Digital Learning Environments), Raine Koskimaa (professor, Digital Culture), Pekka Olsbo (head of the publishing unit, Open Science Centre), and Pasi Tyrväinen (professor, Digital Media). Their multifaceted task involves supporting and adding expertise to the publisher’s role in maintaining quality and ethical publishing and in advancing Human Technology within the university, in Finland, and internationally. In their advisory role, they facilitate the development of the journal from the publishing perspective, thus serving as an instrumental collaborator with day-to-day editorial leadership (i.e., editor in chief and managing editor) of the journal. In many ways, they are to the publisher what the editorial board is to the journal’s editors, although there is considerable overlap in their combined efforts toward the advancement and integrity of the journal.

As technological and global interconnectedness continues to develop in the coming years, and new journals are founded to address niches in the scholarly publishing arena, authors and researchers—and journal publishers, as a community supporting and advocating quality, relevant, and ethical journals—need to remain vigilant against unscrupulous and low-quality journals and publishers. External indexers and accrediting organizations play the essential role of reviewers of ethical and quality publishing practices in journals, particularly young, niche, and OA journals.

ENDNOTES

1. More information on the DOAJ, see https://doaj.org 2. Information on the service that Scopus provides is available at https://blog.scopus.com/posts/is-a-

title-indexed-in-scopus-a-reminder-to-check-before-you-publish 3. The Open Access Scholarly Publishers’ Association (http://thinkchecksubmit.org/check/) provides

advice to submitters on assessing reputable open access journals. 4. See https://www.tsv.fi/julkaisufoorumi/haku.php?lang=en for the Finnish Publications Forum. 5. More information on the Norwegian Register for Scientific Journals, Series, and Publishers is

available at https://dbh.nsd.uib.no/publiseringskanaler/Forside 6. The DOAJ announcement of collaboration with Nordic countries regarding evaluating open access

journals can be found at https://doajournals.wordpress.com/2017/03/31/nordic-research-organizations-support-doaj/

7. A copy of Beall’s List of Predatory Journals and Publishers currently can still be found on the Internet (e.g., http://beallslist.weebly.com/standalone-journals.html), but it is no longer updated.

Role of Assessors of Quality for OA Journals

5

REFERENCES Bartholomew, R. E. (2014). Science for sale [Editorial]. Journal of the Royal Society of Medicine, 107(10), 384–

385. doi: 10.1177/0141076814548526 Beall, J. (2012, September 13). Predatory publishers are corrupting open access [Commentary]. Nature, 489, 179. Beall, J. (2016a). Ban predators from the scientific record [Letter to Editor]. Nature, 534, 326. Beall, J. (2016b). Best practices for scholarly authors in the age of predatory journals. Annals of the Royal

College of Surgeons of England, 98, 77–79. doi: 10.1308/rcsann.2016.0056 Berger, M., & Cirasella, J (2015). Beyond Beall’s list: Better understanding predatory publishers. College &

Research Libraries News, 76(3), 132–135. Butler, D. (2013, March 28). The dark side of publishing. Nature, 495, 433–435. Clark, J., & Smith, R. (2015). Firm action needed on predatory journals: They’re harming researchers in low and

middle income countries most, but everyone must fight back. BMJ (British Medical Journal), 350, h210. doi: 10.1136/bmj.h210

Cook, C. (2017). Predatory journals: The worst thing in publishing, ever. Journal of Orthopaedic & Sports Physical Therapy, 47(1), 1–2.

Gasparyan, A. Y., Yessirkepov, M., Diyanova, S. N., & Kitas, G. D. (2015). Publishing ethics and predatory practices: A dilemma for all stakeholders of science communication. Journal of Korean Medical Sciences, 30, 1010–1016. doi: 10.3346/jkms.2015.30.8.1010

Hansoti, B., Langdorf, M., & Murphy, L. (2016). Discriminating between legitimate and predatory open access journals: Report from the International Federation for Emergency Medicine Research Committee. Western Journal of Emergency Medicine, 17(5), 497–507.

Shen, C., & Björk, B.-C. (2015). “Predatory” open access: A longitudinal study of article volumes and market characteristics. BMC Medicine, 13, Article 230, unpaginated. doi: 10.1186/s12916-015-0469-2

Sorokowski, P., Kulczycki, E., Sorokowska, A., & Pisanski, K. (2017, March 23). Predatory journals recruit fake editor [Commentary]. Nature, 543(7646), 481–483.

Straumsheim, C. (2017, January 18). No more “Beall’s List.” Inside Higher Ed post, retrieved 2 May, 2017, from https://www.insidehighered.com/news/2017/01/18/librarians-list-predatory-journals-reportedly-removed-due-threats-and-politics

Ware, M., & Mabe, M. (2015). The STM Report: An overview of scientific and scholarly journal publishing (4th ed.). The Hague, the Netherlands: International Association of Scientific, Technical, and Medical Publishers.

Authors’ Note All correspondence should be addressed to Pertti Hurme Department of Language and Communication Studies University of Jyväskylä P.O. Box 35 40014 University of Jyväskylä, FINLAND [email protected] Human Technology ISSN 1795-6889 www.humantechnology.jyu.fi

ISSN: 1795-6889


6

Guest Editors’ Introduction

HUMAN–TECHNOLOGY CHOREOGRAPHIES: BODY, MOVEMENT, AND SPACE IN EXPRESSIVE INTERACTIONS

THE EXPRESSIVE POTENTIAL OF HUMAN-TECHNOLOGY CHOREOGRAPHIES The design of anything that is intended for us human beings is fundamentally influenced by how the designers conceptualize what is “human.” Researchers in the field of human–computer interaction happily use human to denote the counterpart of technology. In this tradition, however, human often becomes synonymous with a user of technology. As a result, the emphasis tends to be on the user’s cognitive abilities and successful end-results from interaction. Consequently, the embodied nature of humans—as intentional, moving beings—often receives less focus. We, as editors, but more so as researchers, thus want to promote a shift in focus from what can be done with technology to the inherently expressive processes of how humans move while interacting with technology.

In parallel with the emergence of digital applications in everyday life, there is a need to go beyond the traditional usability approach, to see human beings and the available technology from several points of view. Above all, the discussions inevitably relate to the very core of the nature of humanity. When discussing the relationship between humans and technology, researchers and designers take a stand—either implicitly or explicitly—on what is essentially human and what can be outsourced to technology. © 2017 Antti Pirhonen, Kai Tuuri, & Cumhur Erkut, and the Open Science Centre, University of Jyväskylä DOI: http://dx.doi.org/10.17011/ht/urn.201705272515


Antti Pirhonen Department of Teacher Education

University of Jyväskylä Finland

Kai Tuuri Department of Music, Art & Culture Studies

University of Jyväskylä Finland

Cumhur Erkut Department of Architecture, Design & Media Technology

Aalborg University Copenhagen, Denmark

Expressive Interaction in Human–Technology Choreographies

7

In this thematic issue of Human Technology, we editors turn the focus of human–technology interaction from the traditional cognitive aspects to the moving human being, specifically emphasizing the concept of choreography in design and application. This issue was preceded by another special issue within the same topic area (see Pirhonen, Tuuri, & Erkut, 2016). By choreography we want to emphasize meaningful continua of movement that humans, as individuals or as groups, constitute and experience during interaction with technology (see also, e.g., Loke & Reinhardt, 2012; Parviainen, Tuuri, & Pirhonen, 2013; Tuuri, Parviainen, & Pirhonen, in press).

Our perspective on choreographies in technology design and interaction clearly has been demonstrated as timely and interesting. Through the call for papers on this topic area, a large number of high quality manuscripts were submitted to the journal. Following the review process, we ended up with enough papers to fill two separate thematic issues. In addition, three submissions were shifted to an open submissions issue in Human Technology.1 Another reason for two issues was the variation of topics among the submissions. We found that it was quite effortless to divide the submissions into two categories. The submissions in the May 2016 issue concentrated on interaction design, while the submissions in this issue more specifically focused on expressivity in human–technology choreographies. Therefore, the papers published in the present issue deal with themes such as emotion perception, analysis through dancing, interfaces for musical expression, and the biomechanics of drumming. Interaction design, even if it was the primary focus of the previous choreography-themed special issue, can still be seen at least implicitly as an area that potentially benefits from the insights of the contributions of this issue.

Human–technology choreographies denote a multifaceted approach to human life. As can be seen in the contributions of this issue, the concept may be approached from a wide variety of perspectives. Both the backgrounds of the authors and the application areas are so diverse that, in the end, the common ground that all the texts share the interest in the movements of human beings.

Nearly 3 years ago, we first considered how to continue the work involving human–technology choreographies that formed the basis for our NordiChi ’14 conference workshop. We were convinced then and remain so that movement and the bodily basis of human cognition would be an appropriate approach to conceptualize the relationship between humans and technology. Much has happened in technology and in our culture since that workshop. Indeed, it is often argued in the human–computer research field that the form of technology changes so rapidly that research in that area becomes outdated quickly. However, as we observe the research world around us, it seems our theme has not lost any of its topicality. Rather, the expressive power of the moving human body remains a valuable and valued topic that deserves more attention among designers of technology.

PREVIEW OF THIS THEMATIC ISSUE

Olli Poutanen, Salu Ylirisku, and Petri Hoppu propose a method that highlights the value of first-person enactment in user-centered design. They capture implied choreographies in particular interaction scenarios (drawn from two Microsoft Corporation’s Productivity Future Vision videos) and literally dance these interactions while gathering insights. Their research

Pirhonen, Tuuri, & Erkut

8

design is theoretically and practically relevant for the choreographic approach in interaction design (Parviainen et al., 2013). Theoretically, the authors extend the choreographic approach through applying the hermeneutics of the body; practically, they provide a methodical, movement-based framework for operationalizing the approach. Their work offers much for future intelligent environments designed with this approach.

In their article “Body, Space and Emotion: A Perceptual Study,” Donald Glowinski, Sélim Yahia Coll, Naëm Baron, Maëva Sanchez, Simon Schaerlaeken and Didier Grandjean take a holistic stance on the expression and experience of body movements, specifically focusing on emotional content. In their conceptual framework, they combine the perspectives of the performer and the observer, thus outlining an integrated and interactive model of communication through the body.

In this second thematic issue, we are fortunate to have three articles that relate to embodied musical interaction. The first one, titled “Musical Instruments, Body Movement, Space, and Motion Data: Music as an Emergent Multimodal Choreography,” comes from Federico Visi, Esther Coorevits, Rodrigo Schramm, and Eduardo Reck Miranda. The authors first propose techniques for extracting meaningful information from motion capture data. But the core of this paper lies in their suggestion that artistic practices, such as musical performances, can be utilized in better understanding movement-related data for expressive purposes that ultimately extend beyond musical or artistic applications.

Marcella Mandanici, Antonio Rodà, and Sergio Canazza investigated bodily choreographies in producing musically structured events within a music education-oriented interactive space, explicitly explored through an application titled Harmonic Walk. The authors suggest that movement coordination in such applications relates to entrainment phenomena between the human user and the environmental stimuli in the interactive space. Consequently they propose a framework for designing and assessing motion-based music applications.

John R. Taylor demonstrates yet another different approach to the notion of human–technology choreographies. Playing drums requires learning a large repertoire of choreographed body movements. Applying an information-processing stance as a model of human motor control, the author tackles the biomechanics of a percussive performance in relation to the various developmental goals of a drummer. Motivated by research into computer modeling and as a prequel for future empirical studies, the article outlines a theoretical framework for considering the aspects of movement patterns in the interaction between the drummer and his or her instrument.

Finally, this issue includes a book review. The ubiquitousness of technology and its benefits in 21st century societies undergirds much research and product development. But access to this technology is uneven across the globe and is the subject of the tome Public Access ICT Across Cultures: Diversifying Participation in the Network Society, edited by Francisco Proenza. In his review of this 2015 publication, Sakari Taipale assesses the research regarding workable and problematic provision of public access venues (e.g., government-subsidized telecenters, libraries) in 10 developing countries and emerging economies, as well as various demographic groups, on three continents. He argues the book provides an unprecedented overview of Internet access provision that allows the poorer citizens of the world to participate in the Network Society.

Expressive Interaction in Human–Technology Choreographies

9

ENDNOTE

1. See Human Technology’s website (http://humantechnology.jyu.fi/archive/vol-12/issue-2) for papers by Marc-Eric Bobillier Chaumon, Bruno Cuvillier, Salima Body, & Florence Cros; Elena Márquez Segura, Laia Turmo Vidal, & Asreen Rostami; and Pablo Ventura & Daniel Bisig.

REFERENCES Loke, L., & Reinhardt, D. (2012). First steps in body-machine choreography. In L. Loke & T. Robertson (Eds.),

Australasian Computer Human Interaction Conference, 2nd International Workshop: The Body in Design (OzCHI ’12; pp. 20–23). Sydney, Australia: Interaction Design and Work Practice Laboratory (IDWoP).

Parviainen, J., Tuuri, K., & Pirhonen, A. (2013). Drifting down the technologization of life: Could choreography-based interaction design support us in engaging with the world and our embodied living? Challenges, 4(1), 103–115.

Pirhonen, A., Tuuri, K., & Erkut, C. (2016). Human–technology choreographies: Body, movement, and space [Guest Editors’ Introduction]. Human Technology, 12(1), 1–4.

Tuuri, K., Parviainen, J., & Pirhonen, A. (in press). Who controls who? Embodied control within human–technology choreographies. Interacting with Computers. doi: 10.1093/iwc/iww040

Authors’ Note All correspondence should be addressed to Antti Pirhonen Department of Teacher Education FI-40014 University of Jyväskylä, Finland [email protected] Human Technology: An Interdisciplinary Journal on Humans in ICT Environments ISSN 1795-6889 www.humantechnology.jyu.fi

ISSN: 1795-6889


10

TECHNOLOGY CHOREOGRAPHY: STUDYING INTERACTIONS IN MICROSOFT’S FUTURE

VISIONS THROUGH DANCE

Abstract: In the future, an increasing number of devices will be utilized in concert to support human activities, but little is known about how these interacting multidevice settings should be designed optimally in a human-centered manner. We report on a study in which we took two visions created by the Microsoft Corporation as a starting point. The aim of the paper is to describe a method for user-centered design that extends the ideas of a choreographic approach to interaction design and to demonstrate how micromovement analysis can be conducted in practice. We utilized a structural reorganization of movement continua originally presented in the videos for a first-person enactment of that choreography as a means to understand the kinesthetic quality and the potential of the implied choreographies. The approach underscores the influence of interaction designs on the moving and experiencing body, as well as the potential that the moving and experiencing body has for interaction design. Keywords: interaction design, intelligent environment, choreographic analysis, hermeneutics of the body, embodied mind, dance.

© 2017 Olli Poutanen, Salu Ylirisku, & Petri Hoppu, and the Open Science Centre,

University of Jyväskylä DOI: http://dx.doi.org/10.17011/ht/urn.201705272516


Olli Poutanen Department of Design

Aalto University Helsinki, Finland

Salu Ylirisku SDU Design

University of Southern Denmark Kolding, Denmark

Petri Hoppu Media and Performing Arts

Oulu University of Applied Sciences Finland

Studying Interactions Through Dance

11

INTRODUCTION Interaction design is moving from a single-device focus towards interconnected objects in intelligent environments (Gubbi, Buyya, Marusic, & Palaniswami, 2013). This study focuses on human–computer systems in which interplay occurs between the user and a combination of various form factors—for example, smartphones, tablets, screens, and smart objects, such as office accessories with inbuilt intelligence—that collectively constitute a communicating–actuating network. The enablers of the described interactions are embedded in the systems that are built upon logic that exceeds that of any individual device. The ever-increasing number of sensors, processing power, and communication capabilities of objects enhance everyday objects’ capability of producing a “deeper understanding of the internal and external worlds encountered by humans” (Swan, 2012, p. 217). The main challenge of designing environments that are capable of responding sensibly to the user’s actions is, as Rogers puts it, to make the numbers “intelligible, usable and useful” (Rogers, 2006, p. 412).

In this study, we focus on intelligent environments in which the user is given the role of an active agent who orchestrates the network of smart, interconnected devices to serve his/her needs. We argue that, when defining these systems from the perspective of user-centered interaction design, the moving, sensing, and meaning-making body of researchers and designers should be considered the core of the design research method. We describe an analysis method that quite literally puts the researchers and designers “into the shoes” of the user and increases their awareness on how physical setups of interconnected visual displays, moving application controls, and different haptic–tactile input and visual output mechanisms influence the user’s body during the interaction. Our focus is on the ways in which a designer or a researcher can explore and analyze the qualities of movement experienced by the user during interaction. We describe how we used the method to analyze a conceptual use context. The analysis data come from two Microsoft Corporation’s videos on intelligent environments, namely Productivity Future Vision (2009, 2011).

In this study, choreography is the central concept that serves both as a term of describing body movement in space and time, in which case it is useful in conducting practical movement analysis, and a concept that links our approach to mobile aspects of embodied methodology. Through choreography, movement can be analyzed both as an ephemeral action and a sustained, recurrent structure. The term choreography originates in the domain of dance. It refers primarily to the structuring of movement in space. However, it also implies a profound attentiveness to the body and its movements both on stage and in everyday life, where any movement or mobile action can be turned into choreography when performed by people trained in these activities, such as dance (Thomas, 2013, p. 3). Even within the field of dance, the concept of choreography does not merely refer to an external model or schema for structured actions, but the term has been expanded to encompass its frame and parameters, dancing bodies and their physicality, and processes of learning, as well as the ways in which the choreography is actualized and experienced (Foster, 2011, pp. 215–218). Furthermore, choreography is not limited to dance, but it is used to describe manifold embodied mobile processes, even the functioning of the brain. As J. L. Hanna (2015) suggested, the brain organizes sensory experiences, thoughts, and actions into meaningful entities, such as thinking, speaking, and dancing, which Hanna regarded as a complex choreographic process. Choreography may indeed point to the fact that the mode of cognitive organization is characterized by mobility

Poutanen, Ylirisku, & Hoppu

12

and, therefore, the concept of choreography can be seen as a valid tool for analyzing complex systems of human behavior (Mason, 2009, p. 32). Performed actions, as well as sequences of movement progression, can be encompassed within the term choreography.

The study of movement is not new to interaction design and research. Even as the focus in the field has shifted over decades from usability, user experience, and co-experience to embodied interaction (see Dourish, 2001), all these frameworks have considered the user’s movement as part of the interaction. Understandably, the different frameworks’ relationship to movement, as well as the scope and rigor of movement analysis, vary considerably among these traditions. In some approaches to interaction design, movement has been studied with special attention. This applies especially to movement-oriented interaction design (see Loke & Robertson, 2013). We acknowledge this background, yet we also claim that the role of the physical human body and the ways in which the acting body forms structures of movements in space, in interaction, and in environments with networked technologies has thus far attracted limited attention.

Not surprisingly, the scientific treatment of the human body in interaction design has its roots in the ontological, epistemological, and methodological structures of Western knowledge. Science has been regarded as belonging to the realm of cognition which, in turn, has been treated as nonembodied, that is, a matter of the mind. This, however, has been questioned by scholars from neurologists (e.g., Damasio, 1994) and linguists (e.g., Lakoff & Johnson, 1999) to dance researchers (e.g., Parviainen, 1998). Moreover, philosophers, such as Nietzsche (“there is more reason, sanity and intelligence in your body than in your best wisdom”; 1883/2010, p. 32), Heidegger (1975), and Merleau-Ponty (1945/1965), also aimed at challenging the dominance of logocentric knowledge in Western thinking. Interaction design that factors in the human body, its knowledge, and its versatile intentions in human–technology choreographies may positively contribute to the development of everyday environments.

It is challenging to define the interrelationship between bodies, movement, and objects in interactive environments. A possible starting point for understanding the nature of the interaction and for describing complicated embodied interrelationships is to count human beings as part of the “thing world.” Thus, the body is characterized as the user of things that, in turn, can be considered augmentations of the biological body. The body and the things are thus characterized by constant coevolution in which the reach and the possibilities of the body are defined by the nature of the hybrids created (see Thrift, 2008, p. 10). Because the moving body and its intentional behavior are emphasized as constitutive elements of human–technology interaction in this study, the consequent definitions of interaction can be built upon theories in which the body, movement, and perception are seen as sources of knowledge.

The question of the embodied mind from the perspective of natural sciences has been studied by a number of scholars since the late 20th century. As a biologist, philosopher, and pioneer of the science of embodied cognition, Francisco Varela (1999) applied phenomenological analyses of time-consciousness to neuro-dynamic processes connected to the lived temporal experience of the embodied ego. Influenced by Varela and other scholars of the embodied mind, philosopher Evan Thompson (2007) stated that concepts, language, and thinking are based on embodied information: In other words, the lived body is the foundation of all knowing. Similarly, neurologist Antonio Damasio (1994, p. 118) concurred that the human body, not only the brain, is inextricably and comprehensively intertwined with the mind in the sense that the mind can be seen as embodied. Linguists, such as Lakoff and Johnson (1999), have developed similar themes


13

from the perspective of cognitive science, addressing how language and thinking have their deepest roots in embodied metaphors.

The embodied mind is tightly connected to embodied knowledge, a concept that emphasizes the body as the basis of knowledge. Michael Polanyi (1966) named the embodied dimension of knowing “tacit knowing,” which referred to the kind of knowing that cannot be completely articulated verbally, as it remains rooted in the human bodily abilities and expressions. Embodied knowledge belongs to the realm of nontraditional episteme, in which valid knowledge is not merely propositional. This was further demonstrated by Maxine Sheets-Johnstone (1999, pp. 490–491), who stated that thinking is essentially embodied in a way that is often a matter of movement, not of language, and, as a result, it is not restricted to mental capacities of the brain. Thinking in movement is not the work of a symbol-making body but an existentially resonant body that knows the world through its own kinesthetic qualia as well as the movement of the surrounding world (Sheets-Johnstone, 1999, pp. 516–517).

Embodied or tacit knowledge is often regarded as something that escapes verbal explication, which results in the phenomenon being difficult to recognize and analyze. This, however, is partly a delusion because language is not the only way to articulate interpretations of the reality. In this article, we argue that movement is an integral and natural part of human thinking and that it supports the organization of ideas in human–computer interaction. Movement serves as a means of documenting and reporting results that are easier to understand by looking at the enactment of the movement sequences. In this study, enacting micromovement continua formed an important step in getting access to the kinesthetic experience of human–technology interaction. Furthermore, performing movement sequences with awareness of specific technological contexts made it possible to concentrate on user-centered understanding of the mutual interplay between the user’s body and senses in interaction with technology. Microsoft’s videos Productivity Future Vision (2009, 2011) provided a visual reference for the technological settings. These videos provided comprehensive scenarios of how interactions between a user and a set of devices unfold in space and time. The videos are precise in representing movement continua, which makes them suitable for choreographic analysis. Because the researcher who danced the choreography (author Olli Poutanen) is very familiar with the Microsoft video material that presents the interactions in context, the enactment of choreography in this study also serves as a mediator between the felt essences of movement and the situated interactions.

The aim of the study is to extend the choreographic approach to interaction design introduced by Parviainen, Tuuri, and Pirhonen (2013) and to introduce a research procedure that supports conducting microlevel choreographic analysis in practice. The outcome of a choreographic approach enhances the design of human–technology interfaces that stimulate the imagination of the user with creative movement practices. Ideally, the choreographic approach facilitates the creation of such systems and configurations of networked products, services, and systems that enable users to become more expressive, effective, and productive. According to Parviainen et al. (2013), movement-based interfaces, actions, and the movements themselves may support the creation of sensitive or intelligent user-centered experiences. We find the ideas of choreographic approach very relevant, yet the descriptions provided in the literature do not provide a clear methodical framework for operationalizing the choreographic approach. We adopt the idea of micromovement continua by Parviainen et al. and extend it with another approach called hermeneutics of the body (Hoppu, 1999, 2005). Hermeneutics of the body enabled us to build a framework for conducting first-person enactment and for performing interaction choreography as


14

part of a micromovement analysis. An experiment of utilizing this approach is described and discussed below. The method development presented in this paper is based on a previous experiment (Poutanen, 2015); thus, the experiment is presented in a descriptive manner. The presented method contributes to the understanding of how the kinesthetic quality and potential of the implied interaction choreographies can be addressed through interaction design research.

RELATED WORK

One of the pioneering documented works in the field of prototyping ubiquitous computing environments is reported by Johanson, Fox, and Winograd (2002). Their work outlined both design principles and aspects that need to be considered in evaluation. Their principles for the design of a ubiquitous computing room for teamwork included valuing the collocation of people and the materials they work with, the appraisal of the social conventions of the people, and the aspiration for simplicity. Using these principles, Johanson et al. created an interactive space with novel applications and computing devices that enabled them to study how people used these applications. They studied interaction with a special focus on three generic task scenarios: moving data, moving control, and dynamic application control. Johanson et al. concluded that one of the most critical aspects of the fluency of the space and interactivity is user learning. To facilitate success, users should not be forced to learn more than a minimal number of new controls and modes. We view this perspective as where both the embodied approach and the concept of choreography become very useful, in that designers can become aware of and build systematically on the existing repertoires of bodily conventions of users.

Iqbal, Sturm, Kulyk, Wang, and Terken (2005) studied how ubiquitous technologies could provide support for collocated communication and collaboration in meetings and in teaching. The evaluation method focused on social dynamics, and the goal was on discovering how well a computing environment might function as a proactive “butler” that senses what the users need and then assists them at the right time in the right way. This approach emphasized the intelligence of computing systems over people’s embodied capabilities. Their evaluation built on the collaboration usability analysis (Pinelle, Gutwin, & Greenberg, 2003).

The evaluation of intelligent environments, such as control rooms with complex networked equipment, thus far has been primarily based on the traditional usability and ergonomics emphasis (cf. Savioja, 2014). These evaluations emphasize safety, human well-being, and the attainment of predefined goals. It is understandable that, in production environments, this is a highly feasible approach that conforms to general regulations about work regarding expectations of efficiency. However, when design focuses on private and creative workspaces, a more flexible use of digital resources may lead to better results in terms of work productivity and enjoyment of interaction. In order to achieve these goals, further development of methods and approaches that recognize variation in the users’ needs, as well as differences in the users’ motivations, is needed.

Loke and Robertson (2013) reported a long line of research in movement-oriented interaction design and, on this basis, they formulated a method as presented in their article “Moving and Making Strange” (i.e., moving, sensing, and feeling in order to open up new spaces for design). Central to this method is providing a coherent approach to movement-based interaction design in which movement is considered an input for interaction design. According


15

to Loke and Robertson, the approach allows for a systematic and principled development of movement-based interaction. The approach operates from three central perspectives, those of the mover, the observer, and the machine. The process of moving and making strange consists of seven phases: (a) investigating movement, (b) inventing and choreographing movement, (c) re-enacting movement, (d) describing and documenting movement, (e) analyzing visually and representing movement, and (f) representing machine input and (g) interpreting moving bodies (Loke & Robertson, 2013, p. 7:11). The moving and making strange approach suits well the design of movement-oriented interactive systems.

Kaasinen et al. (2013) approached intelligent environments from a user-centered perspective and built their approach on the concepts of user expectation, user acceptance, and user experience. They recognized the interplay between the concepts of expectation and experience. On the one hand, experiences are affected by individual expectations, and, on the other hand, experiences can become influential in the formation of future expectations (Kaasinen et al., 2013, p. 12). User acceptance, meanwhile, is defined as the intention to use a certain technology or the actual use of that technology (p. 6). The writers pointed out that the majority of user acceptance studies were first conducted in organizational contexts. User acceptance theories have been used to provide an understanding on how workers adopt new information technologies and on how this information could facilitate adoption processes in the target organization. Since then, the arena of user acceptance studies has widened to accommodate complex interrelating systems and consumer-oriented applications. With these reinforcements, Kaasinen et al. argued that user acceptance studies have incorporated some of the critical variables to benefit the studies on intelligent environments (p. 6).

Kaasinen et al. (2013) also noted that users should be provided with a role as co-crafters of their intelligent environments (p. 3). They argued that, as intelligent technologies become ubiquitous and pervasive, the use of intelligent technologies becomes less optional; rather, smart technologies become part of users’ surroundings and daily life. Thus, users should be allowed to participate in the development of these environments because the solutions directly affect their lives (p. 3). Rogers (2006) presented similar ideas and stated that, despite the expected major leaps in technological intelligence and automation, taking the initiative to be creative, constructive, and in control of the interactions within the technological environments is an activity that should belong to the user. Furthermore, technologies could be designed to activate and motivate the user through interactions that creatively, excitedly, and constructively extend what people currently do (Rogers, 2006, p. 406).

Typically, interaction design in intelligent environments is conducted and presented in the form of narratives, such as scenarios (Carroll, 2000), customer journeys, or journey maps (Polaine, Løvlie, & Reason, 2013). These methods omit a wide portion of the actual structures of movements in which people interacting in the technologically mediated environment engage. Choreographic analysis ensures that indirect yet meaningful movements that emerge in the particular local context of use are critically examined.

METHODS AND RESEARCH DATA

For this study, we expanded the choreographic approach to interaction design method of Parviainen et al., 2013) used in a previous experiment (Poutanen, 2015) in which two videos


16

presenting a future view of networked technology by the Microsoft Corporation were analyzed. Both videos are called the Productivity Future Vision: The 2009 version runs 5 minutes and 46 seconds, and the 2011 version has a length of 6 minutes and 18 seconds. These videos narrate how future information networks could be utilized with ubiquitous interactive technologies, clearly emphasizing the benefits of linked data resources, open access, and powerful information processing. The visions are presented as a sequence of different scenes. Together, these two videos can be interpreted as providing views to a day-in-the-life of the two central characters. The scenarios cover places and situations that characterize smart home, intelligent transportation, and smart office, and merge personal and professional communication.

Our analysis of these videos was conducted with the choreographic analysis method extended with the hermeneutics of the body method. The approach yielded new research data in the form of notes and experiences that were utilized in the analysis. For example, the data included notes about desirable and undesirable characteristics of movement continua, such as intuitive movement sequences, logical visual–haptic couplings of augmented and internal feedback, and kinesthetic coherence. The notes were initially made in a format of a dance diary, which was kept throughout the research process. The dance diary was in itself an open-ended but frequent process of writing entries about ideas and experiences related to different steps of the experiment. Connections between the dance diary and the adopted methods—choreographic analysis and hermeneutics of the body—made the dance diary a central tool for reflection of theory in relation to the research data, which consisted of two videos Productivity Future Vision (2009; 2011). The dance diary was also a key process that provided data that allowed documentation and reflection about the hermeneutic process, as experienced by the enacting researcher Olli Poutanen.

Choreographic Analysis

Choreographic analysis (Parviainen et al., 2013) is an analytic approach that addresses diverse movement dynamics and emphasizes the perspective of an embodied and active user. It enables the assessment of interaction choreographies on various temporal and spatial scales. The approach may be used to detect design flaws and highlight successful choreographic designs related to both individual and intersubjective interactions. Choreographies serve as instruments for shifting the focus of design from the shapes and structures of objects to the activity and intended use of technology. Understanding and expressing the user’s movement continua is a practical way to approach the movement-oriented flow of everyday activities (Parviainen et al., 2013). The choreographic approach addresses the embodied dimensions of experience that can be accessed only by an experiencing body, and choreographic analysis hence recognizes the researcher’s body as a legitimate research instrument (cf. Hoppu, 2005).

In practice, choreographic analysis can be described with a three-level structure. The levels are labeled as micro, local, and macro, although all of the levels are interconnected. The microlevel describes the dynamicity of “acting-sensing bodies and enactive minds” (Parviainen et al., 2013, p. 110) and the focus is directed toward the subtleties and habitations of the user’s muscular activity. The spatial focus of microlevel choreographies is the kinesphere, that is, the space within the body’s reach (Parviainen et al., 2013, pp.110–111; also Laban, 1963, p. 85). Different ways of touching, looking, twisting, squeezing, pushing, turning, and so on, are examples of micromovements that are considered in the analysis. These individual micromovements are not


17

studied in isolation but as constituents of micromovement sequences. These movement sequences form the units of analysis for this study, and they are examined in relation to the kinesthetic experience of conducting the movement continua.

Local-level analysis concentrates on the relation between the engaged performer and the “intentional, environment-oriented and social aspects of interaction” (Parviainen et al., 2013, pp.110–111). A local-level analysis aims at clarifying how the flow of the movement continua fits into the user’s situation and addresses usability-related questions, such as the intuitive and engaging character of the interactions. Macrolevel analysis addresses the “socio-cultural effects of the design choices” (Parviainen et al., 2013, p.110). This is a systems-level perspective that helps in addressing the question of choreographic sustainability. Macrolevel analysis operates on a global scale, addressing phenomena both in geographical and virtual space (Parviainen et al., 2013, pp.110–111).

Microsoft’s Productivity Future Visions (2009, 2011) expectedly presented an abundance of visions related to computation, on-demand processing and visualization of data, smart supply chain management tools, and telepresence meeting solutions, all relevant in both local-level and macrolevel choreography analyses. However, because the kinesthetic analysis is the main focus in this study, the local-level and macrolevel perspectives were not aspects of our analysis. More information on local-level analysis can be found in the previous study (Poutanen, 2015, pp. 44–53).

Hermeneutics of the Body

Hermeneutics of the body is a method developed by Petri Hoppu (2005) building primarily on the thinking of Hans-Georg Gadamer (1960/1975). It presents a perspective of science that connects research and dancing. In hermeneutic analysis, the researcher is in contact with his/her object of research (in Gadamer’s work, primarily texts) and, as the researcher gains new insights concerning the object, his/her perceptions and expectations change. The basic idea of hermeneutics can be abstracted as a spiral representing a process that deepens the researcher’s knowledge of the research object (Hoppu, 2005, p.107). With the felt dimensions of movement as the object of research, the researcher’s own body and the memories carried by the body can be seen as primary resources of the study (Sklar, 2000, p. 71).

This entire process of interaction can be regarded as an extension of the human empathic capacity, which, according to de Waal (2008, p. 281), enables a person to access and assess another person’s emotional state and the reasons for the state, and to adopt the other person’s perspective. According to philosopher Edith Stein (1917/1989, p. 20), adopting the other person’s perspective is a special way of “knowing within others,” which differs from conceptual knowledge. The latter knowledge is typically based on verbal communication, whereas the former is that which can be grasped primarily through the experienced, embodied acts with the other. Stein’s view was elaborated by Parviainen (2003, pp. 157–162), who referred to kinesthetic empathy as a key feature in understanding another person’s nonverbal kinesthetic experiences and in acquiring knowledge of his/her movement. When one perceives another person’s movements, one does not merely perceive bodily expressions, but also the living experiences of those movements, though not primordially. Moreover, Dee Reynolds and Matthew Reason (2012) argued that kinesthetic empathy is not limited to human interaction, but it also includes objects and spaces. An object may merge with its human counterpart through an empathic act, of which riding a bike is an example, and a space may have a strong, embodied impact, such as on a person who enters a large room.


18

Typically, researchers have paid little attention to the embodiment of research, despite the fact that corporeality forms an essential part of social reality. According to Thomas (2003, p. 77), little discussion has explored the embodiment of research activities, that is, through practices such as observation, interpretation and/or analysis, and, more importantly, the influence of these on the experiences of both the researcher and the researched (see also Hoppu, 2005, p. 107). However, the body is clearly a means to acquire knowledge of the movement, and—through understanding “qualitative and associative nuances” of the movement—the body provides the researcher with a language to express this knowledge (Sklar, 2000, p. 75). Therefore, the moving and experiencing body is not only a phenomenon under investigation but also an evolving reflexive process of inquiry involving the researcher as a living subject with his/her activities examined from the perspective of embodiment in this approach.

Traditionally, the hermeneutic analysis begins in the field, where a researcher meets practice. During this phase, s/he also articulates his/her experiences in text. In our research, the fieldwork phase refers to the learning and performing a choreography based on the micromovements performed in the Microsoft videos. Here, choreography is seen as a process of both producing and reproducing embodied knowledge—a way of articulating and structuring movement as well as learning and conveying it through empathic acts. In other words, choreography is not limited to something that is merely observed, but it is something that may become a part of a researcher’s body. This resembles Nonaka and Takeuchi’s (1995, p. 64) mode of socialization, where tacit knowledge is conveyed from one person to other people through interaction or observation. Their model has been criticized for being limited (Gourlay, 2006). Nevertheless, adopting tacit knowledge into one’s body is regarded as a significant facet for developing an understanding of the body-movement phenomenon under investigation.

Embodied knowledge needs to be made explicit for people to become aware of it. What Nonaka and Takeuchi (1995, p. 65) called externalization is also a central element of research. In our study externalization encompassed (a) learning to perform the movements, including the continua of body postures, individual limb movements, and sensuous foci reaching out to the surrounding space, (b) understanding meanings of interactions, and (c) interpreting the nuances relating to the microlevel choreographies. In the overall hermeneutic process, embodied knowledge was externalized and documented in active writing, which implies that the researcher constantly compared what he had written regarding evolving experiences, feelings, and skills, and iteratively works towards a deeper understanding of what was examined (Hoppu, 2005, p. 109). The hermeneutic process intertwines human embodiment, its experience, and its interpretation (Hoppu, 2005, p. 107).

The Research Procedure

The research procedure of the micromovement interactions was conducted in five steps: (a) extracting, (b) reorganizing, (c) performing, (d) dance writing, and (e) analyzing.

During the extraction step, the main elements of the choreographies were extracted as separate movements from Productivity Future Visions videos (2009; 2011). A total of 112 individual screenshots depicting individual micromovements were taken and arranged. The screenshots were first arranged into a sequential continuum from a start of an action to its completion. The screenshots were organized on a computer screen so that each movement continuum was organized on a vertical axis, resulting in separate image clusters. Within each


19

image cluster, the first micromovement was set in the uppermost part of the screen. Screenshot clusters representing different scenarios, movement continua, and use contexts were aligned horizontally in rows. This approach provided a wide visual view of the material of the choreography as a whole and thus provided a good starting point for the reorganization of the data.

In the reorganizing step, the previous screenshot clusters were reorganized into three clusters corresponding to the body position of the user during interaction, namely the sitting position, standing position, and walking. The reason for reordering the formerly mixed order was to enhance the flow of activities that composed the choreography, thereby allowing a movement progression from the performance components (i.e., rehearsal and “dancing”) to analysis. Based on body postures, the enactment of the choreography logically flowed through all relevant movement continua first from sitting, proceeding to interactions in a standing position, and moving on from standing to walking, in line with the original interaction.

In the performance step, the enacting researcher performed the rearranged choreography through the chunked movement flows. This performing was supported by a process known as dance writing, meaning developing a script of the micromovements to be performed as well as writing down observations that emerged during the performing. The performing and dance writing steps were carried out alternately. In the rehearsal of the movement continua, the focus of dance writing was on the technical reproduction of the movement: the correctness of the body alignment and executing right micromovements in a right order. Once the choreography was internalized by the researcher, the dance performance process was initiated, with the focus shifting to assessing the bodily experience of the movement. Dancing the movements was thus the premise for creating embodied understanding of interaction and engaging a state of critical thinking where the nuances of interaction could unfold. At this point, a dance diary was used to explicate emerging ideas and interpretations of the kinesthetic experience of human-technology interaction.

Figure 1 illustrates how the movements were performed without representations of technology to better focus on the movements and kinesthetic experience. The enacting researcher performed all the micromovements involved in the specified activity, including the gazing, and imagined the use context that involved the devices, the physical architectonic space and furniture, and the information content around him. This was important because, without keeping the use context in mind, the analysis would only have represented the pleasurability of a certain movement sequence. The researcher’s ability to differentiate transitions from one device to another as well as the spatial organization of the choreography allowed him, for instance, to locate discontinuities in the movement flow and reflect the cause for dysfunction in the suggested micromovement continuum for a specific interaction.

The researcher conducted the choreographed dance within a 7-day timespan of intensive engagement with the movements. The dancing cycle consisted of morning and evening sessions. Writing about the dancing experience was conducted daily within the two sessions, resulting in an iterative writing process typical of the hermeneutic process. The choreography was danced five times a row in each session. A dance diary, written by the enacting researcher immediately after the session, was used for documenting the ideas and findings that emerged during and after the dancing. Diary entries were written in a nonstructured manner. As the writings based on the sessions started to accumulate, several interests and themes started to shape. The practice of writing helped the enacting researcher to explicate kinesthetic sensations and become aware of specific nuances in the choreography. Performing and writing thus constituted a reciprocal process.


20

Figure 1. In the movement continuum example above, the user copies information from one device to

another with gesture commands without actually employing the technology, thus reflecting a perspective of movement experience of using the technology within the imagined use context. The enacting researcher holds his left hand in a pointing position and enacts a sequence in which he directs imaginary information

content on a computer screen. In using his other hand, the user drags the content from a computer to a tablet device. This enactment also demonstrates how the micromovement of dragging is part of a

continuous flow of movement in the overall choreography that progressed seamlessly through the various actions conducted with the imaginary technology surrounding the performer.

Altogether, the enacting researcher performed the choreography 70 times during the entire

enacting period, which means a total of 7840 micromovements performed in the cumulative choreography. One round of performing the full choreography took approximately 2 minutes 30 seconds; thus, the entire study totaled 175 minutes of active dancing, traversing all the studied movements. Going through movements to this extent is not essential for learning an advanced skill, but it nevertheless allows for the attainment of such repeated embodied experience of the studied choreographies that functions as a foundation for drawing conclusions about the kinesthetic experience and about the experienced effectiveness of the performed actions. At the end of the performing and dance writing steps, a final performance was documented on video with three cameras in a television studio. The choreography included the 112 micromovements, flowing from sitting to standing to moving, resulting in a total of 150 seconds of performed choreography, that is, the dance. The video shows the enacting researcher’s body in a quasi-empty studio. The space was furnished with only a chair for presenting movements in a sitting position.

Because the performances were created without physically employing the technologies under investigation, the dance rendered visible the movements that were obscured by the overwhelming visual richness of the original video material that utilized special effects extensively. Thus, the visual documentation of the dance underscored the relationship of reciprocal influence between and among the technological designs and interactive interfaces and a user’s body and his/her movement on the other. The visual performance was, however, only a part of the process. Addressing the experience of performing the movements as if being immersed in the use context formed another challenge.

The research process concluded with the analysis. It would be erroneous to state that it was the last phase, particularly when performance, reflection, and analysis occurred in a hermeneutic loop throughout the study. The analysis process, therefore, unfolded as a reflective and analytical dialogue within the construction of the choreography. In this dialogue, attention was paid to the microlevel phenomena through the definition of movement continua (Poutanen,


21

2015, pp. 38–53). The analytical focus was on the kinesthetic experience on one hand, and on technology design on the other. The notes compiled in the dance diary created by the enacting researcher regarding the microlevel choreographies were grouped and the groups labeled by the enacting researcher during and slightly after the active performing of the choreography. The focus was on discovering the desirable and undesirable characteristics of the movement continua, such as intuitive movement sequences, logical visual–haptic couplings of augmented and internal feedback, and kinesthetic coherence. An example of the imaginary technology and the gestural trajectory are depicted in Figure 2 to illustrate how the enacting researcher conceived the technology in his immediate surroundings in performing the interaction.

Figure 2. A visualization depicting the imaginary technology (i.e., a tablet and a desktop computer)

superimposed onto a frame from the movement continuum used by the enacting researcher in performing a sequence. These visualizations were not part of the performance but are added here to show how the

enacting researcher was imagining the technologies.

FINDINGS The process of reorganizing the micromovements and the iterative processes of rehearsing/dancing and dance writing yielded discoveries in multiple areas. Most importantly, the enactment generated insight into the kinesthetic experience and usability of technologies proposed in the Microsoft visions as well as into generic considerations of how particular technology artifacts constrain, guide, and support movement, positioning, posture, orientation, and sensuous foci. The visual–tactile continua in time and space were found to have a strong significance in regard to the experience of interaction. This outcome points to the importance of how the orchestration of human–computer interaction within a network of heterogeneous, interconnected devices could be made to enhance the user’s kinesthetic experience.

The technological designs in both Productivity Future Vision (2009, 2011) productions are based on five distinct technological artifacts: mobile resources, tabletop screens, smart walls, wearable technologies (glasses and wristwatch), and small-sized smart objects. All these intelligent interactive elements impact the user’s movement, position in space, posture, and sensuous focus, among other embodied experiences. In the Productivity Future Vision videos, many of the artifacts employed have specific vocabularies and typical movement continua for


22

interaction, both individually and when combined with other types of artifacts. One key finding of our research was that combining various artifacts creates the setting for choreographies. Understanding the different choreographic possibilities related to certain technologies and their collaborative use can thus facilitate interaction design in specific use contexts.

Kinesthetically, the enacting researcher experienced the use of small handheld devices as unpleasant and restricting. The experience probably was compounded by the lack of any actual reference point in the enacting researcher’s hand. Larger screens, however, enabled the utilization of broader movement trajectories, allowing the activation of larger areas in the body. The enacting researcher considered these movements more natural and pleasurable. This outcome also could be a question of personal preferences. The smartphone and tablet-size interfaces performed well in situations in which the actual data manipulation was realized with embedded information resources, such as on large wall screens or on the top of a table. The smart handheld devices were experienced as performing the best when operated within the ecology of resources, such as screens shared by connected computing processes (see also Poutanen, 2015, pp. 55–59).

The tactile–visual input–output loops between the user and the technological system can be considered a basic organizing principle of human–computer interaction in the Productivity Future Vision (2009, 2011). The user’s body during the interaction seemed to be related to the spatial–temporal progression of visual foci in space in ongoing interaction. The result of this aspect of the analysis suggests that dynamic relations between the visual and tactile foci have a comprehensive influence on the user’s body during the interaction, which should be further examined. A key finding was related to the role of the user’s sight in interaction; when his/her gaze was traveling across multiple screens provided by a device network, it seemed to have the potential to enable choreographies that activate the user’s body, for instance, enhance shifts between body postures and alignment in space, as a reaction to relocation of visually displayed information in surrounding space. We suggest that designs that assist in shifting visual focus can be intentionally designed to enhance smooth changes in body alignment and posture. On the contrary, superimposed visual and tactile foci, especially when extended over time, can lead to stagnant and unpleasant kinesthetic experiences (see also Poutanen, 2015, pp. 58–59).

Kinesthetic Pleasure

Dance reflections proved to be valuable in recognizing the challenges within the micromovement sequences and, in contrast, finding insightfully designed choreographies that resulted in pleasing kinesthetic experiences. First indications of successful choreographies arose at the beginning of the dance practice. The enacting researcher recognized that it was easy and quick to internalize a particular interaction into dance. Dancing also evoked personal experiences of the pleasurability of particular movement continua. In particular, tapping as a method for drawing data from the device was reported as fun by the enacting researcher. In the dance diary, the enacting researcher described the experience as resembling the act of tapping a floating object so that it submerges and resurfaces with a popping motion.

The kinesthetic feel of the micromovement choreography of a map application that zoomed in response to vertical level alterations of the hand-held smartphone screen also was experienced as pleasurable. The motion would enlarge the map when lifting the device up and to shrink the map when moving the device down (see Figure 3). This movement was among the favorite movement sequences reported by the enacting researcher. The application enabled dynamic exploration of a


23

Figure 3. Example of using level-alterations in scaling the content to a desired level of detail

on a relatively small screen; moving the device upward zoomed in and downward zoomed out.

map on a small-screen device through use of two hands. It recognized sweeps and finger-based zoom commands (two or three fingers) without compromising the usability of the application.

This particular use feature also provided a surprise for the enacting researcher. The visual image of the use situation on video was not appealing at first. However, the related somatic experience of the micromovement choreography of the interaction was experienced as pleasing. Thus, it can be stated that the embodiment of the choreography changed the enacting researcher’s preconception of the use experience of this particular interaction. Enacting also revealed that it is not enough to design good individual movements in isolation: It is necessary to think through how the complete interaction choreography flows as a movement sequence. A clever micromovement idea can be lost within a clumsy movement sequence.

Based on the experience of the enacting researcher, some of the movement combinations required uncomfortable body postures and the suggested micromovement sequences resulted in unnatural bodily positions. For example, the interaction was experienced as unpleasant when the researcher had to stretch his arms towards the side and perform a converging movement. The enacting researcher, a young healthy male, experienced these movements as straining. However, working through the complete dance choreography sensitized the enacting researcher to the differences and similarities across the kinesthetic experience of the movement sequences. Thus, some of movement continua were experienced as complex and physically demanding while others were free flowing. Some movement sequences were experienced as rigid, and their rigidness was highlighted by the surrounding fluent movement sequences. Thus it is possible that technological interactions presented in video productions as smooth and pleasing could, in reality, not be the case. Analysis of technology use through reenactment is therefore a more truthful or accurate means for assessing the quality of experience than is watching a mediated use when the kinesthetic experience is the focus of interest. Usability and Embodied Engagement

In the Productivity Future Vision (2009, 2011) videos, it was suggested that the envisioned interactions enable the user to accomplish technological interactions smoothly. The visions presented rich variations in choreographic strategies that seem to be created in order to avoid interactions with strictly logical means. Usability is taken into account through the optimization


24

of user–system behavior by minimizing the time, effort, and errors by users trying to attain their goals (cf. Nielsen, 1993). Some of the usability issues were easy to identify through the choreographic approach. For example, it was apparent that some of the micromovement sequences were too complex, exemplified by the difficulty in internalizing and performing the sequences. This resulted in extra movements, as compared to other interaction choreographies with a similar intent. By making the usability issues explicit between homologous choreographies, it was possible to locate inconsistencies in the overall choreographic vocabulary.

However, there were some examples of intuitive choreographies that could be figured out just by using the technologies, that is, “learning the possibility of drag and drop interaction can also be learned by accident” (Dance diary excerpt). Usability was also reported as a user-driven recombination of visual displays in space, in that the “wider table-surface-computer provides a larger screen that supports extending the illustration into a form that supports [the activity of the user]” (Dance diary excerpt). A figure representing the transmission of information from one device to another between a table surface computer and a tablet computer device is presented in Figure 4.

In sum, it could be concluded that usability was assessed by the enacting researcher in terms of how a technological system supported the body’s ability to learn, remember, and become skillful in controlling networked resources in the intelligent environment. The researcher’s personal learning process through choreographed reenactments of the micromovements provided a benchmark on how that can be achieved.

Limits of the Choreographic Approach

The choreographic approach had limitations in instances in which particular information perceptualizations had to be considered. Productivity Future Vision (2009, 2011) presented a number of interactions in which the content at the edge of the screen seemingly disappeared from

Figure 4. Example of how devices network spontaneously in Productivity Future Vision and

how the advantages of different forms can be seized; in this figure the surface of a large screen has been taken into use in scaling up content in order to create a more appropriate

visual representation concerning the user’s task at hand.


25

a device and reappeared on another. Because no augmentations or props of any kind were used when reenacting these specific choreographies, interactions in the video data had to be consciously performed in keeping the original context in mind. The problematic character of these events, however, unfolded when the nature of that particular movement continuum was developed into embodied knowledge of the enacting researcher, and thus an active component of the reflection that emerged during the performance. In confronting these types of situations, it is important that the researcher observe closely in the original video the relative position of the devices or whether the target screen is situated behind the sending screen, on top of it, or somewhere else. Moreover, a choreographic analysis is based on subjective experience and the quality of the analysis corresponds with the researcher’s experience and skills. The approach can be considered as artistic research on technology. Further development and adapting the method to a more systematic evaluation of human–computer interaction is needed.

Some usability issues are more challenging to assess through the choreographic approach, however, such as movements that required painstaking care with body positioning. For example, while a menu interface required the user to control his/her hand accurately in an exact position, the danced performance was unable to recreate the role of the physical surfaces in providing physical support for performing the movements accurately. Such realities make the enactment insufficient for providing, for instance, tactile and haptic feedback on the devices with which the user directly interacts.

The choreographic approach conducted with only one person enacting the choreography also overlooks the social aspects of the situations with technology where interaction between different users plays a role. This study was nevertheless an experiment with the goal of developing and understanding what would happen if technology-design critique was based on experiences occurring in continuous movement of a single user in an environment with networked devices, such as visual displays with touch screen interfaces. The study concentrated on the experience of movement and it was realized in a way that explicitly moved technology out of the way of the moving body.

Finally, this research raises concerns regarding whether data can be gathered and analyzed from only single levels of the choreographic approach. It became apparent in our analysis that local-level choreography, that is, collaborative use, also is needed in when investigating the productivity of an activity. For instance, when people use data across devices and work with it collaboratively, the social choreography is heavily influenced by the way in which the users are allowed to make changes to the data on the fly. In addition, collaborative choreographies are closely connected to the perceptual qualities of technology. Further research is needed on the development of a multiple-dancer choreography for learning about the multiactor movement characteristics for investigating the collaborative kinesthetic experience in interactive technology environments. This research approach also would address how it feels to perform particular kinds of movements while others are observing.

DISCUSSION

When performing a well-rehearsed choreography, in this case, based on movement extracted from a fictional representation of human–computer interaction, the enacting researcher is not responding to any new information that might appear on the imaginary screens. Hence, the


26

technique is inward-looking in that it focuses on the embodied experience of using one’s body to control and attend to multiple technology channels in imaginary, ubiquitous environments rather than on the human–computer interaction happening around the body. Thus, the discovery of new affordances, that is to say, new ways of engaging with technology, is not based on feedback from external-to-body devices but rather from the performing and experiencing body.

The choreographic approach directs attention toward personal engagement in the activities of reorganizing, performing, and writing about the choreography, thus fostering the development of a detailed understanding of the kinesthetic experience of the interaction choreographies. As a result, the approach promotes the development of researcher sensitivity, especially to the kinesthetic experience of a movement continuum in which the arrangement of trajectories between visual–tactile foci influences the body posture, alignment, and reach of movement.

Considering the differences between the represented architectonic space in the video representation and the real physical surroundings where the choreography is performed, it may be surprising that the choreographies also facilitate the development of understanding about the interplay between the user and physical contexts of an interaction. The process of enactment had to be carried out in a way that it constructs the context and thus helps the enacting researcher to perform the movements while taking into account the missing elements, such as the devices and characteristics of the physical space in the original use context. The combination of watching the original human–computer interactions and dancing the choreography made it possible to assimilate the experiential and the contextual into an embodied understanding. The approach emphasizes the importance of understanding the embodied mind for the quality of interaction design, and it seems especially well suited for the study of multichannel experiences. As a result, investigating complex interactions through a choreographic approach is neither constrained nor enabled by knowledge or experience of particular technologies. However, the process is fully driven by the development of the choreography that makes visible the demands and delights that the studied design context sets for the experiencing body.

Contemporary visions of intelligent environments are dominated by visual inputs. What if we researchers and designers started to think about movement as an input to interaction design or even as a major factor for the user experience? The development and study of interaction choreographies provides an alternative way to consider users’ preferences in choosing a channel, thereby opening up opportunities to discover how intelligent environments could be improved to provide alternative choices for a diversity of users. In the context of software design, developers typically talk in terms of workflow. This is conceptualized as sequences of tasks leading up to an attained goal. The process of embodying choreographic principles to tap into the body’s knowledge requires time, effort, and creative approaches. However, such investments may be beneficial by increasing opportunities for novel design that addresses better the kinesthetic experience.

The studied videos described people interacting with envisioned future technologies. The research data restricted the choreographic analysis solely to the kinds of interactions that were depicted in the original videos. However, we see that the choreographic analysis can also be used methodically without such detailed examples. For instance, researchers could start from a particular technical possibility, and then pursue investigating the mutual interplay between device networking, technological forms, adaptable spaces, and/or the user’s capacity to move application controls and data across present visual displays. This would allow a greater role and flexibility for embodied improvisation through which novel interaction qualities can emerge spontaneously.


27

The number of ad hoc networks created by various kinds of devices is rapidly increasing. This creates abundant possibilities for combining and recombining microchoreographic continua based on the concept of adaptable technology. It seems a new era has dawned, one in which users have a multitude of possible channels to “resource” (see Ylirisku, Buur, & Revsbæk, 2016a, 2016b). This change may also provide new opportunities for users to choose, develop, and implement preferred personal interaction choreographies, in other words, to engage in an idiosyncratic process of body-making and meaning-making (see Sklar, 2000, p. 74).

The technological contexts presented in the Productivity Future Vision (2009, 2011) videos cannot be accessed; they exist only as visual stories. The depicted technologies and interactions could, however, be projected into a physical setting where an enacting researcher could adapt the choreographic approach with spatially imagined displays, surfaces, and devices. Because the enactment of a choreography as research poses minimal requirements for the physical and technological environments, the ability to learn about the micromovements may prove to be a useful and versatile skill for interaction designers. In line with interaction design patterns in graphical user interfaces, it is feasible to expect that choreographic design patterns for intelligent environments may become commonplace in the future.

CONCLUSIONS

This paper described a choreographic approach to interaction design and introduced a procedure to extract micromovement continua, reorganize movement material into a practicable choreography, conduct first-person enactment of that choreography, and document the analysis. The procedure built on the choreographic approach to interaction design (Parviainen et al., 2013) and extended it by providing an example of how micromovement analysis can be conducted in practice. The choreographic analysis was strengthened with the hermeneutics of the body approach (Hoppu, 2005). The extended method allows operationalizing first-person enactment and performing movement continua related to the human–computer interactions under scrutiny.

The extended choreographic approach for interaction design method starts with documenting movement continua present in human–computer interaction. The material under investigation also can be selected from established technology visions, as was done in the experiment (Poutanen, 2015) analyzed in this study. The movement continua can be video recorded in real use context, or one can design animated situations or document-improvised interactions in a mock-up setting. The key criterion in drawing on video sources is that video depiction presents all individual movements that are constitutive to the intended human–computer interaction. Based on the video representations of specific human–computer interactions, the movement continua are then reorganized to form a coherent choreography that can be rehearsed and enacted efficiently. The reorganization can be carried out with the help of a movement sequence table that is based on screen shots from the video. Screen shots should present understandably and uninterrupted the continua of the represented user’s limbs and changes in body alignment and posture and differentiate between the micromovements constitutive to interaction. The resulting choreography and the use context are then realized through rehearsing the choreography, keeping the use context in mind.

The use context should somehow be present in the movement sequence table so that the images captured from the video representation of the target human–computer interaction


28

serve both rehearsing and embodying information about the use context. Thorough analysis is possible when the choreography is being performed with the use context in mind. Ideally, performing through dance-like movement helps the researcher to organize ideas related both to the experienced movement and to the imagined, technologically augmented space. Reporting the kinesthetic experience related to micromovement continua can be done through writing a dance diary. Expressions of the experiences become more explicit in the course of hermeneutic process that, in this experiment, included rehearsing movement sequences and use contexts with the image movement sequence table and looking at the video excerpts from the original interaction data. Results can be reported as textual descriptions and/or video documentation of the practices and procedures of enacting and commenting on the experience related to the studied movement continua.

The evaluations of intelligent environments have typically concerned safety, human well-being, and the attainment of predefined goals. Choreographic analysis through the hermeneutics of the body approach sheds light on how multiple networked artifacts in ubiquitous computing settings influence the movements and embodied experiences of the user. The choreographic and hermeneutic analyses in this study were based on research data that consisted of futuristic technological visions of interactions within user-centered, intelligent environments. The visions were crafted thoughtfully. Nevertheless, the envisioned interactions included interactions that did not fit well within the movement continuum that they were designed to be part of. Although videos can offer an illusion of smoothness, naturalness, and embodied qualities of interaction, the moving body provides the ultimate reference for human experience of movement.

The extended choreographic approach to interaction design shares many similar features with the moving-and-making-strange approach developed by Loke and Robertson (2013). Both may contain the elements of investigating and choreographing the movement, and they also may include inventing and choreographing movement. Also, visual analysis and representation of movements are applied in both approaches. The major methodological premise for both approaches is the reenactment of movement. This characteristic connects the two approaches methodically and makes them mutually constructive. In future work, this connection should be analyzed more deeply. The choreographic approach to interaction design could benefit from the adoption of an approach similar to the three perspectives—the mover, the observer, and the machine—introduced in the moving-and-making-strange approach. This would enable the choreographic approach to mature as a tool for the design of intelligent environments.

This study focused on a microlevel analysis (Parviainen et al., 2013) that proved useful for depicting the felt qualities in sequences of micromovement interactions. A description and discussion of local-level and macrolevel analyses were not in the scope of this study. However, ideally, microlevel, local-level, and macrolevel choreographic analyses would be applied together. Combined analysis of these interrelated analysis levels remains to be addressed in future studies.

The introduction of new contextual services to be consumed and experienced in increasingly intelligent urban spaces, transportation services, educational facilities, and homes will impose requirements on the methods and methodologies that guide the development of everyday environments in a user-centered, movement-oriented manner. A choreographic approach to interaction design that is extended with the hermeneutics of the body method has the potential to mature into a useful tool, mindset, and method for interaction designers who design intelligent environments.


29

IMPLICATIONS FOR THEORY AND APPLICATION

This study explored what would happen if technology-design critique was based on experiences that explicitly moved technology out of the way of the moving body. While the study is an initial exploration, it emphasizes the value of taking the kinesthetic experience (comprising the felt dimensions of movement and body and the memories and expectations carried by the body) as the starting point for developing and evaluating interactions with technology. The approach is gaining relevance in the context of design of augmented and virtual reality environments, which provide designers with open-ended possibilities for creating new kinds of interaction choreographies. It is also useful for the consideration of how people may combine and recombine microchoreographic continua with available adaptable and networked technologies in intelligent environments. Theoretically the work opens up for the development and study of new kinds of design patterns and patterns of choreographies aimed for kinesthetic appeal.

REFERENCES Carroll, J. M. (2000). Making use: Scenario-based design of human–computer interactions. Cambridge, MA,

USA: MIT Press. Damasio, A. (1994). Descartes’ error: Emotion, reason, and the human brain. New York, NY, USA: Putnam. de Waal, F. B. (2008). Putting the altruism back into altruism: The evolution of empathy. Annual Review of

Psychology, 59, 279–300. doi: 10.1146/annurev.psych.59.103006.093625 Dourish, P. (2001). Where the action is: The foundations of embodied interaction. Cambridge, MA, USA: MIT Press. Foster, S. (2011). Choreographing empathy: Kinesthesia in performance. Abingdon, England: Routledge. Gadamer, H.-G. (1975). Truth and method (G. Barden & J. Gumming, Trans.). London, England: Sheed and

Ward. (Original work published 1960) Gourlay, S. (2006). Conceptualizing knowledge creation: A critique of Nonaka’s theory. Journal of

Management Studies, 43(7), 1415–1436. doi: 10.1111/j.1467-6486.2006.00637.x Gubbi, J., Buyya, R., Marusic, S., & Palaniswami, M. (2013). Internet of things (IoT): A vision, architectural

elements, and future directions. Future Generation Computer Systems, 29(7), 1645–1660. doi: 10.1016/j.future.2013.01.010

Hanna, J. L. (2015). Dancing to learn: The brain’s cognition, emotion, and movement. Lanham, WA, USA: Rowman & Littlefield.

Heidegger, M. (1975). Early Greek thinking (D. F. Krell & F. A. Capuzzi, Trans.). New York, NY, USA: Harper & Row. (Compilation of selected works published in 1940s and 1950s)

Hoppu, P. (1999). Symbolien ja sanattomuuden tanssi. Menuetti Suomessa 1700-luvulta nykyaikaan [The dance of symbols and wordlessness: Minuet in Finland from the 18th century to the present]. Helsinki, Finland: Suomalaisen Kirjallisuuden Seura.

Hoppu, P. (2005). Hermeneutics of the body: Connecting research and dancing in dance research. In I. Björnsdóttir (Ed.), Dance heritage: Crossing academia and physicality (pp. 106–110). Reykjavik, Iceland: Nordic Forum for Dance Research.

Iqbal, R., Sturm, J., Kulyk, O., Wang, J., & Terken, J. (2005). User-centred design and evaluation of ubiquitous services. In Proceedings of the 23rd Annual International Conference on Design of Communication: Documenting and designing for pervasive information (pp. 138–45). New York, NY, USA: ACM. doi: 10.1145/1085313.1085346


30

Johanson, B., Fox, A., & Winograd, T. (2002). The interactive workspaces project: Experiences with ubiquitous computing rooms. IEEE Pervasive Computing, 1(2), 67–74. doi: 10.1109/MPRV.2002.1012339

Kaasinen, E., Kymäläinen, T., Niemelä, M., Olsson, T., Kanerva, M., & Ikonen, V. (2013). A user-centric view of intelligent environments: User expectations, user experience and user role in building intelligent environments. Computers, 2(1), 1–33. doi: 10.3390/computers2010001

Laban, R. (1963). Modern educational dance (2nd ed.). London, England: Macdonald & Evans. Lakoff, G., & Johnson, M. (1999). Philosophy in the flesh: The embodied mind and its challenge to Western

thought. New York, NY, USA: Basic Books. Loke, L., & Robertson, T. (2013). Moving and making strange: An embodied approach to movement-based

interaction design. ACM Transactions on Computer–Human Interaction, 20(1), 1–25. doi: 10.1145/2442106.2442113

Mason, P. (2009). Brain, dance and culture: Broad hypotheses of an intuitive science of dance. Brolga: An Australian Journal About Dance, 3, 27–34.

Merleau-Ponty, M. (1965). Phenomenology of perception (C. Smith, Trans.). London, UK: Routledge & Kegan Paul. (Original work published 1945)

Microsoft. (2009). Productivity Future Vision. Retrieved April 23, 2017, from https://www.youtube.com/watch?v=t5X2PxtvMsU

Microsoft. (2011). Productivity Future Vision. Retrieved April 23, 2017, from https://www.youtube.com/watch?v=a6cNdhOKwi0

Nielsen, J. (1993). Usability engineering. Boston, MA, USA: Academic Press. Nietzsche, F. (2010). Thus spoke Zarathustra (T. Common, Trans.; B. Chapko, Ed.). Paris, France: Feedbooks.

(Original work published 1883) Nonaka, I., & Takeuchi, T. (1995). The knowledge-creating company. How Japanese companies create the

dynamics of innovation. New York, NY, USA: Oxford University Press. Parviainen, J. (1998). Bodies moving and moved: A phenomenological analysis of the dancing subject and the

cognitive and ethical values of dance art. Tampere, Finland: Tampere University Press. Parviainen, J. (2003). Kinaesthetic empathy. Dialogue and universalism, 1(11–12), 154–165. Parviainen, J., Tuuri, K., & Pirhonen, A. (2013). Drifting down the technologization of life: Could choreography-

based interaction design support us in engaging with the world and our embodied living? Challenges, 1(1), 103–115. doi: 10.3390/challe4010103

Pinelle, D., Gutwin, C., & Greenberg, S. (2003). Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration. ACM Transactions on Computer–Human Interaction, 10(4), 281–311. doi: 10.1145/966930.966932

Polaine, A., Løvlie, L., & Reason, B. (2013). Service design: From insight to implementation. Brooklyn, NY, USA: Rosenfeld Media, LLC.

Polanyi, M., (1966). The tacit dimension. Garden City, NY, USA: Doubleday. Poutanen, O. (2015). Embodied interaction choreographies: Kinesthetic approach to intelligent environment

design. Master thesis, Aalto University, School of Art, Design and Architecture. Available at https://aaltodoc.aalto.fi/handle/123456789/19664

Reynolds, D., & Reason, M. (2012). Conclusion. In D. Reynolds & M. Reason (Eds.), Kinesthetic empathy in creative and cultural practices [Kindle Version 1.12.4]. Bristol, England: Intellect.

Rogers Y. (2006). Moving on from Weiser’s vision of calm computing: Engaging UbiComp experiences. Ubiquitous computing, Lecture Notes in Computer Science, 4206, 404–426. doi: 10.1007/11853565_24

Savioja, P. (2014). Evaluating systems usability in complex work: Development of a systemic usability concept to benefit control room design. Espoo, Finland: Valtion teknillinen tutkimuskeskus.

Sheets-Johnstone, M. (1999). The primacy of movement. Amsterdam, the Netherlands: John Benjamins Publishing Company.


31

Sklar, D. (2000). Reprise: On dance ethnography. Dance Research Journal 32(1), 70–77. doi:10.2307/1478278 Stein E. (1989). On the problem of empathy (W. Stein, Trans.). Washington, DC, USA: ICS Publication.

(Original work published 1917) Swan, M. (2012). Sensor mania! The Internet of things, wearable computing, objective metrics, and the

quantified self 2.0. Journal of Sensor and Actuation Networks, 1(3), 217–253. doi: 10.3390/jsan1030217 Thomas, H. (2003). The body, dance and cultural theory. Basingstoke, England: Palgrave Macmillan. Thomas, H. (2013). The body and everyday life. London, England: Routledge. Thompson, E. (2007). Mind in life: Biology, phenomenology, and the science of mind. Cambridge, England:

Belknap Press. Thrift, N. (2008). Non-representational theory: Space, politics, affect. New York, NY, USA: Routledge. Varela, F. J. (1999). The specious present: A neurophenomenology of time consciousness. In J. Petitot, F. J.

Varela, B. Pachoud, & J.-M. Roy (Eds.), Naturalizing phenomenology: Issues in contemporary phenomenology and cognitive science (pp. 266–329). Stanford, CA, USA: Stanford: University Press.

Ylirisku, S., Buur, J., & Revsbæk, L. (2016a, June). Resourcing in co-design. Paper presented at the Design+Research+Society: Future-Focused Thinking conference, Brighton, UK.

Ylirisku, S., Buur, J., & Revsbæk, L. (2016b, November). Resourcing of experience in co-design. Paper presented at the 11th Design Thinking Research Symposium (DTRS’11), Copenhagen, Denmark.

Authors’ Note We express our gratitude to Energizing Urban Ecosystems Programme and WSP in Finland for funding the research. We also thank professors Aija Staffans and Ramia Mazé for their useful comments and support for finalizing the manuscript, and Maria Sannemann, Glen Forde, Heidi Konttinen, Toni Tolin and Max Mäkinen for their kind support in choreography documentation process. All correspondence should be addressed to Olli Poutanen Aalto University Hämeentie 135 C 00076 Helsinki, Finland [email protected] Human Technology ISSN 1795-6889 www.humantechnology.jyu.fi

ISSN: 1795-6889


32

BODY, SPACE, AND EMOTION: A PERCEPTUAL STUDY

Abstract: The present study aims at providing a systematic account of emotion perception applied to expressive full-body movement. Within the framework of the lens model, we identified the decoding process underlying one’s capacity to categorize emotions while watching others’ behaviors. We considered the application of Laban movement analysis, a method focusing on qualitative aspects of movement. An original

© 2017 Donald Glowinski, Sélim Yahia Coll, Naëm Baron, Maëva Sanchez,

Simon Schaerlaeken, & Didier Grandjean, and the Open Science Centre, University of Jyväskylä

DOI: http://dx.doi.org/10.17011/ht/urn.201705272517


Donald Glowinski Neuroscience of Emotion and Affective

Dynamics (NEAD Lab), Faculty of Psychology and Educational Sciences,

and the Swiss Center for Affective Sciences (SCAS), University of Geneva

and

Swiss Integrative Center for Human Health (SICHH), Fribourg

Switzerland

Didier Grandjean Neuroscience of Emotion and Affective



Switzerland

Simon Schaerlaeken Neuroscience of Emotion and Affective



Switzerland

Maëva Sanchez Neuroscience of Emotion and Affective


University of Geneva Switzerland

Naëm Baron Neuroscience of Emotion and Affective


University of Geneva Switzerland

Sélim Yahia Coll Neuroscience of Emotion and Affective



Switzerland

Body, Space, and Emotion: A Perceptual Study

33

experimental setup used a contemporary choreography interpreted by four expert dancers in an environment that restricted their movement to their peripersonal space. Each performance consisted of a subtle or intense emotional interpretation of the choreography (e.g., happiness, anger, surprise, fear, and sadness). Results showed that emotions being expressed in this confined environment could still be identified, categorized, and associated with a profile of movement qualities and specific body parts. Keywords: nonverbal expressive movement, emotion, peripersonal space, lens model, Laban movement analysis, dance, choreography.

INTRODUCTION: BODY, SPACE AND EMOTIONS The body is a tool of unparalleled power for expressing emotion beyond the use of verbal utterances, and this issue has been the focus of numerous research studies (e.g., de Gelder, 2013; Glowinski et al., 2011; Wallbott, 1998). However, the relationship between space and the body’s emotion expression powers has been far less studied. This is due in part to prior difficulties in exploring and recording human interaction or body expression with a high degree of precision and flexibility. Most of the performances recorded for scientific experiments take place within a laboratory setting to allow absolute control over lighting conditions that otherwise might affect the robustness of the tracking. In fact, far fewer studies engage the natural setting of the performance (e.g., open-air performance and theatre). The role of space, and specifically the impact of a small space, on the progression of movement, particularly on the full-body expression of an individual’s emotion, remains to be clarified. A better insight into such relationships is valuable for a wide range of domains, including psychological research on emotional and aesthetical expression or the field of human–computer interaction (Pantic, Pentland, Nijholt, & Huang, 2007).

In a context that combines psychological, aesthetic, and human–computer interaction research, studies focusing on emotion and space often refer to boundary conditions, extreme situations (e.g., flight space), or phobias (e.g., claustrophobia; see Palinkas, 2001). To address the space–emotion relationship, we present a case study of a dance choreography that was performed by professional dancers conveying various expressive emotional intents. This experimental method drew upon previous work by Camurri, Lagerlöf, and Volpe (2003). What characterizes these performances, and distinguishes the novelty of our contribution, is that the space made available to each dancer corresponded to their peripersonal space. This peripersonal space has been approximated by the kinesphere in dance theory as the environmental sphere surrounding the body whose periphery can be easily reached by extending a limb (Laban & Lawrence, 1947; Sutil, 2013). We are interested in understanding how external observers can discriminate the expressed emotions based on the expressive behavior of the dancer within the boundaries of his/her peripersonal space. Our contribution integrated the Laban movement analysis (LMA) categories, used to describe qualitative movement, to fit the decoding process underlying such observers’ emotional categorizations. We also were interested in evaluating other factors’ impact on the emotion recognition process: (a) the observers’ expertise (e.g., does being a trained dancer augment sensitivity to emotion expression?), (b) the performance expressive intensity (e.g., are emotions better recognized when expressed in an emphatic manner?), and (c) the potential role of body parts

Glowinski, Coll, Baron, Sanchez, Schaerlaeken, & Grandjean

34

in emotion recognition (e.g., are there specific relationships between expressed emotions and body parts?). This paper is organized in the following manner. We first review the existing approaches in the literature that attempt to recognize emotions through body postures and movements with a specific focus on dance as a test case. Next, we present the developed experimental framework, which includes the steps dedicated to the recording and presentation of the stimuli and details related to the statistical analysis methods applied to the participants’ answers. This is followed by the presentation of our results: how emotions are recognized and how they can be described in terms of the LMA categories. In the Discussion section, we examine the impact of the participants’ expertise, the intensity of the stimuli, and the relationship between specific emotions and identified body parts. Finally, we conclude this paper by addressing future research directions. Body as a Source of Emotional Expressivity

The body is a key source of information for emotion recognition. An increasing trend in research relates to facial and vocal expression, gesture, and dynamic body motion recognition (e.g., Glowinski et al., 2011). The recent development of low-cost digital image recording equipment, together with the advent of professional motion-capture technologies, has enabled a close analysis of nonverbal modalities in human communication of emotions and, in particular, bodily behavior (Wallbott, 1998). Recently, affective computing, along with the wide range of related application areas, has led the way to meet an increasing demand for the creation of natural, intelligent, adaptive, and personalized multimodal environments (Vinciarelli et al., 2012).

Until the turn of the 21st century, various coding systems were proposed by psychologists. The main focus has been on emotional facial expression due in part to the pioneering work of Ekman, who offered a systematic account for facilitating explicit coding and categorization (FACS; Ekman & Friesen, 1978). A realm of new standards is emerging in this domain, opening opportunities for commercial applications of automatic emotion recognition. Furthermore, alternative approaches have been developed more in recent decades. Research results in psychology suggest, in particular, that body movements do constitute a significant source of affective information (Wallbott, 1998). For example, body gesture, as a complement to facial expressions, can help disambiguate emotional information (de Gelder, 2006; Todorov, Baron, & Oosterhof, 2008). Yet, the vast number and combination of body postures and gestures offers a higher degree of freedom for expressions that can be difficult to easily manage during analysis. No standard body-movement coding scheme equivalent to the FACS for facial expressions exists to facilitate decomposing bodily expression into elementary components. Various systems have been suggested by psychologists (e.g., Bobick, 1997; Dael, Goudbeek, & Scherer, 2013) but none has reached the consensus achieved by the Ekman’s system (Ekman & Friesen, 1978) for facial expression analysis.

Alternative approaches have been developed to fill this need. For instance, research on upper body expressivity took advantage of the clear conceptualization of sign language (e.g., Gunes & Piccardi, 2009). The world’s many sign languages, now being extensively documented, have become a resource for emotion recognition. In sign language, signs made with the hands work in complex coordination with signs made with the face, head movements, torso shifts, gaze, gestures, and mimetic moves. As upper-body movements also correspond to what can be


35

captured easily through a Webcam in the typical working environment (i.e., where people sit in front of their desktop computer), the recognition of such movements has stimulated a realm of applications in this domain. Yet, with the advent of wearable computing, that is, devices worn on the body giving the potential for digital interaction, particular attention is now being devoted to specific limb expressivity (e.g., arms, fingers; see Velloso, Bulling, & Gellersen, 2013). In this context, the lack of systematic coding has been successfully compensated for through a promising alternative approach developed by Caramiaux (2014). Caramiaux, Montecchio, Tanaka and Bevilacqua (2014) investigated and demonstrated how the variability of the body behavior itself can stand as a central cue for capturing expressivity. However, other than the seminal work by Pollick, Paterson, Bruderlin, and Sanford (2001) on motion of knocking, very few experiments have investigated emotional communication through specific limb variations. The key issue in recognition of affective bodily expression is to consider the level of information that determines the quality of movement. The view the body as a whole represents a complete source of affective information is now receiving increased attention in the scientific literature. The recent interest into full-body emotion expressivity can be related to the diffusion of advanced motion capture systems (e.g., Vicon) and especially to the larger dissemination of RGB-depth cameras (e.g., Kinect) that allow for an affordable and relatively fine-tuned tracking of an individual’s body movement.

Existing studies on full-body movement have used coarse-grained posture features (e.g., leaning forward or slumping back) or low-level physical features of movements (i.e., kinematics, see, for example, Bianchi-Berthouze, Cairns, Cox, Jennett, & Kim, 2006). Other approaches have exploited the dynamics of gestures, referring to psychological studies reporting that temporal dynamics play an important role in interpreting emotional displays (e.g., Kapur, Kapur, Virji-Babul, Tzanetakis, & Driessen, 2005).

Towards a Unified Approach

One may note the disparity between the different approaches and, as pointed out earlier, the lack of broader and systematic view to address emotion recognition based on full-body expressivity. A few attempts have drawn inspiration from dance notation and theories (Camurri et al., 2003; Laban & Ullmann, 1971). The key issue is to consider the level of information that determines the quality of movement, that is to say the general characteristics about the way a movement is performed (e.g., the effort dimensions of LMA described below). This level of information may lie between the low-level features (e.g., position of and derivatives in a limb’s trajectory that report a mere displacement and inform about a specific gesture or motion activity) and a high level of information related to the emotional categories that people may use to infer emotional attitudes through observation (Glowinski et al., 2011). In this context, Laban’s (1947) conceptualization has proved useful in modeling what could be an intermediate level of information representing qualitative properties of movement where expressivity is embedded and conveyed. Emotion recognition may ultimately rely on the specific combination of these qualitative properties of movement (Glowinski et al., 2011). Since the pioneering work led by Zhang et al. (2006), an increasing number of computational implementations have been suggested but much more needs to be done on the perceptual side. Thus, our study aims to tackle this aspect by focusing on the perception of emotions


36

expressed through full-body movement in a restricted environment (i.e., the performer’s peripersonal space).

Our specific goal is to provide a systematic account of the decoding process underlying such bodily emotion recognition and observe the respective impact of the observer’s expertise, the intensity of the performance expressivity, and role of body parts in this process. In particular, we are interested in understanding whether an intermediate level of emotional information discernment related to the perception of movement qualities based on the LMA conceptual framework (Laban & Ullmann, 1971) could help in tying the low and high levels of emotional information. We expect this study will shed light on to which level of information may be decisive in researchers’ understanding of the perceptual processes underlying emotion classification.

The research approach to the expression of emotion employed in this study relies on the lens model initially developed by Brunswik (1956), adapted to the performing arts by Juslin and Laukka (2003), and recently reviewed by Glowinski et al. (2014; see Figure 1). According to this model, the analysis of emotion expression must take into account both the sender’s (e.g., the performer) and the receiver’s (e.g., the observer) perspectives. Two types of processes are thus considered: emotion expression/communication through body behavior (e.g., the proximal cues exhibited by a performer) and emotion recognition (e.g., the observer inferences based on the behavioral cues of that performer).

Figure 1. An illustration of the revisited lens model (Brunswik, 1956) integrating the effort

dimensions (i.e., factors and elements) of Laban movement analysis (Laban & Lawrence, 1947). This conceptual framework includes both the point of view of the sender (e.g., the dancer) and of the receiver (e.g., spectator). The respective encoding and decoding processes of emotion expression through body

components can be analyzed in a systematic way. In particular, the attribution of a specific emotion category during the recognition phase of the movement can be related to the way a spectator combines

bodily, effort-based, features of the dancer’s performance.


37

Perceptual Evaluation in Terms of Movement Qualities

An analysis in terms of movement qualities helps identify the differences between the perspective of the person expressing the movement (i.e., the sender) and that of the person observing the same movement (i.e., the receiver). According to Laban (as cited by Sutil, 2013), movement can be experienced intuitively as a continuum, an indivisible flux of changes. Alternatively, movement can be rationalized as a series of snapshots that can be ordered, structured, and formalized as the building blocks of a representation (Glowinski, Camurri, Chiorri, Mazzarino, & Volpe, 2007). This conceptual framework facilitates investigating and understanding the implicit or explicit strategies an observer applies in manipulating the collected snapshots of movement in building up a complete, sensitive, and yet subjective representation of the (performer’s) sequence. Supported by a profound dualistic approach, Laban’s key movement concepts come in pairs of opposites (Laban & Lawrence, 1947). The theory of efforts developed by Laban aims at characterizing such dynamism in relation to four basic properties (effort factors): weight, space, time, and flow (Laban & Lawrence, 1947). Each of these factors is in turn divided into opposed subcategories known as effort elements (heavy–light, direct–indirect, quick–slow, and free–bound). These effort elements allow researchers to understand the fundamental qualitative differences in human movement. As stated by Sutil (2013, p. 5), “The difference between punching someone in anger and reaching for a glass is slight in terms of body organization—both rely on extension of the arm and the same spatial direction of the movement. The weight of the movement and the intensity of the movement are very different, though.”

In this study, the Laban elements (Laban & Lawrence, 1947) are operationalized via measurable descriptions, described as follows:

1. The weight element considers the individual's movements in relationship to gravity and may describe its vigorousness. The two subcategories associated are heavy (i.e., powerful) and light (i.e., delicate).

2. The space element here considers the individual’s movements related to his/her peripersonal space. The two subcategories associated are direct and indirect. Indirect motion is interrupted and roundabout, and direct motion proceeds along a mostly straight line without deviation.

3. The time element is a measure of movement activity speed. The two associated subcategories are quick (i.e., sudden and urgent) and slow (i.e., sustained, continuous, and time stretching).

4. The flow element characterizes the continuity of the movement. The two associated subcategories with this element are free (i.e., a fluid and released movement) and bound (i.e., a controlled or contained movement).

Emotion Categories, Peripersonal Space, and Laban’s Effort Dimensions

Based on the qualitative approach of movement expressivity, recent computational studies have investigated how emotion could be rendered by integrating the effort dimensions (i.e., factors and elements) of Laban movement analysis (Laban & Lawrence, 1947). Initially, the objective in this domain was not to access perceptual processes only but also to create complementary strategies to organize and classify large databases on motion that included more subtle aspects


38

of expressions relating to emotion. A first attempt by Wakayama, Okajima, Takano, and Okada (2010), then Okajima, Wakayama, and Okada (2012), showed that motion retrieval can benefit from the use of a subset of LMA dimensions, especially when searching for data (indications, instances) on body movement in large research databases (Kapadia, Chiang, Thomas, Badler, & Kider, 2013). Recently, Aristidou, Charalambous, and Chrysanthou (2015) inspected the similarities among various emotional states classified according to the arousal and valence of Russell’s (1980) circumplex model and a subset of features that encode stylistic characteristics of motion based on the LMA. Overall, previous experimental results, based on video processing or body limbs’ trajectory dynamics, show that these features can be extracted using the LMA dimensions and thus allow researchers to encode body posture differences depicting emotion expression. The pertinence of Laban’s (Laban & Ullman, 1971) dimensions as descriptors for motion expressivity can be attested further by their use in avatar animation. Since the seminal work of Chi, Costa, Zhao, and Badler (2000), various studies have demonstrated that LMA-derived dimensions can be exploited efficiently in motion parameterization and expression (Zhao & Badler, 2005).

Unfortunately, few studies have considered the application of Laban-based analysis in understanding perceptual processes during emotion recognition. Levy and Duke (2003) used LMA dimensions to score the capacity of nonprofessional dancers to improvise sequences of expressive movement. Correlation analyses in emotion recognition revealed relationships among the emotional states of depression and anxiety and certain movement qualities. Focusing on the specific case of walking, Crane and Gross (2013) explicitly instructed participants to use LMA-based analysis to classify the observed nuances of emotions. Sadness, anger, contempt, and joy were decoded with accuracy that ranged from 74% to 32 %, respectively; for most of the targeted emotions, decoding accuracy was related to the four effort factors (Serino, Annella, & Avenanti, 2009).

Peripersonal Space, Kinesphere, and Emotions

Peripersonal space has been defined in contrast to general space in a way similar to how space is defined in geometry or topology (Serino et al., 2009). In dance theory, this peripersonal space has been approximated further by a kinesphere, which refers to the space occupied by an outreaching body without it moving from one spatial location (Laban, 1966). Specifically, it can be represented as an area within reach of the body’s extended limbs which, when projected in all directions from the body’s center, can be conceived as a totality of movement or a sphere of movement (Sutil, 2013). Therefore, the kinesphere gives an operationalized way to analyze peripersonal space (see Figure 2) and how each individual might explore the surrounding environment to express emotion and, in turn, how such peripersonal space may reciprocally impact on emotional expression.

Dance as a Test Case

Dance appears as an ideal test case to study emotion in relation to movement expressivity. It condenses within a minimal amount of space and time the human capacity to express emotional information. Aesthetically, it is possible to create a minimally expressive dance or a dance that


39

Figure 2. Illustration of the peripersonal space as operationalized by the concept of kinesphere

(Laban, 1966). The figure on the right represents the motion capture of the dancer’s movement (on the left) that was used in the perceptual experiment presented in this paper.

unfolds slowly. Typically, dance provides no functional or utilitarian use of the space (i.e., related to a practical task achievement, such as opening a door or typing on a keyboard), but rather to express oneself. In contemporary Western dance art, however, the aesthetics have turned into quite everyday activities; therefore objects and space may be interrelated in very functional ways. Dance as a test case allows the experimenter to observe movement of an undetermined wide range of complexity that can vary according to the level of the dancer’s expertise (e.g., from beginner to accomplished dancer) as well as according to the choreography. Specific to our research interest, dance also can reveal exceptional and evolved ways of exploiting the surrounding space. By focusing on the expression of emotion, through movement in a limited space or constrained environment, we sought to observe how a dancer faces this challenge: Specifically, we were interested in how the dancer could transform a source of what could appear to most people as a stress or difficulty into a positive experience. Therefore, we experimentally defined a limiting condition where the performer was bounded by a choreography that limited the variety of possible movements within a further, restricted environment that constrained the dancer’s displacement within his/her personal space. As a consequence, we expected that this challenging situation could impact the performer’s creative capacity in expressing emotion through the only dance component that remained freely available: body expressivity. We explored this issue by assessing how external observers perceived such emotional body expression.

Laban Movement Analysis and Dance

The LMA is widely used in dance, either to annotate and generate choreography or to train dancers. With the advent of new systems for motion capture, dancers have shown an increased interest in using these new forms of interaction to map in real-time their expressive body movements to audio or visual feedback. The LMA dimensions have resurfaced as a source of inspiration in capturing key expressive variations in dance performance and for improving the “naturalness” during interaction with automatic systems (Mancas, Glowinski, Volpe, Coletta, & Camurri, 2010). Drawing upon Laban’s approach, Camurri et al. (2003),


40

and Van Dyck, Vansteenkiste, Lenoir, Lesaffre, and Leman (2014) developed a qualitative approach to human full-body movement for emotion recognition. In this study, LMA dimensions were approximated through a combination of low-level physical features to allow for a coarse description of an encoding process of emotion through body behaviors (e.g., emotion arousal revealed by acceleration peaks). Based on a more explicit computational modeling of Effort-Shape features, Alaoui, Caramiaux, Serrano, and Bevilacqua (2012) developed an interactive augmented dance performance that extracts movement qualities (energy, kick, jump/drop, verticality/height and stillness) to generate a visual simulation.

However, these previous works overlooked the experiences that can be gained through perceptual studies based on the LMA. For dance examples, it is useful to consider the key conceptual distinctiveness embodied in this art form. Dance can be considered a specific case of stylized body movements in which the entirety of movement encodes a particular emotion. Stylized motions usually originate in laboratory settings, where subjects are asked to freely act on an emotion without any constraints. Another key distinction may refer to the propositional and nonpropositional aspects of movement (Boone & Cunningham, 1998). Raising one’s hand to indicate stop, for example, may be considered a propositional movement that constitutes established signs to transmit shared meaning. As stated in Camurri et al. (2003), emotions can be expressed through propositional movement (e.g. a clenched fist to show anger or raised arms to demonstrate joy), whereas nonpropositional movements are embodied not in discrete, easily segmented motions but rather through a subtle combination of movement qualities (e.g., lightness or heaviness). From the point of view of a perceiver, this distinction could be interpreted as a shift in attentional focus, whether on the configurational aspects (e.g., gesture as an explicit, well-delimited code) or the dynamic itself (i.e., how one expresses emotional intent). In this study, we focused on the nonpropositional style of movements as represented by dance sequences.

In our research, we considered how external observers combine body movement based on the LMA to recognize the emotional intent of contemporary dancers during their performance. Specifically, we focused on assessing the significance of the time, weight, space, and flow factors featured in emotion recognition by external observers.

METHODS Participants

Dancers

Two female (D1 and D2) and two male (D3 and D4) professional contemporary dancers were recruited for this study. Their average age was 28 years, and they were remunerated 100 CHF for their participation. All participants provided signed informed consent prior to study participation. The protocol for this study was approved by the University of Geneva, Switzerland, at the Faculty of Psychology and Educational Sciences. Dancers were asked to wear tight clothes, their hair was tied up, and all jewelry and accessories were removed for the recordings.


41

Observers

Forty-eight observers, recruited through social networks, participated in the experiment (17 males; age M = 27.77 years, SD = 10.77). Among them, 38 were following or had followed dance classes and 10 never practiced dance. We followed the standard of the Geneva education system in dance to categorize our research participants: The experts in our study (n = 22) were dancers with 8 or more years of practice; those with 1 to 7 years of dance experience were considered novices (n = 16). Individuals with a year or less of dance practice were grouped into the nonexpert condition (n = 10). All observers were competent in French, the language used to collect the data.

Stimuli Recording

MoCap (motion capture) optical reflectors (markers) were positioned on dancers’ body to record their movement (see Figure 3). Each dancer had an equal number of reflectors attached to the left and right halves of their body. However, these reflectors were not symmetrically positioned to allow them to be distinguished more easily by the camera system and to facilitate offline postprocessing of data. Eight Bonita 3 Vicon cameras were used to record dancers’ movements. The motion-capture system sampled the data at 120 frames per second. This material allowed us to record in three dimensions the position of the 30 markers placed on the dancers’ body. By using a body model, the state (orientation and position, where applicable) of each body part was estimated.

Katrin Blantar, a professional choreographer, designed specific sequences of stylistic movements that excluded stereotyped postures that could be perceived as expression of a basic emotion (e.g., upward movements expressing anger). This 30-second microdance (see Glowinski

Figure 3. Disposition of the motion capture optical reflectors on the dancers’ bodies.


42

et al., 2007) was performed in a controlled environment of 3 x 3 m that strictly enclosed the dancer’s kinesphere.

This choreography was first learned by the dancers in a formal way without emotional engagement. Then, during the experiment, each dancer explicitly addressed a scenario for each emotion, meaning that they reviewed the proposed definitions (see Table 1) before integrating that into their expression of the choreography. Each dancer performed each emotional expression twice, at two levels of intensity: first in a subtle way (i.e., low-intensity condition) and then with a more emphasized and demonstrative manner (i.e., high-intensity condition). This distinction was based upon a paradigm used in music to distinguish between levels of expressivity (Davidson, 1993). Dancers were asked not to modify the choreography, but no instructions were given concerning how emotions should be expressed through their dancing that choreography. By the completion of the recording process, each dancer performed a set choreography six times: one that contained no emotional expressivity (neutral, danced twice) and versions that expressed the five emotions (i.e., happiness, anger, surprise, fear and sadness) in both low and high intensity. Thus a total of 12 performances per dancer were recorded, resulting in, overall, about 5 hours of recordings.

Stimuli Preparation The Vicon Nexus software program1 was used to reconstruct and label the data. At the end of the processing, the reflectors were linked to each other to simulate the head, chest, pelvis, arms, and legs (see Figure 4). To create the video stimuli (i.e., the various choreographies performed in abstract representations), we displayed the linked markers through the Vicon Nexus view, using Camtasia software2 (see Figure 5).

Table 1. Definitions of the Five Emotions Used as the Basis for Expressive Choreography and Observer Perceptions (based on Banziger & Scherer, 2007).

Emotion Definition

Anger Feel violent discontent caused by an action deemed stupid or malicious.

Happiness Feel transported by a wonderful event happening in a more or less unpredictable way.

Fear Feel threatened by an imminent danger that could affect one’s survival or physical or mental integrity.

Surprise Feel confronted, often abruptly, with an unexpected or unusual event (without a positive or negative connotation).

Sadness Feel depressed and discouraged by the loss of a relative, an object, or a familiar environment.

Note. The definitions were used by the dancers in creating the emotional expressiveness of the choreography versions for the stimuli preparation; the observers then employed the definitions during the emotion recognition task.


43

Figure 4. A silhouette of a dancer at the end of the Vicon Nexus processing, created through connecting

the data collected via optical reflectors on a dancer’s body.

Figure 5. Screenshot examples of movement sequences of the choreography expressing

happiness, as performed by dancer D1.

Questionnaires

The Qualtrics software program3 was used to create the online questionnaires. Four questionnaires, each containing videos of two dancers, a man and a woman, were created. They were written in French and checked by all authors of the study; all English translations provided


44

for this paper were conducted by the authors. The first questionnaire displayed the performances of dancers D1 and D3, the second of D1 and D4, the third of D2 and D3, and the fourth of D2 and D4. Two videos of D1 expressing sadness were lost due to a technical problem. Thus, the first and second questionnaire presented 22 videos, whereas the other two contained 24 (each video of each of the two selected dancer). All videos were presented, in 428 x 1027 cm format. In each questionnaire, the videos were presented randomly. The observers completed only one of the four questionnaires in order to reduce the duration of the experiment.

The questionnaire was administered in a lab. Observers were asked to complete information regarding their age, gender, and diploma or study degree. Questions followed concerning their dance training (i.e., “Have you already followed dance classes?” “Which kind of dance did you practice?” “How many hours per month?” “How many years of practice?”) and contemporary dance familiarity (i.e., “How frequently do you watch contemporary dance performances?”). Before presenting the videos, the observers were asked to carefully read the definitions of the five emotions they would use in the questionnaire to judge the performances (see Table 1). The neutral condition was simply presented as a performance without clear emotional engagement.

Depending on the questionnaire, 22 or 24 videos were then randomly displayed to each participant. For each video, the participants did not identify the perceived emotions directly but rather by following Dael’s method (Dael et al., 2013), that is, rating the intensity they perceived of each emotional expression from 0 (low intensity) to 100 (high intensity). This 100-point scale enabled a fine-tuned evaluation of participants’ responses and to distinguish better how they recognized emotion. Participants were then instructed, via a forced-choice question, to indicate which body part most captured their attention during the performance, choosing between the head, shoulders, arms, pelvis, or legs. Then, four scales were displayed, ranging from 0 to 100, to measure the various elements of the Laban theory of effort (Laban & Lawrence, 1947): weight (0 = heavy, 100 = light), space (0 = direct, 100 = indirect), time (0 = quick, 100 = slow), and flow (0 = free, 100 = bound). The observers were asked to select these elements intuitively and without overthinking. The questionnaire was completed by the participants in 40 minutes, on average. Figure 6 provides an overview of the experimental protocol adopted in this study.

Statistical Analyses

To analyze the data regarding the participants’ emotion recognition in the performances, their evaluations of the Laban dimensions among the emotions, and their responses regarding emotions and their intensity, and the body parts capturing attention, we used the generalized linear mixed models (GLMMs) statistical method. GLMMs combine the properties of linear mixed models, which incorporate random effects, and generalized linear models, which handle nonnormal data by allowing the researchers to specify different distributions, such as Poisson or binomial (Bolker et al., 2009). By using GLMMs, we could also control for the interindividual variability random effect.

To investigate the contribution of each variable and its interactions, we compared different models, from the most simple (i.e., with one unique variable) to the most complex (i.e., all combinations of variables). Statistical differences were evaluated through Chi-square difference tests. Our fixed effects comprised the specific emotions expressed by the dancers in individual


45

Figure 6. An overview of the experimental protocol adopted in this study, comprising (a) the microdance

sequence of emotionally expressive choreography by professional dancers using motion-capture technology to create the stimuli, and observers’ perception regarding (b) the rating of the perceived

emotions, (c) the body part of the dancer’s representation that drew the participant’s attention during the dance, and (d) the questions regarding the performance qualities and expressivity (LMA elements).

videos (i.e., happiness vs. surprise vs. sadness vs. anger vs. fear vs. neutral), the intensity of the dances (low vs. high), and the dance expertise of the observers (nonexpert vs. novice vs. expert). Our random effect consisted of the interindividual differences of ratings between the observers. Our dependent variables were the observers’ perception responses on the emotional scales (happiness, surprise, sadness, anger, fear, and neutral) that we transformed into binomial variables 1 (the emotion with the relative highest rating) or 0 (for all other emotions that scored under this maximum value) for each trial. Four other dependent variables were the Laban factors (flow, space, time and weight), which were continuous variables ranging from 0 to 100. Finally, five binomial variables concerned the body part (head, shoulders, arms, pelvis, or legs) that captured the most participants’ attention during each performance.

RESULTS

Emotion Recognition, Influence of Expertise, and Relations Between Emotion and Laban’s Dimensions

The dancers’ intended emotion showed a significant main effect in all emotional scales, as displayed in Figure 7: happiness, χ2(5, N = 48) = 92.15, p < .001; sadness, χ2(5, N = 48) = 192.81, p < .001; anger, χ2(5, N = 48) = 52.02, p < .001; fear, χ2(5, N = 48) = 82.47, p < .001); surprise, χ2(5, N = 48) = 28.36, p < .001); and neutral, χ2(5, N = 48) = 30.65, p < .001). As shown

flow

Sudden


46

Figure 7. Histogram representing on the horizontal axis the emotions expressed by the dancers and on the vertical axis the emotions rated by the observers (N = 48). Vertical bars denote a 95% confidence interval.

by contrast analysis, observers perceived significantly more happy, sad, angry, and neutral expressions when the dancers expressed happy, sad, angry and neutral expressions, respectively, than the other emotions: happiness, χ2(1, N = 48) = 69.08, p < .001); sadness, χ2(1, N = 48) = 31.81, p < .001); anger, χ2(1, N = 48) = 62.19, p < .001); and neutral, χ2(1, N = 48) = 25.23, p < .001).

Data did not show any difference in emotion recognition based on level of expertise. No significant interaction between emotion and expertise was observed in the emotional scales: (happiness, χ2(10, N = 48) = 4.52, p = .92; sadness, χ2(10, N = 48) = 11.98, p = .29; anger, χ2(10, N = 48) = 4.40, p = .93; fear, χ2(10, N = 48) = 10.91, p = .36; surprise, χ2(10, N = 48) = 16.96, p = .75); and neutral, χ2(10, N = 48) = 5.14, p = .88).

Data showed that the fear emotion was recognized differently according to the level of intensity (low vs high intensity), χ2(4, N = 48) = 13.13, p < .05 (see Figure 8). As shown by a simple effect, an expressed fear was significantly more frequently perceived as fear when it was expressed with low intensity rather than high intensity (χ2(1, N = 48) = 20.09, p < .001). No other significant interaction was found for the other emotions: happiness, χ2(4, N = 48) = 7.36, p = .12; sadness, χ2(4, N = 48) = 3.97, p = .41; anger, χ2(4, N = 48) = 5.69, p = .22); and surprise, χ2(4, N = 48) = 8.77, p = .07.

Emotion showed a significant main effect in all Laban’s factors: time, χ2(5, N = 48) = 220.32, p < .001; weight, χ2(5, N = 48) = 62.50, p < .001; space, χ2(5, N = 48) = 107.93, p < .001; and flow, χ2(5, N = 48) = 51.51, p < .001). These results are displayed in Figure 9.

anger happiness neutral fear surprise sadness

Expressed Emotion

Per

ceiv

ed E

mot

ion


47

Figure 8. Plot of the interaction between the emotion expressed by the dancers and the observers’ perceived

intensity of the dances in the fear scale. Vertical bars denote a 95% confidence interval.

Figure 9. Histogram representing differences in Laban’s criteria evaluation (speed, weight, extension and

flow) between each expressed emotion (anger, happiness, neutral, fear, surprise and sadness; N = 48). Vertical bars denote a 95% confidence interval.

Time Factor

As shown by the comparisons, anger was rated as significantly quicker, representing sudden and urgent movement, χ2(1, N = 48) = 99.65, p < .001, than the other emotions. Moreover, fear and surprise were rated as significantly quicker, χ2(1, N = 48) = 127.29, p < .001, than the happy, neutral, and sad expressions. Neutral and sad expressions were rated significantly

Laban’s factors

flowspaceweighttime

Eva

luat

ion

of L

aban

’s f

acto

rs


48

slower than happiness, χ2(1, N = 48) = 24.92, p < .001. However, no significant difference was observed between neutral and sad expressions, χ2(1, N = 48) = 1.03, p = .31, or between fearful and surprise expressions, χ2(1, N = 48) = 0.86, p = .36.

Weight Factor

Happiness was rated as significantly lighter than the other emotions, χ2(1, N = 48) = 4.56, p < .05. Conversely, sadness was rated as significantly more anchored and heavier than the other emotions, χ2(1, N = 48) = 19.50, p < .001. No significant difference was observed between anger and neutral, χ2(1, N = 48) = 0.32, p = .57, or anger and surprise, χ2(1, N = 48) = 0.59, p = .44) expressions. However, anger was recognized as significantly more anchored than fear, χ2(1, N = 48) = 5.66, p < .05. No significant difference was obtained between the expressions of neutral and fear, χ2(1, N = 48) = 1.99, p = .16, neutral and surprise, χ2(1, N = 48) = 0.002, p = .96, or fear and surprise, χ2(1, N = 48) = 2.71, p = .10.

Space Factor Sadness was rated as significantly more indirect than the other emotions, χ2(1, N = 48) = 70.25, p < .001. Anger and happiness were rated as significantly more direct than neutral, fear, or surprise expressions, χ2(1, N = 48) = 19.99, p < .001. Between these two groups, happiness was rated as significantly more direct than anger, χ2(1, N = 48) = 8.58, p < .01, and neutral was considered significantly more indirect than the fear or surprise expressions, χ2(1, N = 48) = 7.45, p < .01. However, no significant difference was observed between fear and surprise, χ2(1, N = 48) = 0.99, p = .32.

Flow Factor Happiness was rated as significantly more fluid, that is, referring to a free, relaxed, and uncontrolled movement, than the other emotions, χ2(1, N = 48) = 36, p < .001. By contrast, surprise and sadness were the two emotions recognized as the most bounded, χ2(1, N = 48) = 26.22, p < .001. No significant difference was observed between them, χ2(1, N = 48) = 0.74, p = .39. Moreover, anger was rated no differently than the neutral or fear expressions, χ2(1, N = 48) = 0.34, p = .56 and χ2(1, N = 48) = 0.78, p = .38, respectively, which were also not significantly different, χ2(1, N = 48) = 0.03, p = .86. Relationships Among Expressed Emotions and Body Parts Emotion showed a significant main effect on the dependent variables: the head, χ2(5, N = 48) = 71.23, p < .001; arms, χ2(5, N = 48) = 12.23, p < .05; shoulders, χ2(5, N = 48) = 11.23, p < .05; and pelvis, χ2(5, N = 48) = 12.01, p < .05. However, no significant difference was observed in the legs variable, χ2(5, N = 48) = 6.42, p = .27.

For sake of clarity, we developed a visualization procedure to illustrate how each perceived emotion could be systematically related to a specific body area. Based on the analysis of the contrasts, we individuated three levels of gray-scale intensities (light grey, dark grey, and black) to designate the level of explicit attention the observers noted regarding the head,


49

shoulders, arms, pelvis, and legs with respect to the emotion expressed (see Figure 10). Black indicates the highest significant correlation between an emotion and the observed body part; a light gray code indicates the lowest one. The other body parts, for which correlation was neither the highest nor the lowest, were colored in dark gray. We report in the following the results of the contrast analyses. The observers noted the head as the focus during dances involving fear in comparison to neutral condition: χ2(1, N = 48) = 20.21, p < .001); shoulders (sadness in comparison to surprise: χ2(1, N = 48) = 7.37, p < .01); arms (anger in comparison to fear: χ2(1, N = 48) = 9.47, p < .01); pelvis (happiness in comparison to fear: χ2(1, N = 48) = 8.49, p < .01); legs (anger in comparison to sadness: χ2(1, N = 48) = 3.61, p = .06).

Figure 10. Degree of reported observers’ attention to the different body areas depending on emotions

expressed by the dancers. Neutral and Surprise did not appear to have a clear consensus regarding which body part most observers observed. Rather, the observers had responses across multiple body areas.


50

DISCUSSION In agreement with the literature (Camurri et al., 2003; Crane & Gross, 2013; Shikanai, Sawada, & Ishii, 2013), our results show that the emotions of happiness, anger, and sadness are typically identified correctly by the observers. Of the three, sadness is recognized most often, followed by happiness. This partially reproduces the results of the studies by Camurri et al. (2003) and Dittrich, Troscianko, Lea, and Morgan (1996), in which the highest recognition rate was obtained for sadness, followed by anger and joy. For the neutral performance, in which no emotion was expressed, observers also predominantly perceived it correctly. However, when the dancers expressed fear or surprise, these two emotions were not readily distinguished. This perceptual confusion finds support from two prior studies. In the seminal study by Ekman and Friesen (1974) on emotional facial expressions and in the study of a full-body behavior in a daily life scenario by Meijer (1989), expressions of surprise tended to be recognized as fear. In our study, the perceived expressions of fear and surprise shared most of the Laban factors: time, space, and weight. For the Laban elements associated with time, for example, speed changes were seen as particularly fast for these two emotions and for the weight factor, the performance was perceived as moderately anchored to the ground and light and delicate.

Contrary to our expectations based on earlier studies (e.g., Bläsing et al., 2012; Broughton & Davidson, 2014; Cross, Kirsch, Ticini, & Schütz-Bosbach, 2011), the expertise of the observers had little influence on the emotion recognition in this study. Overall, our results suggest that the affective components of body expression are less driven by expertise in dance; we saw broadly consistent responses across observers, in line with van Paasschen, Bacci, & Melcher (2015). However, the use of a point-light display based on MoCap recordings could have diminished the impact of expertise. This type of minimalistic display transmits a sufficient amount of gestural information for emotion recognition when deployed in a restricted environment, and perhaps even those without dance experience were able to recognize the emotional expressions from these images. It is possible that additional physical evidence (e.g., facial expression and skin texture) may have allowed for one’s dance expertise to become more apparent.

These findings also could be explained by the fact that the criterion chosen to classify our group (i.e., novice, expert, and nonexpert) was not sufficient enough to differentiate the levels of expertise. Our criterion for expertise was based on the standard levels of training period (8 years of regular practice) that Swiss dance institutions suggest to their most motivated students who are preparing for a professional career. Among the 22 experts who participated in our study, only four were dancing professionally. In future studies, we plan to include a larger number of experts, specifically professional dancers, to test whether sensorimotor expertise or aesthetic familiarity may impact emotion recognition in the specific context of this study (see also Bläsing et al., 2012).

Results confirmed that emotions expressed in the high intensity condition (i.e., with a higher emphasis and demonstrative manner) during performances are better recognized than in the low emotional intensity, where emotion would be more subtlety expressed (Burger, Saarikallio, Luck, Thompson, & Toiviainen, 2013). Our results show a statistically significant and a marginal difference, respectively, for fear and surprise in that the emotion recognition was higher in the low intensity condition. However, anger, joy, and sadness were perceived in a similar way for


51

both intensities. This result goes against our expectations based on the literature and the results obtained by Hill and Pollick (2000). These authors showed that a particular increase in the expressive intensity of movements produced a better recognition of joy and anger. In their study, however, the differences between the expressive conditions were achieved through offline manipulation of the recorded point-light display used to render the dancers’ performances.

More noteworthy is that fear was recognized by the observers as a more intense conveyed movement in our context of investigating the body gestures and movements. One can argue that, in real-life conditions, the experience of fear might be related to reduced motion in body parts in order to be less detectable by dangerous animals or conspecifics in a threatening situation. Such fear-related body movement might have been integrated as an internal representation of fear by dancers and spectators, explaining our results.

Results showed that Laban’s effort dimensions can inform on an intermediate level of perceptual processing underlying the emotion recognition. They revealed that the movement qualities can be discriminated in a statistically significant way and be significantly associated with emotion portrayals, thus highlighting their expressive pertinence. Considering the time factor, the anger, fear, and surprise emotions were positively related to quicker element ratings in comparison to the sad and neutral conditions. This is in line with the results in the scientific literature (Camurri et al., 2003; Crane & Gross, 2013; Meijer, 1989). As previous research has shown, fear and surprise cannot be distinguished based on the time factor alone. Concerning the weight factor, performances expressing joy turned out to be lighter and delicate whereas those expressing sadness were seen as firmer and anchored to the ground. However, this single criterion may not be sufficient to discriminate between surprise, anger, and fear. For the space factor, the movement underlying the joy and anger emotions was considered as more direct than those of sadness, confirming recent studies on this specific issue (Crane & Gross, 2013; Shikanai et al., 2013). This factor alone, however, did not allow for any distinction between fear and sadness. Finally, the flow factor was positively rated in the happiness emotion, displaying the LMA elements of relaxed, free, and uncontrolled movement, whereas surprise and sadness seemed to include bound, tense, and controlled types of movement.

These results replicate the findings in Camurri et al. (2003). In addition, our results show that the surprise and fear emotions, which prior search has found are typically confused by observers, could be distinguished on the flow factor: The movement related to fear was rated as more uncontrolled than that of surprise (see also Wallbott, 1998). These findings, in totality, are consistent with Crane and Gross’s observations that emotions displayed by dancers affect movement style in distinctive ways that could be described consistently with a specific combination of the effort dimensions (Crane & Gross, 2013).

An original finding of this study concerns the how various body parts attract the attention of observers according to which emotions they perceive expressed by the dancers. It extends and confirms an original approach developed by Nummenmaa, Glerean, Hari, and Hietanen (2014) to observe how emotions may be preferentially related to specific body parts. Our results concerning the upper body parts (i.e., head, shoulders, arms) and the pelvis—and the absence of any significant effect concerning the legs—are confirmed partially by Sawada, Suda, & Ishii (2003) and by empirical observation from early contemporary dance. In this period, expressions and movements were confined essentially to the upper body, particularly the chest. Legs were less used. In the mid-20th century, choreographer Ted Shawn stated,


52

“The torso must become the most sensitive and expressive part of the body” (Shawn, 1963, p. 63). Our results, however, show that the head attracted more observers’ attention when watching dance performances expressing happiness, anger, or no emotion (the neutral condition) than fear, surprise, or sadness. Arms were primarily observed during performances expressing anger; dances expressing sadness attracted less attention to the arms. Concerning the shoulders, the happy, sad, and neutral conditions attracted the participants’ attention to this area of the body, whereas performances expressing surprise did significantly less. Finally, the pelvis was considered the most salient when watching happy and neutral performances and the least when sad dances were presented. Our results provide a further detailed insight on how differentiated emotion recognition is associated with specific body parts.

The use of peripersonal space (Cléry, Guipponi, Wardak, & Ben Hamed, 2015; Serino et al., 2009) as an environmental restriction to constrain dancers’ expressive movement also is a unique component of this study. It allowed us to evaluate whether expressed motion within a limited environment could still be recognized by observers.

Our findings showed that angry, fearful, happy, surprised, and sad emotions can still be well discriminated in constrained movement. In addition, our findings highlight that not only space-related features but also complementary movement qualities (e.g., flow, weight, and time) characterize the decoding process of emotion expression in a systematic manner within this specific context (see also Taffou & Viaud-Delmon, 2014). To better understand how emotion recognition may vary in relation to a space representation, a future experimental protocol could include systematic manipulation of the space rendering during the stimuli presentation. The same point-light displays could be presented in a virtual environment that differs in terms of space extension (e.g., either strictly matching an individual’s peripersonal space or placing a dancer’s rendered movement within a larger neutral space). On the other hand, these results reveal the dancer’s potential for exploiting his/her peripersonal space for emotion expression. According to Cléry et al. (2015), not only is the type of action performed in the representation of peripersonal space key, but also the emotional consequences of the actions can dynamically modify it. A computational analysis of the Laban effort dimensions may help clarify the encoding processes of emotion through expressive body behavior. Of specific interest will be the relationship among the expressed emotions, the exploration of the confined space, and the qualitative features of movement. An experimental manipulation also could consider the dance performance within a wider variety of environments (e.g., restricted to peripersonal space or to a larger space, such as a theater stage). Finally, this study reveals the potential of using dance as test case for investigating emotion expression and peripersonal space representation.

CONCLUSIONS As novel digital environments increase the degree of freedom in movement expression, ongoing research would benefit from a conceptual framework and set of methodological procedures to consider the implicit effects of the space factor on emotion expressivity and perception. Using dance as a test case and a choreographed performance with a variety of emotional variations, we systematically considered the decoding processes underlying a spectator’s identification of the emotion expressed within the performer’s peripersonal space.


53

Within the integrated conceptual apparatus of the lens model and the LMA, we contribute by offering a better insight into the qualitative features of body expression on which external observers may rely. We further reveal commonalities in perceptual strategies developed by dance experts and nonexperts.

This study sets the agenda for future developments in the thriving and quickly evolving field of motion analysis and on nonverbal communication in technologically mediated environments. Future work includes (a) extending and replicating with a greater number of participants the correlations observed so far, and (b) establishing relationships within the encoding process (i.e., those implemented by the dancers through their body behavior). The computational modeling of LMA features may be exploited in this perspective.

IMPLICATIONS FOR THEORY OR APPLICATION Our study is advancing the technological breakthrough in the analysis of full-body expressivity but also acknowledges the embodied turn toward cognition research, confirming the prominent role of emotion and embodiment as essential components of cognitive processes. We used dance as a test case and bound the dancer’s movements to his/her peripersonal space in order to better study the crucial impact of space constraints on emotional body expressivity and observers’ perception of emotions expressed within those constraints. Drawing upon the Laban movement analysis framework seems an efficient approach for recognizing emotions and revealed underlying perceptual process. Used in a Brunswik’s (1956) lens model, we believe those qualities categorized along the Laban’s effort dimensions are intermediate-level key components in emotion recognition. This study could help advance a new generation of digital environments, allowing for natural and emotionally vivid interactions. Equally, this line of research could facilitate better automated recognition of emotion from bodily movement. In addition, it suggests how dance research can influence a wide variety of disciplines also interested in exploring the perception and interpretation of emotional conveyance.

ENDNOTES

1. More information on the Nexus software is available at http://www.vicon.com/products/software/nexus

2. More information on the video capture software used to display the motion capture video https://www.techsmith.com/camtasia.html

3. More information on the software used to develop the online questionnaire at http://www.qualtrics.com

REFERENCES Alaoui, S. F., Caramiaux, B., Serrano, M., & Bevilacqua, F. (2012). Movement qualities as interaction modality.

In Proceedings of the Designing Interactive Systems Conference (pp. 761–769). Newcastle, UK: ACM Press. doi: 10.1145/2317956.2318071


54

Aristidou, A., Charalambous, P., & Chrysanthou, Y. (2015). Emotion analysis and classification: Understanding the performers’ emotions using the LMA entities. Computer Graphics Forum, 34(6), 262–276. doi: 10.1111/cgf.12598

Banziger, T., & Scherer, K. R. (2007). Using actor portrayals to systematically study multimodal emotion expression: The GEMEP corpus. In A.C.R Paiva., R. Prada, & R. W. Picard (Eds.), Affective computing and intelligent interaction (pp. 476–487). Berlin, Germany: Springer. doi: 10.1007/978-3-540-74889-2_42

Bianchi-Berthouze, N., Cairns, P., Cox, A., Jennett, C., & Kim, W. (2006). On posture as a modality for expressing and recognizing emotions. In Proceedings of the Emotion in HCI Workshop (pp. 74–80). London, UK: University College of London Press.

Bläsing, B., Calvo-Merino, B., Cross, E. S., Jola, C., Honisch, J., & Stevens, C. J. (2012). Neurocognitive control in dance perception and performance. Acta Psychologica, 139(2), 300–308. doi: 10.1016/j.actpsy.2011.12.005

Bobick, A. F. (1997). Movement, activity and action: The role of knowledge in the perception of motion. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 352, 1257–1265. doi: 10.1098/rstb.1997.0108

Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J. S. S. (2009). Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology and Evolution, 24(3), 127–135. doi: 10.1016/j.tree.2008.10.008

Boone, R. T., & Cunningham, J. G. (1998). Children’s decoding of emotion in expressive body movement: The development of cue attunement. Developmental Psychology, 34(7), 1007–1016. doi: 10.1037//0012-1649.34.5.1007

Broughton, M., & Davidson, J. W. (2014). Action and familiarity effects on self and other expert musicians’ Laban effort-shape analyses of expressive bodily behaviors in instrumental music performance: A case study approach. Frontiers in Psychology, 5(10), 1201–1206. doi: 10.3389/fpsyg.2014.01201

Brunswik, E. (1956). Perception and the representative design of psychological experiments. Berkeley, CA, USA: University of California Press.

Burger, B., Saarikallio, S., Luck, G., Thompson, M. R., & Toiviainen, P. (2013). Relationships between perceived emotions in music and music-induced movement. Music Perception: An Interdisciplinary Journal, 30(5), 517–533. doi: 10.1525/mp.2013.30.5.517

Camurri, A., Lagerlöf, I., & Volpe, G. (2003). Recognizing emotion from dance movement: Comparison of spectator recognition and automated techniques. International Journal of Human Computer Studies, 59(1), 213–225. doi: 10.1016/S1071-5819(03)00050-8

Caramiaux, B. (2014). Motion modeling for expressive interaction: A design proposal using Bayesian adaptive systems. In Proceedings of the International Workshop on Movement and Computing (pp. 76–81). Paris, France: ACM Press. doi: 10.1145/2617995.2618009

Caramiaux, B., Montecchio, N., Tanaka, A., & Bevilacqua, F. (2014). Adaptive gesture recognition with variation estimation for interactive systems. ACM Transactions on Interactive Intelligent Systems, 4(4), 1–34. doi:10.1145/2643204

Chi, D., Costa, M., Zhao, L., & Badler, N. (2000). The EMOTE model for effort and shape. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (pp. 173–182). New York, NY, USA: ACM Press. doi: 10.1145/344779.352172

Cléry, J., Guipponi, O., Wardak, C., & Ben Hamed, S. (2015). Neuronal bases of peripersonal and extrapersonal spaces, their plasticity and their dynamics: Knowns and unknowns. Neuropsychologia, 70, 313–326. doi: 10.1016/j.neuropsychologia.2014.10.022

Crane, E. A., & Gross, M. M. (2013). Effort-shape characteristics of emotion-related body movement. Journal of Nonverbal Behavior, 37(2), 91–105. doi: 10.1007/s10919-013-0144-2

Cross, E. S., Kirsch, L., Ticini, L. F., & Schütz-Bosbach, S. (2011). The impact of aesthetic evaluation and physical ability on dance perception. Frontiers in Human Neuroscience, 5(9), 1–10. doi: 10.3389/fnhum.2011.00102


55

Dael, N., Goudbeek, M., & Scherer, K. R. (2013). Perceived gesture dynamics in nonverbal expression of emotion. Perception, 42(6), 642–657. doi: 10.1068/p7364

Davidson, J. W. (1993). Visual perception of performance manner in the movements of solo musicians. Psychology of Music, 21, 103–113. doi: 10.1177/030573569302100201

de Gelder, B. (2006). Towards the neurobiology of emotional body language. Nature Reviews Neuroscience, 7(3), 242–249. doi: 10.1038/nrn1872

de Gelder, B. (2013). From body perception to action preparation: A distributed neural system for viewing bodily expressions of emotion. In K. Johnson & M. Shiffrar (Eds.), People watching: Social, perceptual, and neurophysiological studies of body perception (pp. 350–368). New York, NY, USA: Oxford University Press.

Dittrich, W. H., Troscianko, T., Lea, S. E. G., & Morgan, D. (1996). Perception of emotion from dynamic point-light displays represented in dance. Perception, 25(6), 727–738. doi: 10.1068/p250727

Ekman, P., & Friesen, W. V. (1974). Nonverbal behavior and psychopathology. In R. J. Friedman & H. M. Katz (Eds.), The psychology of depression: Contemporary theory and research (pp. 203–224). New York, NY, USA: Wiley.

Ekman, P., & Friesen, W. V. (1978). Facial action coding system: A technique for the measurement of facial movement. San Francisco, CA, USA: Consulting Psychologists Press.

Glowinski, D., Camurri, A., Chiorri, C., Mazzarino, B., & Volpe, G. (2007). Validation of an algorithm for segmentation of full-body movement sequences by perception: A pilot experiment. In Proceedings of the International Gesture Workshop (pp. 239–244). Berlin, Germany: Springer.

Glowinski, D., Dael, N., Camurri, A., Volpe, G., Mortillaro, M., & Scherer, K. (2011). Toward a minimal representation of affective gestures. IEEE Transactions on Affective Computing, 2(2), 106–118. doi: 10.1109/T-AFFC.2011.7

Glowinski, D., Shirole, K., Torres-Eliard, K., Grandjean, D., Riolfo, A., & Chiorri, C. (2014). Is he playing solo or within an ensemble? How the context, visual information, and expertise may impact upon the perception of musical expressivity. Perception, 43(8), 825–828. doi: 10.1068/p7787

Gunes, H., & Piccardi, M. (2009). Automatic temporal segment detection and affect recognition from face and body display. IEEE Transactions on Systems, Man, and Cybernetics Part B, 39(1), 64–84. doi: 10.1109/TSMCB.2008.927269

Hill, H., & Pollick, F. E. (2000). Exaggerating temporal differences enhances recognition of individuals from point light displays. Psychological Science, 11(3), 223–228. doi: 10.1111/1467-9280.00245

Juslin, P. N., & Laukka, P. (2003). Communication of emotions in vocal expression and music performance: Different channels, same code? Psychological Bulletin, 129(5), 770–814. doi:10.1037/0033-2909.129.5.770

Kapadia, M., Chiang, I., Thomas, T., Badler, N. I., & Kider, J. T. (2013). Efficient motion retrieval in large motion databases. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (pp. 19–28). Orlando, FL, USA: ACM Press.

Kapur, A., Kapur, A., Virji-Babul, N., Tzanetakis, G., & Driessen, P. F. (2005). Gesture-based affective computing on motion capture data. In J. Tao, T. Tan, & R.W. Picard (Eds.), Affective computing and intelligent interaction (pp. 1–7). Berlin, Germany: Springer.

Laban, R. (1966). The language of movement: A guidebook to choreutics. Boston, MA, USA: Plays, Inc.

Laban, R., & Lawrence, F. (1947). Effort. London, UK: Macdonald and Evans.

Laban, R., & Ullmann, L. (1971). The mastery of movement. Boston, MA, USA: Plays, Inc.

Levy, J. A., & Duke, M. P. (2003). The use of Laban movement analysis in the study of personality, emotional state and movement style: An exploratory investigation of the veridicality of “body language.” Individual Differences Research, 1(1), 39–63.

Mancas, M., Glowinski, D., Volpe, G., Coletta, P., & Camurri, A. (2010). Gesture saliency: A context-aware analysis. Gesture in Embodied Communication and Human-Computer Interaction, 5934, 146–157. doi: 10.1007/978-3-642-12553-9_13


56

Meijer, M. (1989). The contribution of general features of body movement to the attribution of emotions. Journal of Nonverbal Behavior, 13(4), 247–268. doi: 10.1007/BF00990296

Nummenmaa, L., Glerean, E., Hari, R., & Hietanen, J. K. (2014). Bodily maps of emotions. Proceedings of the National Academy of Sciences, 111(2), 646–651. doi: 10.1073/pnas.1321664111

Okajima, S., Wakayama, Y., & Okada, Y. (2012). Human motion retrieval system based on LMA features using interactive evolutionary computation method. In T. Watanabe & L. C. Jain (Eds.), Innovations in intelligent machines 2: Intelligent paradigms and applications (pp. 117–130). Berlin, Germany: Springer.

Palinkas, L. A. (2001). Psychosocial issues in long-term space flight: Overview. Gravitational and Space Biology Bulletin, 14(2), 25–33.

Pantic, M., Pentland, A., Nijholt, A., & Huang, T. S. (2007). Human computing and machine understanding of human behavior: A survey. In T. Huang, A. Nijholt, M. Pantic, & A. Pentland (Eds.), Artificial intelligence for human computing (pp. 47–71). Berlin, Germany: Springer.

Pollick, F. E., Paterson, H. M., Bruderlin, A., & Sanford, A. J. (2001). Perceiving affect from arm movement. Cognition, 82(2), 51–61. doi: 10.1016/S0010-0277(01)00147-0

Russell, J. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178. doi: 10.1037/h0077714

Serino, A., Annella, L., & Avenanti, A. (2009). Motor properties of peripersonal space in humans. PLoS ONE, 4(8), e6582. doi: 10.1371/journal.pone.0006582

Shawn, T. (1963). Every little movement: A book about François Delsarte, the man and his philosophy, his science and applied aesthetics, the application of this science to the art of the dance, the influence of Delsarte on American dance. New York, NY, USA: Dance Horizons.

Shikanai, N., Sawada, M., & Ishii, M. (2013). Development of the movements impressions emotions model: Evaluation of movements and impressions related to the perception of emotions in dance. Journal of Nonverbal Behavior, 37(2), 107–121. doi: 10.1007/s10919-013-0148-y

Sutil, N. (2013). Rudolf Laban and topological movement: A videographic analysis. Space and Culture, 16(2), 173–193. doi: 10.1177/1206331213475776

Taffou, M., & Viaud-Delmon, I. (2014). Cynophobic fear adaptively extends peri-personal space. Frontiers in Psychiatry, 5, 122–128. doi:10.3389/fpsyt.2014.00122

Todorov, A., Baron, S. G., & Oosterhof, N. N. (2008). Evaluating face trustworthiness: A model based approach. Social Cognitive and Affective Neuroscience, 3(2), 119–127. doi: 10.1093/scan/nsn009

van Dyck, E., Vansteenkiste, P., Lenoir, M., Lesaffre, M., & Leman, M. (2014). Recognizing induced emotions of happiness and sadness from dance movement. PLoS ONE, 9(2), e89773. doi: 10.1371/journal.pone.0089773

van Paasschen, J., Bacci, F., & Melcher, D. P. (2015). The influence of art expertise and training on emotion and preference ratings for representational and abstract artworks. PLoS ONE, 10(8), e0134241. doi: 10.1371/journal.pone.0134241

Velloso, E., Bulling, A., & Gellersen, H. (2013). AutoBAP: Automatic coding of body action and posture units from wearable sensors. In Proceedings of the Humaine Association Conference on Affective Computing and Intelligent Interaction (pp. 135–140). Los Alamitos, CA, USA: IEEE. doi: 10.1109/ACII.2013.29

Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D’Errico, F., & Schröder, M. (2012). Bridging the gap between social animal and unsocial machine: A survey of social signal processing. IEEE Transactions on Affective Computing, 3(1), 69–87. doi: 10.1109/T-AFFC.2011.27

Wakayama, Y., Okajima, S., Takano, S., & Okada, Y. (2010). IEC-based motion retrieval system using Laban movement analysis. In R. Setchi, I. Jordanov, R. J. Howlett, & L. C. Jain (Eds.), Knowledge-based and intelligent information and engineering systems (pp. 251–260). Berlin, Germany: Springer.

Wallbott, H. G. (1998). Bodily expression of emotion. European Journal of Social Psychology, 28(11), 879–896. doi: 10.1002/(SICI)1099-0992(1998110)28:6<879::AID-EJSP901>3.0.CO;2-W


57

Zhang, S., Li, Q., Yu, T., Shen, X., Geng, W., & Wang, P. (2006). Implementation of a notation-based motion choreography system. In H. Zha, Z. Pan, H. Thwaites, A. C. Addison, & M. Forte (Eds.), Interactive technologies and sociotechnical systems (pp. 495–503). Berlin, Germany: Springer.

Zhao, L., & Badler, N. I. (2005). Acquiring and validating motion qualities from live limb gestures. Graphical Models, 67(1), 1–16. doi: 10.1016/j.gmod.2004.08.002

Authors’ Note The authors thank the dancers, Amory, Katrin, Maraussia, and Phileas, for their performances and Katrin Blantar for the choreography.

The work reported in this paper was supported in part by the National Centre of Competence in Research in Affective Sciences supported by the Swiss National Science Foundation, Grant Number 51NF40-104897 – DG, and by the Swiss Institute of Rome (ISR). All correspondence should be addressed to Donald Glowinski Swiss Center for Affective Sciences University of Geneva, Campus Biotech, bât. H8-2 Chemin des Mines 9, Case postale 60, 1211 Geneva 20, Switzerland [email protected] Human Technology: An Interdisciplinary Journal on Humans in ICT Environments ISSN 1795-6889 www.humantechnology.jyu.fi

ISSN: 1795-6889


58

MUSICAL INSTRUMENTS, BODY MOVEMENT, SPACE, AND MOTION DATA: MUSIC AS AN EMERGENT

MULTIMODAL CHOREOGRAPHY

Abstract: Music is a complex multimodal medium experienced not only via sounds but also through body movement. Musical instruments can be seen as technological objects coupled with a repertoire of gestures. We present technical and conceptual issues related to the digital representation and mediation of body movement in musical performance. The paper reports on a case study of a musical performance where motion sensor technologies tracked the movements of the musicians while they played their instruments. Motion data were used to control the electronic elements of the piece in real time. It is suggested that computable motion descriptors and machine learning techniques are useful tools for interpreting motion data in a meaningful manner. However, qualitative insights regarding how human body movement is understood and experienced are necessary to inform further development of motion-capture technologies for expressive purposes. Thus, musical performances provide an effective test bed for new modalities of human–computer interaction. Keywords: music, movement, performance, musical instrument, motion sensor, score.

© 2017 Federico Visi, Esther Coorevits, Rodrigo Schramm, & Eduardo Reck

Miranda, and the Open Science Centre, University of Jyväskylä DOI: http://dx.doi.org/10.17011/ht/urn.201705272518


Federico Visi Interdisciplinary Centre for Computer

Music Research (ICCMR) Plymouth University

UK

and

Institute for Systematic Musicology University of Hamburg

Germany

Esther Coorevits Institute for Psychoacoustics and Electronic

Music (IPEM) Ghent University

Belgium

Rodrigo Schramm Department of Music

Federal University of Rio Grande do Sul Brazil

Eduardo Reck Miranda Interdisciplinary Centre for Computer

Music Research (ICCMR) Plymouth University

UK

Musical Instruments, Body Movement, Space, & Motion Data

59

INTRODUCTION AND BACKGROUND SCENARIO In the interdisciplinary field of musicological research, the idea that music is a multimodal phenomenon that engages body movement has given rise to a wide range of methodologies for studying musical gestures. Musical gestures are “human body movement that goes along with sounding music … the gestures of those that produce the sounds (the musicians), and the gestures of those that perceive the sounds (the listeners or dancers)” (Jensenius, Wanderley, Godøy, & Leman, 2010, p. 13). On the one hand, the advent of new technologies, such as infrared motion capture, has allowed researchers to observe human movement in detail, extracting precise three-dimensional data and kinematic features of bodily movement. This brought about a corpus of studies where motion analysis is based on the computation of several low-level descriptors—or movement features—that could be linked with musical expression (Godøy & Leman, 2010). For example, acceleration and velocity profiles have been shown to be useful in the study of musical timing (Burger, Thompson, Luck, Saarikallio, & Toiviainen, 2014; Dahl, 2014; Glowinski et al., 2013; Goebl & Palmer, 2009; Luck & Sloboda, 2009). Quantity of motion (QoM) has been related to expressiveness (Thompson, 2012) and has been used to study the dynamic effects of the bass drum on a dancing audience (Van Dyck et al., 2013), while contraction/expansion of the body can be used to estimate expressivity and emotional states (Camurri, Lagerlöf, & Volpe, 2003). More advanced statistical methods, such as functional principal component analysis and physical modeling, have led to midlevel descriptors, including topological gesture analysis (Naveda & Leman, 2010), curvature and shape (Desmet et al., 2012; Maes & Leman, 2013), and commonalities and individualities in performance (Amelynck, Maes, Martens, & Leman, 2014).

On the other hand, gestures in musical performance can be accessed by means of high-level descriptors. Verbal descriptions, subjective experiences, and the musician’s intentions play an important role in daily interaction with music. This is the way performers and audiences naturally communicate about music. Leman (2008b) referred to these descriptions as first-person perspectives on music experience, resulting in intention-based symbolic/linguistic expressions. In the analysis of musical performance, this qualitative approach has been explored profoundly in the studies of, among others, Davidson (Davidson, 2007, 2012; Williamon & Davidson, 2002) and King (2006). In studies such as these, musical gestures are accessed by means of verbal descriptors or directly perceivable movements that appear to be expressive. In that sense, the concept of musical gesture can be useful to bridge the gap between mental and subjective experiences of the performer/listener and the directly observable physical world.

Recent studies have made an attempt to close the gap between these two perspectives in musical performance research by applying a performer-informed analysis (Coorevits, Moelants, Östersjö, & Gorton, 2016; Desmet et al., 2012). In trying to understand the relationship between the physical aspects of movement in space and expressive qualities, the study of musical gestures has resulted in new understandings of the relationship between musician and musical instrument as well. Here, the instrument becomes a natural extension of the musician (Nijs, Lesaffre, & Leman, 2013), being part of the body and hence integrated into the mediation process of communicating musical meaning. In the context of musical practice, an instrumentalist’s gestures bear great expressive potential. Many new interfaces for musical expression (NIMEs) have been developed in the past years and take advantage of this (Miranda & Wanderley, 2006). The development of human–computer interfaces also exploits the expressive potential of musical

Visi, Coorevits, Schramm, & Reck Miranda

60

gestures to enhance the interaction between the digital and the human environments and to create meaningful applications for musical practice (Camurri & Volpe, 2003). Recently, artistic practice has been increasingly adopted as a complementary research method for the arts and humanities, leading to mixed, interdisciplinary methodologies (Smith & Dean, 2009).

In this article, we aim to explore issues related to the representation and mediation of body movement in musical performance through digital data. We do so by adopting an interdisciplinary approach, which involves the theoretical analysis of implications brought about by (a) representing movement through data, (b) defining computable motion descriptors, and (c) employing a case study of a musical composition for viola, guitar, and motion sensors.

The structure of the paper is as follows. We first review and comment on several technological approaches to analysis, computation, and interpretation of movement data obtained from different devices. We then propose some techniques useful for extracting meaningful information from data collected via wearable motion sensors. Successively, we describe a musical performance where motion sensors have been employed alongside musical instruments. The analysis of this case study then leads to a discussion on the interpretation of human expressive movement through musical experience and digitized motion data. Subsequently, we suggest that musical practice can aid the development of new techniques for the interpretation and understanding of movement data for expressive purposes. This can result in valid contributions to the development of motion-based technologies for human-computer interaction beyond music applications.

CAPTURING, STORING, AND MEDIATED MOVEMENT Human movement can be digitally captured and stored via different means for purposes of analysis, description, and notation. In the context of musicological studies, movement has been recorded throughout the years using visual media, such as photography (Ortmann, 1929) and video (Davidson, 1993). More recently, motion capture has become widely adopted as the medium of choice for quantitative studies of human motion. Even though new technologies are emerging, marker-based optical motion capture is still regarded as the most reliable solution for precise, high-speed tracking. Data obtained from these systems is usually in the form of three-dimensional vectors referring to a global coordinate system. Each sample in the data returns three-dimensional information regarding the position of a point (marker) in space in relation to the origin of the Cartesian axes. The origin is defined during the calibration procedure and is usually set in an arbitrary place on the floor within the capture area.

As Salazar Sutil pointed out, the term motion capture (sometimes shortened to MoCap) indicates not only a technological setup but also a “technologized language of movement, involving the formalized description of movement coordinates and movement data for its subsequent computational analysis and ... processing” (2015, p. 198). Compared to photography and film, MoCap is definitely a younger medium. This has obvious technological implications, as MoCap technologies are still being developed and only recently have become more widely accessible to researchers and practitioners. However, the nature of body movement itself makes its mediation somehow still conceptually challenging. Salazar Sutil noted that the conceptualization of corporeal movement often is optically biased, whereas sensations that are unrelated to sight are often neglected. The ubiquity of visual record is certainly a factor in this


61

process. Still, movement cannot be entirely represented and therefore fully understood exclusively by means of visual media. In fact, interpreting human movement objectively as a displacement of body parts in a three-dimensional space would result in a limited interpretation. Merleau-Ponty famously points this out by giving the example of typing:

The subject knows where the letters are on the typewriter as we know where one of our limbs is, through a knowledge bred of familiarity which does not give us a position in objective space. The movement of her fingers is not presented to the typist as a path through space which can be described, but merely as a certain adjustment of motility, physiognomically distinguishable from any other. (Merleau-Ponty, 1945/2002, p. 166)

This is possibly one of the reasons why the use of absolute position in a Cartesian coordinate system imposes some constraints and challenges to high-level analysis of motion data and its use for expressive applications.

In previous works, we used MoCap to carry out experiments aimed at analyzing relationships between body movements and other musical features in instrumental musical performance (Visi, Coorevits, Miranda, & Leman, 2014; Visi, Coorevits, Schramm, & Miranda, 2016). For real-time musical applications, we preferred the use of various wearable sensors because they are easier to transport and use in performance situations. Optical MoCap, on the other hand, is definitely more demanding in terms of portability and setup time. As we will show more in detail, the raw data returned by wearable sensors are intrinsically different from that of MoCap, and this presents some implications for how the data are eventually interpreted and applied. Understanding how to extract meaningful descriptors from such sensors is useful beyond the domain of musical practice because similar technologies are becoming ubiquitous, employed in everyday objects such as mobile devices and game controllers.

Previous research (e.g., Freedman & Grand, 1977; McNeill, 1996) has pointed out that upper body movements are of particular interest when observing expressive behavior. In instrumental musical performance, the upper limbs have a central role; they typically are involved in the main sound-producing gestures. Moreover, in most cases, hands and arms are the primary points of contact between the body of the performer and the instrument. Therefore, in the studies described in this article, we placed the sensor bands on the forearms of the performers. However, as we will show in the following sections, processing data from the inertial measurement units (IMUs) using motion descriptors and machine learning models allowed us to obtain information related to full body movements, which can be used to extract expressive movement features.

In earlier tests (Visi, Schramm, & Miranda, 2014a, 2014b), we made use of fingerless gloves for decreased interference during the musical instrument manipulation. We progressively moved away from gloves in order to obtain an even less obtrusive configuration. We first placed the sensor on the wrists and eventually moved further away from the hands of the performer, onto the upper forearm. Doing so did not reduce the amount of information about hand movements we were able to retrieve. On the contrary, by using multimodal sensing and exploiting the constraints posed by the structure of the limbs and the interdependence of its parts, we were able to estimate various measures describing the movement of both hands.

Initially, we mostly employed IMUs; later we sought to include a form of muscle sensing. This was done in order to address and estimate body movement components beyond those strictly related to displacement in space, such as proprioception and effort qualities.


62

INERTIAL MEASUREMENT UNITS (IMUs) An IMU is a device that incorporates accelerometers and gyroscopes. When these devices are paired with magnetometers, the resulting arrays are known as magnetic, angular rate, and gravity (MARG) sensors as well. These sensor arrays allow for the tracking of acceleration, rotational velocity, and orientation of the object they are attached to relative to the earth’s magnetic field. They are used extensively in aviation, robotics, and human–computer interaction (HCI), and their increasing affordability and small size have made them a very common feature of mobile and wearable devices and other consumer electronics (see Figure 1). Recently, 9 degrees of freedom (9DoF) sensors have become the most widely used type of IMU/MARG. Featuring three types of tri-axis sensors (hence the name), they enable estimating various motion features, including optimized three-dimensional (3D) orientation obtained by fusing together the data from the three types of sensors.

Whereas the raw data obtained using marker-based optical motion capture consist of samples of position based on a 3D Cartesian coordinate system,1 the data returned by 9DoF sensors are usually in the form of three 3D vectors, each one expressing acceleration, rotational velocity, and orientation, respectively. The sensor band we used more recently2 returns acceleration in units of g, rotational velocity in degrees per second, and orientation angles in radians. Orientation also is estimated using a quaternion representation, which—unlike Euler angles—is not subject to problematic singularities such as gimbal lock (Brunner, Lauffenburger, Changey, & Basset, 2015). In addition, the sensor band returns 8-channel electromyogram (EMG) data, which we used to compute descriptors of muscular effort and to estimate the movements of wrists and fingers.

Calculating absolute position from IMU data in real time is technically very difficult if not outright unfeasible, as the operation would require double integration of acceleration data. This would result in a considerable level of residual error because drift would accumulate quadratically.

Figure 1. Some of the wearable devices used in prior research. Clockwise from the bottom-left corner:

Myo armbands, Sense/Stage controllers with wristbands, Axivity WAX9 with silicone wristband, Adafruit 9-DOF IMU Breakout, FreeIMU v0.4.


63

Moreover, it would also be relatively expensive in terms of computation. Madgwick et al. designed computationally efficient algorithms for compensating the residual error (Madgwick, Harrison, & Vaidyanathan, 2011). These can be used for estimating position from IMU data recorded in situations where specific constraints could be exploited, such as gait analysis and cyclic motion.

The data obtained from IMUs are therefore morphologically very different from positional data returned by optical MoCap. The differences in the way movement is tracked and represented by the two technologies has implications on how movement data are eventually interpreted and used, particularly in the context of expressive movement tracking. High-level movement descriptors often are used to extract features from the raw motion data that can help describing the meaning that the movements of the subject convey. This is no trivial task, and various interdisciplinary approaches have been adopted over the past 2 decades. In the following section we look at several motion descriptors most widely used with positional data and discuss how they can be adapted for use with IMU data.

MOVEMENT DESCRIPTORS AND WEARABLE SENSORS: UNDERSTANDING DIGITIZED MOVEMENT QUALITIES

Computable descriptors of human motion are used across several disciplinary fields for various applications, ranging from kinesiology and gait analysis to HCI and gaming. Even though human motion data analysis has become an increasingly active field, there is still little consensus regarding which descriptors and methodologies yield meaningful representations of human body motion.

The MoCap Toolbox (Burger & Toiviainen, 2013) provides a wide range of MATLAB scripts for offline kinematic analysis and visualization. Alternatively, expressive feature extraction and real-time interaction are prominent features of the Eyesweb platform (Camurri et al., 2007).

Going beyond traditional low-level kinematic features has proven challenging, especially when dealing with expressiveness, emotions, affective states, and meaning. Larboulette and Gibet (2015) recently attempted a thorough review of computable descriptors of human motion. This was indeed a useful endeavor; however it showed continued segmentation and that many procedures are either ill-defined or redundant (i.e., similar concepts appear in other research literature under different names).

Most of the descriptors we mention below were conceived using positional data. However, the principles behind their design are nonetheless useful for describing certain movement qualities; therefore, we attempted to adapt them to the data obtained from the IMU/MARG sensors. A series of objects were implemented using Max,3 which was chosen over other programming environments because it allowed for rapid prototyping and testing of algorithms for real-time interaction and for easily integrating them with other music applications. Fluidity and Jerkiness In kinematic analysis, “jerk” is the name given to the third-order derivative of movement position, namely the variation of acceleration over time. The definition of “jerk index” as the


64

magnitude of the jerk averaged over the entire movement (Flash & Hogan, 1985) was used by Pollick, Paterson, Bruderlin, and Sanford (2001) alongside other descriptors to correlate arm movement to basic affective states. This relates to the fluidity or smoothness of a movement—as fluid movement tends to have an even level of velocity and, therefore, low values of higher-order derivatives—that can be used to detect emotionally relevant information in movement data (Glowinski et al., 2011). In fact, roughly speaking, jerkiness could be seen as the inverse of fluidity. Piana, Staglianò, Odone, and Camurri (2016) defined the fluidity index as a local

kinematic feature equal to , where ji is the jerk of the joint4 i. This means that higher

values of jerk correspond to lower fluidity. To estimate jerkiness using 9DoF sensor data instead of positional data, we averaged the

derivatives of longitudinal acceleration returned by the accelerometer ( , , ) into a single jerk index. The resulting value was then combined with the averaged second-order derivatives of the angular velocity returned by the gyroscope ( , , ) and then summed over a time window of length samples:

| | | | | |

3| | | | | |

3.

Coefficients and are weights that balance the data magnitudes obtained from the accelerometer and gyroscope sensors. It is worth mentioning that, in real-world implementations, derivatives are very sensitive to signal noise. Therefore, sensor data may require low-pass filtering before jerkiness can be computed.

From the conceptual framework of Laban effort elements/qualities (Laban & Lawrence, 1947), jerkiness (and its counterpart fluidity) is a useful descriptor that can aid the computational analysis of expressive movements. Laban (as cited in Hackney, 2002) defined four basic effort factors (flow, weight, time, and space); each factor is a continuum between polarities described by effort element/qualities. In particular, flow is related to the continuity and control of the movement. Its polar qualities (free flow and bound flow) have been previously associated with aspects of fluidity in movement. A movement reflecting the free flow effort quality is characterized as fluid, liquid, and outpouring. On the other hand, the bound flow quality indicates containment, restraint, and control. In addition to flow, jerkiness can be related also to the time effort elements. A movement characterized by the sustained effort qualities is expected to have a low level of jerkiness. On the other hand, a movement with the sudden effort qualities (i.e., urgent, quick, staccato) will have most likely a higher rate of change in acceleration and therefore higher levels of jerkiness.

Jerkiness and fluidity, then, can contribute to the analysis and recognition of expressive movement qualities, particularly in multimodal frameworks that involve multiple descriptors and sensing modalities (Camurri & Volpe, 2011; Caramiaux, Donnarumma, & Tanaka, 2015). However, it is important to emphasize that Laban effort elements are qualitative “inner attitudes” of a person moving towards the effort factor. Using computable descriptors should not be seen as an attempt to quantitatively measure the effort qualities but rather as a means to aid in the design of computational models capable of discerning and recognizing different expressive movement behaviors.


65

Quantity of Motion and Overall Motion Energy

Fenza et al. defined QoM as the sum of the Euclidean distances between successive points in a time window (Fenza, Mion, Canazza, & Rodà, 2005) and Glowinski et al. (2011) included a similar measure in their expressive feature set, denoted as overall motion energy. To compute an analogous feature using 9DoF sensor data, we aggregated the magnitude of the variations of the norm of the orientation quaternion (∥ ∥) and of the average acceleration over the three axes ( ). The values for each frame were once again summed over a time window of length N samples:

| ∥ ∥ ∥ ∥ | | |.

Similarly to the previous equation, and were weights to balance individual contributions from distinct sensors.

Contraction/Expansion and Symmetry Contraction and expansion of the body can be computed in different ways. They can be achieved, for example, by calculating the area of bounding shapes (Glowinski et al., 2011), using the contraction index (Fenza et al., 2005) or measuring the volume of a convex hull that encloses the body (Hachimura, Takashina, & Yoshimura, 2005).

When wearing two 9DoF sensors on the forearm, it is possible to project the orientation values over a hypothetical 2D plane in front of the subject and thus obtain approximate coordinates of the points in the plane the arms are pointing to. First, the yaw values for both arms have to be centered while the subject is pointing both arms forward. Then, given θyaw and θpitch as the yaw and pitch angles, respectively (expressed in radians), the coordinates for each point in the plane can be calculated as follows:

,2

2

, 2

22

.

By calculating the Euclidean distance between the two points, it is possible to estimate whether the arms are pointing in opposite directions. When arms are spread wide, the distance between the two points will be at its maximum. In this way, it is possible to have a value that depends on whether arms are wide open or are resting close to the body. This is can be used as an expressive feature, even though it is not as precise as the contraction indexes obtained using an optical motion capture because the values are based on the orientation of the arms and not on their actual position. By comparing the coordinates, we also can see if there is horizontal or vertical symmetry between the arms, which is another useful postural feature that has been previously used for the analysis of expressive movements (Camurri & Volpe, 2011). Periodicity and Rhythmic Qualities: Periodic Quantity of Motion Previous techniques that focused on expressive motion analysis have intensively used the QoM estimation, especially to segment gestures by detecting the resulting motion bell curves


66

generated by a subsequent low-pass filtering process (Fenza et al., 2005). QoM is also useful for detecting the presence of body movements and other related activities during a live performance. It is easy and quick to implement, and it is frequently used as an input for more complex algorithms, such as gesture classifiers based on machine learning.

While the techniques mentioned above could be described as spatial features of the movement, or spatiotemporal in the case of QoM, periodicity is a purely temporal quality. Despite the good estimation of body movement given by QoM, it is not suitable for describing multiple periodic gestures, which are usually associated with the musical rhythmic content. In fact, there is often sensorimotor synchronization between the rhythmic structure of the piece and the periodic motion of the body (Repp & Su, 2013). Periodic quantity of motion (PQoM; Visi, Schramm, & Miranda, 2014a) was proposed as a complementary way to measure periodicity in the movement in relation to the musical rhythm.

The first PQoM implementation was designed to extract periodic motion from the data tracked by optical motion capture systems. The PQoM estimates are obtained from the decomposition of the signal from the motion capture tracker into frequency components by using filter banks (Müller, 2007). The PQoM function uses three main parameters as input: window size, beats per minute (bpm), and the center frequency of each band-pass filter. These parameters are intrinsically associated with the musical content. The window size parameter is used to define the amount of time that should be integrated to compute the final QoM. Once the music tempo is defined in bpm, each center frequency can be expressed by note values (i.e., half note, quarter note, eighth note, etc.) reflecting the rhythmic structure of the music. The PQoM function converts these note values into the center frequencies of each band-pass filter. Each center frequency fc is used to compute the transfer function coeffcients of a respective 4th-order band-pass digital Butterworth filter. Then, the input signal is filtered by a zero-phase forward and backward digital IIR filtering algorithm (non-real-time implementation). Finally, similar to QoM, the filtered values for each sampled frame are summed over a time window of length N (window size) samples. For optical motion capture data, each tracked point is normalized using the origin [0, 0, 0] of the coordinate system as reference,5 and a weight vector

combines distinct points from the body skeleton. It is worth noting that it is possible to ignore specific markers by setting the related coefficient in to zero. Thus, the discrete PQoM can be expressed by

, | 1 | ,

where the vector is the Euclidean norm from the skeleton markers positions, and is the

th band-pass filter operator. The first software implementation of the PQoM was made in MATLAB and can be downloaded as an extension for the MoCap Toolbox6 (Burger & Toiviainen, 2013). The same approach was adopted to estimate the PQoM from data from other sensors. However, a specific vector of coeffcients must be defined regarding each sensor type, and an appropriate norm for the vector must be chosen. For instance, we used a similar approach as in Equation 2, in which our PQoM implementation replaced with the weighted sum of the norm of the orientation quaternion (∥ ∥) with the average acceleration over the three axes.


67

Also, for the use in real-time applications, the PQoM can be implemented using a causal filter (instead of zero-phase forward and backward digital IIR filter). Figure 2 presents the results of the PQoM measure applied to a recorded movement using a motion capture system and an armband device with a built-in IMU/MARG sensor. Figure 2a shows the locations of these sensors (passive markers and armband device). In this example, the musician performed a periodic movement with his arm, syncing it with the half note duration (from 1 to 3 s) and with the quarter note duration (from 5 to 7.5 s). Figure 2b illustrates the respective tracked vectors: motion capture marker (green line, top), acceleration (blue line, center) and orientation (red line, bottom).

The bottom part of this figure shows the PQoM measures extracted from the tracked data. The amount of gesture periodicity over time is indicated by the yellow and light blue regions (higher values) on each matrix. The PQoM estimates from the motion capture marker (Figure 2c) are comparable with the respective PQoM measures extracted from the acceleration (Figure 2d) and orientation (Figure 2e) data. This means that PQoM is robust enough to be measured with different sensors and data types. This flexibility allows for the substitution of very precise and expensive motion capture systems with less accurate—but more affordable—devices, such as IMUs.

Figure 2. Periodic Quantity of Motion (PQoM) computed from motion capture and inertial measurement

unit data. (a) locations of passive markers and armband device. (b) tracking data from sensors (rescaled for better visualization). (c) PQoM from motion capture marker. (d) PQoM from acceleration.

(e) PQoM from orientation. MACHINE LEARNING: MAPPING POSTURAL AND SONIC TOPOLOGIES

Motion descriptors are useful for extracting meaningful features from the raw data, as well as allow for aggregating information relative to all the axes. This helps to move away from a low-level movement representation constrained by the Cartesian coordinate system and to obtain


68

motion data that are less dependent on it. A system based on orthogonal axes is indeed a convenient way to digitize movement. However, a meaningful conceptualization that helps in interpreting the expressivity that body movement conveys may be hindered if subordinated to a highly disciplined method of quantitative representation. In his article about topology and data, Carlsson argued that “coordinates ... are not natural in any sense,... [and] therefore we should not restrict ourselves to studying properties of the data which depend on any particular choice of coordinates” (Carlsson, 2009, p. 256). Moreover, in describing the characteristics of topological methods, he stated that, to obtain knowledge about the data, qualitative information is needed, and this must be established before proceeding with quantitative analysis. Researchers use topology to study intrinsic geometric properties of objects that do not depend on a chosen set of coordinates and this process also has been employed in the analysis of dance patterns (Naveda & Leman, 2010). This approach provides very useful notions for interpreting movement data generated by musical performance gestures. In fact, such body movements are bound to multimodal expressive features, which are inherently qualitative.

To put these concepts into practice, we used machine learning algorithms to define interaction models based on the various postures a musician may adopt during a performance. This was done by asking performers to play freely while wearing two sensor armbands. A small number of postures (4–5) were then defined. This was done by observing recurrent idiosyncrasies and peculiarities of the performance and discussing the qualities of the movements with the musicians themselves to better understand how certain movements relate to each respective instrumental techniques and with the musical features of the pieces performed.7 Sensor data were then sampled repeatedly during each pose in order to train a support vector machine (SVM) classifier. This was implemented using the ml.lib library (Bullock & Momeni, 2015) for Max, which is itself based on the Gesture Recognition Toolkit by Gillian & Paradiso (2014). Every posture was then associated with a set of parameters of a digital sound processing engine. During the performance, the machine learning classifier compared the incoming sensor data stream with the recorded examples, returning the values for the probability (or likelihood) that the current posture of the musician matched one of the defined classes. The values of the probabilities then were used to interpolate between the parameter sets of the sound engine that was used to process the sound of the instrument in real time or to synthesize electronic sounds.

This practical approach resonated with the aforementioned notions of topology because the incoming data were not analyzed quantitatively but rather evaluated in terms of proximity/distance from the predefined postures.8 From this perspective, the sampled postures themselves were topologies determined in relation to qualitative aspects of the movement of that particular performer, thus avoiding dependency from an abstract, artificial coordinate system. The system was instead defined by the idiosyncrasies of the performer.

This approach offers some practical advantages compared to more traditional sensor-to-sound parameter mapping approaches. First, incoming sensor data do not need to be rescaled to the range of the sound parameters they are mapped to. Moreover, the quantitative values of the sensor data can be ignored because the classifier probabilities are used to interpolate multiple sound parameters. This is another advantage because complex mappings can be defined easily and parameter modulation is independent of any coordinate system. Instead, the system quickly adapts to different users, and this is desirable considering the substantially different movements


69

required for playing different instruments and the significant degree of idiosyncrasy that characterizes musical performance.

In the past few years, machine learning techniques increasingly have been employed for interactive computer musical performance. Notable approaches include Wekinator (Fiebrink, Trueman, & Cook, 2009), Gesture Variation Follower (Caramiaux, Bevilacqua, & Tanaka, 2013) and mapping by demonstration (Françoise, Schnell, Borghesi, & Bevilacqua, 2014).

In our early tests, orientation data and an aggregate EMG descriptor for both arms were used as inputs to train the machine learning models. Orientation was chosen because it is not an inertial measurement; therefore, it can be used to describe postures. In addition, EMG data allowed us to consider other characteristics of movement. As pointed out by Salazar Sutil (2015), the perception of body movement involves sensations that go beyond displacement in space, such as interoception and proprioception. Moreover, in their extensive work on the analysis of expressive movement, Camurri and Volpe (2011) defined gesture as a multimodal entity, citing Laban’s theory of effort (Laban & Lawrence, 1947) as a central source of concepts for understanding expressive movement.

CASE STUDY: KINESLIMINA Kineslimina is a musical composition for viola, electric guitar, motions sensors, and live electronics that explores the use of the musicians’ instrumental movements as an expressive medium. Such gestures merge with the other musical features and become an integral part of the score. While playing their instruments, the musicians wear an armband fitted with IMUs, which tracks their movements and sends the motion data to a computer. The computer then processes the movement data and sound, responding with a wide range of dynamics: from subtle timbral alterations that follow the movements of the bow during string changes to deeper resonances when more overt gestures are performed by the musicians. The title is a portmanteau word composed of kinespheres and limina. The concept of kinesphere was defined by Laban (1966, p. 10) as “the sphere around the body whose periphery can be reached by easily extended limbs.” Thus, the kinesphere is a personal space, and how an individual relates and pays attention to it contributes to its delineation. Limina is the plural form of limen, which is a threshold or margin. The piece aims at pushing the boundaries of the personal spaces that surround the musicians during the performance. Throughout the performance, the sound of the instruments is altered, and synthesized sounds are engaged by exceeding the usual extent of instrumental movements. The score of the piece can be considered a script through which a multimodal choreography emerges as the product of learned body schema, altered by the influence of and the reactions to an interactive system. In the ritualized context of musical performance, a nonconventional technology (the sensors) interferes with conventional ones (the instruments), reconfiguring the relationships between the score, the performers, and their tools.

Parviainen, Tuuri, Pirhonen, Turunen, & Keskinen (2013) proposed an approach to interaction design that considers choreography as the holistic, experiential continuum of human movement resulting from the interaction with artifacts. From this perspective, musical instruments, sensors, and movement/sound mappings can be seen as carriers of a set of prechoreographies. The design of these objects (whether material or not, as in the case of software) and the environment where the interaction takes place prechoreographs the


70

performance of the piece. In other words, all the movement opportunities that these objects afford form the basis for the actual choreography that emerges as the score is enacted. Thus, a pre-existent, overarching design influences the movements within the kinesphere. Eventually, the movements that shape the performance exceed what the individual kinespheres can capture. The relations between the two musicians and between the musicians and the audience and the dynamics that arise from these connections are what Parviainen, Tuuri, Pirhonen, Turunen et al. (2013, p. 2) called “local-level movements.”

As noted by Wilson, a traditional musical instrument is not merely a piece of technology: experiential relationships to it are shaped by “the way it is ‘meant’ to be played, the canonic tradition that stands behind it as repertoire, and the normative expressive gestures that are ‘input’ by the player and ‘output’ sonically by the instrument” (Wilson, 2013, p. 426). Bodily relationships with these cultural artifacts are mediated historically and become part of a shared knowledge. Introducing motion sensor technology into this picture adds another layer of complexity, tightly woven to the already established gesture–sound relationships. Figure 3 shows an example of how these different aspects of the piece are interrelated. Towards the end of the piece, the score requires the viola player to repeat an arpeggiated pattern with increasing dynamics, until a chord played by a synthesizer is heard. The part entails repeated bow strokes, and the movement pattern is captured by the sensors placed on the right wrist (Figure 3b). The peaks in the acceleration data control a granular synthesis engine that samples and alters the timbre of the instrument at each peak. At the same time, QoM is computed (red line, Figure 3c)

Figure 3. Excerpt of the viola part of Kineslimina and corresponding kinematic features of the right

hand of the viola player: (a) scored pattern, (b) right hand acceleration of the viola player, and (c) the Quantity of Motion. The green cross indicates when the QoM threshold is crossed and the synthesized

part that closes the composition is played back.


71

and the increase of motion activity introduces other electronic parts, until the QoM data reach a predefined threshold (green cross, Figure 3c) at which point the closing synthesizer chord is triggered and the musicians can move onto the closing notes of the piece. The number of repetitions required to reach this point depends on the movements of the performer, which may vary according to how different musicians interpret the score. The score engenders the instrumental movements required to perform the piece; the movement data alter the sound of the instrument and also impact the structure of the score itself (i.e., the number of repetitions required). This closes a feedback loop in which every part mutually influences the other. The body of the performer is the medium, the locus, where this dynamic entanglement takes place.

This interdependency also affects how motion data are mapped to sound parameters. Mapping sensor parameters to sounds has long been debated in the HCI and NIME research communities (Hunt, Wanderley, & Paradis, 2003). Leman noted that freedom of mapping that characterizes digital musical interfaces “may disturb the sense of contact and of non-mediation” (Leman, 2008a, p. 164). Drawing from an established vocabulary of gesture of a traditional musical instrument and exploiting the constraints that instrumental techniques pose on the body can result in an advantage for obtaining meaningful interactions for expressive musical performance. This approach takes advantage of the ecology of musical instruments (Clarke, 2005) in order to obtain expressive transparency in gesture-sound mapping.

Kineslimina premiered at the Gala Concert of CMMR 2015 (the 11th International Symposium on Computer Music Multidisciplinary Research;9 see Figure 4) and was performed later that same year at MuSA 2015 (the Sixth International Symposium on Music/Sonic Art,10) held at the Institut für Musikwissenschaft und Musikinformatik in Karlsruhe, Germany. From the perspective of the performers,11 the piece reconfigured the relationship between the musicians and their instruments, extending expressive possibilities through their instrumental movements tracked by the sensors. However, this also required the performers to learn new skills and embed them within their existing instrumental techniques. This process became evident during the rehearsals. The performers experienced an increased awareness of the fundamental body schema of their instrument playing, as subtle movements created new sonic results via the motion sensors. This made them pay renewed attention to movements they learned in the early days of their formation as musicians, essential parts of the vocabulary of gestures of their respective instruments. While the musicians learned and became more familiar with the sensors, the system itself continually adapted and adjusted to accommodate the needs of the performers and to better follow their performance styles. As the rehearsals continued, relationships between body movements, instrumental gestures, and sensor data became renegotiated. This did not involve mere parameter adjustments and technical improvements to the sensor system: The process elicited and entailed a careful analysis of the relationship between movements, sound, and score from the privileged perspective of the performers themselves, thus resulting in a useful contribution to research from a practice-led perspective (Sullivan, 2009).

The performers then progressively got to know how the mappings of movement features to sound worked, and how they could explore this unconventional technology in a meaningful way. More than just “sonifying” the movements made by the performers, the sensor system induced the musicians to reconsider their relationships to the performance space. Such space—where the relationships among the players, instruments and audience are located—encompasses a set of conventions and cultural practices that are established and embodied in the performer. This can be compared to what Ervin Goffman (1974) referred to as “the frame,” which is the


72

Figure 4. Kineslimina performance during the Gala Concert of the 11th International Symposium

on Computer Music Multidisciplinary Research (CMMR), 16 June, 2015, Plymouth, UK. perceptual mechanism that indicates the nature and purpose of a behavior and how it is to be interpreted. It is a tool for understanding the implicit agreement between performer and audience on the symbolic status of the performance. From the perspective of a musician, this frame consists of the historically established relationships that transpire between players and instruments. Performance techniques and experience with instruments and instrumental music are habitualized through historical practice, conventions, and education. These relationships are thus part of the embodied knowledge of the performer. In Kineslimina, performing with reconfigured instrument/body/space relationships has made the musicians more aware of other qualities of their movements and their kinespheres.

Laban identified space, weight, time and flow as motion factors toward which performers of movement can have different attitudes depending on temperament, situation, environment and many other variables. The attitudes toward the motion factors he called... Effort. ... Choices are continuously made by all people in motion, consciously or unconsciously, to determine what combinations of these Effort elements will best serve the purposes of their intentness or modify their behavior.... Whatever the action in which the effort combinations appear, the whole biological/psychological system is involved. (Bartenieff & Lewis, 1980, p. 51)

Intentionality is a key aspect in the study of musical gestures; the fact that they are goal-directed actions is an essential quality for the understanding of their expressive qualities (Godøy & Leman, 2010). The effort qualities of a movement are very much the result of this intentionality and they play an important role in the perception and understanding of body movement. During the Kineslimina performances, effort qualities and intentionality appeared amplified by the presence of the motion sensors and their effect on the conventional performance gestures. Moreover, as some audience members commented after the performance, this interplay between the performers and their role in the intersubjective space was transparent through the augmentation of the performers’ musical intentionality.


73

DISCUSSION AND FUTURE WORK We presented techniques for interpreting motion data and discussed the implications that arise when employing motion sensors in conjunction with traditional instruments in musical practice. Within this context, it is clear that body movements go well beyond simply being activators of technological objects, whether these objects are motion sensors or musical instruments. Bodily movement is considered a key element in forming embodied musical meaning (Leman, 2010). However, its role as an important cog of the engine that engenders signification and cognition is obviously not limited to the musical context.

Technological objects have the capacity to entail gestures and store their potential meaning. As we have observed in our research, a musical instrument can be seen as a receptacle of gestures, of kinemes that—through a performance—give rise to a multimodal choreography. From a wider perspective, we could say that objects extend human cognition: they are cosubstantial, continuous, and coextensive elements of minds in action (Malafouris, 2013). Moreover, Wilson stressed the importance of the relationship between instrument technology and the instrumentalist’s pedagogy: “Technology—what the instrument is—is inherently entangled with pedagogy, the historically established relationships found between instrument and instrumentalist” (Wilson, 2013, p. 430; italics in original). In the case study we presented, this “inherent entanglement” encompasses also the score of the piece, which elicited the instrumental movements necessary to execute its parts while, at the same time, being affected by them through the use of motion descriptors. Within this layered process of signification—situated in a cultural ecology and shaped by shared knowledge—the body is the medium where everything takes place. This perspective is akin to Merleau-Ponty’s (1945/2002) phenomenological approach.

Once recognizing the centrality of the body and its movements in the ways humans make sense of the world, it is clear that—in an increasingly pervasive digital “semiosphere”—being able to digitize movement and interpret motion data become of primary interest. However, movement seems to have properties that exceed the system used to represent it. We have discussed the limitations emerging from representing movements exclusively through visual media, and the ubiquity of visual record is certainly a factor in this process. However, solving this bias is one of the challenges that contemporary researchers and practitioners must address in making progress in the discourse on human movement. The development of different computational techniques to describe the qualities of body motion is a necessary step towards more meaningful interpretations of data generated by human movement. However, it is also vital to consider the constraints posed by rigid methods of representation and move towards approaches that allow for addressing the complex, nonlinear phenomena that characterize expressivity.

Using inexpensive and unobtrusive devices, such as 9DoF IMU/MARG sensors, also may help to move the research beyond laboratories. As researchers, we have seen how ecology plays an important role in the way people make sense of music (Clarke, 2005). Similarly, being able to study movement “in the wild” may have considerable implications, as shown in previous studies (e.g., Woolford, 2014).

IMU/MARG sensors provide data that are morphologically distinct from those obtained through optical motion capture. However, it is possible to obtain analogously meaningful information if the data are correctly interpreted. In this context, using machine learning


74

techniques is not only a quicker method toward obtaining complex interaction models that adapt to different individuals, it also provides a way to study and reflect upon the topological qualities of human movement through applied research and practice. More sophisticated algorithms to interpret motion data can help address its complexity, reclaiming the centrality of the body over a rigid representation of data structures.

Extracting expressive movement features from 9DoF data can lead to many other applications, well beyond the field of musical interaction. The ubiquity of 9DoF sensors—which is a very common feature of recent communication and entertainment devices—brings about a vast number of potential scenarios where the techniques we describe can be implemented. Using higher level descriptors to estimate expressive qualities of body movements is a way towards implementing dynamic HCI designs that handle gestures not only as isolated objects of application but as part of longer experiential chains, with multiple layers of significance. This goes beyond the traditional use-oriented approach and is akin to the use of choreographies for interaction design proposed by Parviainen, Tuuri, Pirhonen, Turunen, et al. (2013) and Pirhonen, Parviainen, and Tuuri (2013). It also parallels the work on affective computing carried out by the researchers at InfoMus/Casa Paganini (Glowinski et al., 2011; Piana, Staglianò, Camurri, & Odone, 2013). Both approaches avoid limiting HCI design to goal-directed actions and adopt a more holistic approach that takes into consideration a wider ecology of human movement. Moreover, 9DoF sensors coupled with sound synthesis techniques have already found applications in the field of rehabilitation of stroke patients (e.g., Bevilacqua et al., 2013).

As suggested by a topological approach (Carlsson, 2009), gaining new higher level knowledge from motion data also requires qualitative insights. To access more complex, structural, and subjective qualities that are considerably difficult to model quantitatively, data need to be interpreted through qualitative methods. Intuitions arising from qualitative approaches can contribute to the understanding of how body schema and kinemes work in generating embodied meaning. This can successively inform more advanced computational models able to identify complex and meaningful qualities of human movement. Practice as research can address the need of qualitative insights in the interpretation of motion data. Particularly, music can be an effective test bed, given its multilayered complexities and rich cultural, multimodal qualities. As other projects have previously shown, music and the performing arts can be effective test beds for new modalities of expressive HCI (Camurri, Mazzarino, Ricchetti, Timmers, & Volpe, 2004), and practice-led approaches have yielded technical and conceptual material useful for the development of motion capture technologies (Norman & Blackwell, 2010). Moreover, musicians often are early adopters of new paradigms of interaction that eventually become mainstream (Kirn, 2013). Notable examples are gestural controllers and multitouch technologies, which were adopted by musicians long before they become widespread.

Practice-led approaches are helpful also for carrying out the conceptual work required to make sense of motion data and understand the meanings it can potentially convey. Through their work, some artists seek to affirm the irreducibility of the corporeal presence while simultaneously sublimating it through digital processing (Norman, 2015). This resonates with the rationale behind Kineslimina, and this creative tension can lead to new insights into how movement can carry meaning across physical and digital environments. In Kineslimina, relating motion data to a musical score has shown how multimodal qualities of music are entangled, mutually affecting each other.


75

We expect that future work will adopt this mixed methodology in order to address technical and conceptual aspects related to body movement, motion data, and meaning formation. Other machine learning algorithms will be tested in order to map instrumental gestures to sounds synthesized by means of physical modeling. Sound synthesis techniques based on physical modeling allow artists to generate sounds that resemble those of certain musical instrument families. Working with synthesis parameters facilitates the ability to go beyond the timbral ranges and sonic capabilities of the physical instruments while preserving timbral resemblances to the instrument family. This poses interesting conceptual challenges, as the recognition of timbral qualities of musical instruments relies on a shared knowledge. As we discussed above, the relationships between instruments and instrumental movements is also something encoded in a shared gestural vocabulary. The ecology around musical instruments, their timbres, and their instrumental gestures offers a rich conceptual framework for developing meaningful cross-modal mappings between motion data and synthesis parameters. Cross-modal relations between performance movements and the sonic outcomes will also be inspired by the concept of Uncanny Valley, which was previously adopted in a composition that involved tension and relaxation structures in timbrally varied musical phrases generated by physical models (Bessell, 2011).

From the perspective discussed so far, instrumental music is constituted by abstract structures and performance movements in continual interaction, entangled with technological and cultural knowledge. The concept of choreography appropriately describes the process of multimodal signification that emerges from the performance of a musical score. Body schema and kineme are useful conceptual tools to gain a better understanding of how the body is the central medium in human communication. Movement is a modality of knowledge; therefore, being able to interpret it and represent it through technology offers potential in avenues not yet imagined that should certainly be further explored.

In summary, the article has addressed the challenge of extracting meaningful expressive features from motion data in the context of musical performance. The scope of this work is limited to the context of music and to the case study described. However, the interdisciplinary approach we adopted has led to the design of effective solutions that were implemented in the case study and in other works (Visi, 2017). This has highlighted that qualitative insights into how human body movement is understood and experienced are essential for informing the development of motion capture technologies for expressive purposes, as well as to broaden the discourse on music-related body motion.

IMPLICATIONS FOR EMBODIED HCI In scenarios where computing is becoming ubiquitous, embodied, and considered a fundamental factor for designing interactions with technology (Parviainen, Tuuri, & Pirhonen, 2013), the implications of being able to extract meaningful information from motion data are manifold. Moreover, the study of music-related motion and the analysis of motion descriptors certainly has applications beyond music making. As an example, Bennett, Hinder, and Cater (2016) recently measured periodicity in data obtained from motion sensors applied to rocking chairs in care homes. This was done to help improve the quality of life of residents in dementia care by creating subtle, engaging interactions that support and stimulate memory through music and movement (Bennett et al., 2016). This is but one example of current applications of motion


76

data analysis in conjunction with music. As computing becomes increasingly embedded in the environments people live in, interpreting the data produced by human activity in a meaningful way remains a key issue. This is central for recent trends in computer science, such as affective computing, calm technology, and human-centered machine learning.

ENDNOTES

1. Most marker-based systems also allow capture of 6DoF data (six degrees of freedom, consisting of three-dimensional position and Euler angles) by defining rigid bodies. However, these data are achieved by processing positional data of the single markers grouped into a rigid body.

2. These data were collected with the Myo Gesture Control Armband, produced by Thalmic Labs (https://www.myo.com).

3. Web page of the Max programming environment: https://cycling74.com/products/max/

4. In motion capture terminology, a joint is a point belonging to a three-dimensional representation of a body. Joints are usually defined by positional coordinates.

5. In fact, it is possible to define any arbitrary origin for the coordinate system. [0,0,0] is the default option.

6. The MoCap Toolbox and the PQoM extension can be downloaded from https://www.jyu.fi/ hum/laitokset/musiikki/en/research/coe/materials/mocaptoolbox

7. To view some early tests with different musicians, please go to https://youtu.be/stWI43-EZGA

8. This idea is reinforced by how support vector machines work: “An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall on” (Bullock & Momeni, 2015, p. 4).

9. The Web page of the 11th International Symposium on Computer Music Multidisciplinary Research (CMMR) is available at http://cmr.soc.plymouth.ac.uk/cmmr2015/index.html

10. The Web page of the Sixth International Symposium on Music/Sonic Art is available at http://zilmusic.com/musa2015/

11. Esther Coorevits: viola, motion sensors, live electronics; Federico Visi: electric guitar, motions sensors, live electronics.

REFERENCES Amelynck, D., Maes, P.-J., Martens, J. P., & Leman, M. (2014). Expressive body movement responses to music

are coherent, consistent, and low dimensional. IEEE Transactions on Cybernetics, 44(12), 2288–2301. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/25415938

Bartenieff, I., & Lewis, D. (1980). Body movement: Coping with the environment. London, UK: Routledge.

Bennett, P., Hinder, H., & Cater, K. (2016). Rekindling imagination in dementia care with the resonant interface rocking chair. In Proceedings of the 2016 CHI conference, extended abstracts on human factors in computing systems (pp. 2020–2026). New York, New York, USA: ACM Press. doi: 10.1145/2851581.2892505

Bessell, D. (2011). Ophidian and the Uncanny Valley. In the annual Proceedings of the International Computer Music Conference (ICMC; pp. 346–349). San Francisco, CA, International Computer Music Association, Inc.

Bevilacqua, F., Van Zandt-Escobar, A., Schnell, N., Boyer, E. O., Rasamimanana, N., Françoise, J., Hanneton, S., & Rasamimanana, N. (2013, September–October). Sonification of the coordination of arm movements.


77

Paper presented at the International Conference on Multisensory Motor Behavior: Impact of Sound. Hanover, Germany.

Brunner, T., Lauffenburger, J.-P., Changey, S., & Basset, M. (2015). Magnetometer-augmented IMU simulator: In-depth elaboration. Sensors, 15(3), 5293–5310. doi: 10.3390/s150305293

Bullock, J., & Momeni, A. (2015). ml.lib: Robust, cross-platform, open-source machine learning for Max and Pure Data. In Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 265–270). Retrieved from http://www.nime.org/proceedings/2015/nime2015_201.pdf

Burger, B., Thompson, M. R., Luck, G., Saarikallio, S. H., & Toiviainen, P. (2014). Hunting for the beat in the body: On period and phase locking in music-induced movement. Frontiers in Human Neuroscience, 8(November), 1–16. doi: 10.3389/fnhum.2014.00903

Burger, B., & Toiviainen, P. (2013). MoCap toolbox: A Matlab toolbox for computational analysis of movement data. In R. Bresin (Ed.), Proceedings of the 10th Sound and Music Computing Conference (pp. 172–178). Stockholm, Sweden: KTH Royal Institute of Technology.

Camurri, A., Coletta, P., Demurtas, M., Peri, M., Ricci, A., Sagoleo, R., Simonetti, M., Varni, Volpe, G. (2007). A platform for real-time multimodal processing. In Proceedings International Conference Sound and Music Computing (pp. 11–13). Retrieved from http://www.smc-conference.org/smc07/SMC07 Proceedings/SMC07 Paper 59.pdf

Camurri, A., Lagerlöf, I., & Volpe, G. (2003). Recognizing emotion from dance movement: Comparison of spectator recognition and automated techniques. International Journal of Human-Computer Studies, 59 (1–2), 213–225. doi: 10.1016/S1071-5819(03)00050-8

Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R., & Volpe, G. (2004). Multimodal analysis of expressive gesture in music and dance performances. In A. Camurri & G. Volpe (Eds.), Gesture-based communication in human–computer interaction (pp. 20–39). Berlin, Germany: Springer. doi: 10.1007/978-3-540-24598-8_3

Camurri, A., & Volpe, G. (Eds.). (2003). Gesture-based communication in human–computer interaction. Berlin, Germany: Springer. doi: 10.1007/b95740

Camurri, A., & Volpe, G. (2011). Multimodal analysis of expressive gesture in music performance. In J. Solis & K. Ng (Eds.), Musical robots and interactive multimodal systems (pp. 47–66). Berlin, Germany: Springer. doi: 10.1007/978-3-642-22291-7_4

Caramiaux, B., Bevilacqua, F., & Tanaka, A. (2013). Beyond recognition: Using gesture variation for continuous interaction. In CHI ’13 extended abstracts on human factors in computing systems (p. 2109). New York, NY, USA: ACM Press. doi: 10.1145/2468356.2468730

Caramiaux, B., Donnarumma, M., & Tanaka, A. (2015). Understanding gesture expressivity through muscle sensing. ACM Transactions on Computer–Human Interaction, 21(6), 1–26. doi: 10.1145/2687922

Carlsson, G. (2009). Topology and data. Bulletin of the American Mathematical Society, 46(2), 255–308. doi: 10.1090/S0273-0979-09-01249-X

Clarke, E. F. (2005). Ways of listening: An ecological approach to the perception of musical meaning. New York, NY, USA: Oxford University Press. doi: 10.1093/acprof:oso/9780195151947.001.0001

Coorevits, E., Moelants, D., Östersjö, S., Gorton, D., & Leman, M. (2016). Decomposing a composition: On the multi-layered analysis of expressive music performance. In R. Kronland-Martinet, M. Aramaki, & S. Ystad (Eds.), Music, mind, and embodiment: CMMR 2015. Lecture Notes in Computer Science (Vol. 9617; pp. 167–189). Cham, Switzerland: Springer. doi: 10.1007/978-3-319-46282-0_11

Dahl, L. (2014). Triggering sounds from discrete air gestures : What movement feature has the best timing? In B. Caramiaux, K. Tahiroglu, R. Fiebrink, & A. Tanaka (Eds.), Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 201–206). London, UK: Goldsmiths, University of London.

Davidson, J. W. (1993). Visual perception of performance manner in the movements of solo musicians. Psychology of Music, 21(2), 103–113. doi: 10.1177/030573569302100201

Davidson, J. W. (2007). Qualitative insights into the use of expressive body movement in solo piano performance: A case study approach. Psychology of Music, 35(3), 381–401. doi: 10.1177/0305735607072652


78

Davidson, J. W. (2012). Bodily movement and facial actions in expressive musical performance by solo and duo instrumentalists: Two distinctive case studies. Psychology of Music, 40(5), 595–633. doi: 10.1177/0305735612449896

Desmet, F., Nijs, L., Demey, M., Lesaffre, M., Martens, J.-P., & Leman, M. (2012). Assessing a clarinet player’s performer gestures in relation to locally intended musical targets. Journal of New Music Research, 41(1), 31–48. doi: 10.1080/09298215.2011.649769

Fenza, D., Mion, L., Canazza, S., & Rodà, A. (2005). Physical movement and musical gestures: A multilevel mapping strategy. In Proceedings of Sound and Music Computing Conference, SMC 2005 (unpaginated). Salerno, Italy: SMC

Fiebrink, R., Trueman, D., & Cook, P. R. (2009). A metainstrument for interactive, on-the-fly machine learning. In Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 280–285). Pittsburgh, PA, USA: NIME

Flash, T., & Hogan, N. (1985). The coordination of arm movements: An experimentally confirmed mathematical model. The Journal of Neuroscience, 5(7), 1688–1703.

Françoise, J., Schnell, N., Borghesi, R., & Bevilacqua, F. (2014). Probabilistic models for designing motion and sound relationships. In B. Caramiaux, K. Tahiroglu, R. Fiebrink, & A. Tanaka (Eds.), Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 287–292). London, United Kingdom: Goldsmiths, University of London.

Freedman, N., & Grand, S. (Eds.). (1977). Communicative structures and psychic structures. Boston, MA, USA: Springer. doi: 10.1007/978-1-4757-0492-1

Gillian, N., & Paradiso, J. A. (2014). The gesture recognition toolkit. Journal of Machine Learning Research, 15, 3483–3487. Retrieved from http://jmlr.org/papers/v15/gillian14a.html

Glowinski, D., Dael, N., Camurri, A., Volpe, G., Mortillaro, M., & Scherer, K. (2011). Toward a minimal representation of affective gestures. IEEE Transactions on Affective Computing, 2(2), 106–118. doi: 10.1109/T-AFFC.2011.7

Glowinski, D., Mancini, M., Cowie, R., Camurri, A., Chiorri, C., & Doherty, C. (2013). The movements made by performers in a skilled quartet: A distinctive pattern, and the function that it serves. Frontiers in Psychology, 4(NOV), 1-8. doi: 10.3389/fpsyg.2013.00841

Godøy, R. I., & Leman, M. (Eds.). (2010). Musical gestures: Sound, movement, and meaning. London, UK: Routledge.

Goebl, W., & Palmer, C. (2009). Synchronization of timing and motion among performing musicians. Music Perception: An Interdisciplinary Journal, 26(5), 427–438. doi: 10.1525/mp.2009.26.5.427

Goffman, E. (1974). Frame analysis: An essay on the organization of experience. Boston, MA, USA: Harvard University Press.

Hachimura, K., Takashina, K., & Yoshimura, M. (2005). Analysis and evaluation of dancing movement based on LMA. In IEEE International Workshop on Robot and Human Interactive Communication, ROMAN2005 (pp. 294–299). Nashville, TN, USA: IEEE. doi: 10.1109/ROMAN.2005.1513794

Hackney, P. (2002). Making connections: Total body integration through Bartenieff fundamentals. London, UK: Routledge.

Hunt, A., Wanderley, M. M., & Paradis, M. (2003). The importance of parameter mapping in electronic instrument design. Journal of New Music Research, 32(4), 429–440. doi: 10.1076/jnmr.32.4.429.18853

Jensenius, A. R., Wanderley, M. M., Godøy, R. I., & Leman, M. (2010). Musical gestures: Concepts and methods in research. In R. I. Godøy & M. Leman (Eds.), Musical gestures: Sound, movement, and meaning (pp. 12–35). New York, NY, USA: Routledge.

King, E. (2006). Supporting gestures: Breathing in piano performance. In A. Gritten & E. King (Eds.), Music and gesture (pp. 142–164). Aldershot, UK: Ashgate.

Kirn, P. (2013, May). How music can predict the human/machine future. Lecture presented at the 2013 re:publica conference 2013. Berlin, Germany. Retrieved from https://www.youtube.com/watch?v=Fr_9_6AyCP4

Laban, R. (1966). Choreutics. London, UK: Macdonald & Evans.


79

Laban, R., & Lawrence, F. C. (1947). Effort (Reprinted ed.). London, UK: Macdonald & Evans.

Larboulette, C., & Gibet, S. (2015). A review of computable expressive descriptors of human motion. In Proceedings of the 2nd International Workshop on Movement and Computing ( MOCO ’15; pp. 21–28). New York, New York, USA: ACM Press. doi: 10.1145/2790994.2790998

Leman, M. (2008a). Embodied music cognition and mediation technology. Cambridge, MA, USA: MIT Press.

Leman, M. (2008b). Systematic musicology at the crossroads of modern music research. In A. Schneider (Ed.), Systematic and comparative musicology: Concepts, methods, findings (pp. 89–115). Frankfurt am Main: Peter Lang.

Leman, M. (2010). Music, gesture, and the formation of embodied meaning. In R. I. Godøy & M. Leman (Eds.), Musical gestures: Sound, movement and meaning (pp. 126–153). London, UK: Routledge.

Luck, G., & Sloboda, J. A. (2009). Spatio-temporal cues for visually mediated synchronization. Music Perception: An Interdisciplinary Journal, 26(5), 465–473. doi: 10.1525/mp.2009.26.5.465

Madgwick, S. O. H., Harrison, A. J. L., & Vaidyanathan, R. (2011). Estimation of IMU and MARG orientation using a gradient descent algorithm. In Proceedings of 2011 IEEE International Conference on Rehabilitation Robotics (pp. 1–7). Zurich, Switzerland: IEEE. doi: 10.1109/ICORR.2011.5975346

Maes, P.-J., & Leman, M. (2013). The influence of body movements on children’s perception of music with an ambiguous expressive character. PLoS ONE, 8(1), unpaginated. doi: 10.1371/journal.pone.0054682

Malafouris, L. (2013). How things shape the mind: A theory of material engagement. Cambridge, MA, USA: MIT Press. doi: 10.1007/s11023-015-9356-z

McNeill, D. (1996). Hand and mind: What gestures reveal about thought. Chicago, IL, USA: University of Chicago Press.

Merleau-Ponty, M. (2002). Phenomenology of perception (C. Smith, Trans.). London, UK: Routledge. (Original work published 1945)

Miranda, E. R., & Wanderley, M. M. (2006). New digital musical instruments: Control and interaction beyond the keyboard. Madison, WI, USA: A-R Editions, Inc.

Müller, M. (2007). Information retrieval for music and motion. Berlin, Germany: Springer.

Naveda, L., & Leman, M. (2010). The spatiotemporal representation of dance and music gestures using Topological Gesture Analysis (TGA). Music Perception, 28(1), 93–111. doi: 10.1525/mp.2010.28.1.93

Nijs, L., Lesaffre, M., & Leman, M. (2013). The musical instrument as a natural extension of the musician. In M. Castellegno, H. Genevois, & J.-M. Bardez (Eds.), Music and its instruments (pp. 132–133). Sampzon, France: Editions Delatour.

Norman, S.-J. (2015). Oskar Schlemmer’s programmatic gesture research. In N. S. Sutil & S. Popat (Eds.), Digital movement: Essays in motion technology and performance (pp. 21–34). London, UK: Palgrave Macmillan. Retrieved from http://sro.sussex.ac.uk/56405/

Norman, S. J., & Blackwell, A. F. (2010). Gesture and embodied interaction: Capturing motion / data / value. Leonardo Journal, 43(2), 198–199. doi: 10.1162/leon.2010.43.2.198

Ortmann, O. (1929). The physiological mechanics of piano technique: An experimental study of the nature of muscular action as used in piano playing, and of the effects thereof upon the piano key and the piano tone. Boston, MA, USA: Da Capo Press.

Parviainen, J., Tuuri, K., & Pirhonen, A. (2013). Drifting down the technologization of life: Could choreography-based interaction design support us in engaging with the world and our embodied living? Challenges, 4(1), 103–115. doi: 10.3390/challe4010103

Parviainen, J., Tuuri, K., Pirhonen, A., Turunen, M., & Keskinen, T. (2013). Gestures within human-technology choreographies for interaction design In Proceedings of the 10th International Gesture Workshop and the 3rd Gesture and Speech in Interaction Conference (unpaginated). Tilburg, the Netherlands: Tilburg University. Retrieved from https://jyx.jyu.fi/dspace/handle/123456789/41920


80

Piana, S., Staglianò, A., Camurri, A., & Odone, F. (2013, May). A set of full-body movement features for emotion recognition to help children affected by autism spectrum condition. Paper presented at the 1st International Workshop on Intelligent Digital Games for Empowerment and Inclusion (IDGEI 2013). Chania, Crete, Greece.

Piana, S., Staglianò, A., Odone, F., & Camurri, A. (2016). Adaptive body gesture representation for automatic emotion recognition. ACM Transactions on Interactive Intelligent Systems, 6(1), 1–31. doi: 10.1145/2818740

Pirhonen, A., Parviainen, J., & Tuuri, K. (2013, April–May). Choreographies at home as objects of design. Paper presented at the CHI 2013 Workshop on Methods for Studying Technology in the Home, Paris, France. Retrieved May 26, 2017, from http://studyingthehome.wp.horizon.ac.uk/wp-content/uploads/2012/11/Choreographies-at-Home-as-Objects-of-Design.pdf

Pollick, F. E., Paterson, H. M., Bruderlin, A., & Sanford, A. J. (2001). Perceiving affect from arm movement. Cognition, 82(2), B51–B61. doi: 10.1016/S0010-0277(01)00147-0

Repp, B. H., & Su, Y.-H. (2013). Sensorimotor synchronization: A review of recent research (2006–2012). Psychonomic Bulletin & Review, 20(3), 403–452. doi: 10.3758/s13423-012-0371-2

Salazar Sutil, N. (2015). Motion and representation: The language of human movement. Cambridge, MA, USA: MIT Press.

Smith, H., & Dean, R. T. (Eds.). (2009). Practice-led research, research-led practice in the creative arts: Research methods for the arts and humanities. Edinburgh, UK: Edinburgh University Press.

Sullivan, G. (2009). Making space: The purpose and place of practice-led research. In H. Smith & R. T. Dean (Eds.), Practice-led research, research-led practice in the creative arts: Research methods for the arts and humanities (pp. 41–65). Edinburgh, UK: Edinburgh University Press.

Thompson, M. (2012). The application of motion capture to embodied music cognition research. Unpublished doctoral dissertation, University of Jyväskylä, Finland.

Van Dyck, E., Moelants, D., Demey, M., Deweppe, A., Coussement, P., & Leman, M. (2013). The impact of the bass drum on human dance movement. Music Perception, 30(4), 349–359. doi: 10.1525/mp.2013.30.4.34

Visi, F. (2017). Methods and technologies for the analysis and interactive use of body movements in instrumental music performance. Unpublished doctoral dissertation, Plymouth University, UK. Retrieved from: https://pearl.plymouth.ac.uk/handle/10026.1/8805

Visi, F., Coorevits, E., Miranda, E., & Leman, M. (2014). Effects of different bow stroke styles on body movements of a viola player: An exploratory study. In A. Georgaki & G. Kouroupetroglou (Eds.), Proceedings of the ICMC|SMC|2014 Conference (40th International Computer Music Conference and 11th Sound and Music Computing Conference; pp. 1368–1374). Retrieved from http://speech.di.uoa.gr/ICMC-SMC-2014/images/VOL_2/1368.pdf

Visi, F., Coorevits, E., Schramm, R., & Miranda, E. R. (2016). Analysis of mimed violin performance movements of neophytes. In R. Kronland-Martinet, M. Aramaki, & S. Ystad (Eds.), Music, mind, and embodiment: CMMR 2015. Lecture Notes in Computer Science (Vol. 9617; pp. 88–108). Cham, Switzerland: Springer. doi: 10.1007/978-3-319-46282-0_6

Visi, F., Schramm, R., & Miranda, E. (2014a). Gesture in performance with traditional musical instruments and electronics. In Proceedings of the 2014 International Workshop on Movement and Computing (MOCO ’14; pp. 100–105). New York, NY, USA: ACM Press. doi: 10.1145/2617995.2618013

Visi, F., Schramm, R., & Miranda, E. (2014b). Use of body motion to enhance traditional musical instruments: A multimodal embodied approach to gesture mapping, composition and performance. In Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 601–604). London, United Kingdom: Goldsmiths University of London.

Williamon, A., & Davidson, J. W. (2002). Exploring co-performer communication. Musicae Scientiae, VI(1), 53–72. doi: 10.1177/102986490200600103

Wilson, S. (2013). Building an instrument, building an instrumentalist: Helmut Lachenmann’s Serynade. Contemporary Music Review, 32(5), 425–436. doi: 10.1080/07494467.2013.849871


81

Woolford, K. (2014). Capturing human movement in the wild. In Proceedings of the 2014 International Workshop on Movement and Computing (MOCO ’14; pp. 19–24). New York, NY, USA: ACM Press. doi: 10.1145/2617995.2617999

Authors’ Note The first author would like to thank Andrew Telichan Phillips and Dr. Tae Hong Park at NYU for making the collaboration between the Interdisciplinary Centre for Computer Music Research (ICCMR) and the Music and Audio Research Laboratory (MARL) possible, and for their valuable contributions to the project. Many thanks also to the members of the MARL research group at NYU for their hospitality, friendliness and knowledgeable insights. Special thanks to the musicians that took part in the tests at NYU Steinhardt during summer 2015. The collaborative project between Plymouth University and NYU was generously supported by Santander Universities. This study was funded in part by the FWO-project “Foundations of expressive timing control in music” and it is part of the newly founded MAaV project of the Universidade Federal do Rio Grande do Sul. All correspondence should be addressed to Federico Visi Institute for Systematic Musicology Universität Hamburg Alsterterrasse 1, R111 20354 Hamburg, Germany [email protected] Human Technology ISSN 1795-6889 www.humantechnology.jyu.fi

ISSN: 1795-6889


82

BODILY INTERACTIONS IN MOTION-BASED MUSIC APPLICATIONS

Abstract: Motion-based music applications exploit the connection between body movements and musical concepts to allow users to practice high-level structured elements (e.g., tonal harmony) in a simple and effective way. We propose a framework for the design and the assessment of motion-based music applications by involving outcomes from various disciplines, such as the cognitive sciences and human–computer interaction. The framework has been applied to a working system, the Harmonic Walk, which is an interactive space application based on motion-tracking technologies. The application integrates both digital and physical information by reacting to a user’s movements within a designated 3 x 4 m floor, where six musical chords have been arranged according to a determined spatial positioning. Human choreographies from the user’s coordinated movements to musically structured events are analyzed in order to determine their relationships and to discuss related design issues. Keywords: interactive spaces, reality-based interaction, conceptual blending, implicit knowledge, human choreographies.

© 2017 Marcella Mandanici, Antonio Rodà, & Sergio Canazza, and the Open Science

Centre, University of Jyväskylä DOI: http://dx.doi.org/10.17011/ht/urn.201705272519


Marcella Mandanici Department of Information Engineering

University of Padova Italy

Antonio Rodà Department of Information Engineering


Sergio Canazza Department of Information Engineering


Bodily Interactions in Motion-based Music Applications

83

INTRODUCTION Interactive spaces are two-dimensional surfaces or three-dimensional regions positioned within a range of sensors or a network of control devices for detecting the presence, the position, and/or the gestures of one or more users within the space. Two-dimensional surfaces may be floors, touchscreens, or interactive walls; three-dimensional regions may be positioned anywhere and may be user- or sensor-centered.1 Both two- and three-dimensional spaces belong to everyday life and, consequently, may offer users an immediacy of interaction and effective feedbacks. Applications that employ interactive spaces as their interfaces have the power to mix the reality of the physical space with the richness and variety of a digital domain and, in doing so, to connect the real with the digital via a reality-based interaction style (Jacob et al., 2008). The user acts and moves in the physical space but, through her/his movements, s/he can reach and interact with expressly arranged digital contents, bringing to life a responsive augmented reality. Bill Buxton, while writing his reactive environments design principle 5, said that cameras and microphones used for human–human interaction could also be considered a computer’s eyes and ears, respectively (Buxton, 1997) and thus be employed as well as a means for human–computer interaction. As for cameras, some computer processes may be fed by motion and gestural data; as a consequence, the digital content needs to be arranged within the physical space to be made available to the user. This principle is the core idea of reality-based applications. It describes the birth of interactive spaces where the computer protrudes into reality through digital-content spatial organization, and where the user enters the digital content through bodily motion. As a consequence, bodily motion can be regarded as the leading factor for human–computer interaction in environments such as these.

The aim of this paper is to explore the nature and quality of motion as a means of bodily interaction in environments where music is produced or heard. Movements coordinated with musical events depend on entrainment, which is a complex phenomenon affecting various disciplines, from physics to human physiology. From the biomusicological point of view, entrainment is an organism’s ability to synchronize movements in the presence of an external rhythmic stimulus. Thus, although music is the most studied and complex source of coordinated movements, entrainment is a very common biological occurrence that does not depend on music or musical rhythm only, but also transpires in a great number of acoustical and nonacoustical events (Phillips-Silver, Aktipis, & Bryant, 2010). Consequently, the synchronization required for interaction in musical environments must be considered as a particular case among various other forms of movement coordination found in musical and in nonmusical environments. This could lead to the investigation of the role of entrainment as an overall human–computer interaction theme in the production of human choreographies in interactive spaces.

The paper’s topic is introduced in a theoretical background section, where the fundamental principles of bodily interaction are presented. Blending theory, spatial reasoning, implicit knowledge, and entrainment in its many forms and characteristics are considered the fundamental cognitive processes that govern a user’s behavior in an interactive space. These concepts constitute a set of analytical tools whose reciprocal interrelations form a dynamic system continuously fed by the user’s experiences, which must be regarded as the main agent of the user’s interaction in such environments. Two music analysis subsections focus on rhythmic structure organization and recursive patterns, which primarily influence selective entrainment in musical environments. Harmonic Walk, a motion-based music application

Mandanici, Rodà, & Canazza

84

aimed at the knowledge and practice of tonal harmony, is presented as a useful case study. User movements were analyzed to better understand the relationship between musical structures and human choreographies. Finally, emerging design issues are presented and discussed.

THEORETICAL BACKGROUND Motion can be described as the kinematic relationship between two points in space. Usually, this relationship involves various concepts of physics, such as displacement, distance, speed, acceleration, and so on. When related to a user’s change of position in a musical interactive space, motion implies the existence of the following three conditions:

(a) the spatial positioning of interactive landmarks, (b) a reason to choose one interactive landmark target instead of another (i.e., where to

move), and (c) a reason to move from one interactive landmark to another (i.e., when to move).

Spatial Positioning of Interactive Landmarks In motion-based applications, there is a strong relationship among a physical space, a user’s spatial cognition, and digital contents. Thus, a tool is needed to link these aspects and to manage them in a successful way. An example is provided by Stanza Logomotoria (Zanolla, Canazza, Rodà, Camurri, & Volpe, 2013), a motion-based application aimed at linking a narrative content to sounds. The camera system employed by this application covers a 4 x 3m rectangular floor area. Depending on different purposes, the Zone Tracker application partitions this surface with several masks.2 Each mask provides a generic spatial organization with a number of available landmarks regarding where the content (e.g., audio files, digital sound processing effects, or music composition algorithms) are positioned. The spatial positioning of interactive landmarks may be visible or invisible to the user. When visible, interactive landmarks may be labeled by visual tags or visualized through graphical elements projections. However, the available landmarks must be connected to the content through a spatial organizing principle. This is provided by the conceptual integration between the spatial characteristics of musical features and the actual space. Conceptual integration is a term borrowed from blending theory (Fauconnier & Turner, 2002), which is a framework for human knowledge that suggests the brain constructs information through various forms of integration between two input spaces. The theory defines a four-space model: a generic space, two input spaces, and a resulting blended space. As applied to our research, the generic space is where abstract knowledge about the musical feature is stored. The Input Space 1 is the physical space where interaction happens; the Input Space 2 provides the spatial organization of musical concepts. The resulting blended space is a spatial projection that takes the characteristics from both input spaces to create something completely new (Benyon, 2012). The conceptual blending theory forms the base of many design approaches, for instance in learning applications, where a user’s first-person embodied experiences are used to model physics-related events in augmented-reality environments (Enyedy, Danish, & DeLiema, 2015). In the case of motion-based musical applications, the physical space is


85

matched with the musical concept upon which the application is based (see Figure 1). The relationship between these two input spaces is mediated by a geometrical representation of musical concepts and their spatial projections (Mandanici, Rodà & Canazza, 2015), which provides the sound positions (or sound-processing zones) in the new blended space.

Many musical features, such as harmony or melodic movements, historically have been illustrated by spatial representations. Examples of this kind of representations are Euler’s tonnetz, which literally means “web of tones” and is a spatial schema showing the triadic relationships upon which tonal harmony is based (Euler, 1739); the neumatic notation of Gregorian chant (Strayer, 2013); and chironomy (Carroll, 1955), a gestural system expressing melodic contour variations. These suggest the idea that the spatial positioning of sounds conveys some meaning about their inner nature and element organization and that this meaning can be made available to the user through spatial representations, as can be seen in the modern tonnetz shown in Figure 2.

Where to Move

In order to move in space, a user needs to know where to go. That means that s/he must be informed of where her/his target interactive landmark is. While this information typically is obvious in everyday life, when people’s motion is usually directed to well-known, precise goals, the same cannot be said about a target interactive landmark in an artificial environment. In such a situation, motion implies the creation of a cognitive map by the user, which includes landmarks, way finding, and route segments (Montello, 2001). However, a cognitive map is much more than a simple mental-routing sketch: It includes other nonspatial elements, such as perceptual attributes, emotions, and the system’s feedback. Indeed, the creation of the new blended space allows the user to navigate concepts in the physical space and to move literally inside them, obtaining a different audio feedback according to the occupied zone.

Figure 1. Conceptual blending diagram, adapted from Benyon (2012, p. 223). Starting from the generic space of the musical concept, a conceptual integration is operated between the physical space

and the spatial feature of the musical concept. The resulting blended space is represented on the application interface.


86

Figure 2. The modern tonnetz showing the axis of the circle of fifths (horizontal black dashed line), the axis of minor thirds (diagonal black dotted line), and the axis of major thirds (diagonal

black dashed and dotted line). Each chord is formed by the three pitches at the triangle’s vertices, and each contiguous triangle (chord) shares at least two pitches with its neighbor. The chords

represented by triangles with just one vertex point, have only one note in common. This represents different grades of chords commonalities (Cohn, 1997), which, through

the tonnetz spatial representation, are expressed in a very concise and efficient way.

When applied to music-related research, the coupling between spatial location of concepts (interactive landmarks) and musical-content perception begins to feed the user’s cognitive map of the environment, which then drives her/his decisions on where to move in the artificial environment. This coupling also is driven by the implicit knowledge of musical language, which is a well-known mental process that allows the unconscious acquisition of very complex and structured constructions. The mind is continuously fed by structured stimuli (speech, music, spatial relationships, sensorimotor information, etc.) and, independent of the user’s will, builds an inner knowledge about them (Reber, 1989). As an example, considerable research in the field of music psychology offers evidence that children as young as 4 and 5 years of age (e.g., Corrigall & Trainor, 2010) have wide, implicit harmonic knowledge: They comprehend chord functions, harmonic relations, and perception of regularities of harmonic frames in time. Because tonal music is ubiquitous in the Western music cultural environment, all these characteristics are learned from mere passive exposure (Tillmann, Bharucha, & Bigand, 2000) and so can be considered a cognitive skill common to users who have been exposed to that musical tradition, independent of her/his degree of musical education. Thus, when a user enters a musical interactive space, her/his implicit musical knowledge is elicited by the audio output and can so be accessed, used, and/or modified by the user during the experience and can inform responses regarding where to move within the interactive space. When to Move We have already stated that a user’s implicit knowledge includes musical language structures and spatial cognition. A third important element to consider is entrainment, which is the process responsible of “when” to move in the interactive space. According to Thaut, Mcintosh, & Hoemberg (2015, p. 1), “Entrainment is defined by a temporal locking process in which one system’s motion or signal frequency entrains the frequency of another system.” In physics,


87

entrainment is the frequency alignment of two oscillating bodies on phase or at 180° out of phase. It happens because small quantities of energy transfer from one body to the other until they are fully synchronized. In human beings, the firing frequency of auditory neurons when receiving an external rhythmic stimulus influences the firing frequency of motor neurons, causing over time a coordinated movement (Thaut et al., 2015 p. 2). This resonates with the model of entrainment proposed by Phillips-Silver et al. (2010), according to whom entrainment is composed of three phases:

(a) the rhythmic detection of environmental signals (not only acoustic but also as a byproduct of biological phenomena),

(b) the ability to produce rhythmic signals (not only deriving from musical activities but also from other biological activities), and

(c) the ability to integrate both proceeding phases in order to adjust the output’s rhythm. The main point of this framework is that it does not limit the idea of entrainment to the

basic, regular pulse synchronization generally observed when people clap their hands to their favorite song’s beat in public concerts, when music players align their individual timing on the conductor’s gesture, or when dancers perform the same movement in a strictly rhythmic fashion. Rather, it extends the entrainment range towards a more general and wider number of events, such as environmental signals (seasonal or day/night alternation, weather changes, wind blowing, sea waves crashing) or biologically produced rhythms (e.g., breathing, eating, heart beating, walking, crickets’ chorusing, wolves’ howling). Many living beings’ actions depend on various forms of entrainment with these signals, in both natural and artificial environments. These examples emphasize at least three important aspects of entrainment:

(a) Entrainment may occur in conditions very different from regular rhythmic input (predictive entrainment);

(b) Entrainment is connected not only to “when” to move, but also to “why” to move (selective entrainment); and

(c) Coordination is the condition of success in the activity.

Predictive Entrainment The framework of entrainment proposed above is useful for understanding synchronization in nonmusical reality-based interaction environments, which employ smartphones, tablets, touch screens, interactive walls, and so on. In daily life, synchronization is fundamental to coordinating physical efforts and in helping human communication. Thus, the analogy between real life and digital content, which characterizes such interactive spaces (Jetter, Reiterer, & Geyer, 2013), suggests that synchronization could also play a similar role in the design of applications based on such devices. However, to understand how entrainment works in a musical interactive space, it is necessary to focus not so much on rhythmic input regularities, which can be found in both natural and artificial environments, but rather on rhythmic predictability. For example, an isochronous rhythm has a high degree of predictability because it has an intrinsic regular nature. Nonetheless, a slowing pulse (rallentando) is predictable to some degree, because a listener can follow a previously stored model of rallentando and try to adapt it to the current event sequence (Friberg & Sundberg, 1999). Another example of how humans can adapt entrainment to particular situations is the case of “soft entrainment,” which


88

occurs when small deviations in rhythmic entrainment among different performers in music ensembles are registered. Soft entrainment may occur at various degrees of deviation, depending on the phase of the musical phrase. Yoshida, Takeda, and Yamamoto (2002) reported that synchronization is maximal when musicians aim at the phrase’s climax (tense phase), whereas it deviates more often when approaching the phrase’s end (relaxing phase).

These examples show that entrainment is a dynamic process that involves not only mere beat detection but also a much wider range of musical elements, such as expressive trends, motion patterns, and musical phrase organization. Jones and Boltz (1989) provided an extensive framework of how real-world timing structures are organized in a hierarchical way, thus allowing predictive entrainment to work. They affirmed that the distribution of many natural events’ markers is nested at several timing levels that are consistent with ratio or additive timing transformations. This explains not only why humans can entrain with natural phenomena, such as gradual or abrupt changes of velocity, but also how prediction works when they have to synchronize with multilevel, hierarchically organized time events. For example, musical metric structure starts from a lower level, composed by the smallest rhythmic units and, through successive layers of stratification, reaches more extended musical units, such as musical periods, forms, sonatas, or symphonies. A hierarchical organization such as this allows a subject to have an idea on how musical events are organized and to make a prediction regarding how long s/he has to wait until the expected event occurs. Yet, a wider look at musical entrainment also needs to include the observation that not all kinds of music are strictly based on isochronous pulse. Similarly, not all the parts of a beat-based music are rigorously dependent on beat. To set an example, think about a classic concert’s cadenza, where the soloist leaves the overall ensemble governing pulse, to play freely and express her/his virtuosic ability. In the same way, in the Gregorian chant’s swinging gait, a pulse can sometimes be perceived, but always among many breathing pauses and fermata.3 There are also types of beat-composed music where the pulse is not perceivable at all in the musical output. This is the case of many classical contemporary music compositions, such as Ligeti’s (1966) Lux Aeterna, one of the most popular works of this genre, where the lack of periodic repetitions in the rhythmic pattern prevents any metrical organization of musical elements.

Notwithstanding, all the cited examples show a high degree of predictability in that, even if the events are not subjected to a regular metrical organization, they show some shared musical or nonmusical pattern. In classical music solos and concertos for solo and orchestra, the solo’s cadenza is marked at its beginning by a precise fermata on the second inversion of the I degree at the beginning and by a conclusive dominant chord in root position at the end.4 Thus, two harmonic markers act as strongholds of the relatively beat-free event, allowing the conductor and the whole orchestra to resynchronize their beat at the end of the cadenza. In Gregorian chant, the predictive timing of events is given by the breathing times in the musical phrases, whose code is deeply grounded in physiological, expressive, and melodic structure cues.5 Chaotic mass movements produced by very small musical elements are the result of composition techniques based on mathematical and physical models employed by many 20th-century composers. In Ligeti’s micropolyphony, for example, tendency masks rule the rhythm, density, and pitches of the musical elements (Drott, 2011). The temporal evolution of a tendency mask is perceived by the listener in biological and physics terms, such as growth, proliferation, thickness, fluctuation, and so on. Thus, process endpoints are perceptual landmarks that can be regarded as predictable entrainment anchors.


89

Selective Entrainment In this paper, we call selective entrainment the subject’s ability to focus her/his synchronization activity on a specific environmental signal chosen from among multiple simultaneous rhythmic stimuli. It is easy to notice that, in natural environments, it is common for acoustical or nonacoustical signals to overlap. For example, it is not difficult to imagine that a hunter has to select her/his prey’s biological signals from among all the other signals in the natural environment in order to be successful in the chase. Hence, in a musical interaction event, the subject has to filter the incoming signals to focus on a specific input, depending on the goal s/he wants to achieve. This means that movement is triggered by some environmental changes that the subject is interested in, and that movement depends on the perceptual timing of these changes. Thus, when to move is strictly connected with why to move and the reason to move is the need to remain tuned into important environmental events. This entrainment mechanism, involving contextual awareness and prior skills, is probably one of the most important abilities that a user brings to the digital space (Jacob et al. 2008, p. 2) and which highlights one of the most effective analogies between real-life environments and interactive spaces. Whereas the reacting to environmental signals may have important biological consequences in the physical world, it acquires a completely different meaning in artificial interactive spaces, where signal flow is controlled and where the user’s responsiveness is one of the fundamental aspects of the application design. First of all, if the interaction logic is controlled by the designer, the reaction to the signals is always mediated by implicit knowledge of the user on which the designer can make assumptions based on her/his experience or intuitions. Second, in the case of multilevel signals (e.g., musical input), the user is asked to apply a selective entrainment as s/he decides at which level to synchronize her/his movements with the input. The general idea about entrainment in artificial environments is that movement is always the result of a cognitive-selective process related to previously acquired knowledge. The consciousness about where to point her/his attention is the key element of selective entrainment: How can designers help users in achieving this goal remains a great challenge in reality-based interaction design.

Cognitive Meanings of Coordinated Movement No entrainment activity, whether in a natural or artificial environment, would be successful without coordinated movement. Coordination may have different degrees of accuracy, as in the already cited case of soft entrainment. Nevertheless, it is clear that the success of action depends on the subject’s ability to detect the right rhythmic input and to align her/his behavior to it. Baimel, Severson, Baron, and Birch (2015) underscored how behavioral synchrony in collective activities fosters social coordination and empathic concern, stressing how the outcome of attuning the minds of a group helps its individuals develop abilities. Moreover, Clayton, Sager, & Will (2005) observed that cognitive activities, such as perception, attention, and expectation, depend on entrainment, and that musical entrainment, in particular, helps motor- and self-control development in individuals. These observations foster the idea of entrainment as a general cognitive skill that coordinates the relationships between humans and the world’s events. Considering entrainment in an interactive space, the framework from Phillips-Silver et al. (2010) cited above could be reinterpreted in the following way:


90

(a) The rhythmic detection of environmental signals represents the openness of the subject to receiving information from the system producing the signals;

(b) The user’s ability to produce rhythmic signals means that s/he is able to respond to the signals produced by the application and that s/he understands the interaction mechanisms; and

(c) The ability to integrate both the preceding phases to adjust the output’s rhythm establishes the point in which the user’s response is aligned with the system’s output, confirming that a cognitive relationship is established between the user’s motion and the principles upon which the application is based.

Rhythmic Structure of Music Music is the most powerful source for entrainment because it is based upon a high degree of element organization. The most structured form of music–movement coordination is dance, where single movements or combinations of movements (choreographies) are arranged to follow the musical structure. There are popular dances where movements are rigidly entrained by the steady rhythms and by a simple formal organization of the music. Two examples of this are the traditional Quadrille6 and the folkloristic dances, such as the Cheraw dance or Tinikling,7 where not only the dancers’ movements but also the dancers’ available space is timing-locked by entrainment.

It is not clear why isochronous pulse is so pervasive in many kinds of music. However, the perceptual link between the musical pulse and a fundamental biological element such as the heartbeat could perhaps explain why the role of beat is so important for the organization of musical elements. Nevertheless, it seems that, even when exposed to nonisochronous auditory stimuli, such as Hindustani classical music, subjects respond with an isochronous motor response (Will, Clayton, Wertheim, Leante, & Berg, 2015), as if the need to adapt the auditory stimulus to a previous embodied sense of beat was a necessary perceptual condition. The first level of beat organization in Western music starts from the definition of the musical meter, which depends on the occurrences of melodic elements along the beats timeline. Generally, beats can be grouped into binary or ternary tempi. Every rhythmic unit is further grouped into semiphrases, phrases, and periods, as showed in Figure 3. This hierarchical organization also allows for the detection of other musical elements, such as, for instance, the harmonic rhythm, which depends on the duration of the various harmonic regions. A harmonic region is a musical excerpt that can be harmonized by one musical chord. The chord’s change determines the duration of the various harmonic regions and hence the harmonic rhythm. Recursive Structures in Music Music is characterized by hierarchical structures that are organized in various overlapping layers, expanding from a fundamental, deep structure to reach the complete score level (Schenker, 1979). The collaborative work of music theorist Fred Lerdhal and linguist Ray Jackendoff produced a theory of tonal music organization that showed the commonalities between the structure of


91

Figure 3. Formal organization of a typical music period with phrases, semiphrases, bars with harmonic

functions, and harmonic region durations. In this musical period, the second harmonic region gathers bars Number 2 and 3, which share the same harmonic function (D, dominant). The same happens for bars Number

4 and 5 (tonic function) and for bars Number 6 and 7 (again dominant). As shown in the lower part of the diagram, the extent of the harmonic functions does not correspond to the formal structure of the period. Thus, the harmonic regions (1 + 2 + 2 + 2 + 1 bars) form a perceptual structure that is superimposed to the formal

organization of the period and that establishes a new level of the selective musical entrainment. musical phrases and periods and the structure of linguistic sentences (Lerdahl & Jackendoff, 1985). The linguistic sentence tree structure (i.e., diagram), formed by articles, nouns and pronouns, verbs, prepositions, adjectives and adverbs, conjunctions, and interjections, is similar to Lerdhal and Jackendoff’s hierarchical tree of musical elements grouping, metrical structure, time-span, and “prolongational reduction.” This framework is based on the concept of recursion, which is the repetition of the same pattern at different levels and time scales. As an example, the hierarchical organization of an English sentence is compared to a musical phrase (see Figure 4) showing respectively linguistic and harmonic recursive elements.

Harmonic recursion also plays a fundamental role in harmonic progressions, which are very common, repetitive harmonic patterns used in traditional and popular repertoires. The key concept in tonal harmony is that chords are organized in a hierarchical order where the tonic, dominant, and subdominant chords (i.e., the I, IV and V degrees) play a primary role, whereas chords built on the II, III and VI degrees (i.e., parallel harmonies) play a secondary role. Thus, starting from its simplest form T-D-T (corresponding to the harmonic structure of Schenker’s, 1979, ursatz8) it is possible to expand primary chords with their parallel harmonies to create many alternative harmonic progressions. In Figure 5, the squares represent primary chords and the smaller rectangles are the parallel chords. The diagram shows examples of harmonic progressions obtained by adding parallel harmonies to the IV and I primary chords.

1 T

2 D

3 D

4 T

5 T

6 D

7 D

8 T

eight bars with harmonic functions

semi-phrase semi-phrase semi-phrase semi-phrase

phrase phrase

period

T 1.2 s

D 2.4 s

four semi-phrases

two phrases

one period

five harmonic regions with durations in binary tempo at 100 BPM

T 2.4 s

D 2.4 s

T 1.2 s

*harmonic region overlapping phrase limit


92

Figure 4. An English sentence and a musical phrase presented as six-level syntactic trees. The left area of

the figure presents the English sentence tree. It provides a surface level (i.e., words with their linguistic functions), whereas the right part shows the musical phrase tree with its surface level (i.e., pitches with

harmonic functions; the lowercase letters represent the musical note, and the designations of T, SD, and D are the tonic, subdominant and dominant harmonic functions, respectively). Five other structural levels are shown with related recursive patterns. In the sentence tree recursive patterns can be found at Levels 2 (the girl), 3 (the dress), and 4 (the dress of the girl). Something similar happens in the tonal harmony functions

of the musical phrase when observing the recursive occurrence of the main harmonic relationship, the dominant-tonic (T-D) and tonic-dominant (T-D) harmonic links.

Figure 5. Examples of common harmonic progressions. The Roman numerals represent the musical

chords. The square boxes are the fundamental harmonies (i.e., I = the tonic, V = the dominant, and IV = the subdominant degrees) and the rectangles represent the parallel harmonies (i.e., II = the supertonic and

VI = submediant degrees). The arrows show how the various chords are connected to each other.

A CASE STUDY: HARMONIC WALK In the previous section, a theoretical background on motion and musical structures has been presented and analyzed in defining a framework to understand the elements that trigger movements in musical interactive spaces. The aims of this section are (a) to link the formal characteristics of music to movement patterns derived from a user’s motions in such environments, and (b) to show how such structures shape human behavior during a user’s interaction with musical content. Harmonic Walk has been chosen as a case study because it is based upon one of the most


93

complex and structured features of Western music: tonal harmony. In the next subsections, some motion patterns derived from musical structures are presented, analyzed, and discussed, taking also into account the design optimization of the application. The Application’s General Outline Harmonic Walk (Mandanici, Rodà, & Canazza, 2016) aims at allowing users to experience highly structured musical features through full-body movements in space. It has been designed as a music-learning environment to be applied in teaching music harmony at various ages. Beyond the discovery of several important musical structures, such as chords and harmonic rhythm, the application leads the user to accomplish a complex musical task, such as melody harmonization, in an easy and handy way.9 This is achieved through the adoption of a very simple interaction modality: steps through which the user links the various interactive landmarks.

Related Work A system similar to Harmonic Walk is Harmony Space, designed at the Music Computing Lab of Stanford University (Holland, 1994). The Harmony Space interface is a desktop two-dimensional matrix of pitches that allows the performance of musical chords. The environment is rich and complex and, although it has been designed for learning purposes, it fits an expert-user’s level as well. More recent systems, such as Isochords (Bergstrom, Karahalios, & Hart, 2007) or Mapping Tonal Harmony,10 are also complex environments that require a high degree of knowledge and employ some representation of the harmonic space on a two-dimensional computer screen. Regarding the use of responsive floors for harmonic space representation,11 an example is provided by Holland himself, who tested a physical space extension of his Harmony Space by adding a floor projection and a camera tracking system (Holland et al., 2009). Harmonic Walk has been compared to rhythm games such as Dance Dance Revolution,12 where the user is trained to follow a musical input through visual stimuli. In this game, repeated patterns help movement memorization in a rather mechanical way, independent of the musical features. Harmonic Walk, on the other hand, fosters musical listening and selective entrainment by inviting the user to direct her/his attention to a precise musical target (e.g., harmonic rhythm, musical phrases, a song’s beat).

System Architecture Harmonic Walk employs a ceiling-mounted camera to track the presence and movement of a single user on the floor surface within camera range. Typically, the camera is mounted 3 meters above the floor, which results in a defined rectangular floor surface of 3 x 4 m. The system is composed of two software modules—one for motion data analysis and processing (the Zone Tracker application) and the other for sound production (the Max/MSP patch13) connected through the OSC protocol (Wright, 2005). The system combines motion data with a mask for surface division and for positioning interactive landmarks. As soon as a user enters the Harmonic Walk area, the system detects the zone occupied by the user and sends this information to the Max/MSP patch that provides the audio output.


94

Harmonic Walk Experimental Method A formal test was organized at the Barbarigo Institute in Padova, Italy, in December 2014, with the aim of measuring the power of Harmonic Walk in the field of musical education. The test was carried out with 22 high school students between 16 and 22 years of age, equally subdivided between musicians and nonmusicians, based on whether they were attending a music-based curriculum or a traditional academic curriculum. The popular song “Il ragazzo della via Gluck,” by the Italian singer Adriano Celentano,14 was chosen because of its clear harmonic structure. During the test, only the first phrase of the song was employed. This musical excerpt is composed of 11 harmonic regions built upon 3 chords all belonging to the same key.

The Three-stage Approach The ultimate aim of the Harmonic Walk application is to drive the user towards a tonal melody harmonization, which is a complex multifaceted task for the user. Melody harmonization can engage the user in at least the following three subtasks:

(a) detection of the harmonic rhythm of the melody, (b) knowledge of the harmonic space of the melody, and (c) choice of the chord sequence that can fit the melody.

To check the user’s level of awareness about these three aspects of melody harmonization, a three-stage experimental approach was conducted.

The high school students were tested individually in private sessions where only the music teacher and the test conductor were present. Students were provided with some written instructions about the task to be accomplished and with a short demonstration about the interactive space and the interaction modality. No previous information about the song used in the tests was given to the students.

First Stage: Detection of the Harmonic Rhythm of the Melody The spatial positioning of the interactive landmarks for the detection of the harmonic rhythm of the melody began with the analysis of the perceptual image of a melody by a user. A melody is perceived as a sequence of events (notes) organized on a timeline. The perception of melodic patterns and implicit chord sequences makes the user organize the various notes according to a metrical and harmonic frame (Povel & Jansen, 2002). In our research, this process produced a segmentation of the composition into different harmonic regions, which, in case of one key melody, were all part of the same tonality. To make the Harmonic Walk user aware of this process, we excerpted the song’s audio file where harmonic changes took place, and we assigned each harmonic region to a subsequent target landmark in the interactive space. Adopting the Zone Tracker surface mask depicted in Figure 6(a), the harmonic regions were laid one after the other along the surface’s borders (see Figure 7 for the tag position of the song excerpt used in the experiment). This representation addressed the idea of a conceptual integration between the music structure and the spatial organization of the interactive landmarks discussed above.


95

(a) (b)

Figure 6. The two masks for the floor division in the Zone Tracker application as applied to this Harmonic Walk research. From left to right: (a) a 25-squares zone partitioning, designed for the harmonic changes interaction, with a dark square designating the area activated by a user’s presence; and (b) a ring

with six areas for song harmonization, with the dark area showing the space activated by a user’s presence.

Figure 7. Visual tags of the harmonic regions sequence (white crosses) and of the six chords of the

tonality harmonic space (black crosses, with the fundamental roots, uppercased) and the parallel roots (lowercased) employed in the formal tests of Harmonic Walk. The arrows indicate the starting positions.

Moreover, in this stage of the research, the user did not need to worry about where to move

because the only possible direction was along the tagged path. In the test, the user was asked to link the harmonic regions by stepping to the next position in time with the harmonic change. If s/he was ahead of it, the audio fragments overlapped; if s/he was late, the song was interrupted. However, successful accomplishment implied that the user recognized the harmonic rhythm changes and that s/he was able to apply a selective entrainment to decide the exact moment for moving towards the next song’s excerpt in a seamless manner.

I

V

IV

ii

vi iii


96

Second Stage: Knowledge of the Harmonic Space of the Melody

The spatial organization of interactive landmarks for the exploration of the harmonic space and for melody harmonization derives in a straightforward way from tonal harmony theory (Piston, 1962). A tonality harmonic space is formed by three fundamental (I, IV and V degree) and three parallel (II, III and VI degree) roots. These six chords have meaningful spatial relationships, as shown in the tonnetz representation of Figure 8. To make this spatial disposition available to the user, we employed the circular surface mask depicted in Figure 6(b) and laid the 3 fundamental roots in one half of the ring and the 3 parallel roots in the other, following the same order as in the tonnetz representation (see Figure 7 for fundamental and parallel root disposition on the responsive floor). In the test, the user was asked to enter the ring and freely explore the harmonic space by trying to link the chord sounds to their spatial disposition. Each musical root in the tonality of C major was synthesized using four different wave shapes mixed to form a uniform sound. In this case, the problem of directing the step (i.e., where to move) and when to change to position may have been biased by many factors because both decisions were left to the user.15

Third Stage: Melody Harmonization

For this third task, we used the chord spatial positioning described above. The user was asked to sing the same song chosen for the first stage of experiment (sometimes with the help of the teacher) and to follow its harmonic rhythm in deciding when to move. Moreover s/he had to decide which chord sequence better fit the song’s melody (i.e., where to move) and to combine these two pieces of information in deciding the move. The required movement was rather complex and the timing was very strict. In this case, not only was selective entrainment necessary for focusing on the harmonic rhythm durations, but also predictive entrainment was useful to the subject in preparing the step direction. We assigned 5 minutes to each student to complete the harmonization task. If in this period of time the student was able to identify at least the first seven chord positions useful for the melody harmonization, the task was considered successful.

Figure 8. Tonnetz representation of the six roots of the C major tonality space, with the three

fundamental roots (solid lines) and the three parallel roots (dashed lines).


97

Results At the completion of the first stage of our research, to verify whether the subjects were aware of the harmonic changes that occurred in the first phrase of the song, we provided them with the written words of the song and asked them to underline the syllables where they remembered the harmonic changes happening. The song contained 10 harmonic changes, corresponding to 10 syllables. The average of number of correct harmonic changes detected was 3.6 for nonmusicians and 5.5 for musicians. As can be observed in Figure 9, there was a trend of decreasingly correct answers in the identification of harmonic changes as the song progressed. This could be related to memorization problems or to the fact that the latter part of the phrase contained more complex harmonies.

The second stage of the testing was an exploratory phase with no defined task to perform other than for the subject to investigate chords and their spatial setting within the interactive space. Our observations of users’ behaviors are reported below.

The third stage aimed to assess the subjects’ ability to identify harmonization. The number of subjects who could complete successfully the song harmonization task was just 1 among the 11 nonmusicians but 5 among the 11 subjects in the musicians group. These results show that results show that the simple use of Harmonic Walk, even without providing explicit information to the test subjects, is capable of introducing the students to a complex experience like that of melody harmonization leveraging only on their implicit knowledge of Western tonal harmony. A fuller report and discussion of the collected data and findings can be found in Mandanici et al. (2016).

Figure 9. Diagram of hits of the 10 harmonic changes of the first phrase of “Il ragazzo della via Gluck” by the Italian singer Adriano Celentano, as detected by the musician and nonmusician subjects. The number of hits decreased considerably for both groups from the beginning toward the end of the song’s phrase.

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10

SUBJECTS

HARMONIC CHANGES

Hits for each Harmonic Change

Non Musicians

Musicians


98

Harmonic Walk Experimental Observations Beyond the formal test session presented above, a preliminary experiment of Harmonic Walk was carried out in the spring 2014 in the elementary school of Paderno Franciacorta (Brescia, Italy). The aim of this test was to verify whether children could locate different chords scattered within the interaction space, remember these positions, and find one or more paths to link them (Mandanici, Rodà & Canazza, 2014). The test involved 50 children, aged between 5 and 11 years. In addition, the Harmonic Walk was tested in an informal experimental setup with adults during the Researchers’ Night at the University of Padova (Italy) in September 2014, as well as in several music classes in the above-cited elementary school in 2014. Although both the preliminary testing and the informal tests involved different types of music and processes to assess the subjects’ understanding of harmony, each was in line with the overall goals (i.e., harmonic rhythm interaction, exploration of the harmonic space, and melody harmonization) presented within the formal testing procedures.

Below, we report several observations about the users’ behaviors during the various experimental test sessions (preliminary, informal, and formal). These observations focus on the users’ movements in relation to the structure of the musical elements and to the synchronization mechanism required by the task.

Harmonic Rhythm Interaction The harmonic rhythm interaction required the user to make a step forward when s/he detected a harmonic change. In general, the resulting movement quality was stiffness; it occurred when users related musical harmony in a rather unnatural way. We believe this outcome is related to the musical quality of the song employed in the formal test described above. For example, in one informal test session, we employed instead the first five bars of Beethoven’s Symphony No. 5, where two harmonic changes are marked by a musical fermata (see Figure 10). At these two points (bars 2 and 5), users were facilitated both in the detection of the harmonic changes and in the step movement. In reality, the required synchronization at these two points is much simpler as a result of the fermatas, which allow a much longer release time as compared to typical synchronization times. This is an example of how the musical rhythmic structure can influence the harmonic rhythm interaction.

Figure 10. First five bars of the piano reduction of Beethoven’s Symphony No. 5. The long

release time of the two fermatas at bars 2 and 5 require a much simpler synchronization mechanisms with respect to a stricter pulsed time.


99

Exploring the Harmonic Space

For each Harmonic Walk test, we provided an exploratory phase (second stage of the formal test described above). The following behaviors were observed:

1. Running along the ring. During the informal tests with the elementary school children, the exploration of the harmony was created by tones being presented audibly as the children moved to the interactive landmarks along the ring within the presentation space. Children aged 5 or 6 years were attracted to the sound produced by the ring and simply ran continuously both clockwise and counterclockwise to hear the effect. This observation shows the great communicative power of interactive environments, which have a very strong influence on the younger subjects.

2. Walking along the ring. Adults and high school subjects, when asked to explore the environment, simply walked along the ring. This behavior expresses the influence of the shape of the chord disposition on the users’ behaviors. Harmonic progressions have hierarchical relationships that are not represented in the chord disposition. Thus, the user may assume that following the circle would lead to some useful musical information about the harmonic space, but soon realizes that it is not so.

3. Performing elementary harmonic progressions. Some older users, frustrated by the unsatisfying musical result of the simple circular walk, tried other exploratory strategies by linking only two or three chords at a time. This is a more fruitful approach in that it involves an important exploratory element: listening. While listening, the user can apply her/his implicit knowledge of tonal harmony, which can in turn drive her/his movements. Listening is especially important during the exploratory phase because the user needs to learn to synchronize her/his movements with what s/he perceives from the external environment. Thus, if some strong harmonic relationship is recognized, this allows her/him to stop at that point and repeat the route, thus reinforcing the gained understanding.

4. Exploring newfound progressions following a metrical structure. The harmonic progression path repetition makes the user increasingly confident about which path to follow in the balance of the test, thus allowing some metrical entrainment to emerge. Harmonic progressions organized into a metrical structure open the door to new creative behaviors because they allow new, richer, and original chord progressions to be performed. These can be the basis for new tonal music compositions completely originated from bodily movements.

Melody Harmonization

The melody harmonization tasks described above required the user to perform a true choreography, with a fluent and sometimes elegant but highly constrained motion among the various chord locations. Every user exhibited her/his own motion style while performing a song harmonization, as observed by Holland et al. (2009) in similar conditions. Indeed, it seems that, in spite of its motion constraints, the application was able to evoke in the users the long-term sensorimotor information that characterizes personal movement qualities. Further


100

observations during the tests showed that the ability of completing the melody harmonization task built upon musical memory (i.e., the subject’s ability to remember the song, usually expressed by her/his ability to sing it) and rhythmic, regular movements during the exploratory phase. Musical memory is an important guide because it provides users with handy, basic information to direct movement. Moreover, in all the tests, we observed that users who exhibited a controlled exploratory phase with slow, rhythmic movements were more likely to complete successfully the melody harmonization task.

DISCUSSION Harmonic Rhythm Interaction Harmonic Walk is based on a physical floor interface. As the name suggests, the user interacts with the harmonic regions and chords of an instrumental piece or song through her/his steps, which must be regarded as the only human–computer interaction modality in the Harmonic Walk environment. Kinematic patterns of the gait of an adult human male (Pietraszewski, Winiarski, & Jaroszczuk, 2012) report a stride cycle time (two steps) ranging from a minimum of 0.93 s for high speed gait to a maximum of 1.26 s for low speed gait, with a preferred stride time of about 1 s. These stride times correspond to a musical speed of 60 bpm (beats per minute) for two steps and to a speed of 120 bpm for the single step. Thus, these measurements could be considered the optimal step–beat speed interaction. In the interaction with the harmonic rhythm, the user is not worried about where to move, as s/he has in front of her/him a row of tagged positions, but rather when to move. What triggers the user’s step onward is the prediction of the upcoming end of the present harmonic region. But one of the difficulties of synchronizing the human step with the harmonic rhythm may be that this rhythm usually is much slower than the average stride speed. Imagining the period in Figure 3 as part of a song played at 100 bpm in a binary musical meter (two beats per bar), the harmonic region durations will be 1.2 s and 2.4 s, longer than the preferred average stride and, for the longest durations, even longer the lowest stride limit (1.26 s).

However, the embodied knowledge deriving from such step–harmonic rhythm coordination is a convincing experience of what harmonic rhythm is. Many studies have been done about the relationship between gestures and musical meaning, as documented for instance in Godøy & Leman (2010). From these studies and from personal observation, we derive the idea that the guitarist’s left-hand movements on the fretboard have exactly the same entrainment rhythm we are investigating in our research because the chord changes depend on this movement. We have noticed that when singing with guitar accompaniment, musicians commonly mark the points of harmonic change by head and body movements; the same points correspond to marked accents in the right hand’s strings percussion. This is a strong perceptual marker that has been observed to improve performance participation and enjoyment when playing music in a group. Also noticeable is that playing guitar involves the coordination between the left hand (harmonic rhythm) and the right hand (beat-based rhythmic pattern) and that this entrainment action links the lower level of rhythmic pattern organization to the higher rhythmic level of harmonic structure. This two-level rhythmic performance also could help in the situation—very frequent in popular music—when the change of the harmonic rhythm does not correspond to the phrases and


101

semiphrases organization, as shown in Figure 3, where the asterisk points out the two overlapping musical structures. The phrase change invites the Harmonic Walk user to mark it with a step, while the harmonic rhythm would require no change at all. Nevertheless, it was important to prepare the Harmonic Walk users to perform the harmonic changes through step interaction and to train them to move on the floor to accomplish the melody harmonization task. Using a simple gesture to mark the harmonic changes would not take into account that the harmonic changes must be triggered by occupying different interactive landmarks with precise spatial relationships, and that these landmarks must be reached through steps inside the circular chord mask. Exploring the Harmonic Space The knowledge of the harmonic space is a fundamental prerequisite for melody harmonization. What we asked from the user of Harmonic Walk is not theoretical knowledge but rather an embodied, tacit knowledge that could link her/his previously acquired harmonic experience with the chords s/he heard when occupying one of the six zones of the ring mask of the application. Thus, we posit that the user employs her/his perceptual skills to build a cognitive map of the tonality harmonic space by freely exploring the six chord zones. The freedom in this exploratory phase refers to the lack of external rhythmic constraint deriving from the audio feedback from the system.

The notable part of the exploratory movement analysis was that motion schemas are driven only by the user’s perception while moving in the environment. The path s/he drew during the exploration was like a journal of how her/his brain was working to build the cognitive map of the harmonic space and, consequently, many exploratory strategies may be found. One of these strategies is the experience of recursive harmonic progressions (see Figure 5 for some examples). The progressions can be of various lengths. Nonetheless, recursive harmonic progressions return always on the tonic chord, and this repetition—together with an iterative harmonic rhythm—makes it a kind of firm ground upon which to build melodic variations and refrains.16 The trajectories of the four harmonic progressions are depicted in Figure 11, which shows how they are all concentrated around the V and I degrees. This concentration is the expression of the dominant (V degree) and tonic (I degree) hierarchical role in the tonal harmony. These harmonic progressions reflect good movement patterns because repetition reinforces the perception of harmonic functions and the learning of the spatial relationships among the various chords. Moreover, they require that the path segments among the various chords positions are repeatedly covered and with regularity. Because these progressions are simplified patterns of more extended harmonic progressions, the embodied knowledge deriving from this practice is a very useful background for melody harmonization. Melody Harmonization To perform a good song harmonization with the Harmonic Walk system, the user not only needs to have a clear idea of when to move, but also where to go, just as a dancer following a previously arranged choreography. The information of where to go is embedded within the cognitive map the user builds during the exploratory phase, when s/he has to link the chord sounds with the chord locations. Nevertheless, to succeed in this task implies other important abilities as well, such as musical and spatial skills, path memory, physical awareness of beat, body


102

Figure 11. A spatial representation of four harmonic progressions on the Harmonic Walk interface:

I-V-I (solid line), I-IV.V-I (thin dashed line), I-vi-IV-V-I (bold dashed line) and I-IV-ii-V-I (dotted line). Their trajectories show the concentration of movement around the V and I degree.

movements control, confidence, gait fluency, and the capacity for getting engaged. Moreover, the best results were observed in the nonformal experimental contexts where Harmonic Walk was used during music lessons. Here the melody harmonization was the outcome of a collaborative work, that is, when a group sang the song and members of the group took turns in trying to harmonize it by jumping from one position to the other. Cooperation offers the advantage of sharing a user’s cognitive load of remembering and singing the song. Moreover, singing together with the group engages the user in the time constraints of the group’s singing. This helps the user in her/his effort in achieving the task and rewards her/him in case of melody harmonization completion.

CONCLUSIONS This paper presents a framework for the design and assessment of motion-based music applications, reviewing previous studies conducted in various fields, such as cognitive sciences and human–computer interaction. The framework has been applied to Harmonic Walk, a motion-based music application that uses an interactive space as its interface. Harmonic Walk Future Improvements The discussion presented above highlights some difficulties and unsolicited user reactions. Further, it offers suggestions for application design improvement and more useful utilization practices, some of which are presented here.

In designing entrainment into the harmonic rhythm regions, we used a type of forced entrainment strategy consisting of an abrupt interruption within the audio file when a certain harmonic region ended. Conceivably, a fade-out transition could have helped make the interruption less sharp but, on the other hand, a smoother interruption also might have made the


103

change of harmonic region less clear to the user. Nevertheless, the idea of processing the audio file to help the user predict the end of the harmonic region can be a good way to send the user a warning and to foster her/his listening attention. Various processing techniques could be employed, such as slowing down the playback speed, cross-fading the audio fragments, or using three-dimensional spatialized audio techniques.

We also discussed above that differences between the optimal stride rates and the average lengths of the harmonic regions could make the step synchronization unnatural and perhaps sometimes even clumsy. The problem could be solved by allowing the user to synchronize her/his steps at a quicker pace (i.e., with the song’s beat), marking the harmonic changes with a direction shift, with a path change, or with some other spatial arrangement. Any solution in this vein may lead to interesting choreographies that can enrich the embodied meaning of harmonic rhythm changes. Moreover, because this is a case of multilevel entrainment, other technical possibilities, such as multiuser tracking or gestural tracking, also could help.

The use of Harmonic Walk in music courses or edutainment installations should be considered a “musicking” activity (Rischar, 2003), that is, an activity where every involved person has a creative function, regardless the role s/he plays. Thus, if only one user is tracked by the system, the rest of the group may help her/him in many ways. In nonresearch applications, the entire activity requires the participation of a teacher or leader who must be able to identify the group tasks and to coordinate them. Some useful tasks could involve

(a) beating hands or percussion instrument to the pulse of the song, (b) singing the song, (c) marking the harmonic changes with hand claps or foot stumps, (d) making suggestions to the user regarding where to move and when to move during

the song harmonization, (e) imitating the movements of the user during the song harmonization, or (f) waving in the direction of the required chord.

Following the musicking principles, much music educational technology-enhanced work can be completed in an enjoyable way. For example, crowds of young people engage in public collaborative musical rhythm entrainment activities when dancing together in discos. This kind of shared scenario could be exploited also in educational activities.

Additionally, many harmonic learning systems employ dynamic visualization to help the user link musical chords one after the other (Johnson, Manaris, & Vassilandonakis, 2014). Flashing lights or flickering arrows could indicate to the user the position of the next chord or the direction to move. Whereas such visual tags could surely be a great help for the user, this way of accomplishing the task could become a somehow mechanical activity, requiring no cognitive effort at all. It is clear that a tradeoff must be found between leaving the user alone in front of the harmonization task and suggesting the right solution to her/him. One possible solution would be to offer the user not only a unique chord solution, but also a number of alternatives based on a probabilistic base. Employing some visual projections onto the floor and/or flashing at a rate synchronized with the song’s pulse could also help time coordination.

Finally, we noted above how rhythmic, regular, and slow movements during the exploration phase helped the users complete the melody harmonization task. However, it also may be useful to experiment with meaningful chord progressions, such as linking primary chords with their parallels or practicing the occurrence of harmonic progressions. To further the user’s exploration,


104

the system could play the role of a tutor, constraining the exploration speed, and suggesting the various possible chord progressions. Harmonic Walk Further Development The Harmonic Walk has been tested in various contexts and discussed by users at various levels (i.e., researchers, musicians, high school and elementary teachers). What follows is a summarized excerpt of ideas collected from these subjects for Harmonic Walk’s further development

At the current time, Harmonic Walk is grounded in very precise and limited harmonic rules that significantly influence the application’s design and the user’s interaction. The potential of employing the application with other types of music is an engaging consideration, one which could stress important commonalities among various musical genres. In particular, it is relevant to verify whether the design of Harmonic Walk can be applied in every domain with predictable and repeated patterns, similar to linguistic grammar. For instance, Harmonic Walk is based on a six-chord harmonic space (the ring), derived from the tonnetz, and adapted in the physical space. Although this representation is coherent with the relationships existing among the chords, its use was not found immediately clear to the users. Thus, a discussion and further investigation is necessary to understand if and how abstract concepts can be represented and acted in the physical space.

IMPLICATIONS FOR RESEARCH OR APPLICATION

The theoretical framework presented in this paper identifies three key points for the design of motion-based music applications: (a) the spatial positioning of interactive landmarks that expresses the meaning of a concept through spatial representations; (b) the user’s decision on where to move, related to the user’s spatial knowledge about the represented concepts; and (c) the user’s decision on when to move, which depends on the user’s ability to coordinate her/his motion to the timing of the selected musical event. This approach derives from the interpretation of the spatial and temporal relationships typical of the musical grammar, and thus it can lead to the development of a new type of music applications, characterized by a strong learning power and body involvement. Harmonic Walk offers a novel, creative approach to music understanding, as it allows people to experience the basic concepts of music composition that until now have been accessible only to professionals or skilled amateurs. This is made possible via the physical approach allowed by the Harmonic Walk environment. The harmonic rhythm is embodied in the time-constrained step, whereas the direction of the movements follows the position of the perceived harmonic changes. In a sense, the user play-acts the composition itself, linking her/his musical knowledge to her/his movements. This fully resonates with the pedagogical tradition of music education, which emphasizes the importance of practical experience before theoretical learning (Orff & Keetman, 1977). However, spatial representations with responsive floors and full-body interaction offer a much wider range of possibilities in that they can link the strength of embodied knowledge to the practice of abstract concepts through technology.

Moreover, the above described framework can be considered an overall strategy to model other human–computer interaction applications where spatial relationships and coordinated movement play a fundamental role, such as computer games. The movement


105

patterns resulting from bodily interaction in these environments are the true expression of the cognitive content conveyed by the application.

ENDNOTES

1. An interactive three-dimensional space produced by a full-body sensor like Kinect, is a user-centered

space with the sensor tracking the user’s torso and limbs joints; a little sensor like Leap Motion creates a sensor-centered space where the space formed by an inverted pyramid with the vertex corresponding to the sensor’s position.

2. Zone Tracker is a video application that tracks the user’s position. The video-analysis algorithm analyzes the input images in three steps. First the background is subtracted. Then the resulting black and white images are processed to obtain a well-shaped blob representing the user’s silhouette as seen from above. Finally the blob’s moves are tracked and its two-dimensional barycenter is calculated.

3. A fermata is a sustained note, chord, or rest whose duration is longer than the indicated value. 4. In tonal harmony, each key is represented by three fundamental harmonies: the I degree or tonic, the

V degree or dominant, and the IV degree or subdominant. These harmonies are called fundamental because they summarize all the tonality’s harmonic functions.

5. In the Gregorian chant notation, melodic cues are represented by neumes that are groups of notes tied together to indicate their reciprocal relationships and expressive meaning.

6. The Quadrille is a traditional dance, also called contraddanza or a country dance, popularized in the 19th century United States of America. It is characterized by a very precise geometrical disposition of the dancers in a double row or a square.

7. Cheraw dance (Mizoram, India) and Tinikling (the Philippines) are folkloristic dances characterized by the use of bamboo poles that are beaten on the ground and moved one against the other by two people in a rhythmic way. The dancers jump inside and outside the space between the poles in a highly coordinated way to avoid their feet clashing with the bamboo.

8. In a Schenkerian analysis, the ursatz (fundamental structure) corresponds to the deepest level of a tonal composition and to its most abstract form. The model is grounded on the harmonic linking of Tonic-Dominant-Tonic, with an arpeggiated bass line and a three-note step melody (fundamental line).

9. A video with an example of a collaborative song’s harmonization by a 9-year-old boy with his class group is available at https://youtu.be/c4ru468eqM0

10. Mapping Tonal Harmony Pro information is available at http://mdecks.com/mapharmony.html 11. The term “responsive floor” indicates an area where it is possible to track the presence and movement

of one or more users. Typically this can be done employing essentially two techniques: The first uses a system of sensorized tiles; the second is created through computer vision algorithms. By analogy, in this second case, the term responsive floor is used also if there are no sensors under the floor.

12. Dance Dance Revolution is a popular video game involving music and movement. In this consumer program, the user, moving on a sensorized platform, has to follow arrows that indicate the direction where s/he has to step. The movements are connected to various musical content.

13. Max/MSP is a visual programming language for audio and video production, algorithmic composition and signal processing written by Miller Puckette in 1980. Information is available at https://cycling74.com/

14. References about the Italian singer Adriano Celentano can be found at http://ilmondodiadriano.it/ 15. A video demonstrating the harmonic space exploration by a 9-year-old boy is available at

https://youtu.be/iQlYP5dztDY. This video effectively documents the harmonic space exploration in that the test used for high school students repeated the experimental phase with elementary school students.


106

16. Refrain is a term used both in literature and in music. It defines a repeated musical phrase, like a ritornello, and can be found in various musical styles and forms (e.g., popular songs, chorus, jazz, ancient vocal music).

REFERENCES

Baimel, A., Severson, R. L., Baron, A. S., & Birch, S. A. (2015). Enhancing “theory of mind” through behavioral

synchrony. Frontiers in Psychology, 6, unpaginated. doi: 10.3389/fpsyg.2015.00870 Benyon, D. (2012). Presence in blended spaces. Interacting with Computers, 24(4), 219–226. doi:

10.1016/j.intcom.2012.04.005 Bergstrom, T., Karahalios, K., & Hart, J. C. (2007, May). Isochords: Visualizing structure in music. In Proceedings

of Graphics Interface 2007 (pp. 297–304). New York, NY, USA: ACM. doi: 10.1145/1268517.1268565 Buxton, B. (1997). Living in augmented reality: Ubiquitous media and reactive environments. Retrieved May

28, 2016, from http://www.billbuxton.com/augmentedReality.html Carroll, J. R., (1955). The technique of Gregorian chironomy. Toledo, OH, USA: Gregorian Institute of America. Clayton, M., Sager, R., & Will, U. (2005). In time with the music: The concept of entrainment and its significance

for ethnomusicology. European Meetings in Ethnomusicology, 11, 1–82. Cohn, R. (1997). Neo-Riemannian operations, parsimonious trichords, and their tonnetz representations. Journal

of Music Theory, 41(1), 1–66. doi: 10.2307/843761 Corrigall, K. A., & Trainor, L. J. (2010). Musical enculturation in preschool children: Acquisition of key and

harmonic knowledge. Music Perception, 28(2), 195–200. doi: 10.1525/mp.2010.28.2.195 Drott, E. (2011). Lines, masses, micropolyphony: Ligeti’s Kyrie and the “crisis of the figure.” Perspectives of

New Music, 49(1), 4–46. doi: 10.7757/persnewmusi.49.1.0004 Enyedy, N., Danish, J. A., & DeLiema, D. (2015). Constructing liminal blends in a collaborative augmented-

reality learning environment. International Journal of Computer-Supported Collaborative Learning, 10(1), 7–34. doi: 10.1007/s11412-015-9207-1

Euler, L. (1739). Tentamen novae theoriae musicae ex certissimis harmoniae principiis dilucide expositae [An attempt of new theory of music deriving from the strongest principles of harmony clearly explained]. Petropoli, ex Typographia Academiae scientiarum.

Fauconnier, G., & Turner, M. (2002). The way we think: Conceptual blending and the mind’s hidden complexities. New York, NY, USA: Basic Books.

Friberg, A., & Sundberg, J. (1999). Does music performance allude to locomotion? A model of final ritardandi derived from measurements of stopping runners. The Journal of the Acoustical Society of America, 105(3), 1469–1484. doi: 10.1121/1.426687

Godøy, R. I., & Leman, M. (Eds.). (2010). Musical gestures: Sound, movement, and meaning. New York, NY, USA: Routledge.

Holland, S. (1994). Learning about harmony with Harmony Space: An overview. In M. Smith, A. Smaill, & G. A. Wiggins (Eds.), Music education: An artificial intelligence approach (pp. 24–40). London, UK: Springer.

Holland, S., Marshall, P., Bird, J., Dalton, S., Morris, R., Pantidi, N., Rogers, Y., & Clark, A. (2009). Running up Blueberry Hill: Prototyping whole body interaction in Harmony Space. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (p. 9398). New York, NY, USA: ACM. doi: 10.1145/1517664.1517690

Jacob, R. J., Girouard, A., Hirshfield, L., Horn, M. S., Shaer, O., Solovey, E. T., & Zigelbaum, J. (2008). Reality-based interaction: A framework for post-WIMP interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 201–210). New York, NY, USA: ACM. doi: 10.1145/1357054.1357089


107

Jetter, H., Reiterer, H., & Geyer, F. (2013). Blended interaction: Understanding natural human–computer interaction in post-WIMP interactive spaces. Personal and Ubiquitous Computing, 18(5), 1139–1158. doi: 10.1007/s00779-013-0725-4

Johnson, D., Manaris, B., & Vassilandonakis, Y. (2014). Harmoni Navigator: An innovative, gesture-driven user interface for exploring harmonic spaces in musical corpora. In Proceedings of the International Conference of Human–Computer Interaction (pp. 58–68). New York, NY, USA: Springer International Publishing. doi: 10.1007/978-3-319-07230-2_6

Jones, M. R., & Boltz, M. (1989). Dynamic attending and responses to time. Psychological Review, 96(3), 459–491. doi: 10.1037//0033-295x.96.3.459

Lerdahl, F., & Jackendoff, R. (1985). A generative theory of tonal music. Cambridge, MA, USA: MIT Press. doi: 10.2307/854101

Ligeti, G. (1966). Lux aeterna. London, UK: Edition Peters/Litoff. Mandanici, M., Rodà, A., & Canazza, S. (2014). The “Harmonic Walk”: An interactive educational environment to

discover musical chords. In A. Georgaki & G. Kouroupetroglou (Eds.), Proceedings of the 40th International Computer Music Conference and the 11th Sound and Music Computing Conference (pp. 1778–1785). Athens, Greece: National and Kapodistrian University of Athens.

Mandanici, M., Rodà, A., & Canazza, S. (2015). A conceptual framework for motion based music application. In the IEEE 2nd VR Workshop on Sonic Interactions for Virtual Environments (SIVE 2015; pp. 1–5). Piscataway, NJ, USA: IEEE.

Mandanici, M., Rodà, A., & Canazza, S. (2016). The Harmonic Walk: An interactive physical environment to learn tonal melody accompaniment. Advances in Multimedia, 2, 1–16. doi: 10.1155/2016/4027164

Montello, D. R. (2001). Spatial cognition. In N. J. Smelser & P. B. Baltes (Eds.), International encyclopedia of the social & behavioral sciences (Vol. 12; pp. 14771–14775). Amsterdam, the Netherlands: Elsevier. doi:10.1016/b0-08-043076-7/02492-x

Orff, C., & Keetman, G. (1977). Music for children (Vol. 2): Primary. New York, NY, USA: Schott Music Corporation. Phillips-Silver, J., Aktipis, C. A., & Bryant, G. A. (2010). The ecology of entrainment: Foundations of coordinated

rhythmic movement. Music Perception, 28(1), 3–14. doi: 10.1525/mp.2010.28.1.3 Pietraszewski, B., Winiarski, S., & Jaroszczuk, S. (2012). Three-dimensional human gait pattern. Acta of

Bioengineering and Biomechanics, 14(3), 9–16. doi: 10.5277/abb120302 Piston, W. (1962). Harmony. New York, NY, USA: W. W. Norton. Povel, D., & Jansen, E. (2002). Harmonic factors in the perception of tonal melodies. Music Perception, 20(1),

51–85. doi: 10.1525/mp.2002.20.1.51 Reber, A. S. (1989). Implicit learning and tacit knowledge. Journal of Experimental Psychology: General,

118(3), 219–235. Rischar, R. (2003). [Book review of Musicking: The meanings of performing and listening by C. Small]. Music

Theory Spectrum, 25(1), 161–165. doi: 10.1093/mts/25.1.161 Schenker, H. (1979). Free Composition (Der freie Satz, Vol. 3). New York, NY, USA: Longman. Strayer, H. R. (2013). From neumes to notes: The evolution of music notation. Musical Offerings, 4(1), 1–13.

doi: 10.15385/jmo.2013.4.1.1 Thaut, M. H., Mcintosh, G. C., & Hoemberg, V. (2015). Neurobiological foundations of neurologic music

therapy: Rhythmic entrainment and the motor system. Frontiers in Psychology, 5, Article 1185, 1–6. doi: 10.3389/fpsyg.2014.01185

Tillmann, B., Bharucha, J. J., & Bigand, E. (2000). Implicit learning of tonality: A self-organizing approach. Psychological Review, 107(4), 885–913. doi: 10.1037/0033-295x.107.4.885

Will, U., Clayton, M., Wertheim, I., Leante, L., & Berg, E. (2015). Pulse and entrainment to non-isochronous auditory stimuli: The case of north Indian alap. PLoS ONE, 10(4), Article e0123247. doi: 10.1371/journal.pone.0123247


108

Wright, M. (2005). Open sound control: An enabling technology for musical networking. Organised Sound, 10(3), 193–200. doi: 10.1017/s1355771805000932

Yoshida, T., Takeda, S., & Yamamoto, S. (2002, September). The application of entrainment to musical ensembles. Paper presented at II International Conference on Music and Artificial Intelligence (ICMAI), Edinburgh, Scotland.

Zanolla, S., Canazza, S., Rodà, A., Camurri, A., & Volpe, G. (2013). Entertaining listening by means of the Stanza Logo-Motoria: An interactive multimodal environment. Entertainment Computing, 4(3), 213–220. doi: 10.1016/j.entcom.2013.02.001

Authors’ Note All correspondence should be addressed to Marcella Mandanici Department of Information Engineering University of Padova Via Giovanni Gradenigo, 6 35131 Padova (Italy) [email protected] Human Technology ISSN 1795-6889 www.humantechnology.jyu.fi

ISSN: 1795-6889

www.humantechnology.jyu.fi Volume 13(1), May 2017,109–141

109

DESIGNING A COMPUTER MODEL OF DRUMMING: THE BIOMECHANICS OF PERCUSSIVE PERFORMANCE

Abstract: Becoming a competent musician requires significant practice, including rehearsal of various musical pieces. Complex sequences of musical notes and the associated bodily movements must be choreographed and memorized so that the human body can reproduce these sequences consistently. Such bodily movement occurs within the instrumental performance space, with some instruments, notably the drum set, requiring more bodily movement than most. Choreographed bodily movement in drumming is fundamental for producing the timbral and timing variations crucial in delineating human vs. computer percussive performance. Current computer models designed to simulate percussive performance focus on the cognitive aspects of performance or the musical structure to determine the simulation, while other systems focus on reproducing the physics of musical instruments. The focus of this paper is on the complexities of human movement in drumming, with a view toward proposing, as part of a larger research project, a background understanding and methodology for extracting empirical data from human performance for interactive computer-based percussive performance modeling applications. Keywords: percussion, performance, modeling, drums, biomechanics, computer music.

© 2017 John R. Taylor and the Open Science Centre, University of Jyväskylä DOI: http://dx.doi.org/10.17011/ht/urn.201705272520


John R. Taylor MARCS Institute

Western Sydney University Australia

and

Sydney Conservatorium of Music University of Sydney

Australia

Taylor

110

INTRODUCTION Considerable literature exists concerning the field of computational modeling of expressive music performance (Gabrielsson, 1999, 2003). This research includes a diverse array of approaches, owing to the complexity of human performance (Widmer & Goebl, 2004). Arguably, one of the more complex instruments to computationally model is the drum set because of the complexity involved in playing the instrument. In percussive performance, the interaction between the player and instrument is perhaps the most significant variable in timbre production. This interaction is manifested in different techniques, skill levels, musical knowledge and experiences, and the physical attributes of the performers themselves (e.g., height, body mass, fitness, etc.). The act of musical performance encompasses a variety of contributory aspects (Gabrielsson, 1999, 2003; Palmer, 1997), specific examples of which include the physiological (Fujii, Kudo, Ohtsuki, & Oda, 2009; Lee, 2010), cognitive (Dahl & Friberg, 2004; Laukka & Gabrielsson, 2000; Repp, 1999), technical (Dahl, Großbach, & Altenmüller, 2011), and musical (Repp, 1997), as well as both theoretical and empirical perspectives (Shove & Repp, 1995).

Several aims of music performance modeling have been identified in the literature, encompassing the design of interactive music performance systems, virtual music environments, and compositional software tools. One aim of performance modeling seeks to generate human-like computer performance; therefore, it is useful to consider how human performance is distinct from computer performance. In this research, the analysis regarding human and computer performance addresses particularly the context of modeling percussive performance on a nine-piece drum set comprising bass drum, snare drum, hi-hat, floor tom, low tom, medium tom, high tom, and ride and crash cymbals. Such a drum set configuration is typically used in rock, jazz, and pop music genres. Firstly, the empirical research into the physics of percussion instruments shows that a number of physical factors are involved in timbral variation, such as strike location, construction, material, and so forth (Fletcher & Rossing, 1998; Rossing, 2000; Taylor, 2015). Secondly, timbral variations in drum sounds are important to listeners’ overall perception of music (Rath & Wältermann, 2008). Because playing the drums is a time-sensitive endeavor, the human movement involved in percussive performance can be considered to be “chronemic movement” (Sutil, 2015), in which the qualitative determinations of speed, sustain, attack, or delay (Sutil, 2015, pp. 35–37) in musical and timbral qualities are directly related to instrumental interaction and trajectory control. This article presents part of a wider research investigation into the computational simulation of human percussive performance and presents discussion of relevant literature as a prequel for further empirical work. Why Is Modeling Human Performance on a Drum Set So Difficult? Drumming comprises a variety of set drum patterns and techniques that are learned and performed in different rhythmical and musical contexts, often in an improvisatory manner. To perform these drum patterns and techniques, the drummer must choreograph the human movement of the patterns within the biomechanical constraints of his/her abilities in order to execute them within the rhythmic and time constraints of the music. Each drum must be played optimally at all times, with the performer able to add nuances, such as gestural embellishments or timbral variations that could affect either the timbre or the timing, in each strike. Consequently, a drum performance can

Designing Computer Models of Drumming Performance

111

be regarded as multiple patterns containing choreographed sequences of human movement. A computer model of drumming therefore encapsulates the choreographed movements contained within a performance and the transitional movements between choreographies.

The main aim of this article is to deconstruct and discuss the key aspects of human movement that lead to timing and timbral imperfections in percussive performance on a nine-piece drum set. This analysis enables the identification of a methodology that can be used to analyze human percussive performance with the goal of creating a computer model that represents, musically, the continuum of percussive performance movement in the physical world. More specifically, this analysis identifies ways in which real-world drumming interactions can be captured and how a human might interact with a system that models that interaction. Such a computer model could be used in interactive systems, virtual music environments, and in compositional software tools. This article evaluates methodologies for measuring human movement in order to create a framework that utilizes interactive computer algorithms to simulate percussive performance.

This paper begins with a description of drum rudiments and drummers’ development goals that are fundamental to learning optimal movement and form in drumming. The paper then presents an analytical framework, based upon information processing systems and human motor control, with which to understand the underlying causes of performance variation, particularly regarding instrumental interaction and physical control. This will involve a summary review of the literature in the discipline of biomechanics and the subsequent application of these principles in relation to percussive performance of a nine-piece drum set. This article does not address the different options and timbral and acoustical effects of striking implements (see Halmrast, Guettler, Bader, & Godøy, 2010, pp. 204–207), nor is it intended to be an exhaustive discussion. Many specific aspects have been omitted, including the effect of batter head models on timbre (Henzie, 1960; Lewis & Beckford, 2000); the effect of disuniform tension; potential tonal evolution due to the age (and usage) of the head; tempo (Desain & Honing, 1993); feedback conditions (Brandmeyer, Timmers, Sadakata, & Desain, 2011; Dahl & Bresin, 2001; Pfordresher & Palmer, 2002), and temporal independence (Goebl, 2011). In addition, aspects such as style and genre which, with their obvious contextual performance differences, will not be discussed in detail. Drum Rudiments and Development Goals Drummers develop their technique by learning drum rudiments established by the international drum rudiment committee, part of the Percussive Arts Society. The rudiments currently consist of 40 techniques (Percussive Arts Society, 20141) that often are choreographed independently and have been derived from various musical styles to form a pedagogical method for learning percussion. This method is designed to provide an “orderly progression for the development of physical control, coordination, and endurance” (Carson & Wanamaker, 1984, p. 3). Although not explicitly defined, these development goals can be interpreted and summarized as follows:

Physical control, referring to the performers’ management of stick and instrument interaction, which comprises wrist and hand movement and arm control;

Coordination, referring to the strike accuracy and the performer’s ability to exert physical control over sequences of strikes in different locations; and,

Taylor

112

Endurance, referring to performer attributes, instrumental configuration, and the complexity of piece being performed.

Although these development goals can be considered independent of each other, there is considerable interdependence among the three. One example of this is where a performer has good stick and instrument management but poor coordination. The result is a drummer who could play rhythmic sequences and timbres correctly but not necessarily hit the drum in time. Choreographically, this could be attributed to a disconnect between the sequence design and poor motion control or form. Another example of independent development goals can be observed in a performer’s ability to maintain arm control and coordination during prolonged movements in complex percussive sequences. Control and coordination will deteriorate at varying rates depending on the endurance levels of the performer. Essentially these development goals are individually important to the successful execution of a choreographed movement and contribute towards the overall form of the movement and the sound of the performance. The relationship between the development goals is described in Figure 1.

There is no “magic spot” among these development goals because each drum rudiment requires a unique mix of the three components, depending on the percussionist’s current developmental stage and the demands of the choreographical context. Although these goals are fundamental to the development of a percussionist’s skill, obtaining an understanding of percussive performance by way of deconstructing principles of human movement from these goals is difficult due to the effect of environmental factors on skilled movements (Dahl, 2005). Such factors could include, among others, the effect of temperature and altitude on endurance, auditory feedback on coordination, and stick thickness on physical control. As a result, it is both difficult and impractical to account for all these independent variables.

Figure 1. A diagram outlining the interdependency among the three development goals in drumming.

Individually and collectively, the developmental goals impact performance. Adapted from Carson & Wanamaker (1984).


113

The reduction of independent variables in the analysis of human movement, extending to environmental variables, is not new. In fact the dimensionality of variables in understanding human movement has been the subject of investigation since Nikolai Bernstein first proposed the theory of the degrees of freedom (DOF) in 1967. He theorized that because there are an almost infinite number of ways a movement could be executed through the large network of muscles, joints, and cells in the human body, there are an infinite number of ways that muscles can achieve the different movements.

The control of the nervous system on the musculoskeletal system is highly complex: For any given movement, there are a high number of DOF. This complexity is illustrated during the activation of a single muscular element either in isolation or in any particular sequence (Bernstein, 1967). Thus, if the nervous system controls movement by controlling synergistic groups rather than individual muscles and joints, the number of DOF (and therefore the dimensionality of variables) is reduced (Turvey, 1990). Bernstein (1967) also suggested that sensory feedback from the environment interacts with the nervous system to reduce the number of DOF. Turvey (1990) substantiated the omission of environmental factors within the context of Carson and Wanamaker’s (1984) development goals for this framework by arguing that, “If the environment to which the movement system relates is interpreted as just another large set of variables, then the juxtaposition of an animal and its environment would amplify the problem of degrees of freedom” (Turvey, 1990, p. 940).

Juxtaposing environmental factors onto percussive performance would not only concern human movement and the number of DOF but would necessitate extending environmental variables to the vibrational behavior of each of the nine drums under investigation as well. Because the speed of sound increases with air temperature (Fletcher & Rossing, 1998, p. 70), a bigger picture emerges regarding the inherent difficulty in adequately applying several environmental factors as variables across the different themes noted in this article. In light of this, Turvey’s (1990) position will be considered to be the most appropriate view and, consequently, environmental factors will be considered outside the scope of this discussion.

THE ANALYSIS OF HUMAN MOVEMENT: A THEORETICAL FRAMEWORK In 1982, neuroscientist David Marr presented a tri-level hypothesis by which information processing systems could be analyzed. These levels of analysis can be summarized as follows (Marr, 1982, p. 25):

Computational level: What does the system do? Algorithmic/Representational level: How does the system do what it does? Physical level: How is the system physically realized?

Marr (1982) described how these three levels of analysis are not intrinsically dependent upon one another and that, in some circumstances, analysis can be achieved by using only one or two levels. The choice of analytical level is critical in correctly understanding certain systems. More importantly, Marr described how the computational level of analysis is essential in understanding certain phenomena, particularly where there are significant levels of abstraction between the understanding of a system and the computational representation. Examples of this include a priori understanding of the nature of biological or perceptual

Taylor

114

processes prior to computational representation, rather than by analyzing the computational representation of such process in a given computational environment (Marr, 1982, p. 27).

David Rosenbaum (2010), in his book on motor control, described how Marr’s three analytical levels of information processing systems also represent “the study of human motor control” (p. 4). At the computational level of analysis, Rosenbaum described how, during physical activity, animals and humans plan their movements using what he described as “implicit equations” (p. 5). These implicit equations are derived from Marr’s (1982) computational level, where a system must achieve a function whose representation is often described mathematically. However, for humans and animals, this refers to the mental representation of task to be performed. One example of this is the mental representation a rock climber has of a “dyno” (a jump or leap) to the next position. In the context of percussive performance, this can be a mental representation of an impending drum fill and the drum striking sequence following from the “current position.” Critically, the current position is spatiotemporally unique, thus requiring transitional or linkage movements between sequences. Rosenbaum (2010) noted that the computational level of analysis does not include the execution of the action, which is unsurprising considering the number of DOF.

In applying Marr’s (1982) second level, the algorithmic/representational level, Rosenbaum (2010) noted that a computer’s algorithms are designed to enable a system to undertake their functions with guaranteed success. In the natural world, movements operate in real time (analogous to runtime algorithms) without guaranteed success. As examples, a rock climber might not jump high enough to grab the next hold (and thus fall to the safety net below) and the drummer can hit the wrong instrument or strike the shell of the drum by accident. As Rosenbaum pointed out, each of these real time movements relies upon a procedure, and the person executing the action will draw upon behavior and cognition in order to execute and verify the movement, hence Rosenbaum’s extension of this term as the “procedural level” (Rosenbaum, 2010, p. 5).

Rosenbaum (2010) described the final level of Marr’s (1982) analysis, the implementation level, as the physical aspects of the movement. These biological elements are described by Rosenbaum (2010) as muscle operation and brain activity (e.g., a rock climber will use leg muscles to jump, stretched arms to grab the hold, and fingers and forearms to grip and maintain the hold). For the drummer playing a snare drum followed by a ride cymbal, muscle operation can include the fingers and hand for gripping the stick, adduction of the lower arm for the strike, followed by a lateral rotation and abduction of the arm to reach cymbal height. Such movement can be considered either a choreographed pattern or a transitional or linkage movement. These examples are highly simplified, as it is in this analytical level that the DOF problem is encountered.

Rosenbaum’s (2010) biological adaptation of Marr’s (1982) tri-level analysis provides a solid approach to understanding the movement process. If this three-stage analysis is undertaken in the context of Carson and Wanamaker’s (1984) development goals, it is possible to objectively evaluate existing research and literature on human movement, specifically for percussion. Furthermore, the bottom-up nature of the three analytical levels in relation to performing a drumming action allows for a more comprehensive and structured discussion. This recontextualization is described in Table 1.

Understanding the nature of percussive performance variation requires only the computational level of analysis to gain an understanding of the relevance of human performance on timbre and timing and to uncover critical aspects of human movement in physical performance. Although other additional aspects in the other levels contribute to performance variation, this article presents


115

Table 1. Three Analytical Levels Applied to a Drummer’s Development Goals. Adapted from Carson and Wanamaker (1984) and Rosenbaum (2010).

Level/Goal Physical Coordination Endurance

Computational Planning the control of The physical

movement

Instrumental interaction

Planning coordinated movements Coordinating

simultaneous multiple physical events

Multiple instrumental interactions

Planning movement for improving endurance Economy of movement

Procedural The behavioral and cognitive aspects of carrying out a physical movement, relating to Timbre

Timing

The behavioral and cognitive mechanisms for Measuring current

position Verifying next

movement

Anticipating next timbre/timing

The behavioral and cognitive aspects of improving endurance Performance

psychology

Implementation The physical aspects of carrying out a movement Muscle activity

Brain function

The physical aspects of coordinating multiple instruments Interlimb coordination Muscle activity

Brain function

Physical ways of improving endurance Training Warm up protocols

Performer impairments

rationales regarding why the majority of these are outside of the scope of this investigation due to their highly individual and highly subjective natures, as well as the challenges in adequately proving these.

The first aspect of the framework outside of the scope of this investigation is the behavioral and cognitive aspects of carrying out a physical movement (physical/procedural). This is because behavior and cognition are highly individual, as well as highly dependent on the context of the performance (e.g., genre). An important cognitive element of this analytical level and context includes sensorimotor synchronization (SMS), which is the rhythmic coordination of an action with a regular external event (Repp, 2005). As a result, the computational representation of SMS would be difficult to realize, and the empirical testing required for such a model is outside the scope of this investigation. For further reading on this subject, consider Fujii et al. (2010), Hove, Keller, and Krumhansl (2007), Repp (2005, 2006), Wing, Church, and Gentner (1989), and Wing and Kristofferson (1973a, 1973b).

Another area of the framework outside of investigative scope is the physical aspect of carrying out a movement (physical/implementation), particularly regarding muscle and brain activity. This particular area presents two separate problems. In terms of muscle activity, the most significant modeling challenge lies with the DOF problem and determining which classifiers and representative organizational systems of muscle activation to model. One such solution would be to use a single DOF as a representative for all similar movements in the model. In the case of a drummer, more than one DOF would need to be modeled to cover all

Taylor

116

limbs. In addition, determining the most appropriate DOF for the movement, and even the process of making such assumptions, will produce theoretical shortcomings (particularly for neurophysiologists). Modeling muscle activations also presents problems regarding the relationship between abstracted models of muscle movement and timbre production—a problem that also is found in modeling brain activity. Further reading on muscle activation and brain activity during performance is available in Fujii et al. (2009), Fujii and Moritani (2012a, 2012b), Gabrielsson (2003), and Todorov and Jordan (2002).

The behavioral and cognitive mechanisms associated with performance feedback (coordination/procedural) encompass a range of methods of feedback acquisition. These include auditory, visual, tactile, haptic, and kinesthetic, and combinations of one or more. Each of these individual types of feedback has different effects on cognitive and behavioral mechanisms and varies depending on the performance conditions. With so many combinations of feedback conditions and environmental variables, finding an appropriate representative model is difficult. Additionally, modeling specific effects of certain feedback conditions would have limited practical application. Therefore, aspects of performance are outside the scope of this research and the reader is directed to Brandmeyer et al. (2011), Dahl and Bresin (2001), Fujii et al. (2010), Gabrielsson (2003), Petrini et al. (2009), Pfordresher and Palmer (2002), and Pfordresher and Benitez (2007).

It was noted above that modeling muscle activity was challenging given the DOF problem, the high level of abstraction from timbre production, the timing of both muscle activity and brain function, and the selection of suitable organizational systems for modeling control and muscle activation. This problem is compounded when considering interlimb coordination as a physical aspect of coordinating the strikes of multiple drums (coordination/implementation), particularly in complex tasks such as rhythm production. In creating complex rhythms bimanually, task complexity between the hands (which include cooperative and disjointed tasks) together with the dexterity levels and handedness of the individual will affect the brain’s organizational control of the two hands. In the case of drumming, it is more likely to include leg control for operating the bass drum and hi-hat. This would result in a highly complex study with too many variables to allow for meaningful conclusions relevant to performance modeling. Further reading on this subject, however, is available from Bernstein (1967), Calvin, Huys, and Jirsa (2010), Iannarilli, Vannozzi, Iosa, Pesce, and Capranica (2013), and Kelso, Southard, and Goodman (1979).

Endurance is unique to individuals and can be increased with correct training. However, during performance, endurance can be affected by an individual’s level of physical exertion, which can be mitigated by designing sequences of movement that require less movement or that increase their economy of movement. Other behavioral and cognitive aspects of improving levels of endurance fit firmly within the realms of performance psychology, which are difficult to represent in a computational performance model. Similarly, the modeling of training and warm up protocols also is outside of the scope of this investigation in that they do not bring any direct benefit to the modeled system. No benefit would be gained by modeling a performer with an impairment, such as modeling a drummer with low levels of endurance, because the system would be designed with a level of performer obsolescence, resulting in poor playing after a period of time. Therefore computational, procedural, and implementation levels of analysis relating to endurance are outside the scope of this investigation. However, further reading is


117

available from Abernethy, Hanrahan, Kippers, Mackinnon, and Pandy (2005), Gabrielsson (1999, 2003), and Shaffer (1989).

Thus in the following sections, discussion will focus on physical movement, instrumental interaction, and bodily movement in the context of human movement in the physical world. The aim of this research is to identify a method for analyzing the critical elements of music performance movement for electronic representation in either an interactive music or a virtual system. It is worth noting that, although some aspects of the framework are specifically identified as being outside the scope of investigation, there are overlaps between some of the variables mentioned and aspects of performance that will be discussed in the following sections. Their inclusion within the discussion serves to highlight the complexity of percussive performance and demonstrates the wide reaching implications and importance of the discussion.

CONTROLLING INSTRUMENTAL INTERACTION Why is physical control so important? Striking an object with another object has two repercussions. Firstly, when the struck object produces sound, vibration in the stick travels through the fingers to the hand. In some instances, and depending on the force of the strike and the materials involved, this can extend into the arm. In severe cases, this can cause discomfort (e.g., using a metal bar to strike a large mass of solid metal with extreme force). Secondly, striking an object can cause the striking tool to be deflected away from the surface and, depending on the elasticity of struck materials, the level of deflection will be either minimal (e.g., a hard metal surface) or more significant (e.g., a membrane under tension). Because playing the drums requires striking many objects consisting of different materials, and striking them at different strengths, the amount of vibration experienced in the player’s body varies among the instruments and which, during drum set performance, is exacerbated by deflections of the striking implement caused by different elasticities in the struck surfaces, the angles of the initial strikes, and the strike forces across the individual components of the drum set. Strike location plays a significant role in modal frequency excitation, subsequently affecting the timbre of the drum. Moreover, because playing the drums often requires multiple strikes, it is important for timbral consistency that the drummer maintains physical control of the striking implement across a diversity of potential strike interactions.

Understanding how a performer maintains physical control of a striking implement is important in contextualizing how timing and timbral variations occur in a drumming performance. This information also is useful for developing a performance ontology in which the system either simulates the elements or the results of physical control or transitions into new states as a result of identifying embodiments of physical control as input parameters. This section provides a bottom-up approach to discussing and reviewing the literature concerning instrumental interaction, starting with stick contact and grip, stick rebounds, and preparatory strike movements, to coordinating bodily movement and drum strike trajectories across multiple drums. This approach facilitates a detailed discussion of the complex nature of percussive performance and helps in identifying emergent themes in human percussive performance movement and biomechanics.

Taylor

118

Stick Contact The first point of interaction between a drummer and the drum lies with the stick contact on the surface of the drum, particularly stick contact time. Billon, Semjen, and Stelmach (1996) found, during finger tapping exercises of accented beats, that finger contact time was greater than nonaccented beats. An investigation into stick contact times on a tom-tom by Dahl (1997a) found contact times to decrease with strike force. These stick contact times were measured electrically by using adhesive copper foil on both the surface of the drum and the stick, with data collected for different strength strokes (pp, mp, mf, and ff)2 at the center of the membrane (Dahl, 1997a, pp. 64–65). The results showed that contact time decreased in a nonlinear manner with increased strike strength, ranging from 8 ms to 5.5 ms for the four dynamic levels of strikes in the experiment. Dahl ruled out the surface material and vibrational reflections from the rim of the drum as contributing to this behavior by performing similar strikes on a softer surface (a carpet) and obtaining similar results; he also measured the reflected waves of the drum head on the stick with an accelerometer. Dahl found that the reflected waves were not strong enough to influence the stick motion, but did affect the stick’s bending mode at around 475 Hz.

These findings present an interesting paradox in accented playing and contact times. In some kinematic analyses of percussionists, Dahl found that drumming accents that alternated between hands (i.e., interleaved) were played with increased stroke height (Dahl, 2004). With a correlation between higher preparatory strike heights and striking velocity, including higher dynamic levels (Dahl, 2004, p. 768), Dahl reported that accented strikes tended to have lower stick contact times. In a direct comparison between tapping with a finger and a drumstick on a force transducer, Fujii and Oda (2009) found that there was little difference between tap speed and peak force variability between the finger and stick in 10 s tapping bursts among 17 drummers. However the authors noted that tapping with a stick produced shorter contact times, with a larger peak force and greater stability in the intertap interval, than finger tapping. The authors concluded that the stick “allows drummers to play drums powerfully and stably” (Fujii & Oda, 2009, p. 969). The authors also noted a difference in tap rate and stability between the left and right hand, with the nondominant hand being generally the weaker of the two.

Beyond the practical aspect of force and stability, the player can dampen the vibration of instrument by forcing extended contact with the drum head, thereby adjusting the timbre of the strike. However, due to the generally small contact times with the drum head, such actions must be preparatory and integrated into the strike (Dahl, 2005). The challenge faced by the player, particularly with higher striking velocities, is the deceleration of the drumstick when it makes contact with the membrane and the rejection of the drumstick when the membrane contributes towards the stick’s acceleration in the opposite direction (Wagner, 2006), referred to in drumming as rebound. In the case of a damping effect, the player must exert an opposing force greater than the accelerating force of the membrane and, in the case of a nondamping strike, the player must cease the downward force on the membrane to ensure no further stick contact once the initial opposing acceleration has subsided (e.g., a stroke that can rebound freely). In either case, the stroke is largely determined by the player’s grip on the stick.

The effect of stick grip on the sound characteristics of a drum was investigated by Dahl & Altenmüller (2008a, 2008b), who measured contact force, duration, and pre- and poststick velocity for two different types of grip: a normal grip, where the stick was allowed to rebound freely, and a controlled grip, where the player was asked to stop the stick as close as possible to


119

the membrane after the strike. The authors adopted an approach similar to Wing et al. (1989) in measuring the movement of the stick, index finger knuckle (Metacarpophalangeal or MPC joint), and wrist for both grip types. The authors found that more energy was transmitted to the drumhead in the controlled stroke, with higher peak force and lower contact durations. In addition, the constraining actions of the wrist and MPC joint in the controlled stroke produced a lower poststrike velocity. In order to identify the effect of these grips on the sound of the drum, listening tests were carried out and the normal stroke was considered to have a more full timbre as compared to the timbre of the controlled stroke. The authors noted that this was due to the longer contact durations dampening some modes of the drum but, more interestingly, they “appeared to have affected both the effective mass and possibly also the stick modes” (Dahl & Altenmüller, 2008a, p. 1494).

Different timpani stick grips were analyzed by Bouënard, Wanderley, and Gibet (2008) using motion capture techniques to simulate virtual timpani performance in character animation. This study included analysis of shoulder and elbow motion, which was subsequently simulated using a hybrid approach (Bouënard, Gibet, & Wanderley, 2009) combining inverse kinematics or IK (using equations to determine the joint positions and the position of the stick tip) and inverse dynamics or ID (using position velocity and acceleration to calculate forces and moments). The model was then extended further to include 11 joints with 33 DOF together with the calculation of stick contact information for modal synthesis (Bouënard, Gibet, & Wanderley, 2011), which produced accurate character animations of timpani performance. However this approach was limited to a single drum with the performer using only arms in playing a limited number of drum techniques. Extending this approach to a nine-piece drum set may be computationally impractical, owing to the number of IK, ID, and modal synthesis calculations.

Exclusive of the timbral variations created by the instrumental mechanics, producing an accent or a desired timbre requires preparation on the part of the performer. The performer must be able to adjust (loosen or tighten) his/her grip or adjust the looseness of his/her lower arm (the wrist and MPC joint) to change the interaction between the stick and the drumhead, thus producing variations in timbre. One of the key drivers for grip modification in drumming is to control the amount of rebound. However, as Dahl and Altenmüller (2008b) noted, a grip adjustment for controlling rebound should theoretically be done poststrike because the implicit equation in the preparation for physical movement affects the sound production of the stroke. Stick Rebounds Stick rebounds can have both a positive and negative effect on drumming, depending on the required loudness of the subsequent stroke (Dahl, 2003, p. 11). Furthermore, stick rebound is determined largely by the player’s grip, with looser grips allowing more rebound. In a pilot study on drumming sequences with interleaved accents, Dahl (1997b) found that the stick tip position immediately following an accented strike was heavily influenced by the rebound. Dahl noted that the tip height of the rebound following an accent is “above the optimal starting position for the following soft blow” (1997b, p. 5). In the study, the players’ dampened rebound was manifested in fluctuations in tip position following the accented strikes. This fluctuation resulted from the player exerting force on the stick in opposition to the upward acceleration and then reacting to a small overcompensation before returning to the typical stick motion (albeit with a higher starting position for an unaccented strike). Despite the increased

Taylor

120

(suboptimal) stick height, the time between the onsets of the unaccented strikes (also referred to as the inter–onset–interval, or IOI) appear unaffected, owing to an increase in strike velocity to counteract the height (despite the drummers in the study playing some preaccented strikes early as part of the planning process).

A later study by Dahl presented data addressing this issue, in which larger IOIs occurred in sequences played at softer dynamic levels and at slower tempi (Dahl, 2000, p. 229). In this study, Dahl concluded that the 68% drop in IOI range from ff actions at 200 bpm (beats per minute) to pp actions at 160 bpm was the result of weak rebounds from the softer strikes that, in turn, made “the playing more difficult to control” (Dahl, 2000, p. 232). Generally, notes in drumming can be accented using either a higher dynamic level or prolonged note durations. The former method for accentuation requires a higher preparatory movement. Despite this, Dahl observed that movement increase did not necessarily equate to a delay in the accented note. Instead some of the unaccented strokes following an accent were delayed, although this delay was not consistent. This lack of consistency suggests that rebound control stemming from the greater accented stroke preparatory movement that, when combined with the difficulty in controlling weak rebounds, could account for the “short term variations between adjacent IOIs” or “flutter” (Dahl, 2000, p. 228). This flutter ranged between 2% and 8% of the relative tempo of the subjects and was more noticeable at slower tempi.

In contrast to weaker rebounds, stronger rebounds are more conducive to player control. Furthermore, players can exploit the upward acceleration of stronger rebounds to achieve greater preparatory height with less effort (Dahl, 2000, p. 232). For the moment, ignoring the issue of stroke height apex control, the exploitation of rebounds requires significant planning and has some far-reaching implications on a player’s drum performance. A drum set consists of several drums often on multiple dimensional planes. For example, tom-toms (particularly those mounted on a bass drum) can be adjusted to different angles from the horizontal, depending on preference, their mounting mechanism, and size. Additionally, cymbals tend to be angled to avoid weakening the edge of the instrument from repeated strikes or chewing up the drumstick and to allow the player to strike the bell. From a practical viewpoint, these also can be positioned at any angle on the vertical plane relative to another instrument, depending on personal preference. Thus, the angles of deflection of the rebound can be more or less complementary to a subsequent stroke on another drum, depending on their relative positioning. Furthermore, the angle of deflection on the first drum relies upon the initial stroke angle. This is determined largely by the positioning and deflection angle of precedent strokes (if any), planar positioning of the drum relative to the player, and player posture. It is in this context that instrumental interaction has a significant impact on the execution of transitional movements. This is depicted in Figure 2. For clarity, the strike locations have been placed off center, although the principle applies to a centrally struck drum.

The horizontal drum on the left in Figure 2 is a simplified example of a typical rebound of a drum in a low horizontal plane, similar to the position of a snare drum; the downward stroke tends to be more vertical due to the player’s superior position. The stroke angle is assumed to be closer to the player and the rebound angle away from the player. The angled drum on the right provides an example of a drum closer to the vertical plane and positioned to the right of the player. This example shows the effect of a single strike in a drum fill, where the previously struck instrument was positioned to the left of the player (causing the strike trajectory as illustrated) and


121

Figure 2. An illustration showing a potential deflection angle on a horizontal drum (left) and an angled drum (right). Arrows indicate strokes (downward) and the rebounds (upward). The black arrows represent the three

dimensional Cartesian coordinate system to illustrate the rotational change of the drum head (not to scale). the rebound points towards the next drum target. In this example, the deflection angle is complementary to the following stroke. However, a deflection relies on a number of variables. Figure 3a shows a theoretical ideal of the angle of deflection relative to the strike trajectory. In practice however, the contact between the drum and stick can be forcefully increased during a narrow strike angle coupled with momentum (Figure 3b). In this example, the angle of deflection becomes wider, potentially reducing the complementarity to the subsequent stroke. The prolonged interaction of the stick and drumhead also can affect the timbre of the drum.

Many variables can affect the rebound angle and velocity, several of which already have been discussed. However, it was noted by Wagner (2006) that the rebound speed also depends on the tension of the membrane. Wagner’s experiments on the force, contact time, and acceleration of a drumstick at different membrane tensions demonstrated that an increase in stiffness affects the speed of the transversal wave propagation and internal reflection from the rim. At the same time, this also decreases the contact time in that fewer vibrational modes are excited due to a reduced force pulse (Wagner, 2006).

Drums come in different sizes and with different tensions, so a drum set contains variations in rebound behavior. Although Dahl’s (2000) experiments used two-headed drums, the investigations focused on player movement. Wagner’s (2006) experiments concentrated on the interaction between the stick and drum but used single-headed drums. There currently is no detailed literature investigating stick and cymbal interaction although, given the pivotal movement of a ride or crash cymbal on a stand, it can be assumed that there would be limited interaction with the stick as the cymbal moves away from the stick as a consequence of the downward force of the cymbal movement. In the case of a rebound that is in opposing trajectory to the subsequent strike, Dahl’s (2000) controlled striking experiment demonstrated that the player preemptively compensated for the rebound by using finger, wrist, and arm muscles to counteract the acceleration.

Rebounds also occur in the foot pedals associated with the bass drum and the hi-hat. In the case of a bass drum, the static strike location of the beater on the membrane ensures that rebounds manifest themselves consistently between strikes. The bass drum pedal mechanism also amplifies the rebound because the weight of the beater on the swing arm, combined with the torque from the movement of the foot pedal, tends to push the beater to its default “open” position. As a result, pedal control is important and muscle use is more constant. In the case of the hi-hat, the opposing force to the foot is in the counterweight of the upper hi-hat cymbal, thus

Taylor

122

Figure 3. An ideal deflection of a drumstick on a membrane (a) and the deflection of a drumstick

with an acute strike trajectory and momentum forced contact (b). releasing the pedal opens the hi-hat. Opening the hi-hat generally occurs less often than striking the bass drum in many genres of music so constant pressure is usually applied to the pedal with pressure release opening the hi-hat. The ratio of pedal to hi-hat movement can be adjusted but, in general, controlling this is much easier than controlling a bass drum. Rebounds do not always have an inherently positive effect on playing. Weak rebounds are difficult to control, and opposing rebound trajectories can require either more effort to control or quick reflexes for immediate control. For immediate control, the player must anticipate the rebound and/or modify the preparatory movement of the stroke. Preparatory Strike Movement In Dahl (2000), the comparison between the motion of a drumstick tip and the hand during an accented stroke showed that the hand moved upwards before the tip of the drumstick and that this leading hand movement occurred while the stick was still in contact with the surface of the membrane (Dahl, 2000). The maximum upward velocity of the hand was 2 m/s with a height of 50 cm above the membrane compared with 4 m/s for the tip and a height of 70 cm (Dahl, 2000). Dahl explained that the differential in upward velocity and height means that it is


123

not until the stick has passed its upper turning point [that] an actual force delivery may be applied by the wrist (or fingers) to increase the speed. The result is a “whiplash” of the tip of the drumstick but the motion of the hand is smooth, resembling a fishtail-gesture. This characteristic fishtail motion of the hand in the preparation and delivery of the accented blow is certainly used in other ways in drumming, like reaching a position on another drum far away in ample time. By starting the movement before the last note is finished the player gains time and thereby effort. While the hand and fingers still control the last stages of the present tone the lower and upper arm have already started to move in position for the next. (Dahl, 2000, p. 227)

The fishtail movement described by Dahl is characteristic of several of the findings by Kelso, Buchanan, and Wallace (1991) in which the sequence of strikes resembles prone in-phase (where similar muscles simultaneously contract) movements of the forearm to a metronome. These prone in-phase movements were found to produce a more curvilinear trajectory than antiphase movements (where similar muscles do not simultaneously contract), thus having implications for drumming performance. Firstly, Kelso et al. found that, at a certain metronome frequency, around 1.5 Hz, or 90 bpm, a participant starting in an antiphase manner spontaneously switched to an in-phase movement to keep in time with an increase in metronomic frequency. Kelso et al. noted that, prior to the automatic switch to an in-phase movement, the velocity (which was generally more stable with in-phase movements as compared to antiphase movements) became unstable with a sharp decrease of instability observed shortly after the phase switch. They also observed that, conversely, a participant starting in an in-phase position does not switch to an antiphase position with an increase in frequency. This one-way automatic phase transition suggests that the heterologous nature of the muscle activity in the antiphase movement is less economical, resulting in a decrease in consistency of velocity and, ultimately, comfort. Additionally, the prone in-phase hand positions coupled with the velocity stability allows drummers greater control of the stick-to-hand interaction.

On the subject of movement analysis, two distinguishing features exist between the findings of Kelso et al. (1991) and Dahl (1997b) related to hand movement at the lowest part of the movement (i.e., the stick and drum interaction) and the highest part of the movement (i.e., the fishtail motion at the top of the stroke height). Regarding the stick–drum interaction, when a player starts a movement early (i.e., from the moment of impact), it causes the player to be ahead of time, therefore reducing the need to reactively make inefficient movements. It also allows the player to take advantage of the existing lower and upper arm movement, which is important in instances where the subsequent stroke requires bodily rotation to achieve optimal positioning for the next preparatory movement. For example, this is particularly relevant in sequences involving drums positioned at distance from one another. The greater the distance between the drums, the further the body must move. At higher tempi, it becomes increasingly more challenging to make the movement in the required time.

The second component of the hand movement is the fishtail motion at the upper turning point of the tip. This upper turning point is subject to the least amount of force, and so it is easier to influence. In contrast to the bottom of the strike, there is no rebounding force so the player employs a fishtail motion of the hand to cause the tip to change direction quickly and to move with greater acceleration. Because playing the drums is an ongoing time-sensitive system, both of these two components are necessary for a player’s improved economy of movement.

Taylor

124

The overriding goals of these components draw parallels to Shaffer’s (1989) description of the motor geometry in piano performance:

Getting the fingers to the right locations on an instrument is important but only part of the motor task in playing. The performer can learn to shape the trajectories of movement so as to achieve timing of rhythm and variation of dynamic and tone quality with an economy of motor effort. (Shaffer, 1989, p. 383)

It is evident from both Dahl (1997b) and Shaffer’s (1989) description of musical performance that drumstick management comprises technical elements of playing the drums, particularly the control of rebounds and the control of stick at the height of the strike motions. Technical elements in drumming contribute toward accuracy in timbre production and timing control. Although Dahl (1997b) described variations in the overall motion among the participants (especially at varying skill levels), the curvilinear trajectory followed the findings by Kelso et al. (1991). Bodily Coordination One important concept of choreography that contributes toward motion and form is that of balance arrangement, particularly whether the body is symmetrical or asymmetrical, which is indicative of stability and equilibrium or irregularity and imbalance. Playing the drums requires both bilateral movement (both limbs moving in unison) and unilateral movement (one limb moving at a time). Although drumming can be considered symmetrical (mirrored) or asymmetrical, depending on the combination of individual drums being played (i.e., the context), the process is inherently asymmetrical owing to the configuration of the components of the drum set. This article discusses the effects of the inherently asymmetric environment and how a drummer responds to and uses asymmetry in designing drumming sequences.

Aruin and Latash (1995) investigated the effect of opposing bilateral fast movements on the shoulders (with and without load) of subjects standing on a force platform. They found that anticipatory postural muscle adjustments (APAs) in the trunk and leg muscles were made by the subjects to maintain balance, with adjustments increasing to a maximum when arms were moved in a forward or backward motion and decreasing to no APAs when moving the arm along the sides (i.e., the coronal plane). Furthermore, the authors found no significant difference in muscle adjustment as a result of additional load on the arms. These APAs were evident by changes in the subjects’ anterior, posterior, and vertical centers of pressure and gravity on the force plate prior to the movement.

In the case of drumming, it is quite common for the drummer to be in a seated position with much of the player’s weight supported by the seat. Consequently, the leg muscles play a lesser role in redistributing centers of force and gravity for an APA. The redistribution of weight using the legs is further complicated by their use in applying independent pressure to the hi-hat and bass drum pedals. Consequently, upper body stabilization is carried out by the trunk, specifically the erector spinae (ES) and rectus abdominis (RA), irrespective of the types levels of support in the legs (Aruin & Shiratori, 2003). These findings were supported by Santos and Aruin (2008), who also found that the lateral muscles contributed to upright posture control in feed-forward movements (i.e., movements relying on anticipatory correction), akin to feed-forward movements in drumming and where the level of muscle activation being is directionally specific. With both legs in a fixed position for operating the hi-hat and bass drum, a drummer’s


125

directional posture control is of great importance, particularly in controlling movements requiring axial rotation of the upper body.

Thus APAs in compound multijoint movements—especially those involving changes in direction (Holmes, 1939, pp. 17–19) such as bilateral fast movements of shoulders coupled with point-to-point axial rotation—are critical in maintaining postural stability. However, in addition to bilateral movements, a drummer’s arm movements often are unilateral, are not directly opposing, and are executed at different strengths and speeds relative to the location and distance between subsequent drums to be struck. Where a drummer has different maximum arm heights relative to the horizontal plane, as well as different maximum distances in arm reach required from the center of the torso between strikes, then postural control and stability also affects movement on the vertical (i.e., sagittal) plane. Thus, consequently, a hunched-over position is not conducive to playing strikes at greater heights. With this in mind, it is easy to imagine the variations in the centers of pressure and gravity on a player during the course of a percussive performance. In fact, Alén (1995) suggested similar links between movement and performance variations. In his analysis of the Cuban music genre tumba francesa, particularly a type of performance called a toque macota, Alén described how the large size of a Cuban bulá drum may have affected the performer’s stabilization, requiring torso movements that could contribute towards timing deviations.

Although there are vast differences between the drum set and the bulá, it is conceivable that Alén’s (1995) links also apply to playing the drum set. One theoretical view is that a performer mitigates these effects by maintaining a postural equilibrium, with extreme changes in postural stability countered by APAs stemming from performance planning and musical read-ahead, both of which can be linked to performance skill and having repercussions on musical gesture as a learned deviation.3 In summary, one general rule of drumming performance variation is that the greater the distance and angle of movement (relative to the torso) prior to the strike, the greater the inequality between the opposing reach angle and distance of the other hand, the greater the synchrony/asynchrony of the arm movements, the more complex the biomechanical and neurophysiological process and the increased likelihood of performance variation. Drum Strike Trajectory The trajectory of a drum strike is important in drumming to such an extent that drum strike trajectory was used as an important component in the compositional specification of Karlheinz Stockhausen’s composition Zyklus (1959). As described previously, rebound control can be used to affect the trajectory of the subsequent strike in a sequence of percussive hits. Between rebounds, the player must move the stick from one strike location to another at a speed sufficient for maintaining correct timing. The success of this aim is largely dependent upon trajectory, defined by Abend, Bizzi, and Morasso (1982, p. 331) as “the path taken by the hand as it moves to a new position and the speed of the hand as it moves along the path.”

In their study of hand trajectory to target, Abend et al. (1982) found that the majority of subjects who were asked, with no instruction, to move their hand deliberately to a target, opted for a straight line. With the shortest distance between two points being a straight line, one would expect movements with straight trajectories to have a shorter duration than curved trajectories to the same target. Although this was found to be true, movement duration also is dependent on speed, which Abend et al. found to be more irregular during curved trajectories.

Taylor

126

However, in cases where the average speed was low, even straight trajectories showed irregular speed patterns, suggesting greater difficulty in controlling the movement. In a performance context, a lower movement speed and, therefore, a lower strike velocity, will produce weaker rebounds. Thus, the interaction with the instrument in terms of rebound control and the movement between the strikes is harder for the player to control.

Regarding the irregular speed profiles of the curved trajectories in Abend et al. (1982), it was noted previously that the movement of a drumstick during a strike has curvilinear resemblances due to the phasing of muscle movements (Kelso et al., 1991). However, a connection between the two cannot be drawn because there were differences in planar movement in these studies. The participants in Abend et al. (1982) operated on a horizontal plane, compared to sagittal movements in Kelso et al., (1991) and compared to both sagittal and horizontal movements in Dahl (2000). Despite this, there was a correlation in the increased irregularity in hand speed relative to the antiphase nature of the angular velocity of the shoulder and elbow—in other words, a joint-focused dichotomy with parallels to Kelso et al.’s (1991) muscle synergies.

Drumming invariably uses multiple joints, each with different torques applied from the muscles that, in a multijoint movement, extend to the interaction of other joints and torques in the movement. In the case of multijoint movement, each joint will be subject to different velocity interactions at various points in the movement. Where a trajectory is changed midair and not using a rebound (e.g., at a higher preparatory stick height, as in Dahl et al., 2011), the joint torques will change depending on the new trajectory. Such a movement is subject to interactional forces during the planning and control of the movement—such as the Coriolis, centripetal, and reaction torques (Abend et al., 1982, p. 331)—although the effects of these forces change dynamically over the movement. Hollerbach and Flash (1982) observed such behavior in relation to a curved trajectory where “the velocity interaction torques in fact completely dominate the dynamics at the movement midpoint because the inertial torques go through zero as the movement switches from acceleration to deceleration and the arm is moving the fastest at this point” (Hollerbach & Flash, 1982, p. 76).

In the case of a single stroke, as measured in Dahl et al. (2011), the midpoint would be the arc at the peak of the preparatory movement. In some instances, a change in trajectory at this point would have three benefits. Firstly, this enables a greater preparatory stroke height for the next strike. Secondly, the greater height enables higher maximum acceleration and downward velocity. Thirdly, as a point with the least amount of inertial torque, the player can prepare for the joint torque of the next movement. Such torque control can mitigate timing variation.

In terms of accuracy, it has been found that the trajectory of aimed movement can be learned. These learned trajectory movements were demonstrated by Georgopoulos, Kalaska, and Massey (1981) during a study of aimed movements in Rhesus monkeys. They found that practice over a period of time reduced the mean variability of the trajectory towards a target, together with improved accuracy, irrespective of target location. The implication here is that a human drummer is likely to do the same using the drums as targets. However, as previously noted, drumming requires bilateral and unilateral arm movement, and humans can be either left handed or right handed. Each of these have been demonstrated to be contributing factors towards target accuracy (Garry & Franks, 2000), with increases in reaction time for bilateral strikes with targeting aimed by the weaker hand compared to unilaterally mirrored targeting.


127

The effects of this can be minimized through drum set configuration, with little impact on multijoint bilateral movement.

Although several factors can affect trajectory and control during percussive performance, the most significant factor occurs during multijoint movement, where joint torques impact not only the choice of trajectory but also the control and speed of the movement. In the case of drumming, sequences involving multijoint movements can often include multiple simultaneous planes of motion and axes of rotation. Such an action is illustrated in Figure 4, where a drummer’s movement is described between changes of strike location, from a strike on a snare drum to a strike on a crash cymbal.

In the example in Figure 4, during the movement of the right hand from the starting position (snare drum) to the crash cymbal, there is abduction and extension of the right shoulder on the frontal plane with a posterior axis of external rotation. There is also an elbow and wrist extension on the sagittal plane with a lateral axis of rotation. Assuming no movement to the left arm, then there is also a vertical axis of rotation of the trunk on the horizontal plane to allow the drummer to position the body for reaching the new target. Kinetically, each of these axes of rotation and movement in this multijoint sequence contain torque forces that affect the movement.

If the drummer in the figure had not included a strike at the crash cymbal but a repeat strike to the snare drum, there would have been minimal changes to the existing patterns of joint torque and muscle activation. Additionally, another drum located at the same height as the snare drum, but closer to the crash cymbal, would cause the drummer to make a trunk and shoulder rotation. However, because the drums are at a similar height, there would be less movement over the three planes. Therefore, movements spanning multiple planes of motion and axes of rotation are most likely to affect the movement of a drummer and, subsequently, the timbre and timing variations. Multijoint movements, such as those in Figure 4, are considerably

Figure 4. An illustration showing the typical movements associated with a change of strike location

by a drummer moving from a snare drum strike to a crash cymbal strike.

Taylor

128

problematic to model because there are 17 DOFs in movements of the shoulder, elbow, and wrist: 9 kinematic net moments and 8 dynamic with optimized muscle forces (Chadwick & van der Helm, 2003, p. 15).

This discussion presents some clear difficulties in modeling percussive performance from body, movement, and spatial perspectives. Firstly, stick management plays an important role in the interaction between the stick and the drum in the way that stick contact times can be influenced to alter the vibration of the drum (and the subsequent timbre). Similarly, stick grip influences the rebound of the stick from the drum, which has two effects on drumming: force contact dampening of a drum after a strike and positive and negative rebound use for subsequent strikes, particularly in sequences of drums operating at different angles and locations relative to the torso. Although in most cases stick control can be executed during the strike, due to time constraints much of the rebound and strike control is managed during preparatory movements.

During the downward motion of a strike, a curvilinear trajectory was observed in Dahl’s study (1997b); this can be explained by the phasing of muscle activity in the homologous muscle groups of the arm (Kelso et al., 1991). In-phase muscle activity produced greater arm stability and economy of movement, which is a contributory factor in stick control. At the apex of a strike, a fishtail motion was described (Dahl, 1997b), which further exploits the existing synergy between muscle activities by taking advantage of the upstroke to minimize additional muscle activity in the upper arm. In bimanual and unilateral arm movements, which are common occurrences during drumming, APAs were observed as a means to maintain postural stability. These involved small muscle movements that compensated for changes in force (e.g., changes in the center of gravity) resulting from arm extension. The effect of this, when in a seated position, is that the trunk is responsible for postural stability in the upper body. With more complex arm movements in drumming sequences, compared to the simple arm movements as studied in previous research, the potential need for constant postural anticipation and control was highlighted, particularly in arrhythmic unilateral strikes at nonopposing angles and at various distances from the torso.

TOWARDS A TEMPORAL MOVEMENT CONTEXT This study aimed to assess the current literature relating to human percussive performance on a nine-piece drum set. This was done in order to understand human movement and rehearsal as choreographed motion. The notion of choreography in the real-world context of drumming, as rehearsed sequences of movements, stems from the internationally recognized drum rudiments intended to develop percussionists’ physical control, coordination, and endurance.

In the virtual world context of interactive computer systems, a clear understanding of how these development goals relate to instrumental interaction, biomechanics, and human movement provides an opportunity to explore the interaction possibilities between human and machine. The tri-level Marr/Rosenbaum framework (Table 1) for analyzing information systems and motor control, applied through the lens of the development goals, facilitated a bottom-up analysis of real-world human interaction with drums. The significance of this approach from a human-to-machine interaction perspective is that real-world, bottom-level interactions (i.e., end effector interactions) can have a significant impact on any generated simulation (e.g., force


129

contact affecting the timbre of a drum by altering the drum’s vibrational characteristics). Such conditions may be either too difficult to mathematically represent or computationally too expensive, particularly across the nine individual components of a drum set. Understanding the nuances of real-world interaction provides for more informed decision making when designing interactive virtual systems and human interaction. Design options include substituting mathematical representations (algorithms or subsystems) of some interactions with similar level representations (algorithms or subsystems) of other interactions. The measurement of such interactions may allow design choices that defer interaction from a dynamic mathematical representation to a sensor-based human interface (e.g., measuring force contact duration on a computer-enabled surface), thus reducing computational overhead while simultaneously maintaining real-world interactive authenticity.

The hierarchical nature of a bottom-up analysis allows each real-world interaction to be contextualized within a larger set of movements. In the virtual world, this is equivalent to merging two subsystems to form a larger complex system representative of a more abstract function. From an interaction perspective, this may mean deriving a force contact duration from a series of assumptions about the current state or the context of the system. Such an abstraction could include the representation that stick control is more difficult with weaker rebounds, thus weaker striking leads to longer durations of force contact. However, as each interaction is abstracted to a larger set of movements, human interaction with the system becomes more abstract. Consequently, in designing an interactive system that simulates human percussive performance, there are trade-offs in deferring simulation functions to either horizontally integrated subsystems or abstract layers with regards to the level of similar real-world human interaction with the system. This analytical framework provides a unique way of investigating human percussive performance while concurrently analyzing computational aspects relevant to the system design. The convergence of these two paradigms manifest themselves in system interactivity and how the system represents real-world movements that are inherently both compromised and unique, depending on the vision of the system. Representing Human Movement It is clear that significant issues exist in using the biomechanical considerations of the human body during percussive performance as a method of generating both performance context and in algorithmic control of representative computational musical output. Fundamentally, the main problem in modeling drum set performance is that it is predominantly asymmetrical: The performer’s arm movements (e.g., reach distance, height, and angle) are often unequal, and the rhythmic striking of these can be irregular. The inequality of arm location and irregularity in drumming constantly changes the joint torques and the force interactions that affect trajectory control, movement stability, and postural stability, which subsequently affect strike control, strike accuracy, rebound control, and stick management, ultimately causing variations in timbre and timing. This problem is compounded by an almost infinite number of combinations of movements between Cartesian strike coordinates during drumming and, if one takes into account the DOF problem, there is an extreme abundance of potential system representations. Such an abundance of potential representations would be hard to implement computationally; yet, the selection of a smaller number of representatives is difficult to justify theoretically. As Abend et al. (1982) noted, there would need to be an inverse kinematic transformation of the Cartesian-to-

Taylor

130

joint coordinates and then, using inverse dynamics, the joint torques would need to be calculated. This has significant implications for both the selection of particular variables that would form the basis of any computational model and on the way in which these variables are represented.

One way of representing human body movement in percussive performance would be to examine a specific movement and identify the most likely used DOF in the joints activated during that movement, such as the methods used by Bouënard, Gibet et al., (2011). This would require investigating the effect that each individual joint can have on the overall movement, including the selection of multiple DOFs and subsequent joint angles on the outcome of the movement. Such an approach would allow for an assessment of whether a variation in DOF at a joint closer to the instrument has a greater impact in producing biomechanical errors than variations in DOF at joints closer to the torso. In addition, preferences and/or trends in planar movement for each joint for a given movement could be identified, together with the impact of these planar preferences on biomechanical error. One way of computationally representing this approach is to design an algorithm that uses weighted probability to calculate the likelihood of a selected DOF or angular movement in a given joint. An example of a method of representing this computationally would be a Gaussian distribution of values to represent a joint angle (e.g., shoulder) and a Markov chain to determine the next selected joint angle (e.g., elbow), and so on until a joint angle value is determined for the wrist. At this point, a movement assigned a unique identification number could be used to trigger a predetermined timbral or temporal variation to represent the level of biomechanical error in the movement (as compared to a theoretical ideal).

Determining the probabilities of joint variation within movements would require significant analysis of multijoint, multiplane movement and would require also measuring a quantifiable error from the various movements. Furthermore, deciding which movements to investigate can be problematic in that their relative importance is highly subjective. In addition, identifying preferences or trends in movement at joint level may require significant sample sizes and may generate large quantities of data, particularly should the three axis planar movement be measured at high frequencies. Finally, a link between joint variation and performance variation would need to be quantified and would require multiple methods of analysis, for example, correlating data from joint movement with audio to identify the trends in performance variations associated with combinations joint values. Exactly how the many combinations of joint values represent performance variation also is critical in reproducing human percussive performance in a computer environment, as it relates to a method of controlling one or more aspect of musical parameters, such as timbre or timing.

Another potential method lies in the representation of the drumming techniques by creating an algorithm that represents the DOFs associated with a particular drum rudiment. A skilled drummer will have a standard repertoire of drumming techniques at his/her disposal; so it may be possible to assign various combinations of movement to a given technique that then generates a musical output that closely resembles that technique. However, the execution and application of these techniques will differ across performers and performances, notwithstanding the stylistic differences of the performed music. As a result, the process may produce disjointed sounding performances because the selected techniques are inappropriate, unusual, or humanly impossible for the given musical or performance context. Of course, it may be possible to concatenate algorithmic representations of techniques to form a coherent performance, but that depends upon whether the techniques have unique muscle and joint activations that are reproducible and relevant. The most significant challenge in this approach is identifying and empirically measuring


131

these multiple techniques within a performance and determining a unique movement value. The difficulties associated with collecting joint information as described above are suddenly increased when more variables are introduced into the performance context. It is clear that, in designing a system that simulates human percussive performance, the sheer number of biomechanical and performance considerations pose significant challenges in computationally representing any meaningful or situationally specific interaction. Therefore, it is useful to consider the biomechanical considerations and performance context at a lower level of detail and in the context of playing the drums. So what do we know at this point?

One broader view that can be taken for the purposes of modeling percussive performance is that large multijoint movements operating on multiple planes of motion are more likely to generate performance variations for two reasons. Firstly, a percussive performer playing the drum set is inherently constrained by his/her number of limbs regarding how many instruments can be struck simultaneously. For example, a nine-piece drum set has 64 potential combinations of simultaneous instrument strikes using only the two hands. With the feet fixed in position, the main areas of movement lie in the upper body and torso, which relates to the complexity and equality of bimanual drumming. Secondly, because the number of drums limits the combinations of arm movements, the complexity of the movement is largely affected by prior arm location. Collecting Data from Human Movement The literature discussed in this article present various methods for obtaining observational and empirical data from human percussive performance. Despite varying research aims, these studies reveal important insights into real-world human interaction with drums. In order to virtualize this human–drum interaction, various methods can be exploited for system design and control. The task then is to determine whether the system should be event-driven (i.e., software that changes behavior in line with an event), data-driven (i.e., software whose embedded data controls the flow of the program), or a combination of the two.

An event-driven system would respond to human interaction, such as playing a typical commercially available electronic drum machine in which electronic drum pads measure the strike force and play a sampled drum sound in response. One key consideration of this approach is to ensure that the interaction between human and machine accurately simulates real-world, stick-to-drum interaction. However, most modern electronic drum pads account for this need. In fact, most modern electronic drum sets have begun to incorporate different zones into the drum pads in order to trigger different timbres, thus mirroring the action of physical drums. Obtaining a measurement of contact duration ubiquitously from the drum pad may be useful for filtering a triggered sound to simulate membrane dampening. Pressure sensors mounted in or below the drum stool (similar to the force platform used by Aruin & Latash, 1995) could be used to control additional timbral parameters, although how the center of mass relates to meaningful system output would depend on system representation. Consequently, such functions would produce limited meaningful additional interaction. A completely event-driven system such as an electronic drum kit has two drawbacks. Firstly, it relies on the human user to interact with the system and will only produce sounds relative to the skill level of the user. Secondly, with music being a time-sensitive task, system responses from human interaction would need to be extremely low latency. This may not be possible, depending on the speed of the performance and the number of events that need to be handled.

Taylor

132

A data-driven system could use data captured from actual human performance in order to create an embedded database or for use in real-time interaction. One method of capturing human performance uses infrared cameras and sensors, as demonstrated by Dahl (2004), Dahl et al. (2011), and Kelso et al. (1991). This approach also was used by Bouënard, Gibet et al. (2008, 2011) to capture timpani performances to create a motion database. Although this approach addressed some limitations in physics-based modeling of performers, Bouënard, Gibet et al. noted that the instrument could obstruct the infrared markers. In relation to a nine-piece drum set, this is a significant limitation in that a typical drum set has components at various points around the performer. In addition, Bouënard, Wanderley, Gibet, and Marandola (2011) described limitations in capturing nuances in performance, such as stick grip. With this in mind, real-time control of a percussive system using infrared cameras and sensors would need to take place in an environment devoid of obstructions because, from a human–machine interaction perspective, a significant portion of the stick-to-drum interaction is lost, such as the rebound and force contact.

The descriptions above provide the polar exemplars of event-driven versus data-driven systems, with each having completely distinct aims and outcomes. The event-driven system with the electronic drum kit is typical of a performance system, while the data-driven system with the motion capture is reminiscent of virtual character animation (Bouënard, Gibet et al., 2011). Interactive systems that employ a combination of event-driven and data-driven methods include interactive computer-generated performance tools and electronic composition tools that render a performance based on human interaction. Such systems require a larger amount of abstraction on the data side complemented by human interaction to trigger events and computer state changes. In the case of abstracting data, several methods are available for creating a representation of an aspect of performance. One method of empirical data collection that could help to identify the levels of movement in a drumming performance is the video capture of multiple performances by different performers with the comparative analysis of the movement level in the video across the performers. It also is possible to attach accelerometers to the performer’s hands to measure the amount and direction of their acceleration.

In addition, audio data could help to identify the extent of performance variation by enabling a temporal analysis of performance events and comparing these with elements of the video and sensor data. This could yield information regarding the temporal stability of performance. Although this methodology would facilitate a more generic representation of percussive performance, much information can be obtained from this multimethod approach. Firstly, any empirical performance data obtained could be used in a data-driven model. Secondly, it is possible to infer more generic rules surrounding the use of multiple combinations of instruments and avoid the need for generating multiple variables to cater to the DOF problem. Finally, this approach is more practical because broader observations can be made from a relatively fewer number of participants than would be required to calculate the median joint angle averages for multiple percussive techniques. Therefore, by identifying complex bodily movement in an instrumental performance space, algorithmic logic can be created that can simulate the performance context that forms the fundamental logic of a system that controls the levels of variations simulated in a computer model of drumming. This approach supports rethinking the choreography of performance and its role in designing computer-simulated human interaction, as well as rethinking empirical movement data to contribute towards new concepts of computer generated rhythm systems.


133

Sound generation and instrumental representation are important components of any performance modeling system. With a variety of techniques available for system-modeling consideration, the main considerations are computational overhead, expressivity of the synthesis, and accuracy of the representation (Kahrs & Brandenburg, 1998). The complexity of accurately synthesizing a nine-piece drum kit comprising membranophones and idiophones places some physical modeling synthesis techniques firmly out of scope, particularly when considering the computational overhead and time sensitivity of the system. Consequently, sound generation is more efficient when the computational overhead is transferred to decision making, database matching, and sound playback, as opposed to calculating complex equations and resythesizing the sound at run time. Therefore, using a comprehensive sample database to sonically represent the instruments would augment the realism of the simulation by allowing timbral variations to be linked to inferred representations of performance. Towards a Theoretical Model Adopting a physical-based approach presents two levels of conceptual representation of the system. The first relates to David Marr’s (1982) representational level, whereby the relationships between the samples in a database conceptually represent an instrument. The second level further abstracts performance and presents a more contextual understanding of the variables affecting the relationship between the samples by inferring a relationship between the instrumental representations themselves. This is described in Figure 5, a simplified diagram showing the context of two instrumental representations.

Holistically, the instrumental representations should be part of a larger conceptual construct related to performance context. With this in mind, a theoretical model is presented in Figure 6 that shows a performance model derived from information in the sample data, augmented by representations of performance context.

Figure 5. Intrainstrumental performance context. Pulse-code modulation samples of strikes on a single instrument represent only one occurrence of the performance. Therefore, it is necessary to use the sonic

content of each sample to provide a contextual representation of the instrument, by inferring a relationship between the spectral features of each sample.

Taylor

134

The link between the instruments and the performance context is characterized by the interaction between the performer and the instrument, which is general to all instruments and comprises both intra- and interinstrumental interactions but is unique to the drum set owing to the sheer variety and number of components. The link between instruments and the sample database is an inherent feature of a chosen pulse-code modulation sampling technique and is driven by the diversity of acoustical behaviors of the instrument’s components. However, it is important that the feature extraction and classification methodologies in the sample database are informed by the performance context. Such focus ensures that the parameterization and control of the sample database is consistent with human performance, particularly because the sample database is critical to ensuring an accurate representation of instrumental performance in the model. In addition, performance context is not implied by the presence of a sample database alone. Therefore, the control of the sample database in the performance model must be informed explicitly by the performance context, at both intra- and interinstrumental levels, in order to conform a human performance on the control paradigm.

Figure 6. When using multiple instruments, the intrainstrumental context (described in Figure 5) must be

considered in relation to other instruments (Interinstrument performance context) through the lens of trajectory control, movement stability, etc. Both the intra- and interinstrument context contributes towards

the performance model and towards any novel compositional frameworks.

CONCLUSIONS AND FURTHER WORK This work forms part of a larger research project into the computer simulation of human percussive performance. This article, as a prequel for further empirical work, has presented a theoretical framework for evaluating the key aspects of choreographed percussive movements that are employed by drummers during performance. The discussion and analysis presented in


135

this article forms the basis for an experimental methodology that will inform the analysis of empirical data to be collected.

Therefore, this study set out to investigate and elucidate the fundamental aspects of human movement in real-world drumming on a nine-piece drum set that lead to variations in timing and timbre during a percussive performance. The goal of this study was to explore the ways in which real-world interaction with drums could facilitate the design of an interactive system that modeled human percussive performance and to understand how a human might interact with such a system. The aims of this study were achieved by examining the literature relating to human movement and percussion in the context of a drummer’s developmental goals and in the context of analyzing motor control and information processing systems.

The most obvious finding of this study is that a nine-piece drum set is a physically demanding instrument that involves complex instrumental and biomechanical interactions that must be continuously controlled and, in some cases, preemptively controlled. Given the almost infinite variety of combinations of movements across all limbs, and musical outcomes of human drumming movements, it is not feasible to exhaustively model every performance variable. Consequently, the first task of this article was to address ways of reducing the number of variables. This was achieved by exposing the links between the fundamental attributes of choreographed movements in drumming and how they are understood in the context of information processing systems. This led to the proposal of a theoretical framework based upon Carson and Wanamaker (1984) and Rosenbaum (2010) that drew together two key conceptual levels of human movement: direct physical control and coordinated movements, both of which are highly choreographed. By focusing on literature directly relevant to these two concepts, the scope of this empirical research lies in the disciplines of human movement and percussive performance. By discussing this research through the lens of information processing systems, computer modeling, and biomechanics, the literature reviewed has contributed to an in-depth understanding of human movement in drumming. In addition, the many empirical approaches used in the literature, particularly research that has sought to empirically measure direct physical control and coordinated movement in drumming, were viewed in the context of computer modeling. Consequently, a theoretical model was presented that combines an inferred physical context with a physics-based musical context.

The physical context is present in many of the studies described in this article, with movement being classified into two types: microlevel movements, such as stick contact, grip, and control, and larger localized movements involving multiple joints for rebound control and trajectory modification. These two movement types are inextricably linked because microlevel movements, such as grip modification, can impact larger localized movements, such as rebound control, and larger localized movements, such as trajectory modification, can affect stick contact. Both types of movement directly influence instrumental interaction in the physics-based musical context because they are contributory factors in the control of the timbre and timing variations and the embellishments associated with human percussive performance. Understanding the relationship and effect of microlevel movements and larger localized movements on the musical context provides a preliminary model for guiding experimental setup in reconciling the computational constraints of representing an almost infinite number of variables in human movement while maintaining accurate modeling of percussive performance.

The studies discussed in this article employed a range of empirical approaches to capture microlevel and localized elements of human percussive performance, including audio, video, and

Taylor

136

sensor data. Although some of the biomechanical research investigated multilimb movement, much of the music performance research concerned performance on a single drum rather than across several drums. When microlevel and localized movements are played across multiple drum set components, simultaneous and sequential contexts add an extra, significant layer of complexity. Simultaneous micro- and local-level movements activate complex biomechanical interactions that, depending on the task complexity and direction, may include agonist and antagonist groupings. This may force implementation of separate stick-management techniques and distinct trajectory plans that require anticipatory postural muscle adjustments in the trunk for bodily coordination. Sequential microlevel and local-level movements also activate complex biomechanical interactions during transitional or linkage movements as the stick is moved from one drum to another. Although the biomechanical interactions occurring in sequential movement are similar to simultaneous movement, the interactions relating to trajectory, speed, and rotation are the most important. These factors are less important when a performer is moving between two adjacent drums as compared to two opposing drums. However, further studies could empirically measure such instrumental interaction in the sequential movement of opposing drums. The implication of this is the possibility that a complex system of representation could emerge from relatively simple interactions, although, owing to computational constraints, these may not be satisfactorily simulated.

One of the more significant barriers in adequately capturing drumming performance are the inherent difficulties in capturing a completely seamless stream of drum performance data with infrared sensors, owing to how the drum set is configured around the performer. Yet it is possible to employ combinations of different performance data—such as audio, video, and other sensors (e.g., accelerometers)—and such an approach may allow for variations in real-world applications, while simultaneously presenting opportunities for exploring various system interactions.

Real-world applications using mixed data collection methods could provide more usable data to advance understanding of percussive performances for future computer-based system design or assessment algorithms For example, a real-time interactive teaching system, based upon an electronic drum set, could calculate the timing stability of a drummer (e.g., is the performer consistently playing in time) and offer suggestions for alternative sequences in order to maintain control. In such a system, the human interaction would be similar to that of real-world drumming. Another application could be a real-time interactive improvisational system that intelligently generates timbral variation based upon embedded performance data controlled by strike location on a multizone interface. Such a system also would mimic real human interaction.

However, a system that infers both timbral and temporal variation would need to abandon a real-world-based human interaction to avoid applying concatenating real-world and virtual temporal variations. To simulate timbre and timing variations, such a system would need to be data driven in order to contain a representation of both simultaneous and sequential movement, as well as be event driven in order to dynamically modify the levels of timbral and temporal variation depending on user input. Using a mixed methods approach to data collection, and by focusing on global-level contextual parameters, the computational overhead could be reduced by deferring system control to fewer decision making algorithms with larger databases of micro-level and local-level variables. This approach would free up system resources to concentrate on providing an accurate synthesis by making use of a large database of high-quality samples to improve the representation of the instrument. The goal then would be to ensure contextual consistency in these timbral features and stability in the model’s performance with the additional control of relevant musical parameters.


137

IMPLICATIONS FOR THEORY OR APPLICATION This article has analyzed human percussive performance in the context of computer–based system design in order understand the key variables that make human percussive performance distinct from computer-based percussive performance. Investigating these variables has uncovered a number of studies into biomechanics that have significant bearing on the human percussive performance. Bringing these studies to light in a systematic bottom-up way affords practitioners of the art of drumming to perhaps develop a greater understanding of how human drummers interact with their drums, how the interaction is manifested, and the musical consequences of the interaction. Furthermore, by understanding how humans are biomechanically and psychologically predisposed to certain behavior, for example, when automatically swapping to in-phase tapping or a human’s propensity for curvilinear over straight trajectories, the practitioner may also direct personal development toward improving choreographed sequences as learned skills. In fact, the innovative or avant-garde practitioner may wish to oppose such existential human behavioral and biological traits by developing a new system for percussive performance that runs contrary to the literature discussed herein. The discussion of the literature may also be of relevance to teachers, either developing their pedagogical approaches or considering widening their professional competencies by providing a comprehensive review of key literature as a starting point for further reading.

From a computational perspective, this article has discussed literature that utilized various methods for obtaining empirical evidence from human movement, with a view to understanding specific behavior or creating computer simulations and models. In this article, the evaluation of empirical evidence in the context of human drumming provides greater clarity on those aspects of human performance that should be considered irrelevant for computational modeling, given the constraints in computing, the complexity of the human activity/function/behavior, and the impact of end result. Therefore, the model presented in this article is of particular interest to researchers or system designers who are developing novel compositional tools that aim to simulate drumming on computers and who are considering adopting one or more of the variables inherent in percussive performance.

ENDNOTES 1. Drum rudiments were initially created by the National Association of Rudimental Drummers in

1933 and subsequently expanded from the original 26 rudiments by the PAS. In addition, the 40 rudiments are considered to be a work in progress, thus enabling future development (Carson & Wanamaker, 1984).

2. The musical abbreviations are as follows: pp or pianissimo meaning very soft; mp or mezzo-piano meaning moderately soft; mf or mezzo-forte meaning moderately loud; and ff or fortissimo meaning very loud.

3. Musical gesture is closely associated with the behavioral and cognitive aspects of carrying out a physical movement (physical/procedural analytical level). Consequently, gesture is outside of the scope of this study. However, the reader is directed to Godøy and Leman (2010) for further information.

Taylor

138

REFERENCES Abend, W., Bizzi, E., & Morasso, P. (1982). Human arm trajectory formation. Brain, 105(2), 331–348. Abernethy, B., Hanrahan, S. J., Kippers, V., Mackinnon, L., & Pandy, M. (2005). The biophysical foundations

of human movement (2nd ed.). South Yarra, Victoria, Australia: Palgrave Macmillan. Alén, O. (1995). Rhythm as duration of sounds in Tumba Francesa. Ethnomusicology, 39(1), 55–71. Aruin, A. S., & Latash, M. L. (1995). Directional specificity of postural muscles in feed-forward postural

reactions during fast voluntary arm movements. Experimental Brain Research, 103(2), 323–332. Aruin, A., & Shiratori, T. (2003). Anticipatory postural adjustments while sitting: The effects of different leg

supports. Experimental Brain Research, 151(1), 46–53. Bernstein, N. (1967). The co-ordination and regulation of movements. Oxford, England: Pergamon Press Ltd. Billon, M., Semjen, A., & Stelmach, G. E. (1996). The timing effects of accent production in periodic finger-

tapping sequences. Journal of Motor Behavior, 28(3), 198–210. Bouënard, A., Gibet, S., & Wanderley, M. (2008). Enhancing the visualization of percussion gestures by virtual

character animation. In G. Volpe & A. Camurri (Eds.), Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 38–43). Genova, Italy: Casa Paganini.

Bouënard, A., Gibet, S., & Wanderley, M. (2009). Hybrid motion control combining inverse kinematics and inverse dynamics controllers for simulating percussion gestures. In A. Nijholt, A. Egges, & H. van Welbergen (Eds.), Proceedings of the International Conference on Computer Animation and Social Agents (CASA’09; pp. 17–20). Amsterdam, the Netherlands: Springer-Verlag. Retrieved from https://hal.archives-ouvertes.fr/hal-00408255

Bouënard, A., Gibet, S., & Wanderley, M. (2011). Hybrid inverse motion control for virtual characters interacting with sound synthesis. The Visual Computer, 28(4), 357–370.

Bouënard, A., Wanderley, M. M., & Gibet, S. (2008). Analysis of percussion grip for physically based character animation. In E. Ruffaldi & M. Fontana (Eds.), Proceedings of the International Conference on Enactive Interfaces (pp. 22–27). Pisa, Italy: Edizioni ETS.

Bouënard, A., Wanderley, M. M., Gibet, S., & Marandola, F. (2011). Virtual gesture control and synthesis of music performances: Qualitative evaluation of synthesized timpani exercises. Computer Music Journal, 35(3), 57–72.

Brandmeyer, A., Timmers, R., Sadakata, M., & Desain, P. (2011). Learning expressive percussion performance under different visual feedback conditions. Psychological Research, 75(2), 107–121.

Calvin, S., Huys, R., & Jirsa, V. (2010). Interference effects in bimanual coordination are independent of movement type. Journal of Experimental Psychology: Human Perception and Performance, 36(6), 1553–1564.

Carson, R., & Wanamaker, J. (1984). International drum rudiments. Van Nuys, CA, USA: Alfred Publishing Co. Inc. Chadwick, E. K., & van der Helm, F. (2003). Shoulder biomechanics. In P. Milburn, B. Wilson, & T. Yanai

(Eds.), Proceedings of the International Society of Biomechanics 2003 Congress XIX (pp. 1–30). Dunedin, New Zealand: ISB/University of Otago.

Dahl, S. (1997a). Spectral changes in the tom-tom related to striking force. KTH Royal Institute of Technology, Department of Speech, Music and Hearing, Quarterly Progress and Status Report, 38(1), 59–65. Retrieved from http://www.speech.kth.se/prod/publications/files/qpsr/1997/1997_38_1_059-065.pdf

Dahl, S. (1997b). Measurements of the motion of the hand and drumstick in a drumming sequence with interleaved accented strokes: A pilot study. KTH Royal Institute of Technology, Department of Speech, Music and Hearing, Quarterly Progress and Status Report, 38(4), 1–6. Retrieved from http://www.speech.kth.se/prod/publications/files/qpsr/1997/1997_38_4_001-006.pdf

Dahl, S. (2000). The playing of an accent: Preliminary observations from temporal and kinematic analysis of percussionists. Journal of New Music Research, 29(3), 225–234.


139

Dahl, S. (2003). Striking movements: Movement strategies and expression in percussive playing. Unpublished master’s thesis, Royal Institute of Technology, Stockholm, Sweden.

Dahl, S. (2004). Playing the accent: Comparing striking velocity and timing in an ostinato rhythm performed by four drummers. ACTA Acustica United with Acustica, 90(4), 762–776.

Dahl, S. (2005). On the beat: Human movement and timing in the production and perception of music. (Doctoral dissertation; Record No. TRITA-TMH 2005:5). Stockhom, Sweden: KTH Publikationsdatavas DiVA.

Dahl, S., & Altenmüller, E. (2008a). Motor control in drumming: Influence of movement pattern on contact force and sound characteristics. Proceedings of Acoustics 2008 (pp. 1489–1494). Paris, France: European Acoustics Association.

Dahl, S., & Altenmüller, E. (2008b, June 30–July 4). Movement and grip influence the contact force and sound in drumming. Paper presented at the 155th meeting of the Acoustical Society of America, Paris, France.

Dahl, S., & Bresin, R. (2001). Is the player more influenced by the auditory than the tactile feedback from the instrument? In M. Fernström, E. Brazil, & M. Marshall (Eds.), Proceedings of the COST-G6 Workshop on Digital Audio Effects (DAFx-01; pp. 194–197). Limerick, Ireland: University of Limerick.

Dahl, S., & Friberg, A. (2004). Expressiveness of musician’s body movements in performances on marimba. In A. Camurri & G. Volpe (Eds.), Gesture-based communication in human–computer interaction: Lecture Notes in Artificial Intelligence (Vol. 2915; pp. 479–486). New York, NY, USA: Springer. doi: 10.1007/978-3-540-24598-8_44

Dahl, S., Großbach, M., & Altenmüller, E. (2011). Effect of dynamic level in drumming: Measurement of striking velocity, force, and sound level. In Proceedings of Forum Acusticum (pp. 621–624). Aalborg, Denmark: Danish Acoustical Society.

Desain, P., & Honing, H. (1993). Tempo curves considered harmful. Contemporary Music Review, 7(2), 123–138. Fletcher, N., & Rossing, T. D. (1998). The physics of musical instruments (2nd ed.). New York, NY, USA Springer. Fujii, S., Hirashima, M., Kudo, K., Ohtsuki, T., Nakamura, Y., & Oda, S. (2010). Synchronization error of drum kit

playing with a metronome at different tempi by professional drummers. Music Perception, 28(1), 491–503. Fujii, S., Kudo, K., Ohtsuki, T., & Oda, S. (2009). Tapping performance and underlying wrist muscle activity of

non-drummers, drummers, and the world’s fastest drummer. Neuroscience Letters, 459(2), 69–73. Fujii, S., & Moritani, T. (2012a). Rise rate and timing variability of surface electromyographic activity during

rhythmic drumming movements in the world’s fastest drummer. Journal of Electromyography and Kinesiology, 22(1), 60–66.

Fujii, S., & Moritani, T. (2012b). Spike shape analysis of surface electromyographic activity in wrist flexor and extensor muscles of the world’s fastest drummer. Neuroscience Letters, 514(2), 185–188.

Fujii, S., & Oda, S. (2009). Effect of stick use on rapid unimanual tapping in drummers. Perceptual and Motor Skills, 108(3), 962–970.

Gabrielsson, A. (1999). The performance of music. In D. Deutsch (Ed.), The psychology of music (2nd ed.; pp. 501–602). San Diego, CA, USA: Academic Press.

Gabrielsson, A. (2003). Music performance research at the millennium. Psychology of Music, 31(3), 221–272. Garry, M. I., & Franks, I. M. (2000). Reaction time differences in spatially constrained bilateral and unilateral

movements. Experimental Brain Research, 131(2), 236–243. Georgopoulos, A. P., Kalaska, J. F., & Massey, J. T. (1981). Spatial trajectories and reaction times of aimed

movements: Effects of practice, uncertainty, and change in target location. Journal of Neurophysiology, 46(4), 725–743.

Godøy, R., & Leman, M. (Eds.). (2010). Musical gestures: Sound, movement, and meaning. New York, NY, USA: Routledge.

Goebl, W. (2011, June 26–July 1). Temporarily out of sync: Momentary temporal independence of a solo voice as expressive device. Paper presented at the European Acoustics Association, Forum Acusticum, Aalborg, Denmark.

Taylor

140

Halmrast, T., Guettler, K., Bader, R., & Godøy, R. (2010). Gesture and timbre. In R. Godøy & M. Leman (Eds.), Musical gestures: Sound, movement, and meaning (pp. 183–211). New York, NY, USA: Routledge.

Henzie, C. (1960). Amplitude and duration characteristics of snare drum tones. Unpublished doctoral dissertation, Indiana University, USA.

Hollerbach, J. M., & Flash, T. (1982). Dynamic interactions between limb segments during planar arm movement. Biological Cybernetics, 44, 67–77.

Holmes, G. (1939). The cerebellum of man. Brain, 62(1), 1–30. Hove, M., Keller, P., & Krumhansl, C. (2007). Sensorimotor synchronization with chords containing tone-onset

asynchronies. Attention, Perception, & Psychophysics, 69(5), 699–708. Iannarilli, F., Vannozzi, G., Iosa, M., Pesce, C., & Capranica, L. (2013). Effects of task complexity on rhythmic

reproduction performance in adults. Human Movement Science, 32, 203–213. Kahrs, M., & Brandenburg, K. (1998). Applications of digital signal processing to audio and acoustics. New

York, NY, USA: Kluwer Academic Pub. Kelso, J. A., Buchanan, J. J., & Wallace, S. A. (1991). Order parameters for the neural organization of single,

multijoint limb movement patterns. Experimental Brain Research, 85, 432–444. Kelso, J. A., Southard, D., & Goodman, D. (1979). On the nature of human interlimb coordination. Science,

203(4384), 1029–1031. Laukka, P., & Gabrielsson, A. (2000). Emotional expression in drumming performance. Psychology of Music,

28(2), 181–189. Lee, S. (2010). Hand biomechanics in skilled pianists playing a scale in thirds. Medical Problems of Performing

Artists, 25(4), 167–174. Lewis, R., & Beckford, J. (2000). Measuring tonal characteristics of snare drum batter heads. Percussive Notes,

38(3), 69–71. Marr, D. (1982). Vision: A computational investigation into the human representation and processing of visual

information. Pacific Grove, CA, USA: Brooks/Cole Publishing Company. Palmer, C. (1997). Music performance. Annual Review of Psychology, 48(1), 115–138. Percussive Arts Society (PAS). (2014). PAS international drum rudiments. Retrieved from

http://www.pas.org/resources/education/Rudiments1/RudimentsOnline.aspx Petrini, K., Dahl, S., Rocchesso, D., Waadeland, C. H., Avanzini, F., Puce, A., & Pollick, F. E. (2009).

Multisensory integration of drumming actions: Musical expertise affects perceived audiovisual asynchrony. Experimental Brain Research, 198(2-3), 339–352.

Pfordresher, P., & Benitez, B. (2007). Temporal coordination between actions and sound during sequence production. Human Movement Science, 26(5), 742–756.

Pfordresher, P., & Palmer, C. (2002). Effects of delayed auditory feedback on timing of music performance. Psychological Research, 66(1), 71–79.

Rath, M., & Wältermann, M. (2008). Exploring the perceptual relevance of inherent variability of drum sounds. In R. Kronland-Martinet, S. Ystad, & K. Jensen (Eds.), Computer music modeling and retrieval: Sense of sounds (pp. 303–312). New York, NY, USA: Springer.

Repp, B. H. (1997). The effect of tempo on pedal timing in piano performance. Psychological Research, 60(3), 164–172.

Repp, B. H. (1999). Detecting deviations from metronomic timing in music: Effect of perceptual structure on the mental timekeeper. Perception & Psychophysics, 61(3), 529–548.

Repp, B. H. (2005). Sensorimotor synchronization: A review of the tapping literature. Psychonomic Bulletin and Review, 12(6), 969–992.

Repp, B. H. (2006). Musical synchronization. In E. Altenmüller, M. Wiesendanger, & J. Kesselring (Eds.), Music, motor control and the brain (pp. 55–76). Oxford, England: Oxford University Press.


141

Rosenbaum, D. A. (2010). Human motor control (2nd ed.). Burlington, MA, USA: Elsevier. Rossing, T. D. (2000). Science of percussion instruments. Republic of Singapore: World Scientific Publishing

Company. Santos, M. J., & Aruin, A. S. (2008). Role of lateral muscles and body orientation in feedforward postural

control. Experimental Brain Research, 184(4), 547–559. Shaffer, L. H. (1989). Cognition and affect in musical performance. Contemporary Music Review, 4(1), 381–389. Shove, P., & Repp, B. (1995). Musical motion and performance: Theoretical and empirical perspectives. In

J. Rink (Ed.), The practice of performance (pp. 55–83). Cambridge, England: Cambridge University Press. Stockhausen, K. (1959). Nr. 9 Zyklus für einen Schlagzeuger [Cycle for a Percussionist]. London, England:

Universal Edition. Sutil, N. S. (2015). Motion and representation: The language of human movement. Cambridge, MA, USA: MIT Press. Taylor, J. R. (2015). Ontology of music performance variation (Doctoral dissertation; Record No. 2123/13279).

Sydney, Australia: University of Sydney. Todorov, E., & Jordan, M. I. (2002). Optimal feedback control as a theory of motor coordination. Nature

Neuroscience, 5(11), 1226–1235. Turvey, M. T. (1990). Coordination. American Psychologist, 45(8), 938–953. Wagner, A. (2006). Analysis of drumbeats:Interaction between drummer, drumstick and instrument (Master’s

thesis). Stockholm, Sweden: KTH Royal Institute of Technology. Retrieved from https://www.nada.kth.se/utbildning/grukth/exjobb/rapportlistor/2006/rapporter06/wagner_andreas_06047.pdf Widmer, G., & Goebl, W. (2004). Computational models of expressive music performance: The state of the art.

Journal of New Music Research, 33(3), 203–216. Wing, A. M., Church, R., & Gentner, D. (1989). Variability in the timing of responses during repetitive tapping

with alternate hands. Psychological Research, 51(1), 28–37. Wing, A. M., & Kristofferson, A. (1973a). The timing of interresponse intervals. Perception and Psychophysics,

13(3), 455–460. Wing, A. M., & Kristofferson, A. (1973b). Response delays and the timing of discrete motor responses.

Attention, Perception & Psychophysics, 14(1), 5–12. Author’s Note All correspondence should be addressed to John R. Taylor Western Sydney University The MARCS Institute for Brain, Behaviour and Development Building 1 Bullecourt Avenue Milperra NSW 2214 Australia [email protected] Human Technology: An Interdisciplinary Journal on Humans in ICT Environments ISSN 1795-6889 www.humantechnology.jyu.fi

ISSN: 1795-6889


142

BOOK REVIEW Proenza, F. J. (Ed.). (2015). Public Access ICT Across Cultures: Diversifying Participation in the Network Society. Cambridge, MA, USA: MIT Press, IDRC; 472 pages.

Reviewed by Sakari Taipale Department of Social Sciences and Philosophy University of Jyväskylä Finland

The special needs of the disadvantaged are seldom determinedly promoted, if taken into account at all, in political decision making when public access to computers and the Internet is on the table. Within the covers of the book Public Access ICT Across Cultures: Diversifying Participation in the Network Society, edited by Dr. Francisco Proenza, the reader finds a great collection of studies assessing the socioeconomic impacts of public access to information and communication technology (ICT) on the lives of vulnerable people. The studies represent 10 developing countries and emerging economies on three continents. The overall aim of the book is no less than to gather scientific evidence on what works and what does not in the provision of public access venues (PAVs)—such as telecenters and library access points—that are subsidized by governments all over the world. Individual studies reported in the book are diverse in terms of their methods and commitment to the codes of academic research. Some studies are clearly grounded in social, psychological, and communication theories, while others are more like systematic reports of surveys and interview studies. The chapters are tied together by a loose framework with the aims to “assess impacts with scientific rigor,” “acknowledge the reach and limitation of findings,” and “formulate practical recommendations” (p. 2).

The book is divided into three thematic sections; a fourth section is dedicated solely to a final overview chapter. Below, I highlight topics that are recurrent in the book, and I draw attention to some individual observations that are of particular interest to those interested in ICT, policies, and social equality.

The first section deals with personal achievements reached via PAVs and the impacts of ICT on personal well-being. PAVs must be nearby—close to home or the study or work place—if they are to be used and benefit the user. Many of the chapters’ authors endorse the common understanding that the perceived benefits of cybercafés and other types of PAVs are

© 2017 Sakari Taipale and the Open Science Centre, University of Jyväskylä DOI: http://dx.doi.org/10.17011/ht/urn.201705272521


Book Review

143

primarily related to personal communication and social networking, followed by the possibility for searching for information and/or news. These also are the most commonly reported reasons for visiting PAVs.

This section also considers the urban–rural differences in the use of PAVs. For instance, Mazimpaka, Mugiraneza, and Thioune note in their study that, in Rwanda, government-funded telecenters in rural areas are better equipped to provide ICT training, compared with profit-based cybercafés and firms in larger towns and cities that offer Internet access alongside their primary services, such as photocopying and printing. The crucial role of PAVs in small cities and rural communities—and for rural–urban migrant workers—is also highlighted by Proenza et al. in their study, carried out in China. These studies also show that PAVs function as strong social markers, separating people who have no other choice in accessing the Internet from people who can use the Internet at work, for example, and visit PAVs for convenience.

The second section of the book is devoted to social inclusion and social networking. The section begins with the study by Larghi et al. that focuses on low-income youth in Argentina. This study shows how market-based solutions, like in Rwanda, do not meet the needs of marginalized communities. The authors rightly point out that cybercafés are becoming the facilities of marginal neighborhoods, not least because of the fast diffusion of affordable personal ICTs, such as smartphones. Another study worthy of closer attention concerns Malaysia. Aziah Alias et al. provide evidence of the benefits of cybercafés in fostering users’ social connectedness and their feeling of empowerment in rural regions. Their chapter points toward Internet-enabled smartphones as alternatives to PAVs, especially when the reasons for Internet use relate to personal communication and social networking.

The third section deals with the impact of public ICT access on women. This section, in particular, underscores that the use of PAVs reflects the many existing gender inequalities within and across countries. In fact, gender inequalities may be even more pronounced and present in the young-male-dominated PAVs than in societies in general. The chapter by Phillippi and Peña skillfully shows that women use PAVs differently than men in Chile, in that women engage Internet resources more often for the common good of the family. Among other things, women visit PAVs to organize family matters and children’s education over the Internet. The chapter by Dacanay, Luz Feranil, Silverio, and Taqueban brings to the fore a truly disadvantaged group by examining Burmese female migrant workers and their possibilities to access the Internet in a Thai border town. In order to avoid mistreatment by local police and to communicate freely in their own language, these migrant workers either must find a Burmese-friendly PAV, which are few, or, more commonly, rely on their personal mobile phones.

The book ends with an extensive summary compiled by the editor and his collaborating authors. Despite the apparent differences among the chapters, Proenza et al. succeed in identifying some cross-cutting themes, which they then translate into discreet policy recommendations. The authors argue that PAVs deserve explicit political support and powerful advocates who can make the venues safe for the most vulnerable groups (e.g., women, ethnic minorities). These advocates also could promote the functional uses of ICTs (instead of entertainment) at PAVs and help establish mutually beneficial partnerships between PAVs and educational institutes.

Public Access ICT Across Cultures opens many windows on the status and meaning of cybercafés and other types of PAVs, many of which have already vanished from developed countries. It seems that the functional resilience of cybercafés and telecenters depends not only

Taipale

144

on paying customers but perhaps increasingly on the use of personal communication technologies and citizens’ opportunities for gaining ICT skills through formal education, which then reduces the need for the ICT training offered by PAVs. The book also reveals, although rather implicitly, that Internet-enabled smartphones have notably different affordances compared with the desktop computers offered by PAVs. Unlike PAV facilities, smartphones are not very practical for accomplishing more complex or time-consuming tasks, such as completing an online job application.

While there is considerable variation in the length of chapters, an overload of tables, and a scarcity of theoretical discussion, this book presents an unprecedented overview of public Internet access provisions. The language of the chapters is consistent in style and easy to read, owing much to the painstaking editorial work of this impressive book. The book is recommended reading for anyone interested in the complex marriage of social, economic, and digital inequalities and policies promoting universal access to ICTs. Human Technology ISSN 1795-6889 www.humantechnology.jyu.fi

Human Technology

www.humantechnology.jyu.fi

[email protected]

Submissions: humantechnologypublishing.jyu.fi

Human-Technology Choreographies: Body, Movement, &...

Documents

Transcript of Human-Technology Choreographies: Body, Movement, &...