Playing a Virtual Piano with Dynamics

Daniel SalzStanford Universitydasalz@stanford.edu

Farhan AzamStanford Universityfazam@stanford.edu

Abstract

This project sought to create an intuitive pianoplaying experience through VR that allows usersa cheaper and mobile alternative to that of realpianos. We used Leap Motion controls for ac-curate finger tracking and the Unity engine forcreating the piano playing scene. We were ableto successfully implement finger tracking to pressthe keys of the piano. User feedback dependedon how advanced their piano skills. Novice usersactions were easily recreated whereas advancedusers could not get the feedback they wanted. Thedemo currently works better as a prop as opposedto a real piano, though future improvements couldlead to a more authentic playing experience.

1. Introduction and Motivation

Virtual Reality (VR) opens the opportunity tointeract with real world devices that would other-wise be difficult to obtain or realize. With this inmind, we wish to create an interactive virtual pi-ano that allows users with a Leap Motion deviceand VR headset to play as though they were infront of a real piano. This project is significantbecause it would not only allow a new popula-tion of people to play the piano, but also makeit easier for current piano players to practice andplay while on the go. Many people want to learnhow to play instruments, but the financial burdenof buying and maintaining an instrument can be

a significant obstacle. For example, even an al-ready purchased piano requires tuning that costshundreds of dollars. Mobility is also a significantproblem; pianos are large and extremely cumber-some. Instruments like these are difficult to trans-port without other machinery. Most people can’tmake space for something like a piano, especiallyon the move. A virtual piano would allow usersto avoid these cost, mobility, and space issues.With the widespread use of VR, we hope to cre-ate a simple VR experience in which a growinguser base can play a piano almost anywhere. Ourwork would allow an entire new population accessto pay the piano without the need to leave theirhomes, pay for lessons or rent rooms. Furtherwork could include incorporating a virtual assis-tant that dictates the user to continue improving,removing the necessity of a piano teacher.

We hope to create a stimulating experience thatcaptures the same realities of sitting in front ofa piano and playing. The virtual piano will re-spond to user’s fingers accurately hitting the keysby using hand and finer motions tracked by theLeap Motion device. The Leap Motion device isa hardware sensor that takes these inputs and re-quires no physical contact. None of the currentiterations of virtual pianos accurately track the ve-locity of finger tracking to dynamic sound of thekeys, something extremely important for all songsof all genres. Doing this requires fine tuning ofthe sounds of notes dependent on the velocity atwhich users move their fingers to press keys. This

1

would be the largest contribution from this projectas current VR pianos do exist, but they miss onthis vital aspect of playing the instrument. Theend product should be a fully interactive pianothat mimics real life without the tactile responseaspect. Future work could be down to incorporatetactile feedback which would complete the entireexperience of playing a piano, but now in VR.

In summary our contributions are as follows:

1. Incorporate a a piano into a VR unity en-vironment that allows for tracking of fingermovements to play.

2. Using velocity of finger movements tointroduce dynamics of the notes beingpressed.

2. Related WorkOur virtual piano will build upon and improve

on work with hand tracking, virtual pianos, andother virtual instruments.

2.1. Hand and Finger Tracking

The most important aspect of the project is pre-cise hand and finger tracking while also removingthe need of intrusive hardware. We could haveused haptic gloves, but this wouldn’t fit all users.The Leap Motion is one-size-fits-all and providesan accurate interface that is only inaccurate up to.5mm [6]. This provides confidence that the LeapMotion will be sufficient to get accurate measure-ments for playing virtual instruments. A roughidea of the Leap Motion device and its usage isshown in Figure 1. How to utilize these measure-ments is also essential. A previous EE267 projectintegrated the leap motion for several movementssuch as hand translation, rotation, key tapping,and dynamic movement of the fingers [3]. Theseare all essential movements for playing the pianoand will be heavily utilized in our own implemen-tation of the leap motion device.

Figure 1: The Leap Motion is a small USB pe-riphery device that captures hand motion using IRlights and LED’s.

2.2. Current VR Musical Instrument Implementa-tions

Currently most instrument implementations inVR are simple string or tapping based instru-ments. These are beneficial because they allowa tipping off point for the beginning of musicalexperiences in VR, however, they do not con-tain many of the complexities that are involvedwith transferring a real musical instrument to VR.These projects though do give a framework onhow to approach bringing music to VR [2, 4, 7]

2.3. Current implementations of VR Piano

The best implementation of a viewable VR pi-ano project currently is from the Beijing Instituteof Technology. They also used the Leap Motionfor finger tracking and successfully implementedthe piano playing features. Their main issues withthe final product were: no force feedback mak-ing it difficult for users to locate their hands whenplaying quickly and no real collision making cer-tain actions difficult to interpret in the game en-gine. Our implementation hopes to do the sameas this project, but also add dynamics to the keys[5].

3. Implementation3.1. Unity and Piano Integration

While we could create our own interactable pi-ano object for Unity, this wouldn’t be practicalor time-effective. Therefore, we were limited bythe assets offered in the Unity Asset Store. Fortu-nately, we found a musical keyboard asset com-

plete with animations and musical notes corre-sponding to the individual piano keys. Origi-nally, the animations and notes were triggered bypointer clicks, so we had to modify the scripts totrigger the notes upon collision with other objects.

3.2. Unity and Leap Motion Integration

The integration of Leap Motion and Unity wasdifficult at first because the Leap Motion is onlycompatible with certain VR headsets like the Ocu-lus Rift and HTC Vive. Also, the Leap MotionUnity Asset only works for Windows, so we hadto develop on the EE 267 Lab Computers. Orig-inally, we were planning to use the Oculus Go tocreate a more seamless VR experience, but theOculus Go doesn’t support Leap Motion integra-tion. Therefore, we have to fall-back on the Cus-tom View Master Head Mounted Device (HMD)built for EE 267. More details and specificationsof this can be found on the EE 267 website. Theexperience was run on a Windows PC in Unity,and the LCD display within the HMD was con-nected as an external display. Using concepts thatwe learned in EE 267 and the scripts provided bythe TA’s, we could use stereo rendering as picturebelow.

Figure 2: There are two images rendered, onefor each eye in the View Master. The discrepan-cies between these two images give the illusion ofdepth.

3.3. Finger Tracking and Collisions

There are a total of 4 hand models in eachscene. There is a rendered hand model that youcan see for each hand, and a physical interactionbased model that is hidden from the user. Thephysical based model contains collider shapes,outlined in green in the image below. These col-lider shapes are used to demarcate when objectscollide with one another.

Figure 3: Each of the individual bones of the fin-gers have colliders objects around them, as wellas the wrists and palms.

Once the Leap Motion hand models were in-tegrated with Unity, it was time to finally testthe collision between the keyboard buttons andhand models. While the key pressed animationswere correct, the sound dynamics were incorrect.Specifically, some of the notes didn’t stop playingonce collision between the piano key and handended. Our initial guess was that the Leap Mo-tion hand models used a sweep-based continuouscollision detection algorithm [8]. However, theUnity implementation of this algorithm doesn’taccount for the angular momentum of objects, andthe large performance overhead associated withthis algorithm may have caused it to miss somecollision enter/exits. As a solution, we called afunction that performed sound fading shortly af-ter the note was played, simulating correct sounddynamics.

3.4. Sound Fade Dynamics

To create realistic sounding piano notes, soundfade must be incorporated after the key is nolonger pressed. There are different curves usedfor sound fade, including linear, logarithmic, ex-ponential and s-curves. In this experience, astraightforward linear sound fade is used. Thealgorithm for the linear sound fade is presentedbelow:

Figure 4: In this algorithm, the volume of theaudio source is linearly decreased over time.Time.deltaTime records the time elapsed betweenthe previous frame and the current frame.

3.5. Visual Feedback

An extra feature added was visual color feed-back. When the keys are pressed by the user,they light up as a visual feedback. Our purposefor doing this is that both visual, haptic, and au-ditory feedback are a important components ofcommercial VR systems [9]. While we couldn’timplement haptic feedback, we thought that col-ored keys could be a feasible and worthwhile ad-ditional feature. You can observe this implemen-tation below.

Figure 5: The individual piano keys light up withdifferent colors when the user presses them.

Additionally, this visual component provides anice segue into for future work in creating tutori-als by lighting up the keys that users should presswhen playing a new song.

4. Experiments4.1. User Feedback

Our experiments were limited to our group andthe few people who tried our experience duringthe EE 267 Final Project Demo Session. Most, ifnot all, users were novice piano players. Manypeople were pleased with our demo. However,there were some hit-or-miss experiences depend-ing on how users position themselves.

The Leap-Motion controls require users to bepositioned accurately above the sensors. If usersdo not place themselves correctly, the demo feelsstrange due to the visual feedback of finger move-ment being different than their own. Further workcould employ strategies in Unity to mask thefinickiness of the Leap Motion interaction withthe piano. Another option could be to add an in-teractive tutorial at the beginning of a session thatdemonstrates the best way to position their handsor even have some kind of object in the scenethat helps users with the positioning of their handswhile playing.

Novice users in general were happy with thedemo to play a few keys with their fingers. Thisis where the demo excels since it can very eas-ily track the independent fingers and the collisionwith keys works very well when only one fingeris used to play. This was not the same thoughfor advanced players. They reported normal handand finger movements they usually employ whenplaying piano did not work as well as they wouldhope. The main issue it seemed is that it was diffi-cult to perceive how close to the piano their handswere and when simply lowering a finger to play akey it would either not reach or they would puttheir hands too close causing several unwanted

keys to play. Another issue they reported wasthe inability to rest their hands on the keys sincethe colliders immediately start playing the soundwhen a finger pressed on the keyboard. A solu-tion to this in the future could be to incorporatethe velocity of the finger as a threshold before thefinger is pressed.

From this user feedback we evaluated that ourdemo as it stands now works more as a prop pianorather than an alternative to real piano playing.Novice users would enjoy it significantly more asthe demo works fine with how they typically em-ploy their fingers, for example playing the songChopsticks works very well as opposed to any-thing that requires more than one finger on eachhand.

4.2. Qualitative Evaluation

The robustness of the experience depended alot on how users placed their forearms/wrist. Ifusers placed their forearms/wrists relatively highabove the Leap Motion, the Leap Motion wasable to accurately tracked their hands and fingers.However, if their forearms/wrists right above theLeap Motion, there hands would be placed atawkward and unnatural positions in the VR expe-rience. We believe this is reasonable, since whenpeople are taught to play the piano, they are ofteninstructed to keep their wrists raised. You can seeboth of these positions in Figure 6 below.

Figure 6: One the left, the user has the wristscloser to the Leap Motion device, resulting in aninaccurate depiction of the hand models. On theright, wrists are raised, so the Leap Motion is ableto get a full view of both hands.

5. ConclusionOur finished product resulted in a demonstrable

prototype of a VR piano experience. Users maytake some time to adjust to the hand placement atfirst, but after a few minutes of playing with it,they were capable of playing beginner songs withreasonable results. Advanced users though wereon the opposite spectrum and did not enjoy thedemo nearly as much. The lack of tactile feedbackand inaccuracies in the Leap Motion to track themuch faster motion of advanced players left themwanting more. Our motivation for this projectwas to provide an alternative to a real piano, asit stands this is not the case. Small improve-ments though in future work though could create amore realistic experience and provide users a mo-bile substitute for real pianos. What we learnedthrough this implementation could be applied tocreating other non-wind type instruments in VR.Creating a VR drums experience not to be too faroff from the same methods employed for the pi-ano and we expect it to be more realistic as play-ing the drums already requires the use of a con-troller like interface.

6. Future WorkIn the short term, we could definitely make

some modifications to our prototype to create amore robust experience. For starters, the ViewMaster HMD requires the use of one’s hand at alltimes. Therefore, the user isn’t able to simulta-neously play the piano with both hands and usethe View Master rendering. To solve this issue wewould need to create a head strap similar to othercommercial VR headsets, allowing for a more im-mersive experience.

Additionally, we would also like to make thehand tracking more robust. To do that, we wouldneed to provide the Leap Motion device with anunobstructed view of both of the user’s hands.To do this, we could attach the Leap Motion de-vice to the View Master HMD, as some othercommercial VR headsets do. That way, the user

can place their wrists/forearms as high or low asthey please. This would require more calibrationwithin Unity, but users would be happy to do thisat the beginning of a session if it meant more nat-ural movement from increased finger tracking ac-curacy.

To create a more realistic sound fade, we couldconsider implementing a logarithmic curve soundfade. This more closely mimics how human earsperceive sound. This would be a nice touch foraudiophiles and musicians who wish to use thisexperience in a way that more accurately imitatesreal pianos [10].

Adding tutorials is a simple addition that couldbe easily implemented by utilizing the lights onkeys we already have. The addition of tutori-als could provide a cheap experience for usersto learn as opposed to paying for real lessons orlearning from videos which is not nearly as intu-itive.

References[1] Play Music with AR Technology! Re-

trieved April 26, 2019, from https://www.music-everywhere.co/ (n.d).

[2] Peterson, C. Video Series: Taking VRGuitar to a Whole New Depth. Retrievedfrom http://blog.leapmotion.com/video-series-taking-vr-guitar-whole-new-depth/ 2017.

[3] Lin Shao. Hand movement and ges-ture recognition using Leap MotionController. Retrieved from http://stanford.edu/class/ee267/Spring2016/report_lin.pdf(2016).

[4] Vestergaard, J. Building a virtual realitymusic instrument. Retrieved from https://medium.com/@jespervega/building-a-virtual-reality-music-instrument-d97bdf9fc5d42015.

[5] W. Qiao, R. Wei, S. Zhao, D. Huo and F. Li,A real-time virtual piano based on gesturecapture data, 2017 12th International Con-ference on Computer Science and Education

(ICCSE), Houston, TX, 2017, pp. 740-743.doi: 10.1109/ICCSE.2017.8085592, 2017.

[6] Guna, Joe & Jakus, Grega & Pogacnik,Matevz & Tomazic, Saso & Sodnik, J. AnAnalysis of the Precision and Reliabilityof the Leap Motion Sensor and Its Suit-ability for Static and Dynamic Tracking.Sensors (Basel, Switzerland). 14. 3702-20.10.3390/s140203702, 2014.

[7] J. Fillwalk, ChromaChord: A virtualmusical instrument, 2015 IEEE Sym-posium on 3D User Interfaces (3DUI),Arles, 2015, pp. 201-202, 2015. doi:10.1109/3DUI.2015.7131770

[8] Larsen, E. Collision detection for real-timesimulation. Case study, Reseach and Devel-opment Department, SCEA, 1999.

[9] Wendy J. Adams, Iona S. Kerrigan, ErichW. Graf, Efficient Visual Recalibration fromEither Visual or Haptic Feedback: The Im-portance of Being Wrong Journal of Neuro-science 3 November 2010, 30 (44) 14745-14749; DOI: 10.1523/JNEUROSCI.2749-10.2010

[10] Langford, S. Digital Audio Editing. Burling-ton: Focal Press. pp. 47-57, 2014.