Investigating Gestures on ElasticTabletops

Dietrich KammerChair of Media DesignTechnische Universitat Dresden01062 Dresden, Germanydietrich.kammer@tu-dresden.de

Thomas GrunderChair of Media DesignTechnische Universitat Dresden01062 Dresden, Germanythomas.gruender@tu-dresden.de

Fabian GobelChair of Media DesignTechnische Universitat Dresden01062 Dresden, Germanyfabian.goebel@tu-dresden.de

Rainer GrohChair of Media DesignTechnische Universitat Dresden01062 Dresden, Germanyrainer.groh@tu-dresden.de

Copyright is held by the author/owner(s).TEI’14, Feb 16 – Feb 19, 2014, Munich, Germany.

AbstractThis work in progress investigates gestures on tabletopswith elastic displays that allow temporary deformations ofthe surface. While tabletops with rigid interactive surfaceshave been subject of the research agenda of tangible,embedded, and embodied interaction for a considerableamount of time, we review novel systems that exploit thethird dimension offered by tabletops with elastic surfaces.In addition, we propose a tentative interaction syntax. Ina user study, we compare push gestures on elastictabletops with swipe gestures on a multi-touch display.

Author KeywordsElastic displays; Tabletops; Haptic interaction; Naturalinteraction; Elastic gestures

ACM Classification KeywordsH.5.2 [Information interfaces and presentation (e.g.,HCI)]: User Interfaces: Input Devices and Strategies,Interaction Styles.

General TermsHuman Factors; Interaction Design; Elastic Displays

IntroductionElastic displays address a challenge that current tabletopsystems are facing: the lack of differentiated haptic

feedback. While a very direct and natural interaction with2D content is possible with standard rigid multi-touchsurfaces, they only provide uniform feedback for the usershands. Elastic tabletops promise to remedy this lack oftactile and spatial experience. However, other thandisplays that can be permanently deformed, they alsomaintain consistency of shape and thus enable viewing aninterface in a standard and well-established way. Incontrast to gestures performed in mid-air, elastic displaysadd a third dimension to touch while maintaining hapticfeedback.

First, we give a brief review of related work in the domainof elastic displays, distinguishing the domain from otherresearch in the field of deformable displays (cp. [1]).Second, we propose a tentative interaction syntax forelastic gestures. Our pilot study investigates whether anelastic display holds advantages over a multi-touchtabletop and shows that a number of challenges remain toestablish this novel type of interactive surface.

Related WorkRecently, researchers have started to focus on interactivesurfaces other than flat and rigid ones [1, 12]. While thereis a considerable body of work in the literature concerningmalleable displays [5, 10] and actuated displays [9], theknowledge about elastic displays that feature onlytemporary deformations is scarce [6, 2].

One of the first elastic displays presented is the“Khronos” projector by Cassinelli and Ishikawa [3]. It is avertical installation of a deformable tissue that is used tofast-forward to a certain position in a video when touchedand depressed. With the “deformable workspace” acomprehensive system for manipulating virtual 3D objectson vertical elastic displays is available [13]. However, in

this paper we focus on horizontal tabletop systems withelastic displays.

An elastic display that allows varied haptic feedback is“MudPad” [7]. Although it is used as horizontal display,its size is relatively small. One of the first publishedsystems that exhibited a tabletop system with an elasticdisplay is “DepthTouch” [11]. The “Obake display” is aprototype devised at MIT media lab that demonstratesvarious interactions with a silicone based screen [4]. Theproposed interaction language features variouscombinations of intruding and extruding the elasticdisplay, which are addressed in the next section.

Interaction Syntax for Elastic GesturesElastic displays afford novel types of interaction usinghands. Although the gestures themselves are not elastic,we use the term elastic gestures. A systematic overview ofinteraction techniques with elastic displays is still lackingin the literature. For multi-touch gestures, varioushigh-level abstractions exist, e. g. based on a semioticanalysis [8]. As a prerequisite for such future work in thedomain of elastic displays, we propose a tentativeinteraction syntax for elastic gestures. We identified threemain categories: push, pull, and touch (see Table 1).

Pushing the surface produces valleys and requires acertain amount of strength from the user depending onthe depth of pushing (see Figure 1). Pulling an elasticsurface requires not only strength but also a certainamount of training and dexterity (see Figure 1). In ourexperience, most new users have difficulties performingthis interaction. The third category is touch interaction,which is comparable to standard multi-touch interaction.In theory, all multi-touch gestures possible on a rigidplanar surface can also be performed on an elastic screen.

However, there is another parameter: pressure. A touchgesture can be performed with different degrees of forceand thus in different depth levels. Difficulties in draggingon an elastic display have been described by [2].

Figure 1: Interacting with anelastic display using push andpull gestures.

Categories PUSH PULL TOUCH

ObjectManipulation

Indirect &direct

Indirect &direct

Direct

Static Hit Flip Tap, Hold

Dynamic Multiplehandspushingsurface

Multiplehandspullingsurface

Multi-touchgestures withdifferentpressure

Joining &splitting ofpulled areas

Combination of techniques

Table 1: Classification of interaction syntax for elastic displays

Another important issue is the way objects are beingmanipulated. While touch interaction is commonly basedon direct manipulation (i.e. touching) of virtual objects onthe surface, pushing and pulling an elastic display can alsobe used to indirectly collect and disperse objects byexploiting natural physics.

All of the main categories of the proposed elastic gesturescan either be of static or dynamic nature (cp. [14]). Themain property of static gestures is that no continuousmovement of the user is necessary. For pushing, a simplehit (or bump) on the elastic surface causes a slightvibration, similar to ripples on a water surface. In the caseof pulling, letting the surface flip down by quickly letting

go of the cloth causes a similar effect. In the case ofstandard touch interaction, tap or hold gestures areconsidered to be static (cp. [14]). In contrast to hittingand flipping an elastic display, only visual feedback can beprovided by the application. Static gestures are often usedto perform selections, activate menus, or confirm actions.

Dynamic gestures also address the movement whilepushing or pulling the cloth in order to manipulateobjects. The depth of a push or height of a pull areimportant parameters that can be used by an applicationfor different purposes. A single user can employ two handsto push or pull the cloth at multiple locations (see Figure1). In the case of multi-users more than two areas can bemanipulated in parallel (see Figure 1). It is conceivablethat multiple pulled areas can be brought together or bepulled away from each other (cp. [4]). As stated before,all dynamic multi-touch gestures are also conceivable onan elastic tabletop. Moreover, a combination of pulling,pushing, and touch interaction is possible.

While considerable knowledge exists on multi-touchgestures [8, 14], there is little or no evaluation of howthey are performed on an elastic display. Furthermore, thediverse pushing and pulling gestures possible on an elastictabletop have not been thoroughly investigated yet (cp.[2]).

Study on Memorability and LearnabilityWe compared the interaction using a standard rigidmulti-touch surface with a tabletop using an elasticdisplay. In our pilot study we investigated the followingunderlying hypotheses: [H1] Interaction with an elasticdisplay supports finding and memorizing different layers ofinformation (memorability), [H2] The interaction withmetaphors based on depth levels is easier to learn and

understand using an elastic display (learnability).

ParticipantsThere were 21 participants (6 female), from 24 to 36years averaging at about 29 (SD = 2.97). All but onewere familiar with multi-touch tabletops (7 or more hours)and only 2 with interacting on an elastic display. Most ofthe participants (14) knew how to interact with the elastictabletop (1-2 hours experience), but were no professionalsin dealing with it.

Figure 2: Study setup withelastic tabletop (includinginternal setup of projector, kinectsensor, and mirror) andmulti-touch tabletop system.

ApparatusWe used a commercial multi-touch tabletop with physicalsize of 115 x 85 x 95 cm (width, length, height) and ourcustom built elastic tabletop with physical size of 115 x 85x 107 cm (width, length, height). Both test applicationswere implemented using Java and the Processing librariesfor visualization.

ProcedureParticipants were presented with the task of finding andmatching images of fruits (see Figure 2). While one imageshowed the target, the other two had to be manipulateduntil the target was found. There were 8 images in eachstack and all of the 8 fruit cards had to be found in eachof the 5 experiment blocks. An intermediate break of 15seconds took place between the blocks. Hence, everyparticipant had to solve a total of 40 trials on each systemand fill out a survey afterwards. Before starting theexperiment, every participant did 5 trials for training.

The task in the study was to manipulate two stacks ofimages in order to find a given target image, similar to thepopular memory card game. Since stacks in reality alsohave a certain depth, stacks should provide an appropriateinteraction metaphor to compare multi-touch and elastictabletop interaction. Only push gestures from our

interaction syntax were used on the elastic tabletop. Onthe multi-touch display, we also aimed for continuousmanipulation gestures and hence implemented swipegestures to the left and right to browse the image stacks.In both cases the image stacks were explored with aninteraction technique appropriate to the technology. Halfof the participants started with the elastic tabletop andthe others with the multi-touch tabletop. This procedureavoided a bias towards one of the tabletops in the data.The Items in the stack were ordered randomly on the leftand on the right side, but fixed overall for both systems.

Data CollectionWe collected task completion times and conducted aNASA-TLX questionnaire to identify task difficulties (seeFigure 3). A custom survey elicited subjective opinionsregarding the two systems.

ResultsIn general, the task was solved faster on the multi-touchsurface (t-test, mean = 1.2 seconds faster, t(20) = 4.14,p<.001) and judged more efficient in the survey. However,the elastic interaction was regarded easier to understandand to learn (p = .012). In both cases, the error rate wasthe same (zero). The NASA-TLX survey supported thefindings in the data. The multi-touch system was ratedeasier to use (p = .005) and less demanding in a physicalway (p<.001). Furthermore, we found a tendency towardsa higher frustration with the elastic tabletop (p<.010).The data also showed that participants with equalexperience in both systems were less frustrated with theelastic tabletop than with the multi-touch system. Forthis group, the task itself was regarded easier to solvewith the elastic tabletop. However, it was rated physicallymore demanding and slower to solve than on themulti-touch tabletop.

While our study showed first in-dications that learning and un-derstanding of an elastic table-top is easier, it also showedthat the multi-touch interactiontechnique was faster. Hence,we could find support for oursecond hypothesis, but no ev-idence that our first hypoth-esis holds true. One reasoncould be the study design, sincethe multi-touch interaction mayhave led to a memorization ofthe position of the items as well,however, not in depth, but inhorizontal positions.The frustration with the elastictabletop as reported by the par-ticipants stems mainly from jit-tering in the tracking system.The result was a significantlyhigher amount of card turns onthe elastic tabletop. With amore robust tracking hardware,we hope to be able to remedythis source of frustration. Like-wise, the precision of the depthinteraction will then close thegap to multi-touch technology.

0

2

4

6

8

10

12

14

16

18

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

0

2

4

6

8

10

12

14

16

Mental

Demand

Physical

Demand

Temporal

Demand

Performance Effort Frustration

Elastic Tabletop

Multi-touch Tabletop

Individual study participants

Mean completion times for 40 trials (seconds)

Rating (0=lowest, 20=highest)

Figure 3: Mean completion times of the 21 study participants (error bars show standard deviations) and results of the NASA-TLXquestionnaire (mean values across 21 participants, error bars show standard deviations).

Conclusions and Future WorkIn this report on our work in progress, we proposed atentative interaction syntax for elastic tabletops. Athorough and systematic evaluation of all possiblegestures is subject of future work. While our pilot studydoes not yet show significant benefits over standard rigidmulti-touch surfaces, we believe that more robusthardware and suitable applications is going to prove that amore natural and intuitive interaction is possible withelastic tabletops.

AcknowledgementsThomas Grunder has been supported by the EuropeanSocial Fund and the Free State of Saxony.

References[1] Alexander, J., Brotman, R., Holman, D., Younkin,

A., Vertegaal, R., Kildal, J., Lucero, A. A., Roudaut,A., and Subramanian, S. Organic experiences:(re)shaping interactions with deformable displays. InCHI ’13 Extended Abstracts, ACM 2013, 3171–3174.

[2] Bacim, F., Sinclair, M., and Benko, H.Understanding touch selection accuracy on flat andhemispherical deformable surfaces. In Proc. of the2013 Graphics Interface Conference, GI ’13, CanadianInformation Processing Society 2013, 197–204.

[3] Cassinelli, A., and Ishikawa, M. Khronos projector. InACM SIGGRAPH 2005 Emerging technologies, 2005.

[4] Dand, D. Obake: Interactions with a 2.5D elasticdisplay, 2013.

[5] Follmer, S., Johnson, M., Adelson, E., and Ishii, H.deForm: an interactive malleable surface forcapturing 2.5D arbitrary objects, tools and touch. InProc. of the 24th annual UIST, ACM 2011, 527–536.

[6] Gruender, T., Kammer, D., Brade, M., and Groh, R.Towards a design space for elastic displays. In ACMSIGCHI CHI - Workshop: Displays Take New Shape:

An Agenda for Future Interactive Surfaces (Paris -France, 2013).

[7] Jansen, Y., Karrer, T., and Borchers, J. MudPad:tactile feedback for touch surfaces. In CHI ’11Extended Abstracts, ACM 2011, 323–328.

[8] Kammer, D., Wojdziak, J., Keck, M., Groh, R., andTaranko, S. Towards a formalization of multi-touchgestures. In ACM ITS, ACM 2010, 49–58.

[9] Leithinger, D., and Ishii, H. Relief: a scalableactuated shape display. In Proc. of the fourth TEI,ACM 2010, 221–222.

[10] Matoba, Y., Sato, T., Takahashi, N., and Koike, H.ClaytricSurface: an interactive surface with dynamicsoftness control capability. In ACM SIGGRAPH 2012Emerging Technologies, ACM 2012, 6:1.

[11] Peschke, J., Gobel, F., Grunder, T., Keck, M.,Kammer, D., and Groh, R. DepthTouch: an elasticsurface for tangible computing. In Proc. of theInternational Working Conference on AdvancedVisual Interfaces, ACM 2012, 770–771.

[12] Steimle, J., Benko, H., Cassinelli, A., Ishii, H.,Leithinger, D., Maes, P., and Poupyrev, I. Displaystake new shape: an agenda for future interactivesurfaces. In CHI ’13 Extended Abstracts, ACM 2013,3283–3286.

[13] Watanabe, Y., Cassinelli, A., Komuro, T., andIshikawa, M. The deformable workspace: Amembrane between real and virtual space. In 3rdIEEE International Workshop on HorizontalInteractive Human Computer Systems, 2008.TABLETOP 2008 (2008), 145–152.

[14] Wobbrock, J. O., Morris, M. R., and Wilson, A. D.User-defined gestures for surface computing. In Proc.of the SIGCHI Conference on Human Factors inComputing Systems, ACM 2009, 1083–1092.