Mimetic Interaction Spaces : Controlling Distant …grisoni/iui0314-rateauA.pdfMimetic Interaction...

6
Mimetic Interaction Spaces : Controlling Distant Displays in Pervasive Environments Hanae Rateau Universite Lille 1, Villeneuve d’Ascq, France Cite Scientifique, 59655 Villeneuve d’Ascq [email protected] Laurent Grisoni Universite Lille 1, Villeneuve d’Ascq, France Cite Scientifique, 59655 Villeneuve d’Ascq laurent.grisoni@lifl.fr Bruno De Araujo INRIA Lille 40, avenue Halley - Bat A - Park Plaza 59650 Villeneuve d’Ascq [email protected] ABSTRACT Pervasive computing is a vision that has been an inspiring long-term target for many years now. Interaction techniques that allow one user to efficiently control many screens, or that allow several users to collaborate on one distant screen, are still hot topics, and are often considered as two different ques- tions. Standard approaches require a strong coupling between the physical location of input device, and users. We propose to consider these two questions through the same basic con- cept, that uncouples physical location and user input, using a mid-air approach. We present the concept of mimetic interac- tion spaces (MIS), a dynamic user-definition of an imaginary input space thanks to an iconic gesture, that can be used to define mid-air interaction techniques. We describe a partic- ipative design user-study, that shows this technique has in- teresting acceptability and elicit some definition and deletion gestures. We finally describe a design space for MIS-based interaction, and show how such concept may be used for multi-screen control, as well as screen sharing in pervasive environments. Author Keywords gestural interaction ; mid-air gestures; contactless interaction ACM Classification Keywords H.5.2 User Interfaces: Ergonomics, Evaluation / methodology, Interaction styles, User-centered design General Terms Human Factors; Design; Measurement. INTRODUCTION Grasping the mouse, or touching the pad, is currently, by far, the most common way to start interacting with an application. Such paradigms imply both proximity between user and inter- active system. For interaction situations in which distance be- tween user and screen can not be avoided(e.g distant screen), Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full cita- tion on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- publish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. IUI’14, February 24 - 27 2014, Haifa, Israel Copyright c 2014 ACM 978-1-4503- 2184-6/14/02..$15.00. http://dx.doi.org/10.1145/2557500.2557545 and instrumented interaction may be difficult to deploy (pub- lic displays), or limiting (family in front of connected TV, work meetings, etc . . . ), mid-air gestural interaction appears to have great potential for such contexts. Pervasive environments are contexts in which fluid inter- action has a key role to play for M. Wieser’s vision to be reached. We need always-available, (ideally) low- instrumented, interaction techniques, that would permit users interacting with several displays; we also need techniques that allow collaboration in the same room for a given task on the same display. Mid-air interaction still has several drawbacks that are not overcome yet; moreover it is still poorly under- stood, quite apart from elementary tasks [13]. A common (wrong) approach is to think about mid-air gestures as ”touch at a distance”, as stated in [14]. We generalize, in this article, the idea of predefined plane for mid-air interaction with distant display, and present the con- cept of MIS gestures. Instead of interacting in a pre-defined static space, we allow the user to create and delete his own interaction space at any time and place thanks to a simple gesture that mimics the interaction space. This article first presents a user study that provides some el- ements of knowledge about how users, in a participative de- sign approach, would potentially use such systems. In our results, we show that users validate the idea of a planar MIS, and that most users that run the experiment instinctively state that plane position is user-defined and dynamic (can be both created and deleted). We also show that users easily integrate mental representation of interaction MIS, since user-defined deletion gestures take plane location into account. Finally, we provide guidelines for MIS gestures in mid-air interaction techniques. We also describe the design space as- sociated to the presented concept, and describe the proof of concept of MIS interaction that illustrates 2 key scenarios. RELATED WORK Although the proposed concept is novel, we can find in liter- ature other works that relate, on some aspects, to MIS. Mid-air Interaction. Several virtual interaction volume techniques have been proposed in the past years. All in dif- ferent contexts of use and with different properties.

Transcript of Mimetic Interaction Spaces : Controlling Distant …grisoni/iui0314-rateauA.pdfMimetic Interaction...

Mimetic Interaction Spaces :Controlling Distant Displays in Pervasive Environments

Hanae RateauUniversite Lille 1, Villeneuve

d’Ascq, FranceCite Scientifique, 59655

Villeneuve d’[email protected]

Laurent GrisoniUniversite Lille 1, Villeneuve

d’Ascq, FranceCite Scientifique, 59655

Villeneuve d’[email protected]

Bruno De AraujoINRIA Lille

40, avenue Halley - Bat A -Park Plaza 59650 Villeneuve

d’[email protected]

ABSTRACTPervasive computing is a vision that has been an inspiringlong-term target for many years now. Interaction techniquesthat allow one user to efficiently control many screens, or thatallow several users to collaborate on one distant screen, arestill hot topics, and are often considered as two different ques-tions. Standard approaches require a strong coupling betweenthe physical location of input device, and users. We proposeto consider these two questions through the same basic con-cept, that uncouples physical location and user input, using amid-air approach. We present the concept of mimetic interac-tion spaces (MIS), a dynamic user-definition of an imaginaryinput space thanks to an iconic gesture, that can be used todefine mid-air interaction techniques. We describe a partic-ipative design user-study, that shows this technique has in-teresting acceptability and elicit some definition and deletiongestures. We finally describe a design space for MIS-basedinteraction, and show how such concept may be used formulti-screen control, as well as screen sharing in pervasiveenvironments.

Author Keywordsgestural interaction ; mid-air gestures; contactless interaction

ACM Classification KeywordsH.5.2 User Interfaces: Ergonomics, Evaluation /methodology, Interaction styles, User-centered design

General TermsHuman Factors; Design; Measurement.

INTRODUCTIONGrasping the mouse, or touching the pad, is currently, by far,the most common way to start interacting with an application.Such paradigms imply both proximity between user and inter-active system. For interaction situations in which distance be-tween user and screen can not be avoided(e.g distant screen),

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected]’14, February 24 - 27 2014, Haifa, Israel Copyright c© 2014 ACM 978-1-4503-2184-6/14/02..$15.00.http://dx.doi.org/10.1145/2557500.2557545

and instrumented interaction may be difficult to deploy (pub-lic displays), or limiting (family in front of connected TV,work meetings, etc . . . ), mid-air gestural interaction appearsto have great potential for such contexts.

Pervasive environments are contexts in which fluid inter-action has a key role to play for M. Wieser’s vision tobe reached. We need always-available, (ideally) low-instrumented, interaction techniques, that would permit usersinteracting with several displays; we also need techniques thatallow collaboration in the same room for a given task on thesame display. Mid-air interaction still has several drawbacksthat are not overcome yet; moreover it is still poorly under-stood, quite apart from elementary tasks [13]. A common(wrong) approach is to think about mid-air gestures as ”touchat a distance”, as stated in [14].

We generalize, in this article, the idea of predefined plane formid-air interaction with distant display, and present the con-cept of MIS gestures. Instead of interacting in a pre-definedstatic space, we allow the user to create and delete his owninteraction space at any time and place thanks to a simplegesture that mimics the interaction space.

This article first presents a user study that provides some el-ements of knowledge about how users, in a participative de-sign approach, would potentially use such systems. In ourresults, we show that users validate the idea of a planar MIS,and that most users that run the experiment instinctively statethat plane position is user-defined and dynamic (can be bothcreated and deleted). We also show that users easily integratemental representation of interaction MIS, since user-defineddeletion gestures take plane location into account.

Finally, we provide guidelines for MIS gestures in mid-airinteraction techniques. We also describe the design space as-sociated to the presented concept, and describe the proof ofconcept of MIS interaction that illustrates 2 key scenarios.

RELATED WORKAlthough the proposed concept is novel, we can find in liter-ature other works that relate, on some aspects, to MIS.

Mid-air Interaction. Several virtual interaction volumetechniques have been proposed in the past years. All in dif-ferent contexts of use and with different properties.

Hilliges et al.[7] propose a static extension of the 2D displaythat allows the user to perform 3D gestures above the screen.There is a direct mapping between the hand above the surfaceand the output (shadows displayed). As long as the systemcan detect the user’s hands, the user can manipulate objectsof the 3D scene. In that case, the interaction volume is static,always active and of a predefined size (here the screen size).

In [6], Gustafson et al. propose a system with no visual feed-back. Screen is replaced by short term memory. The userdefines dynamically the space in which he wants to interactwith a non-dominant hand posture as a reference point. In-teractions start with a posture and stop when the user releasesthe pose. Three studies show that the more time spent, themore degraded memory. But using the non dominant handas a reference point improves performance. In our conceptof MIS, the short term memory is maintained by the visualfeedback and the reference point is not compulsory anymoreas showed in [2].

The work in [10] presents a virtual touch panel named Air-Touch Panel. The user has to form an L-shape with his lefthand to define a virtual panel and then can interact with anAirTouch panel-based intelligent TV: changing channels andvolume. In this work, the panel has a pre-defined size theuser cannot control but he can define the position and the ori-entation of the panel. Our work is a generalization of theAirTouch Panel concept.

MIS interaction. To our knowledge, there is no existingstudies on how users may create or delete interaction sub-spaces. However, some work on multitouch interaction cangive some hints.

In [15] several participants conceived imaginary areas aroundthe screen with particular properties, as clipboard or a trashcan. Similarly, some of them also imagine invisible widgetsand reused them. The mental representation of invisible in-terfaces is not unnatural or too much exotic to users. In thissame study, participants mostly preferred one-hand gesturesas in [11] for the efficiency/simplicity and energy saving.

In [10] the authors also conducted two studies. The first is re-lated to what kind of click gesture will be more appropriate.Results showed that, considering the average miss-clicks, thetapping gesture is the worst, the left hand click is the moretiring and a specific gesture, which is stretching the thumbaway from the index, has the highest satisfaction rate. Inter-estingly, in [4], the air tap is the preferred gesture to click inmid-air. The second study investigates the more appropriatesize of panel to avoid miss click and satisfy user’s comfort.The 24” panel was the more appropriate size. Concerning thesize, in [9], Kattinakere et al. study and model a steering lawfor 3D gestures in above-the-surface layers, resting the handon the surface. Results suggest that a layer should be at least2 cm thick and that steering along more than 35 cm generatesmore errors.

Methodology for Eliciting gestures. We chose to carry outa gesture elicitation study, as in several prior work, in orderto see how potential users could use the MISs, and what theycould expect.

The methodology proposed by Nielsen et al. in [12] con-sists in identifying ”the functions that will be evoked” by thegestures, which are in our work a creation, click and dele-tion functions, then, finding ”the most appropriate gesture foreach of those functions” by an analysis phase of the gesturesperformed by the users. In [15], Wobbrock et al. conducteda similar study in the context of gesture-based surface com-puting. They identified 27 common commands and the par-ticipants had to choose a gesture for each of these. In [11],which is the follow up of [15], the authors concluded ”thatparticipatory design methodologies [. . . ] should be appliedto gesture design”.

Gesture classification. Cadoz [3] suggested a classificationregarding the function of the gestures which are complemen-tary and dependent : semiotics (modification), ergotic (per-ception) and epistemic (communication). But these are notappropriate for our domain. Karam and Schraefel proposeda classification adapted to HCI based on gesture styles : de-ictic, manipulative, semaphoric(with a cultural meaning likethumb up for OK), gesticulation(conversational gesture), signlanguage, multiple(combined) gestures styles. Aigner et al.presented in [1] a modified taxonomy of Karam and schrae-fel [8] adapted to gesture elicitation study in mid-air withoutspeech command or sign language.

MIMETIC INTERACTION SPACES: CONCEPT DESCRIP-TIONWe present here the concept of Mimetic Interaction Spaces(MIS) and MIS gestures. A MIS is a delimited sub-spaceof the user’s space, used to perform interaction gestures. Itcan be of arbitrary dimension i.e. a 1D curve, a 2D shapeor a finite volume depending on the application. The chosensub-space is simple enough so that it is possible to evaluatewhether or not user’s hand is within this sub-space and if ges-tures shall be taken into account for interaction or not. TheMIS gestures are defined as the set of user’s gestures whichcan be performed within such space, to interact with it, as wellas to create or delete it. It may relate (but not necessarily) to aphysical object, or an imaginary representation of it. By ges-turing on or in the MIS, the user can interact with a distantscreen, e.g control a mouse cursor on an invisible touchpad(planar MIS). We think this concept is interesting becauseit is more specific than the standard understanding of mid-air interaction, while obviously leaving quite an interestingdesign space to distant display control (shape type, dimen-sion, space localization regarding user and display, multiplespaces, etc. . . see further description in this article).

Formal Definition of MIS, and MIS-based interaction tech-niqueFrom the concept of MISs, we first define formally a MIS asa virtual object with four characteristic components detailedas follows: geometric definition (GD), input reference frame(IRF), action reference frame (ARF), interaction attributes(IA). Each of these components are described below. We de-fine a MIS-based interaction technique as a particular set ofthese four components.

Geometric definition (GD)We defined here the elementary geometric aspects of a MIS:shape, orientation, scale, position. They are expressed rela-tive to the input frame of reference of the MIS they describe.

Input Reference Frame (IRF)This is the coordinate frame that links MIS to the physicalworld in which the user evolves. In the general case, a MIScan be anchored to an entity of the real world, possible en-tities being user’s body or a part of it (e.g hand, head,. . . ),or any identified object or the world (fixed position). If thisentity moves, then the MIS moves as well. A MIS may havemultiple IRFs. Then, a main IRF must be declared for the pri-mary properties. Plus, it can be changed during interaction,using specific command gesture associated to the MIS.

Action Reference Frame (ARF)This is the coordinate frame that links MIS to the display withwhich user is willing to interact. A MIS can have multipleARFs. A default ARF is defined, that may be changed duringinteraction.

Interaction Attributes (IA)The interaction attributes gather all properties that may benecessary to define the interaction technique based on theMIS defined by a set (GD, IRF, ARF). They may relate tohuman factors, data acquisition specificity, or any additionalelement that needs to be taken into account to define interac-tion technique. Such attributes may vary, both from numbers,types and values, depending on the interaction techniques wetarget.

USER STUDYOur user study was designed to collect gesture data that couldbe used to define MISs and question the users on what theycould expect of MIS interaction. In order to perform suchstudy without suggesting any solution, we decided to simulatedistant screens using a large curved screen (5.96 meters by2.43 meters).Using such environment, we are able to projectimages of displays at different locations and of different sizes.By doing so, we expected to represent daily scenarios in anabstract way such as using a computer screen or a television athome or in collaborative working sessions. . . . The remainingof the section describes our experimental protocol and how itrelates to our concept of MIS interaction. With their agree-ment, all participant sessions have been videotaped using aKinect camera in front of the user and a video camera on theside recording a different point of view and sound.

ProtocolParticipants had to define 90 areas corresponding to projectedvirtual screens of two different sizes : 32 inches and 55inches. Each virtual screen were displayed 3 times at 15 dif-ferent positions on the large screen. They could take a breakevery 10 trials to avoid fatigue.

For each trial, participants had to define, by a gesture or aposture, an area they thought was the most relevant and com-fortable to control the shown virtual screen. Then they hadto touch it as if they were interacting with the virtual screen.They were told the virtual screen could be either a computer

screen or a television. The only constraint was that they werenot allowed to walk but they could turn around. After therepetitive trials, they were asked to tell which gesture theypreferred during the experiment. Then they had to imagine agesture they would perform to delete an area they have previ-ously defined.

Participants18 participants volunteered for the study (4 female). 8 par-ticipants worked in HCI. They were between the ages of 22and 43 (mean: 27.6). Two participants were left-handed andone was ambidextrous. All participants used a PC and 39 %of them used tactile devices almost everyday (mostly smart-phones).However, only 28 % of the participants played videogames regularly. Even if they were not gamers, all of themhad already tried and knew 3D gestures using the Wiimote,the Kinect, the Eyetoy or the PS Move.

Gesture ClassificationTo classify the different gestures performed by the partici-pants, we used the gesture taxonomy proposed by the Aigneret al. [1] and depicted in Figure 1. This taxonomy proposesfour different classes of gestures: pointing, semaphoric, pan-tomimic and iconic.

Figure 1. Classification used to analyse the gestures made in the userstudy.

While pointing gestures are mostly used to name an object ora direction, semaphoric gestures are gestures that are mean-ingful. There are static semaphoric gestures like the thumb-up posture that means ”OK”, and dynamic semaphoric ges-ture like waving the index finger sidewards to mean ”no”.Note that these meanings are strongly dependent of the cul-tural background and experience of the user. Pantomimicgestures refer to gestures used to mimic an action like grab-bing an imaginary object and rotating it. Finally iconic ges-tures represent informative gestures. They inform about theproperties of an object like specifying a size or a shape.There are static iconic gestures and dynamic gestures. Unlikesemaphoric gestures, no common knowledge of the user’spast experience is needed to understand these kind of ges-tures.

ResultsThis section presents the results and observations of ourstudy. We decouple our analysis into three parts related to theMIS interaction basic steps which are: the gestures to create

it, how users can interact with it and finally how participantspropose to delete it.

Interaction space creation gestureWe analyzed the video of each participant and described eachgesture performed along the 90 trials of the experiment us-ing the gesture taxonomy presented by Figure 1 and comple-mented with the information about which hands were used,hand postures and the relationship between the location ofthe gesture and the user field of view or any significant bodypart. We choose to discard any isolated gesture performed orslightly different variants from the same gesture.

Figure 2. Frequent creation gestures proposed by the user: defining arectangular area using one or both hands (top) and using an openinggesture in its field of view with diagonal or horizontal symmetric gesture(bottom).

Looking to the set of the 33 gestures performed by all users,71 % of them describes an area that can be assimilated to aplane. We noticed that 89 % of users performed iconic dy-namic gestures, representing 60 % of all the gestures. Theymostly represent rectangular shapes (66 %) or opening ges-ture (28 %) along a line or diagonal delimiting the size of aframe as depicted by Figure 2. Circular motions such as cir-cles and waving in front or around the user were less common(9 %).

Regarding hand usage, we noticed that 33 % of them exclu-sively defined gestures using one hand, 33 % using both handsand 33 % mixing both approaches while performing the sev-eral trials. While all unimanual gestures were mainly done us-ing the dominant hand, most of bimanual gestures describedsymmetrical movements or poses. Only three users presentedgestures following the asymmetric bimanual Guiard model[5]. While performing the gestures, we noticed that most ofparticipants used a reduced set of hand poses shown in Figure3. Index finger pointing to the screen, and mimic of a pencilwere prominent among participants (77 %) compared to bothL shape (27 %) and open flat hand postures (33 %).

About display position influence, we noticed that most of theparticipant aligned their field of view prior to start the ges-ture by rotating both the head and body. However, 39 % ofthe users depicted gestures in a fixed position regarding theirbody. The preferred approach (61 % of users) was to createvertical planes aligned with the field of view or the projectedscreen by drawing rectangles or defining static frames. Inthe case of horizontal or oblique planes independently of the

Figure 3. The 3 main hand postures. From left to right: pointing toa given direction, flat hand posture defining a spatial reference, two Lhand postures delimiting an area.

screen position or field of view user was never looking at hishands while performing the gesture.

Interacting on a MISFor each trial, we asked the participants to touch or interacton the previously defined interaction area. They mainly sim-ulated drawing or small push actions close to the area definedas shows Figure 4. Users touched the imaginary space usingtheir dominant hand, except one with both hands. We noticedthree different major hand poses: pointing using the indexfinger, pointing using a flat hand and pushing using an openhand with a percentage of 56, 22 and 17 respectively. Peopleusing an open or a flat posture tend to push, grab or swipeclose to the MIS definition. While participants using theirindex finger tried to mimic drawing short scribbles or pushsmall imaginary buttons. These behaviors showed a strongmaterialization of the MIS as a physical tool.

Figure 4. Common touch gestures proposed by the subjects: pointing ona vertical or horizontal imaginary area and touching the non dominanthand as a reference.

Deleting a MISAt the end of experiment, we asked participants to proposea delete gesture considering that their interaction zone cre-ation was persistent. Looking to the 23 gestures collected, wenoticed a strong usage of pantomimic gestures since most ofusers materialized the interaction MIS. 23 % of the proposalsdo not fit in this classification such as leaving the interactivearea, waiting for it to disappear, drawing a cross or using theinverse of creation movement. For users that used non domi-nant hand as a support to interact, the area shall disappear justby removing the hand. Figure 5 illustrates the main proposedgestures.

Figure 5. Participants delete gesture proposals: pushing the area withone hand, closing the MIS using both hand or throwing it away to a givenlocation.

ObservationsFrom the current user study, we can highlight the followingobservations and remarks to implement MIS based applica-tions and better take advantage of the design space offered bysuch concept.

Make MIS planar, and dynamic : most of users sponta-neously create planar MISs, and take for granted that they canspecify them in arbitrary position, without any experience.

User tends to turn in the direction of the screen : in thatcase, MIS tends to be vertical, and directly relates to the fieldof view of user. In case where users do not orientate them-selves in the direction of the screen, MIS is created horizon-tally, for indirect interaction.

Gesture for creating and deleting MISs can be parameter-ized gestures: for most users, these gestures specify both acommand (e.g create subspace) and some parameters of thecommand (e.g some geometric features such as MIS locationfor creation), in the same gesture.

User has proper mental perception of MISs he/she createsSince all users provided delete gestures that start in a locationdevoted to the MIS that was previously created. The MISbecame real.

DESIGN SPACEFrom previous experiment observations and the MIS formaldefinition, we explore the design space according the fourcomponents defining a MIS. The mentioned variations canbe combined to provide a large and flexible set of mis-basedinteraction techniques.

On Geometric Definition One specific shape could repre-sent one specific range of possible actions. A plane may referto a 2D control of a cursor, whereas a sphere, for example,may suggest a rotation control of virtual objects. As well, aparticular orientation may refer to a particular action. Differ-ent dimensions could allow more or less accuracy.

On Input Reference Frame Attaching a MIS to the world asa reference frame of input links it to the world. Even if theuser moves, the MIS will not. If the MIS is associated andlinked to the user, the latter can move around the environmentkeeping the MIS next to him at the same position regardinghis position. The MIS could also be attached to a physicalobject. The MIS will remain attached to the object and thencan be shared in a collaboration context.

On Action Reference Frame As explained in the section ,Action Reference Frame links MIS to the display it controls.It can be associated to a static display or the ARF can alsobe associated to moving display. In this latter configuration,whatever the position of the display, the MIS still controls it.The ARF may be re-affected in a multiple displays configu-ration.

On Interaction Attributes In this section, only very few”properties” of the MIS are addressed. These attributes mayenable bimanual gestures, tuning of the sensitivity of the MIS,relative or absolute mapping, 3D or 2D input . . .

MIS PROOF OF CONCEPTFollowing the observations resulting from our user study, wedevised an application as a proof of concept to let one or moreusers interact with one or more distant displays.

Several key scenarios were possible to implement regardingboth the number of users and the number of screens. Theone user interacting with one screen scenario, the one userwith multiple screens scenario and the multiple users withone screen scenario.

The application consisted of providing to two users the ca-pacity to control and share the mouse cursor between severaldisplays allowing to interact with any content displayed bythe screens. We chose to implement a planar MISs solutiondefined by rectangular gestures since such gestures were themost common among our user study. The application was im-plemented as a daemon sending mouse inputs directly to theoperating system (Microsoft Windows 7).

To track the user’s gestures, we chose to rely on a wirelessmagnetic based tracking system i.e. Liberty LATUS sys-tem from Polhemus complemented with a button to emulatethe mouse click as depicted in Figure 6. Such solution waspreferred to non intrusive tracking solutions such as the Mi-crosoft Kinect depth sensor, in order to obtain reliable posi-tions and orientations of the user’s hand. However, our MISconcept could be used in a more pervasive environment usingseveral cameras to track users in a non-intrusive way. All in-put data were streamed to our software daemon using a TUIOclient approach.

Figure 6. The user is only equipped with (a) a tracker and (b) a wiredbutton.

The details of the implementation are discussed in the follow-ing chronologically from creation gesture to deletion gesture.

ApplicationThe detection of a MIS creation gesture is made through 3steps analyzing the user’s hand motion. First, both the begin-ning and the end of a gesture are triggered based on thresholdvalues over the hand acceleration. All the positions and ori-entations retrieved in between these two events are recordedtracking user gestures. The second step is the computationof the plane thanks to a least square method. We then definethe origin, the normal and construct the reference frame of theplane from the average of the orientation vectors of the user’shand during the gesture to get the ”up direction” (i.e y-axis)and the ”right direction” (i.e x-axis) as depicted by Figure7. The dimensions are computed by projecting the gesture

points on the newly defined plane and computing the alignedbounding box on its reference frame. Finally to detect rect-angular shape creation gesture, we use the 1$ recognizer onthe 2D path corresponding to the projection of the 3D handpositions on the pre-computed plane. A pop-up on the screeninforms the user the MIS is created.

Figure 7. The frame of reference of a MIS

Once the MIS is created, each 3D position received is thentreated regarding the MIS. When the hand is near enoughfrom the MIS, we allow the user to control the mouse cursorwith his hand. The mapping between the hand position in theMIS and the mouse cursor position on the screen is absolute.

Currently this proof of concept was defined to track two usersmax and interact with two screens. When the MIS is cre-ated by a user, it is automatically attached to the closer screenregarding the user’s position. The directional swipe gestureallows to change such default binding.

To delete such space, we choose to detect horizontal swipegestures starting within the MIS and finishing out of it with agiven velocity and along the x-axis of the plane.

CONCLUSIONWe presented elements of knowledge about mid-air interac-tion with distant displays. We introduced the concept of MISgestures, that we think is a flexible approach to mid-air inter-action within pervasive environments, as the associated de-sign space is quite large. We showed that MIS gestures are,to the highest acceptability, planar and dynamic. The appli-cation developed allows to see few interesting possibilitiesamong all of possible MIS-based interaction techniques.

As future work, a final complete set of questions related toMIS is the practical application of such concept to collabo-rative, co-located interaction contexts, e.g. such as commandcenters. Studies of uses of MIS within such contexts wouldbe interesting in order to understand how to take the best fromthe presented concept, adapted to collaborative environments.

Also, an in-depth study of the possible applications of MISmay highlight, within all mid-air possible interaction con-texts, some specific subsets, that opens new research direc-tions. Interaction techniques, visual feedback, reachable in-teraction precision taking into account distance of view, areinteresting questions in this context.

REFERENCES1. Aigner, R., Wigdor, D., Benko, H., Haller, M.,

Lindbauer, D., Ion, A., Zhao, S., and Koh, J. T. K. V.

Understanding mid-air hand gestures: A study of humanpreferences in usage of gesture types for hci. Tech. Rep.MSR-TR-2012-11, Redmond, WA, USA, Nov 2012.

2. Balakrishnan, R., and Hinckley, K. The role ofkinesthetic reference frames in two-handed inputperformance. In Proceedings of UIST ’99, ACM (NewYork, NY, USA, 1999), 171–178.

3. Cadoz, C. Le geste canal de communicationhomme/machine: la communication instrumentale. TSI.Technique et science informatiques 13, 1 (1994), 31–61.

4. Camp, F., Schick, A., and Stiefelhagen, R. How to clickin mid-air. In Proc. of HCII 2013 (July 2013), 78–86.

5. Guiard, Y. Asymmetric division of labor in humanskilled bimanual action: The kinematic chain as amodel, 1987.

6. Gustafson, S., Bierwirth, D., and Baudisch, P. Imaginaryinterfaces: spatial interaction with empty hands andwithout visual feedback. In Proceedings of UIST ’10,ACM (NY, USA, 2010), 3–12.

7. Hilliges, O., Izadi, S., Wilson, A. D., Hodges, S.,Garcia-Mendoza, A., and Butz, A. Interactions in theair: adding further depth to interactive tabletops. InProc. of UIST ’09, ACM (NY, USA, 2009), 139–148.

8. Karam, M., and m. c. schraefel. A taxonomy of gesturesin human computer interactions. Technical report,University of Southampton, 2005.

9. Kattinakere, R. S., Grossman, T., and Subramanian, S.Modeling steering within above-the-surface interactionlayers. In Proceedings of CHI ’07, ACM (New York,NY, USA, 2007), 317–326.

10. Lin, S.-Y., Shie, C.-K., Chen, S.-C., and Hung, Y.-P.Airtouch panel: A re-anchorable virtual touch panel. InProceedings of ACM Multimedia 2013 (ACM MM),ACM (october 2013), 625–628.

11. Morris, M., Wobbrock, J., and Wilson, A. Understandingusers’ preferences for surface gestures. In Proceedingsof GI ’10 (Toronto, Canada, 2010), 261–268.

12. Nielsen, M., Moeslund, T., Storring, M., and Granum, E.A procedure for developing intuitive and ergonomicgesture interfaces for hci. In Proc. of the 5th InternationGesture Workshop, GW 2003 (2003).

13. Ren, G., and O’Neill, E. 3d selection with freehandgesture. Computers & Graphics 37, 3 (2013), 101 – 120.

14. Wigdor, D., and Wixon, D. Brave NUI World:Designing Natural User Interfaces for Touch andGesture, 1st ed. Morgan Kaufmann Publishers Inc., SanFrancisco, CA, USA, 2011.

15. Wobbrock, J., Morris, M., and Wilson, A. User-definedgestures for surface computing. In Proc. of CHI ’09,ACM (NY, USA, 2009), 1083–1092.