Guidelines for Gestural Tablet Interaction

C O V E N T R Y

U N I V E R S I T Y

Faculty of Engineering and Computing

Department of Computing and the Digital Environment

Creative Computing

303CDE - Individual Project

Guidelines for Gestural Tablet Interaction

Author: Thomas Davies

SID: 2179955

Supervisor: John Halloran

Submitted in partial fulfillment of the requirements for the Degree of Bachelor of Creative

Computing

Academic Year: 2010/11

Declaration of IndividualityThis project is all my own work and has not been copied in part or in whole from any other source

except where duly acknowledged. As such, all use of previously published work (from books,

journals, magazines, internet, etc) has been acknowledged within the main report to an entry in the

References list.

I agree that an electronic copy of this report may be stored and used for the purposes of plagiarism

prevention and detection.

I understand that cheating and plagiarism constitute a breach of University Regulations and will be

dealt with accordingly.

Signed: Date:

Office Stamp

Copyright

The copyright of this project and report belongs to Coventry University.

Declaration of Individuality i

AbstractThe tablet devices market is growing substantially and it expected to sell 140 million devices in 2014

and therefore there is a need for interaction guidelines that span both applications and platforms.

This report presents a set of guidelines for gestural tablet interaction based on experienced and

inexperienced participants who were asked to create gestures for common computer actions. 18

participants created 324 gestures for 18 actions; gestures that were analysed and categorized before

being rated on their appropriateness for the intended action. This data provided the foundation of the

set of guidelines for gestural tablet interaction that I created and were then validated by another set

of users.

Abstract iii

Table of ContentsAdditional Materials on the CD viii

Acknowledgements ix

Introduction & Background 1

Background to the Project 1

Recent History 1

Market Potential 2

Method of Study 2

Findings 3

The Need for a Gesture Set 4

Conclusion 5

Section Summary 6

Methodology 7

Introduction 7

Researching Ethics 7

Guessability Testing 7

Inter-Rater Agreement 11

Gesture-Meaning Association Testing 12

Section Summary 14

Formative Investigation 15

Introduction 15

Guessability Study 15

Inter-Rater Agreement 15

Section Summary 17

Findings 19

Introduction 19

Guessability Study 19

Rating Study 22

Table of Contents v

Section Summary 25

The Gesture Set 27

Introduction 27

Guidelines for Gestural Tablet Interaction 27

Section Summary 32

Summative Investigation 33

Introduction 33

Gesture-Meaning Association Testing 33

Alterations to Original Methodology 34

Results 34

Findings 35

Section Summary 37

Discussion & Conclusion 39

Conclusion 39

Value & Benefit 39

Implications 40

Challenges 40

Future Work 40

Project Management 43

Supervisor Meetings & Progress 43

References 45

Bibliography 47

Appendix A 49

Guessability Study Documentation & Forms 49

Appendix B 85

Gesture Screenshots 85

Appendix C 123

Rating Study Documentation & Forms 123

Appendix D 127

vi Guidelines for Gestural Tablet Interaction

Gesture Data Charts 127

Appendix E 137

Gesture-Meaning Association Results 137

Appendix F 139

Supervisor Correspondence 139

Table of Contents vii

Additional Materials on the CD

viii

Acknowledgements‘Technology alone is not enough’

Steve Jobs

This thesis would not have been possible without the support and guidance of Dr John Halloran who

has assisted this project from the very beginning, through his enthusiasm and immense knowledge.

I would also like to express my gratitude to all the Creative Computing staff who have played an

instrumental role in my development over the past three years.

Finally, I wish to thank my parents who have always supported me and ensured that I had every

chance of fulfilling my dreams.

Acknowledgements ix

1. Introduction & Background1.1. Background to the Project

The launch of the iPad in early 2010 ushered in a new class of tablet devices that are radically

different than the earlier tablet PCs of the 1990s and early 2000s. Styli and desktop-based operating

systems have been replaced by multi-touch displays and mobile operating systems. After 40 years of

using the same human-computer interaction paradigms created by the infamous Xerox PARC team of

the 1960s and 1970s, change has come in the form of light, compact, and efficient modern tablet

devices (Saffer 2008: 3). This shift in paradigm is different however to that of the desktop era where

only two major platforms led the way - Windows and Mac OS - as multiple companies have ventured

into the tablet sector. Users will be introduced into this new method of interaction via a fragmented

sector in both hardware and software and therefore it is important to define a set of multi-platform

guidelines and standards as the methods we create today will define the landscape for decades to

come (Saffer 2008: 4). 2011 will be a defining year for tablet devices as many manufacturers will be

launching their products, aiming to eradicate Apple's huge market share.

1.2. Recent History

RIM

HP WebOS

Android

AppleiPad releasedApr 2010

January 2010

April 2010

July 2010

October2010

January 2011

April 2011

July 2011

October 2011

HP TouchPad announcedFeb 2011

Playbook announcedSep 2010

Honeycomb OS releasedFeb 2011

Galaxy Tab releasedNov 2011

iPad 2 releasedMar 2011

HP TouchPad releasedSummer 2011

Motorola Xoom releasedFeb 2011

Galaxy Tab 10 releasedSummer 2011

Playbook releasedApr 2011

Figure 1.1 Timeline of recent history in the tablet sector

In September 2010, Research in Motion (RIM) introduced their rival to Apple's iPad; the PlayBook.

The OS will not be based on RIM's smartphone operating system but on QNX Software Systems' OS

which 'is designed specifically for the tablet size computer' (BBC News 2010). QNX was acquired by

RIM in April 2010, 5 months before the PlayBook announcement (RIM 2010). The PlayBook -

Chapter 1: Introduction & Background 1

designed for the enterprise sector - is expected to be 'commercially available early in 2011' (BBC

2010).

While Samsung released their Galaxy Tab in November 2010, it was using an Android build that was

designed for smartphones and not tablets and therefore cannot be considered as a legitimate tablet

device. Therefore, throughout 2010, Apple dominated the sector with a 'market share in tablet

computers at 95 percent in the third quarter' (Bloomberg 2010).

The next major tablet announcement took place on February 2nd, 2011 when Google announced the

Honeycomb update to their Android OS that was targeted specifically for the tablet form factor

(Google 2011). This Google update then resulted in several manufacturers; Samsung, Motorola and

Asus announcing that their Honeycomb-based tablets will launch in the first half of 2011. One week

after the Google announcement, HP revealed plans for their recently acquired Palm webOS mobile

operating system. Along with updates to their smartphones, HP introduced the TouchPad; a webOS-

based tablet device that will launch in Summer 2011 (Palm 2011).

On March 2nd, 2011 Apple announced the second iteration of the iPad and that it would be available

in the North American market on March 11th and on March 25th for the international market. Apple

announced their second tablet device significantly later than all competitors, yet were able to launch

the product significantly before all other manufacturers - except Motorola with their Xoom tablet.

Therefore, 2011 will see four mobile operating systems vying for market share; Apple's iOS, webOS,

Android and RIM's Tablet OS. While three of those OSs are not licensed to run on third-party

hardware, Android is open-source and is currently used by several hardware manufacturers;

fragmentation will occur on both the hardware and software aspects of tablet devices.

1.3. Market Potential

The iPad became commercially available on April 3rd, 2010 and sold its millionth device 28 days after

launch. Comparatively, it took 73 days for the original iPhone to reach the 1 million mark. (Yahoo

News 2010). By the end of 2010 - 9 months since launch - Apple had sold over 14 million devices

(Apple 2011). Meanwhile, research firms 'forecast 57 million tablets will be sold' in 2011 and expects

the sector to reach 171 million devices in 2014 (Fox Business 2011) amounting to one in five adults

owning a tablet device (Palm: 2011).

1.4. Method of Study

The fundamental basics of both the International Usability Partners and the University of

Washington’s study consists of this ‘guessability study methodology’ (Wobbrock et al 2009: 1). As the

University of Washington used live hardware, users were shown ‘the effect of gestures’ via an on-

screen animation and then asked to perform a gesture to invoke that effect. However, the

International Usability Partners study utilised a ‘lo-fi prototype simulating a touchscreen’ with before

and after screenshots, asking users how they would produce what was on the after screenshot

(Mauney et al 2010: 2) as to ‘focus on the user’s intuitive behaviour’ (Mauney et al 2010: 13). Dan

Saffer also agrees that it is important to ‘employ the knowledge and intuition of those who will use it’

2 Guidelines for Gestural Tablet Interaction

and similarly suggests using this method, or reversing it by demonstrating ‘a gesture and see what

feature users would expect that gesture to trigger.’ (Saffer 2008: 28).

For the specifics, both studies chose to remove all bias by having ‘no elements specific to Windows

or the Macintosh’ while ensuring that ‘the commands were application-agnostic’ (Wobbrock et al

2009: 2). This is important because the studies wanted to limit the amount of gestures that were

learnt from previous experiences rather than being purely intuitive.

The University of Washington were concerned about the lack of research and that ‘gestures have

been defined by system designers’ and not users, therefore they believe that it is right to ‘consider

the types of gestures people make without regard for recognition or technical concerns.’ (Wobbrock

et al 2009: 1). While the International Usability Partners study selected a range of participants, both

experienced and inexperienced the University of Washington study only ‘recruited non-technical

people without prior experience using touch screens’ as they felt this was the only subset of people

that ‘would behave with and reason about interactive tabletops differently than designers and system

builders.’ (Wobbrock et al 2009: 2). Therefore, it was interesting to note that the International Usability

Partners research found it difficult - even with inexperienced users - to understand when a participant

performed a gesture that was naturally intuitive due to ‘local and media exposure of

touchscreens.’ (Mauney et al 2010: 12).

1.5. Findings

The International Usability Partners study found that on tablet-sized devices there was ‘frequent

usage of the ‘tap’ gesture’ suggesting ‘participants imagined a button or context menu to trigger the

required action with a tap.’ (Mauney et al 2010: 10). Their study produced different results to the

University of Washington’s research into surface computers where ‘users rarely care about the

number of fingers they employ’ (Wobbrock et al 2009: 1). While tablet devices and surface computers

differ in size, tablet devices are large enough to take several inputs at once similar to surface

computers.

An issue with the International Usability Partners report is that is doesn’t specifically state how the

proposed gesture set was developed. Questions remain as to whether it was solely based on the

analysis of their user studies, or a combination of that and a designer’s perspective. Yet, the

University of Washington clearly states that the final gesture set ‘was developed in light of the

agreement participants exhibited in choosing gestures for each command.’ So, if ‘more participants

used the same gesture for a given command, the more likely that gesture would be assigned to that

command’ and the study actually found that the ‘user-defined gesture set emerged as a surprisingly

consistent collection founded on actual user behavior’ (Wobbrock et al 2009: 2).

While the International Usability Partners study was purely quantitative; producing only data from the

gesture testing, the University of Washington used ‘Likert scales concerning gesture goodness and

ease’ (Wobbrock et al 2009: 3) and utilised the think aloud method to gauge how users were feeling

when creating the gestures.


1.6. The Need for a Gesture Set

While the likes of Apple and Google have interface guidelines that they advise third-party developers

to follow, the majority of the guidelines focus on the aesthetics of the application and not the

interaction techniques. In 2000, Jef Raskin stated that modern computers were 'a feast for the eyes,

with glorious colorful backgrounds and three-dimensional tromp l'oeil', a fact still true for modern

tablet devices. Yet while aesthetics play a role in the overall experience, 'the quality of any interface is

ultimately determined by the quality of the interaction between one human and one system' (Raskin

2000: xix).

1.6.1. Habituation

After prolonged and persistent practice, people will habitually complete a task without consciously

thinking about it. Therefore the gestures that are developed for tablet devices will become habit for

users, whether they are good or bad.

The issue with developing habit for bad practices is that 'you cannot break a habit by an act of

violation' but simply by 'a time-consuming training process' (Raskin 2000: 20). It is therefore pivotal

that the correct gestures are learnt by users otherwise they will struggle to habituate to another set of

gestures.

1.6.2. Inconsistent Gesture Usage

Many current iPad apps are creating different methods to complete a similar action, yet this does not

benefit the end user as 'having multiple options can shift your locus of attention from the task to the

choice of method' which drastically hinders the user's experience of the product (Raskin 2000: 20).

Altering a method for an action can act as an unique selling point and is actively promoted in

advertising as shown by Apple's original iPhone adverts that highlighted key interaction styles and

techniques in Figure 1.2.

Figure 1.2 Apple’s ‘How to’ iPhone advert (Apple 2007)

While this differentiates manufacturers from each other, the practice is detrimental to users as

'inconsistent gesture usage can lead to frustration, confusion and usability problems' (Ginsburg 2010:

22).


1.7. Conclusion

When discussing the need for gestural standards, Dan Saffer imagines a scenario where physical

movements would be required to turn on the light in a darkened room. With no indication about what

gesture is required, you attempt several random movements in an attempt to turn on the light. While

this might be an 'imaginable future scenario' for free-form gestural interfaces, these sorts of

scenarios are already appearing for touch-based systems.

Dan Saffer suggests setting up 'some sort of standards board, similar to what the Web has with the

W3C', a board that has helped define the future of the web across a multitude of devices. But, if we

do not work with the individual companies - such as Apple, Google and HP - they will simply

continue to create gestures and try and force them onto end users as standards.

The concern on individual companies leading the charge for gestural standards is the possibility of

them patenting their creations. Saffer states that the actual gesture probably can't be patented as it

fails two US patent requirements; it's natural and too obvious. However, Apple is currently attempting

to patent 'a gesture tied to a specific action' - specifically the Pinch to Shrink gesture - and if this is

accepted, then they will be able to license the patent or force infringing parties to stop using the

gesture. This would then force other manufacturers to create their own gestures so that they won't

infringe on others' patents and will then look to protect these gestures in retaliation. While Saffer

attempts to reassure us by stating that 'patent holders do not have to enforce their patents', they still

have the option of suing infringing parties in their arsenal.

Ctrl + C is the standard command to copy an item in Windows and it is shared by all third-party

Windows developers; the command is even similar on Mac OS X with Command + C (Where

'Command' replaces 'Ctrl' for all commands on Mac OS X). With further patent claims by tablet

manufacturers and operating system developers, users will have to learn 'new commands for actions

such cut-and-paste for every program' and platform (Saffer 2008: 174-176).

If we don't act soon, users will be unable to go from platform to platform and utilise their habitual

knowledge as they will have to 'remember a crazy number of gestures for common actions' (Saffer

2008: 174). We are on the brink of a new computing paradigm which could radically change how

people interact with technology and therefore we should create a set of standard gestures that can

work across different platforms and applications.


1.8. Section Summary

1.8.1. The Market

• The iPad was launched in 2010 and this spurred many other manufacturers to produce their own

competing tablet devices.

• Major companies launching tablet devices in 2011 include Apple, HP, Motorola, RIM, Samsung and

Acer.

• The tablet market is very different to the early desktop computing landscape. Multiple companies

have entered the tablet market, and the dominance of a couple of platforms seems unlikely.

• Industry experts expect the fledgling market to grow substantially over the next 5 years, with the

sector selling over 150 million tablet devices in 2014.

1.8.2. Prior Research

• Tablet devices received little exposure in academia.

• Previous academic research is limited to touch-based phones and surface computers.

• Research involved real users and potential users throughout the entire study.

• They noted the lack of user involvement in current gestural creation.

1.8.3. The Need for Guidelines

• Gestures used by the end users will become habit, whether they are good or bad practice.

• Breaking a habit is challenging and can only be achieved by a time-consuming process.

• Inconsistent gestures have already appeared across both applications and platforms.

• Manufacturers and software developers are keen to protect their intellectual property and have

begun patenting their creations and suing infringing parties. Action/gesture relationships are a key

issue in these lawsuits.


2. Methodology2.1. Introduction

This section discusses the methodology for the entire study, detailing the processes behind the three

stages of creating a set of guidelines for gestural tablet interaction. As shown in Figure 2.1, the

individual stages lead on and inform the proceeding stage.

Inter-Rater Agreement

Guessability StudyGesture-Meaning Association Test

Figure 2.1 The 3 stages of the research

2.2. Researching Ethics

Collecting information from the public can be challenging as they are often concerned about how

their data will be used. Therefore, it is pivotal that I handle their data with care; informing them that it

will be used anonymously. Moreover, making participants more comfortable benefits me as the

researcher because ‘it will yield better results, since the subjects will likely open up more if they know

that they (and their data) are being treated well.’ (Saffer 2007: 74).

2.3. Guessability Testing

2.3.1. Process

Recruit Particpants

Planning

Create Discussion Guide

Guessability Test

Think Aloud Protocol

Observation & Questioning

Data AnalysisPresentation of

Data

Figure 2.2 Guessability testing process

2.3.2. Guessability Method

Prior to undertaking the test, the participants completed a consent form that confirmed they were

willing to participate in the study and that the results from their tests will be used in the report

anonymously.

Following this, they completed a short questionnaire that documented their demographics in terms of

age and gender along with information about their experience using tablet devices. While there was

no specific age range requirements, it will be interesting to see how different subsections of people

could possibly create different gestures. The results from questioning the users’ experience would

ensure I have a 50/50 split; half experienced and half inexperienced.

Chapter 2: Methodology 7

When conducting the actual tests, subjects were shown a short animation - as shown in Figure 2.3 -

that displayed the outcome of the gesture for that action. To ensure that the participants were not

guided to perform a specific gesture, the animations were single slides and lacked any animation. For

example, for the Zoom-in action, the video didn’t slowly transition the smaller squares into larger

squares as this could result in the users performing a gesture based on the video and not based on

intuition.

Slide 1 - Before Screenshot Slide 2 - After Screenshot

Figure 2.3 Zoom-in action slides for video

This approach is similar to the University of Washington’s study where after the animation, the

participants were shown ‘the objects as they were before’ the action was performed and then they

were expected to perform the gesture they felt would invoke that desired action (Wobbrock et al

2009: 4). The before screenshot was placed as the background of a drawing app - called Doodle

Buddy - on the iPad that allowed for multi-finger input and following the drawing of the gesture, I

would take a screenshot of the canvas. After each gesture, the participants were asked to rate their

created gesture based on the statement: ‘The gesture I picked is a good match for its intended

purpose’ and rated it on a 5-point likert scale from Strongly Disagree to Strongly Agree. The rating -

along with the recorded video of the participants performing the gestures and screenshots - will be

used in analysing the results from this test.

2.3.3. Participants

For most usability studies, Jakob Nielsen recommends 5 users as they find 90% of issues. This study

however, isn’t a typical usability study and therefore there isn’t enough data available to set a

precedent.

The International Usability Partners study used 40 participants per country, with 20 experienced and

20 inexperienced users. They gave no reason as to why they selected that number and I also feel that

40 participants are too many for my project due to time and resource constraints.

I attempted to find similarities between this method and other types of usability studies. Card sorting

seems similar as it looks for similarities in user’s decisions with the aim of producing a taxonomy and

gaining understanding of their mental models. For card sorting, Nielsen suggests 15 users. As I also

have two types of users; experienced and inexperienced I would need around 30.


When testing multiple categories, you don’t need to test 15 users per group as there will be overlaps

in the results. As Nielsen discussed the 5 person rule for most usability studies, he stated that you

only need 3 users per category when testing 2 groups. That amounts to 60% of the original 5 person

rule. Therefore, the recommendation would be to use 18 participants; 9 experienced and 9

inexperienced.

Usually, the selection ‘of participants whose background and abilities are representative of your

product’s intended end user is a crucial element of the testing process,’ yet one main aim of this

study is to discover how outcomes from experienced and inexperienced users differ (Rubin 1994).

Moreover, the modern tablet market is still maturing and there is no knowledge of what specific types

of people currently use tablet devices and who will in the future.

2.3.4. Equipment & Materials

Device Usage

iPad For showing the short animations and for completing all the guessability testing

Panasonic Lumix TZ7 For recording the participants performing the gestures

Table 2.1 Set of actions to be used in the Guessability Testing

For the study, the participants will be using an Apple iPad. Unlike the International Usability Partners

study which utilised webcam rig that recorded the piece of paper that was placed on an immovable

object, I feel it is important to allow the user to fully embrace a tablet by being able to grasp and hold

the physical object in a way that is natural to them. By making participants feel more relaxed by

allowing them to use the tablet, the results extracted from the study will be more reliable. I will use an

iPad application that allows me to place an image - for this study; a screenshot - in the background

and then ask the participant to enact a gesture that they feel will accomplish the given task.

2.3.5. Environment

To further allow participants to relax I will use a guerrilla usability testing style. One method is known

as coffee shop testing where ‘researchers go to their local coffee haunt in search of study

participants.’ (Ginsburg 2010: 181) This method is also beneficial to me as I lack the big budget

required to have a usability lab and entering such labs can often be a daunting task for any

participant. The consumer electronics companies are advertising these products with relaxing

imagery; lounges and living rooms with the users resting their feet on a table. I aim to mimic this calm

environment to ensure my results are reliable.

2.3.6. Think Aloud Protocol

The guessability method yields a lot of quantitative data, however to fully understand users’ mental

models, qualitative data is necessary. During the studies, participants were asked to think aloud as

this will allow us to ‘obtain rich qualitative data that illuminates user’s mental models’ (Wobbrock et

al. 2009: 1). Utilising this method allowed us to have participants ‘express their confusion, frustration

and even their delight’ as well as discuss their thought process and reasoning behind the

gestures .they were performing (Rubin 1994: 218). Moreover, it also allowed me to ask more informed

questions after each gesture as I had a basic understand as to why they felt that gesture should

invoke a specific action.


2.3.7. Questioning

Working in collaboration with the think aloud protocol, questioning participants allowed us to ‘fully

understand the “why” behind performance and preference’ (Rubin 1994: 219). I also ensured that I

was asking open-ended questions to ‘elicit more and better information’ than closed-ended

questions as they can suggest to the participants that you are only looking for a limited answer that

rarely yields enough qualitative information. Moreover, the open-ended questions ‘help prevent

“leading the witness”’ which is especially critical when creating a set of guidelines for interaction

(Goodwin 2009: 60).

2.3.8. Set of Actions

Based on the International Usability Partners and University of Washington studies, along with Dan

Saffer’s Designing Gestural Interfaces, I selected 18 actions that are required for every day

computational tasks. The 18 actions are shown in Table 2.2.

No. Action No. Action

1 Select single object 10 Back

2 Select multiple objects 11 Forward

3 Zoom-in 12 Rotate

4 Zoom-out 13 Open menu

5 Help 14 Close menu

6 Print 15 Move object

7 Undo 16 Cut

8 Redo 17 Copy

9 Go home 18 Paste

Table 2.2 Set of actions to be used in the Guessability Testing


2.4. Inter-Rater Agreement

2.4.1. Process

Recruit Particpants

Planning

Create Discussion Guide

Inter-Rater Agreement Test


Data

Figure 2.4 Inter-Rater agreement process

2.4.2. Testing Method

The inter-rater agreement method is similar to the categorisation tests that are utilised in icon testing,

in which participants are asked to ‘arrange icons into meaningful groups’ to measure the ‘users’

recognition of the images and also their understanding of the concepts represented by the

images’ (Horton 1994: 301).

Following the guessability study, I had 18 gestures per action and I had to decipher a single gesture

from all of those options that would be the sole representation for a specific action. For certain

actions, many gestures were very similar, yet had individual traits that also made them slightly

different. Therefore, I had to find a consensus, where multiple people could categorise the gestures,

allowing me to recognise the similarities in participants’ groupings. These similarities would allow me

to understand different participants’ mental models which will provide a solid foundation when

crafting the set of guidelines for gestural tablet interaction.

However, participants would have to categorise all of the gestures in one attempt, and with 18

gestures per action, with a total of 18 actions that would require the participants to view 324

gestures. Moreover, there was concern about participants forgetting why certain gestures were

placed into certain categories as they progressed through all 324 gestures. Users would also

probably review already-categorised gestures which would further prolong the test.

Therefore, I decided to initially run a pilot test for this categorisation method so this method’s

effectiveness could be evaluated. The pilot would involve 3 participants and individually, they were

asked to group 90 gestures (based on 5 actions, each with 18 gestures), placing gestures that they

felt were similar together.

2.4.3. Data Analysis

I will be utilising the Fleiss’s Kappa method as it allows me to measure category agreement between

several raters. The result from the Fleiss’s Kappa measure will be a number between 0 and 1, where a

higher number would suggest more of an agreement between the raters, therefore suggesting a

better agreed category.

Based on this category score, I will be able to select the action/gesture pairings that are agreed upon

by the participants of the inter-rater agreement study; resulting in a more reliable set of guidelines for

gestural tablet interaction.


2.5. Gesture-Meaning Association Testing

2.5.1. Introduction

The legitimacy of gestures hasn't been tested until recently. While there are many precedents in other

aspects of usability study design, there is no common way to test a set of gestures to see whether

the relation with the desired action is intuitive.

Icons and gestures are, by definition, similar as they are signs or symbols that refer to a specific item

or action. However, icons benefit from being a visual symbol that is placed on the visual user

interface while gestures are invisible until the gesture is performed.

2.5.2. Process

Recruit Particpants

PlanningGesture-Meaning Association Test


Data

Figure 2.5 Gesture-Meaning Association process

2.5.3. Participants

When Jakob Nielsen conducted icon intuitiveness tests, a small set of users were invited to

participate; around 3 - 5 users. While this is a small number, it was important to select the correct

users as ‘no testing is effective without adequate test subjects.’ (Horton 1994: 293) and ensure that

the test subjects reflect the tablets’ user base. Moreover, William Horton suggests that the

participants shouldn't have ‘inappropriate knowledge of the product’ (1994: 293), however we can

not ignore the fact that Apple - along with other manufacturers - are teaching people how to use the

products through their multi-million dollar advertising campaigns and have to accept that this could

affect some users’ decisions.

2.5.4. Testing Method

There are two main methods to test the effectiveness of icons and they can be adapted to suit

gesture evaluations too. William Horton suggests usability testing and image-meaning association

analysis.

When conducting usability testing, icons are placed 'in a prototype of the user interface and we test

whether users can do meaningful work with the system.' This allows us to analyse how well the icons

integrate 'with other aspects of user-interface design' (Horton 1994: 301). Even though this can also

be done with lo-fidelity prototypes, it won't validate the set of gestures I create. To inform the design

of the gestures I utilised a tablet device that participants used to create gestures based on a desired

action. If this method was selected, we would merely be recreating the exact same test from the

previous stage. Therefore, to validate the gesture set we need to reverse this concept and ask a

different set of participants what they expect the gesture to do. For icon design, this is known as

image-meaning association which measures 'the degree to which the graphic chosen for the icon

represented the intended concept' (Nielsen 1995). For the purpose of gesture testing this method will

be known as gesture-meaning association.

As shown in the following chart, there are several options for gesture-meaning association. This

project used the gesture-meaning matching method as it provided us with more data; allowing us to


'observe which are matched first and which later' and 'which are matched quickly and which are

matched only by a process of elimination' (Horton 1994: 300). This knowledge was fundamental in

ranking the gestures in terms of user acceptance.

Method Description

Open-ended meaning-for-

gesture

Shows the participant the gesture and asks “What does this gesture mean?”

This tests the association a user already has for a gesture.

Multiple-choice meaning-

for-gesture

Presents the user with a gesture and a list of possible meanings for the user to pick

from. This requires four or five plausible choices for the testing to be valid.

Gesture-for-meaning Asks the user to identify which of several gestures best represents a concept. Can use

it to test for completely different gestures or test different gestures for a single action.

Gesture-meaning matching Provides participants with gestures and verbal labels for the same concepts but in a

different order. Allows us to observe how quickly and accurately users match the image

with the appropriate label.

Table 2.3 Gesture-meaning association options (Horton 1994: 297-300)

Therefore, I had 32 pieces of paper; 18 slips that listed the actions and another 18 that showcased

the gesture. The action slips were placed on one side while the gesture slips were placed on the

other; random in order. I then, individually, asked the three participants to match the action/gesture

pairs.

2.5.5. Data Analysis

For the final aspect of this report, I analysed two performance measures as documented by William

Horton (1994: 302):

Method Description

Accuracy of selection Following the completion of the exercises, I checked the action/gesture relationships

that were successfully paired and asked the participants about the pairings that were

incorrect to discover why they felt their was a relationship between that specific action

and gesture

Speed with which

selections are made

I also noted the length of time it took for each pair to be chosen; paying special

attention to the pairings that seemed to be created through a process of elimination

Table 2.4 Data analysis on the results from the Gesture-meaning association tests



2.6.1. General

• There are three stages to this research; guessability study, inter-rater agreement testing and then

gesture-meaning association testing.

• These individual stages led on from each other and informed the proceeding stage.

2.6.2. Guessability Study

• 18 participants individually created gestures for 18 commonly used computational actions.

• 50% of the participants had prior tablet experience, while the other half hadn’t used a tablet device

before.

• The participants used an iPad to draw the gestures on, and a screenshot was taken for each

gesture created while the entire test was video recorded.

2.6.3. Inter-Rater Agreement

• It is a categorisation test that is used in icon usability testing.

• Gestures for specific actions from the guessability study were similar, yet had unique traits that

might differentiate them from one another to a variety of users.

• Participants are asked to arrange the gestures created from the guessability study into categories.

• Will allow me to understand users’ mental models.

• The test would require each participant to watch all 324 gestures and categorise them. I was

concerned over the possibility of participant fatigue.

• Decided to run a pilot with 3 participants who would group 90 gestures based on 5 actions that

each had 18 individual gestures.

• If successful, the whole test will be initiated.

2.6.4. Gesture-Meaning Association Testing

• Currently, there is no precedent in the process of validating gestures.

• This test is also based on icon usability testing.

• 32 slips of paper will be placed in front of a participant; 18 actions and 18 gestures.

• The 3 participants will be asked to match a gesture to a specific action.

• Will record accuracy of selection and speed with which selections are made.


3. Formative Investigation3.1. IntroductionIn the previous section, I introduced the methodologies that would be utilised throughout this

research; guessability study, inter-rater agreement and gesture-meaning association testing. Section

3 is an account on the first two stages; the guessability study and the inter-rater agreement.

It will detail the issues discovered with the inter-rater agreement pilot that took place with three

participants and how I revised that stage of the research to ensure that I would still produce similar

ranking results to inform the final part of the project.

3.2. Guessability Study

Figure 4.1 Subject B participating in the Guessability Test

As shown in Appendix A, I documented all 18 tests through consent and feedback forms, along with

a monitoring sheet where I detailed my observations while the participants were creating their

gestures. Appendix B showcases all 324 gestures that were created by the 18 participants.

To ensure participants were relaxed and in an environment where they felt they would use a tablet

device, I allowed the individual participant to decide where their test would take place. By using 18

separate locations, I had to continually setup the camera to record the iPad screen, ensuring the

environment was well lit and that the device was clearly in shot. Yet, if I had utilised a single location

for the tests, I would have saved a considerable amount of time as the test layout would have already

been setup. While I believe the inconsistent locations provided more accurate results, the additional

time did lead to increased participant stress due to the build-up process of setting up the test.

3.3. Inter-Rater Agreement

During the methodology section, I discussed the need to have a consensus on the type of gestures

being performed - a tap, swipe, curve etc - and therefore decided to use the inter-rater agreement

method. Yet there were concerns about the complexity and time-consuming nature of the method

and as a result, I chose to run a pilot.

For the pilot, I selected 3 participants to group 18 number of gestures from a set of 5 actions, with a

total of 90 gestures overall. Individually, each participant was asked to watch a gesture video and

Chapter 3: Formative Investigation 15

group the gestures on how they saw fit. This began successfully, however participants started to

struggle as they reached the thirties and were constantly reviewing previous gesture videos to decide

what group the current gesture belongs to. This was concerning because the study had amassed 324

gestures among 18 participants and 18 actions, yet the participants were struggling with only 10% of

all gestures in the pilot. Therefore, it was decided that I should terminate the pilot and consider other

options.

3.3.1. 2nd Stage Revision

Instead, for each action I grouped the 18 gestures based on the type of gesture. Following this, I

asked 3 people to rate the effectiveness of the gesture for that action; the same question that I asked

the original guessability study participants. By averaging these ratings, I was able to have a

consensus on whether specific gestures were a good match for the intended action. To stem the

possibility of fatigue, participants were asked to rate only one action at any one sitting, amounting to

only 18 gestures at a time. This process therefore, required a significant amount of time and lasted

two weeks.

Figure 4.2 A rater participating in the replacement rating study

By combining my grouping data along with their ratings I was able to extrapolate which type of

gestures were accepted among participants. This data then allowed me to create the set of gestures.




• This stage took place without any major issue.

• Allowing the participant to decide the test location resulted in a time-consuming setup process for

every test. This led to slightly increased participant stress.

• 324 gestures were created by 18 participants.

• Appendix A documents the consent, feedback and expert forms for all 18 participants.

• Appendix B shows all 324 gestures that were created.

3.4.2. Inter-Rater Agreement

• During the early stages, this stage was progressing well.

• Raters struggled with the test when they reached around the 30th gesture of the test.

• They began re-watching previous gesture videos to ensure they would put the current gesture in

the correct group.

• There are over 300 gestures, yet the raters were struggling with only 10% of the overall gesture

amount.

• Decided to terminate the pilot and select another method.

3.4.3. 2nd Stage Revision

• I grouped the gestures based on the type of gesture - for example: tap, swipe, pinch.

• 3 raters would rate each individual gesture on it’s relevancy to the specific action.

• Asked the same question as the guessability study participants - ‘The gesture they picked is a

good match for its intended purpose’

• Coupling the gesture groupings with the rating scores allows me to identify good matches for

action/gesture relationships.

• This stage will inform the design of the set of guidelines for gestural tablet interaction.

Chapter 3: Formative Investigation 17

4. Findings4.1. Introduction

Chapter 3 discussed how the guessability study and inter-rater agreement pilot progressed. I also

examined the issues concerning the inter-rater agreement pilot and why there was a need for another

approach as the raters were struggling to complete the entire test.

In this chapter, I will discuss the results from the first two stages; guessability study and the inter-

rater agreement replacement test. The second stage was replaced with a basic rating routine that

didn’t require the raters to go through the entire study in one sitting.

4.2. Guessability Study

Appendix B documents the type of gestures created by the participants for each action.

Figure 4.1 A selection of gestures for the Help action (top row) and the Close menu action

4.2.1. Desktop Metaphors

Like other studies, I discovered that many participants were relying on the desktop-based metaphors

that have been prominent for over 30 years. Even though I reiterated the multi-touch capability of the

device - even demoing it to them prior to the user study - many users still only utilised a tap; akin to a

single click or double click on a desktop mouse. This finding was also noted in the University of

Washington study where ‘participants still often thought of the desktop paradigm’ (2009: 8), while my

study found that 23% of all gestures were either single tap or double tap. This statistic is quite

concerning as it could result in a reliance of desktop metaphors that were designed over 30 years

ago and fail to take advantage of the unique multi-touch display that blankets the front of modern

tablets.

4.2.2. The use of Letters and Symbols

With several actions, many participants utilised letters or symbols as their choice of gestures; with in

many cases being the most popular choice. This was a very interesting discovering as it was a huge

shift from the desktop-based gestures that were more common for other actions in this study.

Chapter 4: Findings 19

It is also very interesting to note how participants were drawing letters or words to describe the

action they desired during the studies; letter or words associated with desktop-based shortcuts.

Tap and Hold

ʻPʼ Letter

Multi-Tap

Tap

Swipe

0 2 4 6 8 10 12Strongly Disagree Disagree Neutral Agree Strongly Agree

For example, when asked what gesture they would choose to print a document, 6 out of 18

participants decided to draw the letter ‘P’ on the screen. Even though the letter ‘P’ is a rather

common shortcut on the desktop - and would probably be used by inexperienced desktop users;

one participant even utilised the more obscure shortcut letter ‘Y’ for redo.

However, once participants thought of using a letter for an action, it seemed to be the default gesture

for subsequent actions that were deemed challenging by the participants. This is in sharp contrast to

the University of Washington study where they found the single tap gesture was utilised for difficult

actions instead (Mauney et al 2010: 10).

Moreover, I feel apprehensive in using symbols or letters as gestures due to their ‘potential for

showing differences across cultures’ and therefore gestures will need to be adapted across cultures

and countries (Mauney et al 2010: 6). Yet, I can’t dismiss the overwhelming support for symbolic

gestures that were created by the participants of this study while developing my own gesture set.

4.2.3. Interaction with other objects

When I asked Subject M what gesture they felt was suitable for printing, they performed a swipe

gesture in the direction of the printer in their bedroom. Even though only one participant executed

this gesture, I found it to be rather significant. When I asked him to explain their choice, Subject M

stated that it felt natural to gesture in the direction of his printer, yet would not feel comfortable

performing a similar move with a mouse on a desktop computer. Unfortunately, he was unable to

further explain his choice of gesture, however, while observing the process, it seemed like he had

forgotten about the constraints of desktop computing - one of the points I hoped all participants

followed - and imagined how he would wish to interact with future technology.

Figure 4.2 Print action gestures


Aza Raskin believes that the job of an interface designer is to remove as much of the interface as

possible so ‘you don’t even realize you are using’ an interface, because ‘the best interface is no

interface' (Raskin 2008). It is therefore, these types of gestures that make ‘invisible interfaces’ a viable

option in tablet devices - more so than on the desktop - due to the range of possible input methods.

This finding is both a positive and negative result from my insistence of participants deciding on

where the tests should take place - based on where they felt comfortable - as this result wouldn’t

have been discovered in a closed lab environment, yet the lack of printers in other participant’s

locations mean’t that this outcome was not possible for them.

4.2.4. Smartphone experience and advertising influence

Many of the participants for this study were under 30 and were therefore aware of touchscreen

devices; many having used a touch-based smartphone within the past couple of years. Therefore,

even without having experience on a tablet device, many participants were aware of gestures utilised

on the iPhone and Android-based smartphones; gestures that are still relevant for tablets.

Moreover, many participants stated that the gestures they performed for certain actions, were based

on the methods seen in television advertising for smartphones; especially the iPhone. While 50% of

participants were required to have no tablet-based experience, the ubiquitous nature of touch-based

smartphones influenced the results of this study.

4.2.5. Struggling to perform gestures

Occasionally, some participants were unable to perform a gesture for a specific action. Before the

guessability study, I presumed that some users would be unable to execute gestures for cut, copy

and paste as I deemed them to be the most difficult actions in the set. However, it was interesting to

discover that Participant C - an inexperienced, over 50s participant - was unable to perform both the

zoom-in and zoom-out actions. The overwhelming majority of participants used the pinch gesture; a

gesture that wasn’t performed by British users in a 2010 study (Mauney et al 2010) and therefore

suggests that people can quickly adapt to different gestures for commonly used actions at this early

stage in tablet computing. Yet, the inability of Participant C to create gestures for these actions and

her concern for being unable to complete these tasks shows how the intimidating and inaccessible

nature of desktop systems still remain for tablet devices, in regards to basic computer users who lack

the experience of smartphone devices and aren’t influenced by the manufacturers’ adverts.


4.2.6. Need for Context Sensitive Menus

For many actions, participants requested context sensitive menus that would be displayed following

a basic gesture. For the cut, copy and paste actions, 70% of participants asked for context sensitive

menus following gestures such as: tap and hold, multi-tap, and tap as shown in Figure 4.3.

6%13%

12%

6%

35%

29%

Tap and Hold Multi-Tap Tap Letters & Symbols Other No Attempt

The IUP study also recognised similar results as their ‘participants consistently requested [context

sensitive menus] for some actions, and they are particularly useful for novice users’ and therefore

they created two gestures - for certain gestures - for both novice and expert users.

While this study was trying to discover what gestures best suit certain actions, it is also important to

uncover what actions are difficult to convey through a gesture and what ones require additional on-

screen support to be completed.

4.3. Rating Study

4.3.1. Relationship between total number of similar gestures and user agreement

Prior to the Rating Study, I expected that the gesture types that were performed the most would be

rated with a higher score. While this was the case for the majority of actions, some saw the opposite.

Tapping the top-right and swiping were the most popular gestures for the Close Menu action;

accounting for over 60% of all created gestures. However, in the Rating Study, the Swipe gesture

received a score of 2.9 approval and Top-Right Tapping had an approval rating of 3.7, yet it was the X

gesture that received the highest approval score. Drawing an X on the menu received a score of 4.5,

even though only one participant in the Guessability Study choose that gesture.

Figure 4.3 Percentage of gesture types for cut, copy & paste actions


0

1

2

3

4

5

6

Swipe ʻXʼ Letter Multi-Tap Pinch Tap Top-Right No AttemptAgreement Average Total No. of Gestures

When creating the set of gestures, issues like these drastically affect what gestures I should select for

a specific action. I can not create a gesture by combining two gestures as it will lead to a sign that

was neither created by users nor approved by them.

4.3.2. Advertising Influence

It was interesting to note the very high approval rating of 5 gestures across 5 actions; gestures that

are widely used in advertising smartphones and tablet devices. These gestures received an approval

rating of over 4.5 for their relevant action.

Action Dominant Gesture

Select single object Tap

Zoom-in Reverse-pinch

Zoom-out Pinch

Back Swipe

Forward Swipe

For example, the pinch and reverse-pinch gestures scored a 4.6 approval rating for the zoom-out and

zoom-in actions respectively; compared to the 1.5 approval rating average of the other two gestures

created by the Guessability Study participants.

This finding was also noted by the IUP study and they questioned ‘whether the gestures performed

were naturally instinctive’ due to the obvious influence from advertising. Moreover, it was these

gesture/action combinations that were popular among both experienced and inexperienced

participants; suggesting that people are able to learn and recall gestures that they have seen before;

even without prior experience of performing the gesture (Mauney et al 2010: 12).

Figure 4.4 Comparison between Agreement Averages and Total No. Of Gestures for Close Menu

Table 4.1 List of popular gestures among advertised gestures


4.3.3. Approval of Symbols & Letters as Gestures

In section 4.2.2, I discussed my concern over some participants using letters as the default gesture

when they reached a difficult action; therefore not challenging themselves to create other options.

However, the Rating Study participants approved of the use of letters as gestures for certain actions

as shown in Figure 4.5 which details the approval ratings of the Paste action gestures.

0

2

4

6

8

Tap and Hold ʻVʼ Letter ʻPʼ Letter Curve Multi-Tap Tap No AttemptAgreement Average Score Total No. of Gestures

As witnessed with this example, the ‘P’ letter received a low approval rating of 1.3. However, if we

combine this information with data from the Print action, we discover that the Rating Study

participants rate letters associated with the relevant desktop shortcuts higher than ones that are not.

0

2

4

6

8

Tap and Hold ʻPʼ Letter Multi-Tap Tap Swipe

Agreement Average Score Total No. of Gestures

Figure 4.5 High approval rating of the ‘V’ Letter

Figure 4.6 High approval rating of the ‘P’ Letter for the Print action




• There seemed to be a reliance on desktop-based metaphors as 23% of all gestures were either

single-tap or double-tap.

• The use of letters and symbols for gestures was very popular. Associated letters were mainly based

on the desktop shortcut rather than the action word. Eg. V for the Paste action instead of P.

• For the Print action, one participant swiped the text towards the physical location of his printer in

their room.

• Many participants used gestures they have learnt from smartphone advertising campaigns as these

gestures are still relevant on tablet devices.

• The request for context-sensitive menus was overwhelming. For the cut, copy and paste actions,

70% of participants asked for a context-sensitive menu. This suggests that not all actions can be

easily defined by a single gesture.

4.4.2. Rating Study

• Certain gestures that were more popular for specific actions didn’t necessarily score a good

approval rating.

• Sometimes gestures that were created only once would score a high approval rating. Eg. ‘X’

gesture for the Close menu action was implemented by one participant in the guessability study,

yet scored an approval rating of 4.5 out of 5.

• Widely advertised gestures scored an approval rating of 4.5 out of 5 or above by the raters. Eg.

Pinch for the Zoom-out action and Swipe for the Back and Forward actions.


5. The Gesture Set5.1. Introduction

Chapter 4 explored the results from both the guessability study and the rating study, examining the

most popular gestures created by participants and how popularity doesn’t necessarily improve a

gesture’s approval rating among other users. I discussed the role of advertising and how it could

have affected how instinctively the gestures were performed. Moreover, I discovered that using letters

as gestures for certain actions were very popular, yet are not adopted by current tablet devices.

Figure 5.1 details the guidelines for gestural tablet interaction that I created based on the findings

from the two prior procedures; the guessability study and then the following rating study.

5.2. Guidelines for Gestural Tablet Interaction

Action Proposed Gesture

Select single object Tap item

Select multiple objects Tap and hold one item and sequentially tap other items

Chapter 5: The Gesture Set 27


Zoom-in Reverse pinch with two fingers on the canvas

Zoom-out Pinch with two fingers on the canvas

Help Draw symbol ‘?’ anywhere on the canvas

??Print Double-tap anywhere on the canvas to open a context sensitive menu

Shortcut: Draw letter ‘P’ anywhere on the canvas

Context-Sensitive Menu P

Context-Sensitive Menu P



Undo Double-tap anywhere on the canvas to open a context sensitive menu

Shortcut: Draw letter ‘Z’ anywhere on the canvas

Context-Sensitive Menu Z

Context-Sensitive Menu Z

Redo Double-tap anywhere on the canvas to open a context sensitive menu

Shortcut: Draw letter ‘Y’ anywhere on the canvas

Context-Sensitive Menu Y

Context-Sensitive Menu Y

Go home Swipe down with one finger on the canvas to reach the bezel of the device

Back Swipe with one finger from left to right on the canvas

Page 2 Page 1Page 2 Page 1



Forward Swipe with one finger from right to left on the canvas

Page 1 Page 2Page 1 Page 2

Rotate Tap and hold the corner of the item and curve in the desired direction

Hello

HelloHelloHello

HelloHello

Open menu Double-tap anywhere on the canvas to open the menu

MenuMenu

Close menu Draw letter ‘X’ inside of the menu box

MenuMenu

\ XMenuMenu

\ X



Move object Tap item and then drag to the desired location

Cut Double-tap anywhere on the canvas to open a context sensitive menu

Shortcut: Draw letter ‘X’ on the item to be cut

Context-Sensitive Menu X

Context-Sensitive Menu X

Copy Double-tap anywhere on the canvas to open a context sensitive menu

Shortcut: Draw letter ‘C’ on the item to be copied

Context-Sensitive Menu C

Context-Sensitive Menu C

Paste Double-tap anywhere on the canvas to open a context sensitive menu

Shortcut: Draw letter ‘V’ on the location where the item is to be pasted

Context-Sensitive Menu V

Context-Sensitive Menu V

5.2.1. Need for Shortcut options

As 6 actions required a context-sensitive menu to be displayed following a gesture, there was a need

to create shortcut gestures for these actions that would provide advanced users immediate access to

that specific feature. The shortcut gestures were based on the letters used in desktop shortcuts as

these are already well known and do not depend on cultural knowledge.



• Created a set of guidelines for gestural tablet interaction.

• 18 gestures crafted for 18 actions.

• Based on results and findings from the guessability study and rating study.

• 6 actions have shortcut gestures associated with them as well because their primary gesture

launches a context-sensitive menu.


6. Summative Investigation6.1. Introduction

The previous section introduced the set of guidelines for gestural tablet interaction that I had created

based on the guessability study and rating test. I created a set of 18 gestures that were individually

paired with a specific action; some functions requiring two gestures as the primary gesture only

displayed a context-sensitive menu.

As I have created the set of gestures, Chapter 6 details the findings and results from the validation

stage which utilised the gesture-meaning association testing method. The results from this section

will determine if the action/gesture relationships - and therefore the overall gesture set - are clear and

incontestable.

6.2. Gesture-Meaning Association Testing

As shown in Appendix D, I conducted a Gesture-Meaning Association test with three participants

where they paired 18 common computer actions to 18 gestures.

Figure 6.1 Raters during the Gesture-Meaning Association testing

Chapter 6: Summative Investigation 33

6.3. Alterations to Original Methodology

It was originally planned that all created gestures would be assessed through the Gesture-Meaning

Association method, however not all the gestures created can be adequately assessed using this

procedure. This concerns the gestures that require a context-sensitive menu to be displayed

following a double-tap gesture and therefore these will not be assessed during this stage. However,

all actions that utilised a context-sensitive menu, also had a shortcut gesture associated with it - for

example: the letter ‘P’ for printing - and these will be assessed.

6.4. Results

The results from the gesture-meaning association test are shown in Table 6.1 and it details the

number of participants who correctly matched the action/gesture relationship and the order in which

a pairing was created.

Participant 1Participant 1 Participant 2Participant 2 Participant 3Participant 3

Action/Gesture Pairings Matched? Order Matched? Order Matched? Order

Select single object ✔ 4 ✔ 1 ✔ 4

Select multiple objects ✔ 5 ✔ 9 ✔ 8

Zoom-in ✔ 1 ✔ 3 ✔ 2

Zoom-out ✔ 2 ✔ 4 ✔ 3

Help ✔ 8 ✔ 10 ✔ 5

Print ✔ 9 ✔ 11 ✔ 10

Undo ✔ 11 ✔ 8 ✔ 9

Redo ✘ - ✘ - ✔ 12

Go home ✔ 17 ✔ 16 ✘ -

Back ✔ 3 ✔ 5 ✔ 6

Forward ✔ 6 ✔ 6 ✔ 7

Rotate ✔ 10 ✔ 7 ✔ 13

Open menu ✔ 15 ✔ 13 ✘ -

Close menu ✔ 13 ✔ 12 ✔ 11

Move object ✔ 7 ✔ 2 ✔ 1

Cut ✔ 16 ✘ - ✔ 16

Copy ✔ 12 ✔ 14 ✔ 14

Paste ✔ 14 ✔ 15 ✔ 15

Table 6.1 Gesture-Meaning Association test results


6.5. Findings

6.5.1. Gestures & Actions that were Easily Paired

Out of the 18 actions, 9 of them - shown in Table 6.2 - were very easily paired with the correct

gesture by all participants.

Easily Paired ActionsEasily Paired ActionsEasily Paired Actions

Select single object Select multiple objects Help

Zoom-in Zoom-out Rotate

Back Forward Move object

Table 6.2 Easily paired actions

While the majority of those gestures are already well known and accepted gestures - thanks to

manufacturers’ advertising - I was pleased to see the Help action be easily paired. This ? gesture is

currently not implemented by any major tablet OS - iOS, Android and WebOS - yet the universal

symbol seems to be recognised and supported by all participants - across all three stages - in this

report.

6.5.2. Difficult Actions & Gestures to Pair

During this process, I discovered that there were three actions that all participants had difficulty in

pairing with a gesture; Redo, Go home and Open menu. While the Redo and Open menu gestures are

founded in desktop-based knowledge, the Go home gesture is rather distinct, even if it mimics the

minimizing gesture on Windows and Mac OS X.

Figure 6.2 Minimizing effect on Mac OS X

Therefore, the Go home action was one of the final actions to be paired with a gesture; often being

chosen through a process of elimination. However, while discussing the action following the test,

participants understood the reasoning behind the gesture, but two testers admitted that they are

concerned by the hidden nature of the gesture.

All of the shortcut gestures created as part of the set of gestures were based on the desktop

shortcuts for that action, yet the ‘Y’ Redo shortcut is only known to Windows users as Mac OS X

utilises another letter for the Redo function on their computers. This issue was only raised because

two of the Gesture-Meaning Association participants were Mac OS X users and therefore were

unable to associate the ‘Y’ gesture to the Redo action.

6.5.3. Unfounded Concerns over Cut, Copy and Paste

Due to the trouble creating the gestures - both during the Guessability Study and the gesture

production stages - for Cut, Copy and Paste, I predicted that these three actions would be difficult to

pair up with the correct gestures. Yet, while these actions were some of the last to be paired -


probably due to the simplicity of many other action/gesture pairings - two participants successfully

paired these actions and gestures together. The participants were pleased that the gestures utilised

their prior computing knowledge; echoing the support from a participant in the University of

Washington study who stated ‘anything I can do that mimics Windows—makes my life

easier’ (Wobbrock et al 2009: 8).

6.5.4. Confusion between Gestures & Actions

As participants always had access to every action and gesture, they were able to change their

decision at any time during the process. This allowed me to note any incident where gestures were

paired with incorrect actions, only to be later paired with the correct action; informing me of possible

confusion when the gestures are used as a group rather than as individuals.

There was slight confusion with the ‘P’ gesture, as two participants originally paired the gesture with

the Paste action instead of the correct Print action. This was before the participants realised that the

shortcut gestures were based on their desktop counterparts and therefore the pairings were rectified

before the end of the test.



6.6.1. Alterations to Original Methodology

• When creating the set of guidelines for gestural tablet interaction, 6 gestures had a primary gesture

that displayed a context-sensitive menu. To assist advanced users, a shortcut gesture was

included for those actions.

• The primary gesture for the context-sensitive menu is difficult to validate because there is no direct

connection between the action and the gesture.

• The 6 gestures that displayed a context-sensitive menu were not validated, but the secondary

shortcut gestures were validated using the gesture-meaning association test.

6.6.2. Findings

• 9 action/gesture relationships were correctly paired very easily by all participants, 8 of which are

actively used in manufacturers’ advertising campaigns.

• The Redo, Go home and Open menu actions were very challenging for participants to correctly

pair. The Redo shortcut is based on a Windows-only shortcut that isn’t available on Mac OS X.

Other actions seemed to be paired by a process of elimination at the later stages of the gesture-

meaning association test.

• Cut, copy and paste actions were correctly paired by 2 participants. They were pleased they could

utilise their desktop knowledge when using a tablet device.

• Initial confusion between certain gestures and actions; two users initially placed the P gesture with

the Print action, before correcting their mistake and placing it with the Paste action.


7. Discussion & Conclusion7.1. Conclusion

I have presented a study that examined and understood what gestures users associate with specific

actions, leading to the creation of a set of guidelines for gestural tablet interaction. During the

guessability study, I gathered 324 user-generated gestures across 18 actions, by 18 participants.

These were then examined and grouped by gesture type, and given an approval score based on the

appropriateness of the action/gesture relationship. Following this, I created a set of 18 action/gesture

pairings which were then validated by another set of participants by utilising the gesture-meaning

association method.

It is hugely important that we can establish a set of common gestures shared across platforms and

applications to eradicate the possibility of a fragmented interaction style on this new computing

paradigm. If tablets follow a similar path as desktop computers, then they have the possibility of

being in widespread usage for over 30 years. While desktop computers have improved their graphics

and industrial design, the fundamentals of desktop computing hasn’t changed since their formation

at Xerox PARC in the 1960s and 1970s. Therefore, it is pivotal that before we see widespread

adoption of tablet devices, we must create a set of common gestures that can last a generation.

7.2. Value & Benefit

With several manufacturers gearing up to launch and release their tablet devices this year, I have

created a set of platform-agnostic gestures, based on the input of 18 original participants who

produced over 320 individual gestures across 18 actions. This report has shown that there is no need

for platform-specific gestures and that a common set can be shared across a multitude of platforms.

This therefore will allow users to move from platform to platform and utilise their habitual knowledge

of a universal gesture set.

Throughout this report, I emphasised the importance of user involvement as it allowed me to

understand what gestures felt natural for them to perform, as well as understand participants’ level of

action/gesture pairing acceptance. Users being the driving force behind the creation of the set of

guidelines for gestural interaction should be an obvious choice, yet this approach is absent from

current tablet implementations.

Therefore, I involved current and potential tablet device owners, asking them to create, assess and

validate the set of 18 gestures. Yet, this report discovered more than just 18 gestures, but provided

information on action/gesture relationships that are not approved by users. This means we have the

knowledge of what gestures to avoid in future so users don’t learn incorrect gestures and develop

habit for bad practices.

It is important to note that the set of gestures has been validated by another set of participants,

which shows that these action/gesture relationships have been accepted and understood by a

Chapter 7: Discussion & Conclusion 39

number of participants. Interestingly, the University of Washington study ignored this step and just

released their set of gestures for surface computers based on a similar guessability study. While a

gesture might be natural to draw, it doesn’t necessarily mean that the gesture can be interpreted as a

depiction of a specific action.

7.3. Implications

Moreover, we witnessed gestures that are not included in current tablet devices; especially the use of

letter or symbols. This discovery allows for the possibility of a huge number of potential gestures

based on letters and symbols. It allows users to utilise their pre-existing computing and cultural

knowledge on tablet devices; reducing the learning process of new users. However, it must be noted

that while the majority of letter-based gestures created in this report are based on universally-

accepted desktop shortcuts, future letter-based gestures need to be tested for cultural differences.

During the Guessability Study, we discovered that many participants were asking for context-

sensitive menus to be displayed for certain actions; suggesting that not every function can be

depicted as a gesture. Therefore, there needs to be a common gesture that will provide further

actions and it could be argued that this gesture would be the most vital one as it will provide access

to a number of actions.

7.4. Challenges

7.4.1. Involvement of Users

My research was very user-driven, requiring them to participant throughout all three stages of the

project. This was very time-consuming and challenging for both the participants and myself, yet

without them, my report would have been worthless. Finding 18 participants for the guessability

study that were both available and fit the set criteria was difficult as many potential participants were

also completing academic projects.

7.4.2. Failed Pilot Study

The failed inter-rater agreement pilot study also added additional pressure as it required me to

research another method, design the test and then run it, thereby increasing the workload. This issue

wasn’t anticipated at the start of the project; only arising at the beginning of February. While I

overcame the issue - by finding raters eligible to participate within a week’s notice - it still brought

unneeded pressure on both the report and my other academic responsibilities.

7.5. Future Work

7.5.1. Individuality of Gestures

Even though we have created and validated individual gestures for this report, the gestures were not

tested as an overall set. Like the International Usability Partners study, some participants created

'similar, or even the same, gestures … for dissimilar actions' (Mauney et al 2010: 13) and this

therefore affected which gestures would be tied to a specific action - sometimes with the most

popular one not being selected.


Even though the study was conducted in real-world environments - based on where the participants

felt comfortable and where they would consider using a tablet device - it remained platform and

application agnostic in an attempt to limit the affect of pre-learned skill. However, there is a need to

test the gesture set in real applications and with proper use cases to justify it's implementation in real

system. Testing the gesture set in real-world applications has the opportunity of yielding different

results to a study that was performed in isolation.

7.5.2. Different Types of Users

Many different subset of users are having their first computer experiences on tablet devices.

However, this study was based mainly on the limited age range of 20-25 - with a few exceptions -

and this ignores many other important age groups. Some old age pensioners have seen the potential

of tablet devices that portray a friendlier and more user-friendly image to that of standard desktop

computers (Wipf 2010), while the tactile nature of the devices are enticing toddlers to interact with

them (Stewart 2010). Moreover, further research should discuss accessibility issues and involve

disabled participants in creating and evaluating a gesture set. Even though basic applications have

been created for certain disabled users, there is a need for an accessible gesture set that spans

platforms rather than just individual applications (Apple 2011).

7.5.3. Further Analytical Work

As I was unable to fully utilise the inter-rater agreement method due to the inability of the raters to

complete the pilot task, the use of analytical data is very limited in this report. The replacement

method only asked the raters to rate the gestures based on the appropriateness of the action/gesture

relationship, and the three raters’ scores were then merged to create an average score. Future

research should discuss and examine the need for increased analytical data and how it could assist

in the creation of a more solid set of guidelines for gestural tablet interaction.

7.5.4. Reliance on Desktop Knowledge

Further research should also discuss the impact of users’ reliance on desktop-based metaphors. To

limit the affect of cultural differences, I based the shortcut gestures on the desktop-shortcut

equivalent and not on the action word; for example: V for the Paste action instead of P. However, we

need to examine whether this has any detrimental effects on tablet interaction and whether it limits

the overall scope of interaction styles and techniques on tablet devices.

Chapter 7: Discussion & Conclusion 41

8. Project Management8.1. Supervisor Meetings & Progress

The notes from the meetings with my supervisor can be found in Appendix F.

Prior to the Interim Report, I had scheduled 4 meetings with my supervisor to discuss the progress

that I had made on this final year project. We continued to have meetings throughout the academic

year; roughly every 2 weeks. These meetings were pivotal in ensuring that my project remained on

track as mini-deadlines were set up, so that my supervisor could see the project develop over time.

Throughout the year, I attended all scheduled meetings with my supervisor.

8.1.1. 26th January

Due to illness, I was unable to attend this meeting. However, I emailed my supervisor concerning the

progress that I had made; informing him of the slight additions to the guessability study and that I

was beginning the process of writing up the Methodology section.

8.1.2. 9th February

We discussed the final two stages of the research; inter-rater agreement and the validation test. I

raised concerns about the inter-rater agreement method as it would be rather time-consuming for the

participants because it would take several hours for an individual to complete. Therefore, we decided

to run a pilot test and John Halloran requested it to be completed by the next meeting.

The literature review - the results of which can be found in the Introduction & Background section -

was completed prior to this meeting. As stated in my Interim Report, it was a challenge to find

relevant literature as it is an emerging field. Therefore, I was limited to just two academic papers and

one published book that discussed relevant gestural systems. Yet, the first section of this project

benefits from the literature that discussed the forging of the desktop 30 years ago as many issues

facing that generation still concerns the fledging tablet market.

8.1.3. 23rd February

During this meeting, I brought back the results from the pilot study. As mentioned in Section 3, the

pilot study was unsuccessful and therefore I had to select another approval rating method. As I was

approaching the end of February, it was very concerning that I had to plan and complete another

approval rating test and therefore we decided to take a slightly more simpler approach where raters

would rate each gesture based on how well it represented the intended action.

Moreover, I also completed the first draft for the Introduction & Background section, along with the

Methodology section. My supervisor read through the draft during the meeting and informed me of

the importance of referring back to the purpose of this research; the purpose has to permeate

throughout the entire document.

Chapter 8: Project Management 43

8.1.4. 9th March

The results of the new approval test were explored during this meeting. I informed my supervisor of

the method utilised and how many participants were involved, as well as discussing how the results

would be expressed in the project documentation. My supervisor also asked for a draft project by the

next meeting in two weeks time.

8.1.5. 23rd March

My supervisor reminded me of the four components that make up this module; the Interim Report,

Presentation, Report, and Product Quality. As the Product Quality mark scheme wasn’t available

online, my supervisor told me to request the mark scheme from Richard Rider. The supervisor also

asked for a full draft of the report and that it be emailed to him for the 12th April.

I emailed Richard Rider, and he uploaded the mark scheme online for all students. Even though I

emailed my supervisor on the 11th April, he did not receive the report. Therefore, John Halloran

contacted me on the 14th April asking why he was yet to receive the document. Unfortunately, I do

not know why my supervisor didn’t receive the draft, yet we arranged to have a meeting on the 18th

April.

8.1.6. 18th April

For this meeting, I brought along the Results and Conclusion sections for my supervisor to review.

While reading these sections, my supervisor was concerned that it was difficult to see progression

from section to section and suggested that I had included a brief review and introduction at the start

of each section, along with a summary at a section end.

My supervisor asked me to implement these changes and send him a final draft over the next few

days for him to review and reply with feedback. On the 20th April, I emailed him with my final draft

and he stated that the report was progressing very well; only suggesting minor alterations. He

reiterated the importance of binding and submitting the report as early as possible to avoid any

issues during the final stages.


ReferencesApple, 2011. iPad: Year One. [Online Video].

Available from: http://www.youtube.com/

watch?v=HpiVeC1Z3yI

[accessed 10 April 2011]

Apple, 2011. Apple Reports First Quarter

Results. [Webpage].

Available from: http://www.apple.com/pr/

library/2011/01/18results.html

[accessed 16 February 2011]

Apple, 2007. How To Advert. [Video].

Cupertino, California : Apple.

BBC News, 2010. Blackberry launches

Playbook rival to iPad. [Webpage].

Available from: http://www.bbc.co.uk/news/

technology-11416190


Bloomberg, 2010. Apple Increases Tablet Share

to 95 Percent as Android Slips. [Webpage].

Available from: http://www.bloomberg.com/

news/2010-11-02/apple-increases-tablet-

share-to-95-percent-as-android-slips.html


Ginsburg, S., 2010. Designing the iPhone User

Experience: A User-Centered Approach to

Sketching and Prototyping iPhone Apps. 1 ed.

Upper Saddle River, NJ: Addison Wesley.

Goodwin, K., 2009. Designing for the Digital

Age: How to Create Human-Centered Products

and Services. Indianapolis, IN: John Wiley &

Sons.

Google, 2011. Android Event. [Online Video].


watch?v=RfJuigJebRg


Horton, W., 1994. The Icon Book: Visual

Symbols for Computer Systems and

Documentation (Wiley Technical

Communication Library). 1 ed. New York: John

Wiley & Sons.

Mauney, D., Wirtanen, A., Capra, M., 2010.

Cultural similarities and differences in user-

defined gestures for touchscreen user

interfaces. CHI EA '10 Proceedings of the 28th

of the international conference extended

abstracts on Human factors in computing

systems.

Nielsen, J., 1995. Icon Usability. [Webpage].

Available from: http://www.useit.com/papers/

sun/icons.html


Palm, 2011. HP webOS Think Beyond Event.

[Online Video].


watch?v=VnjwG7Z8AM8


RIM, 2010. Research in Motion to Acquire QNX

Software Unit from Harman International.

[Webpage].

Available from: http://press.rim.com/

release.jsp?id=3766


References 45

Raskin, A., 2008. Don't Make Me Click. [Online

Video].


watch?v=EuELwq2ThJE


Raskin, J., 2000. The Humane Interface: New

Directions for Designing Interactive Systems. 1

ed. Reading, Mass.: Addison Wesley.

Rubin, J., 1994. Handbook of Usability Testing:

How to Plan, Design and Conduct Effective

Tests (Wiley Technical Communications

Library). New York: John Wiley & Sons.

Saffer, D., 2008. Designing Gestural Interfaces:

Touchscreens and Interactive Devices. 1 ed.

California, United States: O'Reilly Media.

Saffer, D., 2009. Designing for Interaction:

Creating Innovative Applications and Devices

(Voices That Matter). 2 ed. New Riders.

Stewart, T., 2010. The iPad as a serious

business tool for toddlers. [Webpage].

Available from: http://econsultancy.com/uk/

blog/6329-the-ipad-as-a-serious-business-

tool-for-toddlers


Wipf, A., 2010. Over-65 Audience. [Webpage].

Available from: http://culturalfuel.net/

2010/04/20/ipad-a-push-for-silver-surfing-

testing-the-ipad-with-a-65-target-audience/


Wobbrock, J. O., Ringel Morris, M., Wilson, A.

D., 2009. User-Defined Gestures for Surface

Computing. CHI '09 Proceedings of the 27th

international conference on Human factors in

computing systems.

Yahoo News, 2010. iPad sales cross million

twice as fast as original iPhone. [Webpage].

Available from: http://news.yahoo.com/s/

ytech_gadg/20100503/tc_ytech_gadg/

ytech_gadg_tc1901



BibliographyBenyon, D., Turner, P. & Turner, S., 2004.

Designing Interactive Systems: People,

Activities, Contexts, Technologies. 1 ed.

Harlow, England: Addison Wesley.

Cooper, A., Reimann, R. & Cronin, D., 2007.

About Face 3: The Essentials of Interaction

Design. 3Rev Ed ed. Indianapolis, IN: John

Wiley & Sons.

Kortum, P., 2008. HCI Beyond the GUI: Design

for Haptic, Speech, Olfactory, and Other

Nontraditional Interfaces (Interactive

Technologies). Amsterdam: Morgan Kaufmann.

Moggridge, B., 2006. Designing Interactions.

Cambridge, Mass.: MIT Press.

Nielsen, J., Mack, R. L., 1994. Usability

Inspection Methods. New York: John Wiley &

Sons.

Shneiderman, B., Plaisant, C., Cohen, M. &

Jacobs, S., 2009. Designing the User Interface:

International Version: Strategies for Effective

Human-Computer Interaction: Strategies for

Effective Human-Computer Interaction:

International Edition. 5 ed. Pearson Education.

Bibliography 47

Appendix AGuessability Study Documentation & Forms

Participant A

Appendix A 49

Participant B

Appendix A 51

Participant C

Appendix A 53

Participant D

Appendix A 55

Participant E

Appendix A 57

Participant F

Appendix A 59

Participant G

Appendix A 61

Participant H

Appendix A 63

Participant I

Appendix A 65

Participant J

Appendix A 67

Participant K

Appendix A 69

Participant L

Appendix A 71

Participant M

Appendix A 73

Participant N

Appendix A 75

Participant O

Appendix A 77

Participant P

Appendix A 79

Participant Q

Appendix A 81

Participant R

Appendix A 83

Appendix BGesture Screenshots

Participant A

Select single object Select multiple objects Zoom-in

Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 85

Open menu Close menu Move object

Cut Copy Paste


Participant B


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 87


Cut Copy Paste


Participant C


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 89


Cut Copy Paste


Participant D


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 91


Cut Copy Paste


Participant D


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 93


Cut Copy Paste


Participant E


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 95


Cut Copy Paste


Participant F


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 97


Cut Copy Paste


Participant G


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 99


Cut Copy Paste


Participant H


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 101


Cut Copy Paste


Participant I


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 103


Cut Copy Paste


Participant J


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 105


Cut Copy Paste


Participant K


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 107


Cut Copy Paste


Participant L


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 109


Cut Copy Paste


Participant M


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 111


Cut Copy Paste


Participant N


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 113


Cut Copy Paste


Participant O


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 115


Cut Copy Paste


Participant P


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 117


Cut Copy Paste


Participant Q


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 119


Cut Copy Paste


Participant R


Zoom-out Help Print

Undo Redo Go home

Back Forward Rotate

Appendix B 121


Cut Copy Paste


Appendix CRating Study Documentation & Forms

Appendix C 123

Appendix C 125

Appendix DGesture Data Charts

Select single object

Tap

Multi-Tap

Circle

0 4 8 12 16Strongly Disagree Disagree Neutral Agree Strongly Agree

Select multiple objects

Multi-Tap Consecutive

Multi-Finger Simultaneously

Number Items

Circle

Tap Consecutive


Appendix D 127

Zoom-in

Finger Spread

Zoom-Bar Swipe

Multi-Tap

No Attempt

0 2 4 6 8 10 12 14 16Strongly Disagree Disagree Neutral Agree Strongly Agree

Zoom-out

Finger Spread

Zoom-Bar Swipe

Tap

No Attempt



Help

ʻ?ʼ Symbol

Tap

Multi-Tap

ʻHʼ Letter

No Attempt


Print

Tap and Hold

ʻPʼ Letter

Multi-Tap

Tap

Swipe


Appendix D 129

Undo

ʻZʼ Letter

Swipe

ʻ^ʼ Symbol

Curve

Multi-Tap

Tap

Tap and Hold

Shake Device


Redo

Swipe

Reverse ʻZʼ Letter

ʻ^ʼ Symbol

Curve

Multi-Tap

Tap

Tap and Hold

ʻYʼ Letter

Shake Device



Go home

Swipe Down

ʻHʼ Letter

Swipe Left

Tap Top-Right

ʻXʼ Letter

5 Finger Pinch

Tap and Hold


Back

Swipe Up

Swipe Left

Arrow

Tap Left


Appendix D 131

Forward

Swipe Down

Swipe Right

Arrow

Tap Left


Rotate

Multi-Finger Curve

Corner-Grab Curve

Single-Finger Curve

No Attempt



Open menu

Multi-Tap

Tap and Hold

ʻMʼ Letter

Swipe

Tap

No Attempt


Close menu

Swipe

ʻXʼ Letter

Multi-Tap

Pinch

Tap Top-Right

No Attempt


Appendix D 133

Move object

Swipe

Tap and Hold > Swipe

Arrow Swipe

Tap and Hold > Tap


Cut

Tap and Hold

ʻXʼ Letter

ʻCʼ Letter

Swipe

Scrub

Multi-Tap

Tap

No Attempt



Copy

Tap and Hold

ʻCʼ Letter

Swipe

Curve

Multi-Tap

Tap

No Attempt


Paste

Tap and Hold

ʻVʼ Letter

ʻPʼ Letter

Curve

Multi-Tap

Tap

No Attempt


Appendix D 135

Appendix EGesture-Meaning Association Results

Participant 1Participant 1 Participant 2Participant 2 Participant 3Participant 3

Action/Gesture Pairings Matched? Order Matched? Order Matched? Order

Select single object ✔ 4 ✔ 1 ✔ 4

Select multiple objects ✔ 5 ✔ 9 ✔ 8

Zoom-in ✔ 1 ✔ 3 ✔ 2

Zoom-out ✔ 2 ✔ 4 ✔ 3

Help ✔ 8 ✔ 10 ✔ 5

Print ✔ 9 ✔ 11 ✔ 10

Undo ✔ 11 ✔ 8 ✔ 9

Redo ✘ - ✘ - ✔ 12

Go home ✔ 17 ✔ 16 ✘ -

Back ✔ 3 ✔ 5 ✔ 6

Forward ✔ 6 ✔ 6 ✔ 7

Rotate ✔ 10 ✔ 7 ✔ 13

Open menu ✔ 15 ✔ 13 ✘ -

Close menu ✔ 13 ✔ 12 ✔ 11

Move object ✔ 7 ✔ 2 ✔ 1

Cut ✔ 16 ✘ - ✔ 16

Copy ✔ 12 ✔ 14 ✔ 14

Paste ✔ 14 ✔ 15 ✔ 15

Appendix E 137

Appendix FSupervisor Correspondence

25th January - Thomas Davies to John Halloran // Subject: 26th January Attendance

John,

I'm sorry but I am not feeling too well and therefore I don't think I will be able to make it to

tomorrow's meeting.

This week I have started the guessability study and have completed four tests at the moment. As I

wasn't too sure about just having the before and after screenshot, I am showing participants an

animation of the action (Eg. Moving an object action will show the object moving across the screen)

prior to them creating the gesture. The gesture is still drawn on top of the before screenshot though.

The entire process is being recorded so I can ensure all gestures and the order in which they occur

are documented. I am also observing the participants and having them think aloud.

I have also nearly completed the methodology section of the write up.

Regards,

Thomas

Appendix E 139

9th February - John Halloran to Thomas Davies // Subject: 9th February Meeting Notes

IMAGE DEFINITION

inter-rater agreement

get a panel of people working independently

show them video of gestures

ask them to create clusters where they think the gesture is the same

get THEM to come up with clusters; diff raters' clusters should agree

ask raters for confidence levels when they put an item in cluster

(5 point Likert scale)

they can have a 'misc' pile

cronbach's alpha inter rater reliability (google)

PILOT STUDY

10 gestures

5 users

(raters would need to have some experience of this type of computing)

VALIDATION

image-meaning association - validation

aim for 2 weeks on the pilot

---------------------------------------------------

Dr John Halloran BSc, MSc, PhD


Coventry University

Coventry CV1 5FB

john DOT halloran AT coventry DOT ac DOT uk

ext 7631


23rd February - John Halloran to Thomas Davies // Subject: 23rd February Meeting Notes

change study

pairwise rater agreements within categories

18 categories

2 x 9

maximum of 9 users (say) - 2 each

- stats. difficult! needs SPSS; and what is the right stat.

- I'll do a bit of research - looks like fliess of cohen (not cronbach)

- work out the logic of the thesis

- we are looking for a standard set - intuitive and recognisable

- so what?

- supports the thesis that physical knowledge is basic, universal, not learned so easily deployed for

this interaction style and validates interaction style (lit about people who say it is 'natural' / innate vs

those who say rubbish it is all learned)

- support developers

- supports users

- where manufacturers deviate it is going to be harder to learn

- independent user test of the results

j

---------------------------------------------------



Coventry University

Coventry CV1 5FB


ext 7631

Appendix E 141

9th March - John Halloran to Thomas Davies // Subject: 9th March Meeting Notes

'Eyeball' shows inter-rater agreement within categories

'do you believe this gesture suits the desired action' - Likert 1-5

3 raters

we need a chart, as I scribbled

need a test for statistically significant agreement between raters

I do not know what this test is!

Next time 23rd March

- I want to see a draft dissertation

- if not done, put placeholder content e.g. 'What is going to appear here is as follows'

- Get the logic of the thesis and how the studies support it

- I am away 4th-11th inclusive (but not otherwise)

- aim at a longer supervision on 30th

1. we show that there is statistically significant agreement between raters

2. implement the gesture set

- description / storyboard

3. validate it

- 'image meaning test'

---------------------------------------------------



Coventry University

Coventry CV1 5FB


ext 7631


23rd March - John Halloran to Thomas Davies // Subject: 23rd March Meeting Notes

Make sure there is a full draft dissertation on my desktop 9am Tues 12th April for comment

Keep hammering the research question, what it is, where we are with it, what got established so far

and what's next and why it's relevant

Then we will arrange a meet in week 18-21 if need be

Check the project guide esp the marking criteria

remember there are FOUR components - brief (10), poster (10), report (40), 'product/quality' (40)

what are the forms? request from Richard Rider

---------------------------------------------------



Coventry University

Coventry CV1 5FB


ext 7631

| DREAMSEQUENCE | THE ANNUAL CREATIVE COMPUTING SHOW | Friday 6 May 16:00, Humber

Theatre, Coventry University | www.dreamsequence.co.uk |

Appendix E 143

14th April - John Halloran to Thomas Davies // Subject: Project

Tom

I have had nothing from you, which is uncharacteristic. I am guessing this could be server rejection

but am a bit worried it might be illness for e.g.

Let me know what's happening. We'll probably need to meet next week

Regards

John

15th April - Thomas Davies to John Halloran // RE Subject: Project

John,

Thanks for emailing me. I emailed it in the early hours of Tuesday but it obviously didn't go through.

I'm not at home right now so I won't be able to resend it until the evening. I understand that it will be

difficult for you to read through it now though so don't worry.

In that email, I also asked if it would be possible for you to act as one of my references for my

application to City London. There are no immediate time constraints for the reference though. I've

asked Peter to be my second reference at the same time - he got my email!!

Thank,

Thomas

15th April - John Halloran to Thomas Davies // RE RE Subject: Project

OK. Yes happy to be a ref

What about meeting next week? Say Mon; I could do 15:00 or 12:00

Regards

John

15th April - Thomas Davies to John Halloran // RE RE RE Subject: Project

12:00 will be great.

Thanks,

Thomas

15th April - John Halloran to Thomas Davies // RE RE RE RE Subject: Project

OK confirmed

See you then

Regards

John


20th April - Thomas Davies to John Halloran // Dissertation

John,

Here is the link to my dissertation - http://files.me.com/thomasdavies/e9jcls

Also, as you didn't receive my original email, here is also the reference document for the City

University post-grad course - http://files.me.com/thomasdavies/eautlv

Regards,

Thomas

20th April - John Halloran to Thomas Davies // RE Dissertation

just confirming I've received this - I can download the doic no problem

expect feedback by close of play tomorrow

Regards

John

20th April - John Halloran to Thomas Davies // RE RE Dissertation

Thomas

This is looking excellent. The only issue is the contents page doesn't match the sections - in

particular I can't see a project management section, and the page nos look wrong. Make sure the

project management section appears and that the sections in general are consistent with the

contents page.

I trust you to do this without further guidance. I suggest you try to get it printed and bound before

next week and submit asap which I guess will be Tuesday. I am confident that you know what the

procedure is.

Good Luck

Regards

John

Appendix E 145

http://files.me.com/thomasdavies/e9jcls

http://files.me.com/thomasdavies/e9jcls

http://files.me.com/thomasdavies/eautlv

http://files.me.com/thomasdavies/eautlv

Guidelines for Gestural Tablet Interaction

Documents

Transcript of Guidelines for Gestural Tablet Interaction