Personal interaction via multi-touch gesturesA step toward NUI

Alessio Malizia, Associate ProfessorAndrea Bellucci, Ph.D. candidate

Dept. of Computer Science, Universidad Carlos III de Madrid, Spain

SynopsisResearch areasBackgroundProblem(s)Question(s)HypothesisContributionExpected result

Research areas

Human Computer InteractionNatural User InterfacesInteraction/UX Design

Human Factors

1960s –CLICommand Line InterfaceMemorization/Recall

1980s –GUIGraphic User InterfaceRecognition (recall = shortcuts)

2000s –NUINatural User InterfaceRecall? Gestures are like shortcuts

Exploration

TESIS: Turn Every Surface into an Interactive SurfaceDemo @ACM Int. Conference ITS 2011

Don’t touch me!Demo @ACM Int. Conference AVI 2010

Problem(s)

Question(s)

• How can I create an interface in which users may quickly gain expertise using actions that feel natural to them? (research question 1)

• How can I be sure that I succeeded in creating an interface that feels natural to the user?(research question 2)

Hypothesis

• Natural properties are related to universals

• The concept of affordance is crucial for defining a widget-level architecture for NUI– Interface as a Service– Widgets expose their affordances

• Personalization and adaptation/evolution as a step toward natural interaction

I’d like to convince you(thesis)

• Gestures (as signs) are something alive, which change depending on cultural and human aspects, time and context. Therefore users’ personalization can be a step to address natural interaction.

• Refined research question 1: to what extend personalization of multi-touch gestures can be valuable for enabling natural interaction?

Natural properties and Universals

• Concept of perfect natural properties from metaphysics– Perfect natural properties are shared among all the elements

of a set

• For we humans, perfect natural properties are our perception faculties (hearing, feeling by touch, seeing, tasting, smelling) – There are also acquired skills...

• Could there exist such a thing as a natural gesture? – Take into account users innate and learned abilities

Which one is natural?

Zooming in real life: bring the newspaper closer

iPhone: pinch for zooming

Samsung Galaxy s2: holding down on the screen with two fingers and moving your phone back and forth

Affordance

• Why affordance is important for natural interaction?

– Gibson (1977) defined affordances as all action possibilities latent in the environment

– Affordances as perceived action possibilities (Norman, 1988)

– Affordance is the visual clue to the function of the object (an opportunity for action)

Design without affordance

You can move an object in single-pixel increments by nudging it.

Touch and hold the object with one finger, and then use another finger to swipe across the page in the direction you want the object to move.

From iPad’s Pages gestures list

A good affordance

What happens if you attempt to make users interact in a way which is not natural to them?

Beyond affordance languages

Personal/Private

Public/Shared

Small

Medium/High

Personalization

Adaptation/Evolution

Form factor

Display

Expected result