Deformable and shape-changing user interfaces

Kasper Hornbæk, University of Copenhagen

apple.com

wallviz.dk

Samsung, CES 2013

PaperPhone, CHI 2012

Samsung, CES 2014

MorePhone

MorePhone, CHI 2013

PneUI, UIST 2013

PneUI, UIST 2013

Rasmussen et al., CHI 2012

44 papers on shape change

44 papers on shape change

Little UX

ghost-fet.org

ghost-fet.org

Dr. Kasper Hornbæk Dr. Esben W. Pedersen Giovanni M. Troiano John Tiab

UserExperience

Technology Design

GHOST

UserExperience

Technology Design

GHOST

Study 1:Guessability of Gestures

Study 2:Crowd-Sourced Reactions

Study 3:Repository Grid Evaluation

Troiano, Pedersen & Hornbæk (AVI 2014)

Gestures were recorded with four cameras

Gestures were recorded with four cameras

10 page coding manual

Many%varia)ons%

Influence of Elasticity

Study 1:Guessability of Gestures

Study 2:Crowd-Sourced Reactions

Study 3:Repository Grid Evaluation

Pedersen, Subramanian & Hornbæk (CHI 2014)

Pedersen, Subramanian & Hornbæk (CHI 2014)

Shape-changing Handheld

Shape-changing Handheld

Parameters of shape change

1.5x

3x

5x

pi

-pi/2

pi/2

15 mm

20 mm

60 mm

n = 3

n = 8

n = 14

n = 1

n = 2

n = 3

towards (18 mm)

away (18 mm)

13 mm/s

0 1 2 3 64 5

53 mm/s

0 1 2 3 64 5

Area Curvature Amplitude Zero-crossing Speed Tapering Corner bend

LowM

ediumH

igh

Hemmert et al. Gomes et al. Roudaut et al.

Parameters of shape change

1.5x

3x

5x

pi

-pi/2

pi/2

15 mm

20 mm

60 mm

n = 3

n = 8

n = 14

n = 1

n = 2

n = 3

towards (18 mm)

away (18 mm)

13 mm/s

0 1 2 3 64 5

53 mm/s

0 1 2 3 64 5

Area Curvature Amplitude Zero-crossing Speed Tapering Corner bend

LowM

ediumH

igh

Hemmert et al. Gomes et al. Roudaut et al.

Parameters of shape change

1.5x

3x

5x

pi

-pi/2

pi/2

15 mm

20 mm

60 mm

n = 3

n = 8

n = 14

n = 1

n = 2

n = 3

towards (18 mm)

away (18 mm)

13 mm/s

0 1 2 3 64 5

53 mm/s

0 1 2 3 64 5

Area Curvature Amplitude Zero-crossing Speed Tapering Corner bend

LowM

ediumH

igh

Hemmert et al. Gomes et al. Roudaut et al.

Parameters of shape change

1.5x

3x

5x

pi

-pi/2

pi/2

15 mm

20 mm

60 mm

n = 3

n = 8

n = 14

n = 1

n = 2

n = 3

towards (18 mm)

away (18 mm)

13 mm/s

0 1 2 3 64 5

53 mm/s

0 1 2 3 64 5

Area Curvature Amplitude Zero-crossing Speed Tapering Corner bend

LowM

ediumH

igh

Hemmert et al. Gomes et al. Roudaut et al.

Use scenariosNotification Approach

Use scenariosNotification Approach

187 participants

AttrakDiff2 Animacy SAM UC Specific

Zero crossing

Area

Curvature

Amplitude

Tapering

Corner bend

High

Low

Prag

mat

ic Q

ualit

y

Zero crossing

Area

Curvature

Amplitude

Tapering

Corner bend

High

Low

Prag

mat

ic Q

ualit

y

Structured

Simple

Practical

Predictable

PQ

Zero crossing

Area

Curvature

Amplitude

Tapering

Corner bend

High

Low

Prag

mat

ic Q

ualit

y

Zero crossing

Tapering

Area

Curvature

Amplitude

Corner bend

High

Low

Hed

onic

Qua

lity

Zero crossing

Tapering

Area

Curvature

Amplitude

Corner bend

High

Low

Hed

onic

Qua

lity

Captivating

Stylish

Premium

Creative

HQ

Corner Bend

27 %less urgent than the three most urgent shape changes

Corner Bend

27 %less urgent than the three most urgent shape changes

Corner Bend

27 %less urgent than the three most urgent shape changes

Amplitude Zero crossingArea

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

area

Area

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

area

Amplitude

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

area

amplitude

Amplitude

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

area

amplitude

Corner bend

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

area

amplitude

Corner bend

corner bend

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

area

amplitude

corner bend

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

area

amplitude

corner bend

taperingcurvature

zero crossing

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

area

amplitude

corner bend

taperingcurvature

zero crossing

Calm

Aroused

PleasentUnp

leasan

t

III

III IV

alarmedtense

angryafraid

annoyeddistressed

frustrated

miserable

sadgloomy

depressed

bored

droopytired sleepy

relaxedsatisfied

at easecontentserene

gladpleased

happy

delighted

excited

astonishedaroused

area

amplitude

corner bend

taperingcurvature

zero crossing

Study 1:Guessability of Gestures

Study 2:Crowd-Sourced Reactions

Study 3:Repository Grid Evaluation

Kwak, Markopolos, Bruns & Hornbæk (DIS 2014)

Bootcamp I: Master Students Industrial Design

Goal:

Define interaction possibilities and pilot workshop approach

Results:

10 prototypes

- 3 -

isolated pairs of user input and system output. However, when the properties of a product, such as its shape, change our perceptions will follow these changes through a process of adaptation [12]. A product that changes its shape can potentially give rise to a feedback loop playing on the action-perception possibilities, offering a dynamic interaction between the action of the user and the reaction of the product, which has been named interactive materiality [34]. Our perspective on designing shape-changing interfaces and evaluating emerging interactions takes this broader perspective.

The focus is on interaction supported by manipulation, gesture, and touch, with medium sized (hand scale) objects. Lederman and Klatzky [28] provided a systematic framework for analyzing hand movements and manipulations that are conducted in the context of the active explorations people make in order to perceive with touch, what Gibson [12] called active touch. Specifically, Lederman and Klatzky [28] identified the necessity and demonstrated the appropriateness of a small set of stereotypical movements for perceiving the shape of different objects: lateral motion, pressure, static contact, unsupported holding, enclosure, contour following, function test, and part motion test. This framework is not directly related to shape-changing interfaces or current models thereof; yet it helps generate design ideas and identify ways one may engage with shape-changing interfaces.

Design Processes In a one week-long design boot camp, we invited 20 advanced Master level students in industrial design to use a parallel design process to develop shape-changing interfaces that feature variations on the interactive action-perception loop. The shape-changing interfaces were to sense the explorations that the user makes through active touch [12] and adapt their physical properties. Students

were instructed to design artifacts that support different explorations of users along the classification of typical movement patterns for exploratory procedures (EPs). A collection of ten shape-changing artifacts was created (Figure 1). Videos of the artifacts can be viewed at http://www.anonymous.com.

These first design explorations were followed by a more sustained and in-depth design process, where a design researcher synthesized the initial design concepts, redesigned the initial artifacts in order to create a collection of six shape changing interfaces featuring very distinct experiences of shape-change.

The processes described above resulted in six artifacts, illustrated in Figure 2 through Figure 7. The accompanying video details the interaction possible with the artifacts. These six artifacts were used in the study presented next.

Figure 1. First generation of shape-changing interfaces, exploring dynamic orientation, form, volume, texture, and spatiality in one and two degrees of freedom. The bottom-

right figure shows two artifacts.

Figure 2. PIEGA explores dynamic changes in form as a response to static contact and consists of a cloth surface

attached in the center to a servomotor. The rotation of the servomotor results in folds in the cloth surface. A capacitive sensor is used to determine user presence and static contact.

Figure 3. GATO explores dynamic changes in texture as a response to static contact and consists of a platform with spring steel loop ‘hairs’ that lowers and rises to alter the length of the ‘hairs’; short hairs are rougher to the touch than longer hairs.

A capacitive sensor is used to determine user presence and static contact.

COGNITIVE PSYCHOLOGY 19, 342-368 (1987)

Hand Movements: A Window into Haptic Object Recognition

SUSAN J. LEDERMAN

Queen’s University at Kingston, Ontario, Canada

AND

ROBERTA L. KLATZKY

University of California at Santa Barbara

Two experiments establish links between desired knowledge about objects and hand movements during haptic object exploration. Experiment 1 used a match- to-sample task, in which blindfolded subjects were directed to match objects on a particular dimension (e.g., texture). Hand movements during object exploration were reliably classified as “exploratory procedures,” each procedure defined by its invariant and typical properties. The movement profile, i.e., the distribu- tion of exploratory procedures, was directly related to the desired object knowl- edge that was required for the match. Experiment 2 addressed the reasons for the specific links between exploratory procedures and knowledge goals. Hand move- ments were constrained, and performance on various matching tasks was as- sessed. The procedures were considered in terms of their necessity, sufficiency, and optimahty of performance for each task. The results establish that in free exploration, a procedure is generally used to acquire information about an object property, not because it is merely sufficient, but because it is optimal or even necessary. Hand movements can serve as “windows,” through which it is pos- sible to learn about the underlying representation of objects in memory and the processes by which such representations are derived and utilized. o 1987 Academic

Press. Inc.

When we feel extremely helpless in a situation, we commonly say, “My hands are tied!” Indeed, it is hard to imagine a world in which we cannot feel the soft fur of a kitten or even tie our shoelaces. Yet, psy- chology has often portrayed the hand as a second-class citizen. Research

The research reported in this paper was supported by the Natural Sciences & Engineering Research Council of Canada (Grant A9854 awarded to SJL) and by the National Science Foundation (Grant BNS84-21340 awarded to RLK). Reprint requests may be sent to SJL, Psychology, Queen’s University, Kingston, Ontario, Canada K7L 3N6 or to RLK, Psy- chology, University of California, Santa Barbara, CA 93106. Order of authorship does not reflect relative contribution; both authors contributed equally to the work. We thank An- drew Currie for his considerable contribution; he helped to prepare the stimulus objects, ran the experiments, scored some of the videotapes for our reliability checks, collated and analyzed much of the data, and provided valuable comments in general discussion.

342 OOlO-0285/87 $7.50 Copyright 0 1987 by Academic Press, Inc. All rights of reproduction in any form reserved.

Design Interaction Behaviour

Repertory Grid Study

These two seem insecure

This one seemsdetermined

Insecure Determined

Constructs

ConstructsFriendly/Distant

Indifferent/Attentive

Honest/Underhand

Random/Predictable

Sturdy/FragileMechanical/Organic

Cautious/Ambitious

Insecure/Determined

Unconcerned/InsecureVulnerable/Strong

Surly/Sensitive

Rebellious/Cooperative

Stubborn/Compliant

Obedient/Recalcitrant

Open/MysteriousBoisterous/Introvert

Social/Introvert

Skittish/Affectionate

Frightened/Defensive

Aggressive/Coy

Weak/Powerful

Playful/Calm

Restful/Hasty

Downhearted/HopefulHappy/Sad

Future Work

- How do we map deformation to (sound) parameters?- Participatory Design with 10 musicians as co-designers.

Deformable UI for real people

Workshops continued

Deformable and shape-changing user interfaces

Kasper Hornbæk, University of Copenhagen