Enhancing the interaction space of a tabletop computing system to design paper prototypes for mobile...

Enhancing the interaction spaceof a tabletop computing system

to design paper prototypesfor mobile applications

Master ThesisFrancesco Bonadiman

Introduction

Blended Prototyping

● tabletop system based on hand-drawn paper sketches

Blended Prototyping

○ converted into digital versions→ projected on the table→ expanded further

○ transformed into testable applications→ on a mobile device

Blended Prototyping

○ transformed into testable applications→ on a mobile device

● enhances development of mobile applications○ accelerates early design phases

Blended Prototyping

Hardware

● video projector

○ projects mobile frames & prototypes

Hardware

● video projector

● webcam

○ recognizes barcodes of screens

Hardware

● video projector

● webcam

● DSLR camera

○ shoots HQ pics of the sketches

Hardware

● video projector

● webcam

● DSLR camera

○ shoots HQ pics of the sketches

● tablet

○ allows to perform actions on the prototypes

Software

● Java application

○ controls behavior of projector & cameras

Software

○ identifies barcode markers on paper sheets

Software

○ digitize sketches & perform actions on them

Software

○ digitize sketches & perform actions on them

● Barcode (similar to QR-code)

○ wider → optimized for webcam

○ gives sheet’s position & rotation on table

Context

Advantages of Paper Prototyping

● potential of Paper-Prototyping & mobile devices = combined

● first impression of not-yet-developed product

● paper → cheap, fast & intuitive (Snyder)

○ no unimportant details → iterative refinement (Nielsen)

○ quickly create multiple design alternatives (Landay)

● same usability issues as Hi-Fi discovered

○ benefits of early usability data = 10+ times bigger (Snyder)

● encourages collaboration & critiques →

interdisciplinary & creative (no “gaps”)

Advantages of the System

● testing within real-life scenarios (de Sá / Carriço)

→ in thousand of different usage conditions

● testing within real-life scenarios (de Sá / Carriço)

→ in thousand of different usage conditions

● add code & functionalities

○ define dynamic interface behavior

○ smooth transition to development

Core features

Via TABLET → Determine widgets & semantics

Core features

● manually define “hotspots” on prototype

● turn these into different design patterns

Core features

● create links between the prototypes

● remove components & connections

Core features

● create links between the prototypes

● remove components & connections

● convert into working code

Problem

● several devices + media → different degrees of fidelity

○ paper & office supplies → computers & smart objects

Problem

BUTChanging fidelity of tool & modality of interaction

→ disrupts creative design process

Problem

BUTChanging fidelity of tool & modality of interaction

→ disrupts creative design process

→ shifting continuously confuses users

Fidelity Clash

● combination of low- & high-tech solutions = puzzling

Fidelity Clash

○ one single user stops ideation process

→ to perform any action

Fidelity Clash

○ breaking of collaborative & creative moment (Snyder)

Fidelity Clash

● users distracted → do not interact anymore

○ get unfocused → lose the “flow”

Fidelity Clash

● users distracted → do not interact anymore

○ get unfocused → lose the “flow”

→ perceived as too technical, isolating & distracting

Objectives

● replace high-tech interactions with low-tech approaches

Objectives

● new interaction techniques → not to shift fidelity of media

Objectives

THEN● find & implement alternative solutions to tablet

Objectives

THEN● find & implement alternative solutions to tablet

● low-tech approaches to define “hotspots” & perform actions

○ keep interaction techniques learnable & usable

Enhanced System

1. digitize a screen by taking a picture

2. duplicate a whole screen

3. detect a component (button, image or textbox)

4. connect two screens (link a button to next screen)

5. remove a component from a screen

6. remove a connection between two screens

7. remove a whole screen

Alternatives considered

● Voice recognition

● Object recognition

● Gesture recognition

● Special pen

● Special button

● Colored objects

● Barcode recognition

● Transparent layers

Alternatives considered

● Voice recognition

● Object recognition

● Gesture recognition

● Special pen

● Special button

● Colored objects

● Barcode recognition

● Transparent layers

● most approaches = unfeasible

○ noisy / confusing environment

○ need of not-yet-existing hardware

○ time constraints

Color Detection

BarcodeRecognition

A. Color Detection

● no specific hardware

● dynamic & “playful” interaction

A. Color Detection

Concept: different color = different component

A. Color Detection

→ colored markers + semi-transparent BBPapier

→ users can see through without “ruining” sketches

A. Color Detection

→ colored markers + semi-transparent BBPapier

→ users can see through without “ruining” sketches

→ JavaCV & OpenCV

→ Procedure

● control card = “toolbox” → paint with markers● same color → fill up the chosen component

→ Procedure

● digitize the sketch → creating threshold○ average color in toolbox → range (+30/-30)○ exterior pixels discarded → central values

→ avoid mistakes & colors overlapping

→ Procedure

● digitize the sketch → creating threshold○ average color in toolbox → range (+30/-30)○ exterior pixels discarded → central values

→ avoid mistakes & colors overlapping

● threshold = calculate component’s RGB value● any pixel within → associated to specific UI component

→ Approximation

Once pixels & colors detected

● algorithm recognizes contours of shape

● approximates it to a rectangle

● component created & projected

→ Approximation

Once pixels & colors detected

● algorithm recognizes contours of shape

● approximates it to a rectangle

● component created & projected

Need for filling component

● algorithm not efficient only for outline

● rough drawings typical of sketching

B. Barcode Recognition

Barcode marker = tool for specific operations

● take pictures of device

● connect screens

● recognize & delete components

● copy screens

● take pictures of device

● connect screens

● recognize & delete components

● copy screens

Algorithm continuously running

→ checks if new barcodes are detected

→ Tools

Further improvements

Sidebar → alternative approach for some functionalities

Sidebar → alternative approach for some functionalities● rectangular transparent area + at top of table● if barcode placed inside → operation performed

Clock (progress indicator) → added onto center-top area

Clock (progress indicator) → added onto center-top area● immediate visual feedback to users● if appears → users realize they are performing an action

● 3 seconds = time-frame users might change their mind→ effective to avoid accidental mistakes

Evaluation

Modality A: Sidebar

● digitize a screen → taking a picture

○ by placing single barcode inside

Modality A: Sidebar

● digitize a screen → taking a picture

○ by placing single barcode inside

● duplicate a screen

○ by placing two barcodes inside

○ one is already digitized (source)

○ one is empty (destination)

Modality A: Colors

● toolbox projected

→ boxes for button, image, textbox…

Modality A: Colors

● BBPapier over toolbox → paint with color

● same color → BBPapier over component

Modality A: Colors

● BBPapier over toolbox → paint with color

● same color → BBPapier over component

● digitize screen with toolbox visible on table

→ colored square displayed to represent component

Modality A: Colors

To connect two screens

● same + use FROM → TO box

To delete component → digitize again

● physically remove colored BBPapier

● or cover it with white paper

To remove connection

● same with button-connector

Modality B

● camera tool to digitize

● copy tool to duplicate

● handles to detect component

● arrow tool to connect screens

● rubber tool to delete○ component○ connection○ screen (only way)

Dependent Variables

● measure overall success of new interaction techniques

○ if quick & easy-to-use without distracting user

○ which of the two modalities = most effective

Dependent Variables

● prove if efficient & offers satisfactory user-experience

Dependent Variables

● prove if efficient & offers satisfactory user-experience

4 parameters:

→ Quickness → Ease-of-use → Distraction → User-Experience

Quickness & Ease-of-Use

Quickness

● time for task completion

● evaluation is videotaped

Quickness

● Effective Time

○ duration of task

→ without system errors

→ avoids malfunctionings

Quickness

● Effective Time

Ease-of-use

● % successfully completed tasks

● ratio calculated by checking interactions executed perfectly

Quickness

● Effective Time

Ease-of-use

● % successfully completed tasks

● ratio calculated by checking interactions executed perfectly

○ users make no mistakes

○ no significant problems

○ not need any help / hint

Distraction

Two factors: quickness & workload index (RTLX)

● unweighted (Raw) version of NASA Task Load Index (TLX)

● 6 subscales:

○ Mental, Physical, Temporal Demand; Performance; Effort; Frustration

Distraction

Two factors: quickness & workload index (RTLX)

● unweighted (Raw) version of NASA Task Load Index (TLX)

● 6 subscales:

○ Mental, Physical, Temporal Demand; Performance; Effort; Frustration

● own survey → filled in by users after every task

● 7-points linear scales → then averaged = RTLX

● the lower → the less demanding & distracting task is

User-Experience

● dimensions of AttrakDiff Survey → studies by Hassenzahl

○ “how users rate the usability and design of your interactive product”

User-Experience

● dimensions of AttrakDiff Survey → studies by Hassenzahl

○ “how users rate the usability and design of your interactive product”

● own custom version → filled in after finishing whole study

● 28 7-points semantic differential scales

○ opposite adjectives at both poles ("good - bad" / "human - technical")

○ implicitly divided into 4 dimensions (Pragmatic Quality, Hedonic Quality - Identity, Hedonic Quality - Stimulation, Attractiveness)

Independent Variables

● Task: 1 to 7

● Modality: A | B

● Group: AB | BA

● Area of expertise: HCI | IT | Other

● Sketching familiarity: 1 to 5

● Mobile familiarity: 1 to 5

● Tabletop use: Yes | No

● Lighting conditions: Day | Night

Experiment Design

● two different design solutions → A/B Testing

● “within-subject” design → every user tests both versions

○ no interactions in common → no influence

Experiment Design

● two different design solutions → A/B Testing

● “within-subject” design → every user tests both versions

○ no interactions in common → no influence

● to avoid possible bias = AB & BA groups

User Demographics

● one user at a time → focus on interactions

○ easier to observe & more difficult to get biased

User Demographics

● one user at a time → focus on interactions

○ easier to observe & more difficult to get biased

● 24 participants → 11 females & 13 males

● half aged 18-24, 10 aged 25-34, two older

● 7 background in HCI, 8 IT, rest other expertise

● via Facebook groups & personal connections

● reward = tasty gift or house utensils

Analysis of the Results

Results

● reviewed videos from GoPro

● investigate how long took users to complete tasks

● detect when they had troubles & needed help

Results

● reviewed videos from GoPro

● investigate how long took users to complete tasks

● detect when they had troubles & needed help

● SPSS → ONE-WAY ANOVA test

○ ideal for type of Dependent Variables (continuous)

○ valid for any group of users of study

○ robust to violations in underlying assumptions

Quickness

● Modality B faster than Modality A

○ time for A = 40.1s, B only 23.25s

○ statistically significant (p < .05)

Quickness

● Effective Time (without system errors)

Quickness

● Effective Time (without system errors)

→ color detection = long process

Ease-of-use

● Modality A easier to use than B

Ease-of-use

● Modality A easier to use than B

● perfectly accomplished tasks:

○ ratio: A = 84%, B = 60%

Distraction

● A almost as demanding as B

○ A RTLX index = 1.90

○ B RTLX index = 1.84

Distraction

● A almost as demanding as B

○ A RTLX index = 1.90

○ B RTLX index = 1.84

● On a 7-points scale →

not statistically significant

(F1,286 = 0.289, p = 0.591)

User-Experience

● users feel “assisted by the product”

○ usability can be improved

User-Experience

● users are “stimulated by the product”

○ users can identify with it

○ room for hedonic improvement

User-Experience

● users are “stimulated by the product”

○ users can identify with it

○ room for hedonic improvement

● system considered as “rather desired”

○ attractiveness above average

Single Tasks Analysis

Task 1 Analysis

● sidebar (A) better

○ faster

○ easier to use

○ less demanding

Task 1 Analysis

● sidebar (A) better

○ faster

○ easier to use

○ less demanding

● camera tool slower

○ no clue on how /

where to place it

Task 2 Analysis

● sidebar (A)

○ easier to use

Task 2 Analysis

● sidebar (A)

○ easier to use

○ faster (only E.T.)

○ causes several

system errors →

see later

Task 2 Analysis

● sidebar (A)

○ easier to use

○ causes several

system errors →

see later

● RTLX == copy tool

Task 3 Analysis

● handles (B)

○ faster

○ less demanding

○ easier to use

Task 3 Analysis

● handles (B)

○ faster

○ less demanding

○ easier to use

● color detection

○ 3 tasks in one

○ digitize + create

Task 4 Analysis

● arrow (B)

○ faster

○ less demanding

Task 4 Analysis

● arrow (B)

○ faster

○ less demanding

○ where?

Task 4 Analysis

● arrow (B)

○ faster

○ less demanding

○ where?

● color detection

○ easier to use

○ create + connect

Task 5 Analysis

● rubber (B)

Task 5 Analysis

● rubber (B)

○ bug found!

Task 5 Analysis

● rubber (B)

○ bug found!

● color detection (A)

○ easier to use

○ less demanding

○ more accurate

Task 6 Analysis

● rubber (B)

○ faster

○ less demanding

Task 6 Analysis

● rubber (B)

○ faster

○ less demanding

○ easier to use

○ experience?

Interesting facts

● people with background in HCI

→ performed more easily & faster

● good sketching abilities

→ same + found system more attractive

Interesting facts

● people familiar with mobile devices

→ faster + system less distracting

Interesting facts

● people familiar with mobile devices

→ faster + system less distracting

● during day-time → better performance & lower RTLX

Considerations

● accuracy = ratio of task’s precision

○ global accuracy = 86%

○ color detection = 67%

→ more delicate + error-prone

Considerations

● barcodes + hands into digitized image → distraction + inexperience

Considerations

● barcodes + hands into digitized image → distraction + inexperience

● if BBPapier wavy → grey area projected

○ projector’s brightness → shadow (“ghost shape”)

Projector’s beam

● when toolbox at sides of projection → outer areas appear darker

○ either no color is detected

○ “fake” grey color recognized → multiple components created

Projector’s beam

● most system errors

○ excessively intense light causing reflections on table

Projector’s beam

○ excessively intense light causing reflections on table→ closest barcodes could not be detected→ devices continuously refreshing

Projector’s beam

○ excessively intense light causing reflections on table→ closest barcodes could not be detected→ devices continuously refreshing

○ “effective time” → task completed if interaction is correct

Conclusions

Which better?

○ easier to use

○ takes longer time

Which better?

○ easier to use

● barcode recognition (B)

○ definitely faster

○ harder to understand

Which better?

○ easier to use

● barcode recognition (B)

○ definitely faster

○ harder to understand

● both have similar workload

○ around 1.9 out of 7

○ positively low →

not too distracting51

Ideal implementation

1-2. sidebar → digitize & duplicate screens

2. copy tool too → duplicate without moving papers

3-4. both approaches to detect components & create connections

a. color detection = from scratch + multiple tasks at same time

b. handles & arrow = fast & easy-to-use

5-6-7. rubber tool to remove components, connections & screens

→ implement separate ones to avoid possible mistakes

Future Work

● visual feedback when projections disappear from table

Future Work

● icon near clock → tells which action is being performed

Future Work

● exploring color detection further

○ better algorithms to calculate the threshold

○ better ways to detect shapes through contours

Future Work

● barcodes = built with wood / opaque plastic → reduce reflections

Future Work

● compare interaction techniques with original system

Future Work

● compare interaction techniques with original system

● test system in real-life scenario → inside company or startup53

Thanks!Any questions?

images from pixabay.com & papers by Bähr

Enhancing the interaction space of a tabletop computing system to design paper prototypes for mobile...

Technology

Transcript of Enhancing the interaction space of a tabletop computing system to design paper prototypes for mobile...

Evotech Prototypes

Prototypes slideshow

furniture / prototypes

Tabletop Controller

Tabletop Learning

Tabletop Mushroom Cultivation - Meetupfiles.meetup.com/1497456/Mushrooms - Tabletop... · Tabletop Mushroom Cultivation Alson Sears, D.V.M. Christopher Keller awsears@hotmail.com

Tabletop Displays

Valkyrie - Tabletop

Tabletop Rules

Paper prototypes

Laguna 'Mini' Tabletop and Tabletop Pump Manual · 2001-07-25 · Mini Tabletop Pump Pompe Laguna Mini Tabletop Mini-Bomba Tabletop Mini Tabletop Pump Tabletop Pump Pompe Laguna Tabletop

TABLETOP EXERCISE

System Prototypes

Refs.requirements Prototypes

Prototypes, Prototypes, Prototypes

Enhancing Quality of Argumentation in a Co-located Collaborative … · 2016-11-04 · Enhancing Quality of Argumentation in a Co-located Collaborative Environment through a Tabletop

Prototypes Borowik

Tabletop Bandsaw

Thesis Prototypes

Tabletop Solutions