Enhancing the interaction spaceof a tabletop computing system
to design paper prototypesfor mobile applications
Master ThesisFrancesco Bonadiman
2016
Introduction
1
Blended Prototyping
● tabletop system based on hand-drawn paper sketches
2
Blended Prototyping
● tabletop system based on hand-drawn paper sketches
○ converted into digital versions→ projected on the table→ expanded further
2
● tabletop system based on hand-drawn paper sketches
○ converted into digital versions→ projected on the table→ expanded further
○ transformed into testable applications→ on a mobile device
Blended Prototyping
2
● tabletop system based on hand-drawn paper sketches
○ converted into digital versions→ projected on the table→ expanded further
○ transformed into testable applications→ on a mobile device
● enhances development of mobile applications○ accelerates early design phases
Blended Prototyping
2
Hardware
● video projector
○ projects mobile frames & prototypes
3
Hardware
● video projector
○ projects mobile frames & prototypes
● webcam
○ recognizes barcodes of screens
3
Hardware
● video projector
○ projects mobile frames & prototypes
● webcam
○ recognizes barcodes of screens
● DSLR camera
○ shoots HQ pics of the sketches
3
Hardware
● video projector
○ projects mobile frames & prototypes
● webcam
○ recognizes barcodes of screens
● DSLR camera
○ shoots HQ pics of the sketches
● tablet
○ allows to perform actions on the prototypes
3
Software
● Java application
○ controls behavior of projector & cameras
4
Software
● Java application
○ controls behavior of projector & cameras
○ identifies barcode markers on paper sheets
4
Software
● Java application
○ controls behavior of projector & cameras
○ identifies barcode markers on paper sheets
○ digitize sketches & perform actions on them
4
Software
● Java application
○ controls behavior of projector & cameras
○ identifies barcode markers on paper sheets
○ digitize sketches & perform actions on them
● Barcode (similar to QR-code)
○ wider → optimized for webcam
○ gives sheet’s position & rotation on table
4
Context
5
Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
6
Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
● first impression of not-yet-developed product
6
Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
● first impression of not-yet-developed product
● paper → cheap, fast & intuitive (Snyder)
○ no unimportant details → iterative refinement (Nielsen)
6
Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
● first impression of not-yet-developed product
● paper → cheap, fast & intuitive (Snyder)
○ no unimportant details → iterative refinement (Nielsen)
○ quickly create multiple design alternatives (Landay)
6
Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
● first impression of not-yet-developed product
● paper → cheap, fast & intuitive (Snyder)
○ no unimportant details → iterative refinement (Nielsen)
○ quickly create multiple design alternatives (Landay)
● same usability issues as Hi-Fi discovered
○ benefits of early usability data = 10+ times bigger (Snyder)
6
● encourages collaboration & critiques →
interdisciplinary & creative (no “gaps”)
Advantages of the System
7
● encourages collaboration & critiques →
interdisciplinary & creative (no “gaps”)
● testing within real-life scenarios (de Sá / Carriço)
→ in thousand of different usage conditions
Advantages of the System
7
● encourages collaboration & critiques →
interdisciplinary & creative (no “gaps”)
● testing within real-life scenarios (de Sá / Carriço)
→ in thousand of different usage conditions
● add code & functionalities
○ define dynamic interface behavior
○ smooth transition to development
Advantages of the System
7
Core features
Via TABLET → Determine widgets & semantics
8
Core features
Via TABLET → Determine widgets & semantics
● manually define “hotspots” on prototype
● turn these into different design patterns
8
Core features
Via TABLET → Determine widgets & semantics
● manually define “hotspots” on prototype
● turn these into different design patterns
● create links between the prototypes
● remove components & connections
8
Core features
Via TABLET → Determine widgets & semantics
● manually define “hotspots” on prototype
● turn these into different design patterns
● create links between the prototypes
● remove components & connections
● convert into working code
8
Problem
● several devices + media → different degrees of fidelity
○ paper & office supplies → computers & smart objects
9
Problem
● several devices + media → different degrees of fidelity
○ paper & office supplies → computers & smart objects
BUT
9
Problem
● several devices + media → different degrees of fidelity
○ paper & office supplies → computers & smart objects
BUTChanging fidelity of tool & modality of interaction
→ disrupts creative design process
9
Problem
● several devices + media → different degrees of fidelity
○ paper & office supplies → computers & smart objects
BUTChanging fidelity of tool & modality of interaction
→ disrupts creative design process
→ shifting continuously confuses users
9
Fidelity Clash
● combination of low- & high-tech solutions = puzzling
10
Fidelity Clash
● combination of low- & high-tech solutions = puzzling
○ one single user stops ideation process
→ to perform any action
10
Fidelity Clash
● combination of low- & high-tech solutions = puzzling
○ one single user stops ideation process
→ to perform any action
○ breaking of collaborative & creative moment (Snyder)
10
Fidelity Clash
● combination of low- & high-tech solutions = puzzling
○ one single user stops ideation process
→ to perform any action
○ breaking of collaborative & creative moment (Snyder)
● users distracted → do not interact anymore
○ get unfocused → lose the “flow”
10
Fidelity Clash
● combination of low- & high-tech solutions = puzzling
○ one single user stops ideation process
→ to perform any action
○ breaking of collaborative & creative moment (Snyder)
● users distracted → do not interact anymore
○ get unfocused → lose the “flow”
→ perceived as too technical, isolating & distracting
10
Objectives
● replace high-tech interactions with low-tech approaches
11
Objectives
● replace high-tech interactions with low-tech approaches
● new interaction techniques → not to shift fidelity of media
11
Objectives
● replace high-tech interactions with low-tech approaches
● new interaction techniques → not to shift fidelity of media
THEN
11
Objectives
● replace high-tech interactions with low-tech approaches
● new interaction techniques → not to shift fidelity of media
THEN● find & implement alternative solutions to tablet
11
Objectives
● replace high-tech interactions with low-tech approaches
● new interaction techniques → not to shift fidelity of media
THEN● find & implement alternative solutions to tablet
● low-tech approaches to define “hotspots” & perform actions
○ keep interaction techniques learnable & usable
11
Enhanced System
12
Tasks
1. digitize a screen by taking a picture
13
Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
13
Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
3. detect a component (button, image or textbox)
13
Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
3. detect a component (button, image or textbox)
4. connect two screens (link a button to next screen)
13
Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
3. detect a component (button, image or textbox)
4. connect two screens (link a button to next screen)
5. remove a component from a screen
13
Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
3. detect a component (button, image or textbox)
4. connect two screens (link a button to next screen)
5. remove a component from a screen
6. remove a connection between two screens
13
Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
3. detect a component (button, image or textbox)
4. connect two screens (link a button to next screen)
5. remove a component from a screen
6. remove a connection between two screens
7. remove a whole screen
13
Alternatives considered
● Voice recognition
● Object recognition
● Gesture recognition
● Special pen
● Special button
● Colored objects
● Barcode recognition
● Transparent layers
14
Alternatives considered
● Voice recognition
● Object recognition
● Gesture recognition
● Special pen
● Special button
● Colored objects
● Barcode recognition
● Transparent layers
● most approaches = unfeasible
○ noisy / confusing environment
○ need of not-yet-existing hardware
○ time constraints
14
Color Detection
A
BarcodeRecognition
B15
A. Color Detection
● no specific hardware
● dynamic & “playful” interaction
16
A. Color Detection
● no specific hardware
● dynamic & “playful” interaction
Concept: different color = different component
16
A. Color Detection
● no specific hardware
● dynamic & “playful” interaction
Concept: different color = different component
→ colored markers + semi-transparent BBPapier
→ users can see through without “ruining” sketches
16
A. Color Detection
● no specific hardware
● dynamic & “playful” interaction
Concept: different color = different component
→ colored markers + semi-transparent BBPapier
→ users can see through without “ruining” sketches
→ JavaCV & OpenCV
16
→ Procedure
● control card = “toolbox” → paint with markers● same color → fill up the chosen component
17
→ Procedure
● control card = “toolbox” → paint with markers● same color → fill up the chosen component
● digitize the sketch → creating threshold○ average color in toolbox → range (+30/-30)○ exterior pixels discarded → central values
→ avoid mistakes & colors overlapping
17
→ Procedure
● control card = “toolbox” → paint with markers● same color → fill up the chosen component
● digitize the sketch → creating threshold○ average color in toolbox → range (+30/-30)○ exterior pixels discarded → central values
→ avoid mistakes & colors overlapping
● threshold = calculate component’s RGB value● any pixel within → associated to specific UI component
17
→ Approximation
Once pixels & colors detected
● algorithm recognizes contours of shape
● approximates it to a rectangle
● component created & projected
18
→ Approximation
Once pixels & colors detected
● algorithm recognizes contours of shape
● approximates it to a rectangle
● component created & projected
Need for filling component
● algorithm not efficient only for outline
● rough drawings typical of sketching
18
B. Barcode Recognition
Barcode marker = tool for specific operations
19
B. Barcode Recognition
Barcode marker = tool for specific operations
● take pictures of device
● connect screens
● recognize & delete components
● copy screens
19
B. Barcode Recognition
Barcode marker = tool for specific operations
● take pictures of device
● connect screens
● recognize & delete components
● copy screens
Algorithm continuously running
→ checks if new barcodes are detected
19
→ Tools
20
Further improvements
Sidebar → alternative approach for some functionalities
21
Further improvements
Sidebar → alternative approach for some functionalities● rectangular transparent area + at top of table● if barcode placed inside → operation performed
21
Further improvements
Sidebar → alternative approach for some functionalities● rectangular transparent area + at top of table● if barcode placed inside → operation performed
Clock (progress indicator) → added onto center-top area
21
Further improvements
Sidebar → alternative approach for some functionalities● rectangular transparent area + at top of table● if barcode placed inside → operation performed
Clock (progress indicator) → added onto center-top area● immediate visual feedback to users● if appears → users realize they are performing an action
21
Further improvements
Sidebar → alternative approach for some functionalities● rectangular transparent area + at top of table● if barcode placed inside → operation performed
Clock (progress indicator) → added onto center-top area● immediate visual feedback to users● if appears → users realize they are performing an action
● 3 seconds = time-frame users might change their mind→ effective to avoid accidental mistakes
21
Evaluation
22
Modality A: Sidebar
● digitize a screen → taking a picture
○ by placing single barcode inside
23
Modality A: Sidebar
● digitize a screen → taking a picture
○ by placing single barcode inside
● duplicate a screen
○ by placing two barcodes inside
○ one is already digitized (source)
○ one is empty (destination)
23
Modality A: Colors
● toolbox projected
→ boxes for button, image, textbox…
24
Modality A: Colors
● toolbox projected
→ boxes for button, image, textbox…
● BBPapier over toolbox → paint with color
● same color → BBPapier over component
24
Modality A: Colors
● toolbox projected
→ boxes for button, image, textbox…
● BBPapier over toolbox → paint with color
● same color → BBPapier over component
● digitize screen with toolbox visible on table
→ colored square displayed to represent component
24
Modality A: Colors
To connect two screens
● same + use FROM → TO box
To delete component → digitize again
● physically remove colored BBPapier
● or cover it with white paper
To remove connection
● same with button-connector
25
Modality B
● camera tool to digitize
● copy tool to duplicate
● handles to detect component
● arrow tool to connect screens
● rubber tool to delete○ component○ connection○ screen (only way)
26
Dependent Variables
● measure overall success of new interaction techniques
○ if quick & easy-to-use without distracting user
○ which of the two modalities = most effective
27
Dependent Variables
● measure overall success of new interaction techniques
○ if quick & easy-to-use without distracting user
○ which of the two modalities = most effective
● prove if efficient & offers satisfactory user-experience
27
Dependent Variables
● measure overall success of new interaction techniques
○ if quick & easy-to-use without distracting user
○ which of the two modalities = most effective
● prove if efficient & offers satisfactory user-experience
4 parameters:
→ Quickness → Ease-of-use → Distraction → User-Experience
27
Quickness & Ease-of-Use
Quickness
● time for task completion
● evaluation is videotaped
28
Quickness & Ease-of-Use
Quickness
● time for task completion
● evaluation is videotaped
● Effective Time
○ duration of task
→ without system errors
→ avoids malfunctionings
28
Quickness & Ease-of-Use
Quickness
● time for task completion
● evaluation is videotaped
● Effective Time
○ duration of task
→ without system errors
→ avoids malfunctionings
Ease-of-use
● % successfully completed tasks
● ratio calculated by checking interactions executed perfectly
28
Quickness & Ease-of-Use
Quickness
● time for task completion
● evaluation is videotaped
● Effective Time
○ duration of task
→ without system errors
→ avoids malfunctionings
Ease-of-use
● % successfully completed tasks
● ratio calculated by checking interactions executed perfectly
○ users make no mistakes
○ no significant problems
○ not need any help / hint
28
Distraction
Two factors: quickness & workload index (RTLX)
● unweighted (Raw) version of NASA Task Load Index (TLX)
● 6 subscales:
○ Mental, Physical, Temporal Demand; Performance; Effort; Frustration
29
Distraction
Two factors: quickness & workload index (RTLX)
● unweighted (Raw) version of NASA Task Load Index (TLX)
● 6 subscales:
○ Mental, Physical, Temporal Demand; Performance; Effort; Frustration
● own survey → filled in by users after every task
● 7-points linear scales → then averaged = RTLX
● the lower → the less demanding & distracting task is
29
User-Experience
● dimensions of AttrakDiff Survey → studies by Hassenzahl
○ “how users rate the usability and design of your interactive product”
30
User-Experience
● dimensions of AttrakDiff Survey → studies by Hassenzahl
○ “how users rate the usability and design of your interactive product”
● own custom version → filled in after finishing whole study
● 28 7-points semantic differential scales
○ opposite adjectives at both poles ("good - bad" / "human - technical")
○ implicitly divided into 4 dimensions (Pragmatic Quality, Hedonic Quality - Identity, Hedonic Quality - Stimulation, Attractiveness)
30
Independent Variables
● Task: 1 to 7
● Modality: A | B
● Group: AB | BA
● Area of expertise: HCI | IT | Other
● Sketching familiarity: 1 to 5
● Mobile familiarity: 1 to 5
● Tabletop use: Yes | No
● Lighting conditions: Day | Night
31
Experiment Design
● two different design solutions → A/B Testing
● “within-subject” design → every user tests both versions
○ no interactions in common → no influence
32
Experiment Design
● two different design solutions → A/B Testing
● “within-subject” design → every user tests both versions
○ no interactions in common → no influence
● to avoid possible bias = AB & BA groups
32
User Demographics
● one user at a time → focus on interactions
○ easier to observe & more difficult to get biased
33
User Demographics
● one user at a time → focus on interactions
○ easier to observe & more difficult to get biased
● 24 participants → 11 females & 13 males
● half aged 18-24, 10 aged 25-34, two older
● 7 background in HCI, 8 IT, rest other expertise
● via Facebook groups & personal connections
● reward = tasty gift or house utensils
33
Analysis of the Results
34
Results
● reviewed videos from GoPro
● investigate how long took users to complete tasks
● detect when they had troubles & needed help
35
Results
● reviewed videos from GoPro
● investigate how long took users to complete tasks
● detect when they had troubles & needed help
● SPSS → ONE-WAY ANOVA test
○ ideal for type of Dependent Variables (continuous)
○ valid for any group of users of study
○ robust to violations in underlying assumptions
35
Quickness
● Modality B faster than Modality A
○ time for A = 40.1s, B only 23.25s
○ statistically significant (p < .05)
36
Quickness
● Modality B faster than Modality A
○ time for A = 40.1s, B only 23.25s
○ statistically significant (p < .05)
● Effective Time (without system errors)
○ time for A = 29.24s, B only 18.38s
○ statistically significant (p < .05)
36
Quickness
● Modality B faster than Modality A
○ time for A = 40.1s, B only 23.25s
○ statistically significant (p < .05)
● Effective Time (without system errors)
○ time for A = 29.24s, B only 18.38s
○ statistically significant (p < .05)
→ color detection = long process
36
Ease-of-use
● Modality A easier to use than B
37
Ease-of-use
● Modality A easier to use than B
● perfectly accomplished tasks:
○ ratio: A = 84%, B = 60%
○ statistically significant (p < .05)
37
Distraction
● A almost as demanding as B
○ A RTLX index = 1.90
○ B RTLX index = 1.84
38
Distraction
● A almost as demanding as B
○ A RTLX index = 1.90
○ B RTLX index = 1.84
● On a 7-points scale →
not statistically significant
(F1,286 = 0.289, p = 0.591)
38
User-Experience
● users feel “assisted by the product”
○ usability can be improved
39
User-Experience
● users feel “assisted by the product”
○ usability can be improved
● users are “stimulated by the product”
○ users can identify with it
○ room for hedonic improvement
39
User-Experience
● users feel “assisted by the product”
○ usability can be improved
● users are “stimulated by the product”
○ users can identify with it
○ room for hedonic improvement
● system considered as “rather desired”
○ attractiveness above average
39
Single Tasks Analysis
40
Task 1 Analysis
● sidebar (A) better
○ faster
○ easier to use
○ less demanding
41
Task 1 Analysis
● sidebar (A) better
○ faster
○ easier to use
○ less demanding
● camera tool slower
○ no clue on how /
where to place it
41
Task 2 Analysis
● sidebar (A)
○ easier to use
42
Task 2 Analysis
● sidebar (A)
○ easier to use
○ faster (only E.T.)
○ causes several
system errors →
see later
42
Task 2 Analysis
● sidebar (A)
○ easier to use
○ faster (only E.T.)
○ causes several
system errors →
see later
● RTLX == copy tool
42
Task 3 Analysis
● handles (B)
○ faster
○ less demanding
○ easier to use
43
Task 3 Analysis
● handles (B)
○ faster
○ less demanding
○ easier to use
● color detection
○ 3 tasks in one
○ digitize + create
43
Task 4 Analysis
● arrow (B)
○ faster
○ less demanding
44
Task 4 Analysis
● arrow (B)
○ faster
○ less demanding
○ where?
44
Task 4 Analysis
● arrow (B)
○ faster
○ less demanding
○ where?
● color detection
○ easier to use
○ create + connect
44
Task 5 Analysis
● rubber (B)
○ faster (only E.T.)
45
Task 5 Analysis
● rubber (B)
○ faster (only E.T.)
○ bug found!
45
Task 5 Analysis
● rubber (B)
○ faster (only E.T.)
○ bug found!
● color detection (A)
○ easier to use
○ less demanding
○ more accurate
45
Task 6 Analysis
● rubber (B)
○ faster
○ less demanding
46
Task 6 Analysis
● rubber (B)
○ faster
○ less demanding
● color detection (A)
○ easier to use
○ experience?
46
Interesting facts
● people with background in HCI
→ performed more easily & faster
● good sketching abilities
→ same + found system more attractive
47
Interesting facts
● people with background in HCI
→ performed more easily & faster
● good sketching abilities
→ same + found system more attractive
● people familiar with mobile devices
→ faster + system less distracting
47
Interesting facts
● people with background in HCI
→ performed more easily & faster
● good sketching abilities
→ same + found system more attractive
● people familiar with mobile devices
→ faster + system less distracting
● during day-time → better performance & lower RTLX
47
Considerations
● accuracy = ratio of task’s precision
○ global accuracy = 86%
○ color detection = 67%
→ more delicate + error-prone
48
Considerations
● accuracy = ratio of task’s precision
○ global accuracy = 86%
○ color detection = 67%
→ more delicate + error-prone
● barcodes + hands into digitized image → distraction + inexperience
48
Considerations
● accuracy = ratio of task’s precision
○ global accuracy = 86%
○ color detection = 67%
→ more delicate + error-prone
● barcodes + hands into digitized image → distraction + inexperience
● if BBPapier wavy → grey area projected
○ projector’s brightness → shadow (“ghost shape”)
48
Projector’s beam
● when toolbox at sides of projection → outer areas appear darker
○ either no color is detected
○ “fake” grey color recognized → multiple components created
49
Projector’s beam
● when toolbox at sides of projection → outer areas appear darker
○ either no color is detected
○ “fake” grey color recognized → multiple components created
● most system errors
○ excessively intense light causing reflections on table
49
Projector’s beam
● when toolbox at sides of projection → outer areas appear darker
○ either no color is detected
○ “fake” grey color recognized → multiple components created
● most system errors
○ excessively intense light causing reflections on table→ closest barcodes could not be detected→ devices continuously refreshing
49
Projector’s beam
● when toolbox at sides of projection → outer areas appear darker
○ either no color is detected
○ “fake” grey color recognized → multiple components created
● most system errors
○ excessively intense light causing reflections on table→ closest barcodes could not be detected→ devices continuously refreshing
○ “effective time” → task completed if interaction is correct
49
Conclusions
50
Which better?
51
Which better?
● color detection (A)
○ easier to use
○ takes longer time
51
Which better?
● color detection (A)
○ easier to use
○ takes longer time
● barcode recognition (B)
○ definitely faster
○ harder to understand
51
Which better?
● color detection (A)
○ easier to use
○ takes longer time
● barcode recognition (B)
○ definitely faster
○ harder to understand
● both have similar workload
○ around 1.9 out of 7
○ positively low →
not too distracting51
Ideal implementation
1-2. sidebar → digitize & duplicate screens
52
Ideal implementation
1-2. sidebar → digitize & duplicate screens
2. copy tool too → duplicate without moving papers
52
Ideal implementation
1-2. sidebar → digitize & duplicate screens
2. copy tool too → duplicate without moving papers
3-4. both approaches to detect components & create connections
a. color detection = from scratch + multiple tasks at same time
52
Ideal implementation
1-2. sidebar → digitize & duplicate screens
2. copy tool too → duplicate without moving papers
3-4. both approaches to detect components & create connections
a. color detection = from scratch + multiple tasks at same time
b. handles & arrow = fast & easy-to-use
52
Ideal implementation
1-2. sidebar → digitize & duplicate screens
2. copy tool too → duplicate without moving papers
3-4. both approaches to detect components & create connections
a. color detection = from scratch + multiple tasks at same time
b. handles & arrow = fast & easy-to-use
5-6-7. rubber tool to remove components, connections & screens
→ implement separate ones to avoid possible mistakes
52
Future Work
● visual feedback when projections disappear from table
53
Future Work
● visual feedback when projections disappear from table
● icon near clock → tells which action is being performed
53
Future Work
● visual feedback when projections disappear from table
● icon near clock → tells which action is being performed
● exploring color detection further
○ better algorithms to calculate the threshold
○ better ways to detect shapes through contours
53
Future Work
● visual feedback when projections disappear from table
● icon near clock → tells which action is being performed
● exploring color detection further
○ better algorithms to calculate the threshold
○ better ways to detect shapes through contours
● barcodes = built with wood / opaque plastic → reduce reflections
53
Future Work
● visual feedback when projections disappear from table
● icon near clock → tells which action is being performed
● exploring color detection further
○ better algorithms to calculate the threshold
○ better ways to detect shapes through contours
● barcodes = built with wood / opaque plastic → reduce reflections
● compare interaction techniques with original system
53
Future Work
● visual feedback when projections disappear from table
● icon near clock → tells which action is being performed
● exploring color detection further
○ better algorithms to calculate the threshold
○ better ways to detect shapes through contours
● barcodes = built with wood / opaque plastic → reduce reflections
● compare interaction techniques with original system
● test system in real-life scenario → inside company or startup53
Thanks!Any questions?
images from pixabay.com & papers by Bähr
Top Related