Context Aware Paper- based Review Instrument: A User Study Presented by: Dipti Shah, Hoda Homayouni,...
-
date post
21-Dec-2015 -
Category
Documents
-
view
215 -
download
0
Transcript of Context Aware Paper- based Review Instrument: A User Study Presented by: Dipti Shah, Hoda Homayouni,...
Context Aware Paper-Context Aware Paper-based Review Instrument: based Review Instrument: A User StudyA User Study
Presented by: Dipti Shah, Hoda Presented by: Dipti Shah, Hoda Homayouni, Daniel BelcherHomayouni, Daniel Belcher
AgendaAgenda
Introduction of CAPRIIntroduction of CAPRI Related Works and MotivationRelated Works and Motivation Experimental DesignExperimental Design DiscussionsDiscussions
ObjectiveObjective
Evaluating CAPRI (in terms of Evaluating CAPRI (in terms of efficiency and perceptual efficiency and perceptual measures) by comparing it to measures) by comparing it to two other review methodstwo other review methods
Traditional Review SystemTraditional Review System
SortingSorting Memory workMemory work Errors Errors TimeTime CollaborationCollaboration
What is CAPRI ?What is CAPRI ?
Context Aware Paper Based Context Aware Paper Based Review InstrumentReview Instrument
Tangible and augmented user Tangible and augmented user interface for architectural interface for architectural document review document review
CAPRI SystemCAPRI System
Table topTable top PlanPlan PapersPapers ProjectorProjector CameraCamera Computer with a Computer with a
software software program (Java)program (Java)
CAPRI ProcessCAPRI Process
Input InterfaceInput Interface Medium Medium
DetectorDetector Context Context
InterpreterInterpreter Output InterfaceOutput Interface
Input InterfaceInput Interface
Menu SelectionMenu Selection
Grab and MoveGrab and Move
Related WorksRelated Works
CAPRI is an CAPRI is an augmentedaugmented and and tangibletangible environment for environment for facilitating facilitating documentdocument reviewreview using design objects that are using design objects that are the fundamental computational the fundamental computational elements. elements.
Document Review SystemsDocument Review Systems
Document review is a Document review is a cognitively challenging process.cognitively challenging process.
The “hypertext-like” aspect of The “hypertext-like” aspect of CAPRI decreases the reviewers’ CAPRI decreases the reviewers’ cognitive load. It also saves the cognitive load. It also saves the reviewer time and effort in reviewer time and effort in sorting and presenting the sorting and presenting the relevant details.relevant details.
Document Review SystemsDocument Review Systems
Architectural documents are Architectural documents are often reviewed by different often reviewed by different people. Thus, the issue of people. Thus, the issue of authority always exist in design authority always exist in design review process.review process.
CAPRI affords diverse teams to CAPRI affords diverse teams to directly and intuitively interact directly and intuitively interact with the design document.with the design document.
Document Review SystemsDocument Review Systems
Related WorksRelated Works JANUSJANUS is based on a model of a is based on a model of a
computer supported design that computer supported design that utilizes a knowledge-based utilizes a knowledge-based approach to link a graphic approach to link a graphic construction system to construction system to hypertext.hypertext.
JANUSJANUS
Tangible User InterfacesTangible User Interfaces
Tangible User Interfaces (TUI) Tangible User Interfaces (TUI) are interfaces that couple digital are interfaces that couple digital information to physical objects information to physical objects and environments. and environments.
By using socially accepted media, By using socially accepted media, CAPRI relies on natural CAPRI relies on natural behaviors to support a complex behaviors to support a complex task.task.
Tangible User InterfacesTangible User Interfaces
Related WorksRelated Works The The metaDESKmetaDESK
system system focuses on the focuses on the use of tangible use of tangible objects as objects as driving driving elements of elements of HCI.HCI.
Augmented Reality InterfacesAugmented Reality Interfaces
Augmented Reality (AR) is an Augmented Reality (AR) is an emerging platform for emerging platform for interacting with computers, interacting with computers, contextualizing virtual objects in contextualizing virtual objects in a physical world.a physical world.
CAPRI is an interface that has CAPRI is an interface that has almost no learning curve. almost no learning curve. Unifying the tasks space, it Unifying the tasks space, it encourages focus on the design encourages focus on the design discussion instead of the discussion instead of the mechanics of interactions.mechanics of interactions.
Augmented Reality InterfacesAugmented Reality Interfaces
Related WorksRelated Works ATIATI is an interface is an interface
device for distributed device for distributed design review that design review that fuses input and output fuses input and output that can be handled, that can be handled, passed back and forth, passed back and forth, inspected visually, inspected visually, positioned relative to positioned relative to other objects and other objects and even assembled or even assembled or disassembled disassembled
with them. with them.
MotivationMotivation
Predictions based upon the CAPRI Predictions based upon the CAPRI claims:claims:
More efficient (less time, better More efficient (less time, better performance, less errors)performance, less errors)
Less cognitive loadLess cognitive load Higher level of collaborationHigher level of collaboration
Experimental Design OverviewExperimental Design Overview
Three Conditions:Three Conditions: Condition 1: Paper-Based Condition (PB – 1)Condition 1: Paper-Based Condition (PB – 1) Condition 2: Computer-Supported Condition (CS -2)Condition 2: Computer-Supported Condition (CS -2) Condition 3: CAPRI Condition (CP – 3)Condition 3: CAPRI Condition (CP – 3)
45 Tasks (15 per condition)45 Tasks (15 per condition)
10-15 Subject (projected) = 100-150 data points per condition.10-15 Subject (projected) = 100-150 data points per condition.
Data recorded:Data recorded: Performance measures (time and error) recorded.Performance measures (time and error) recorded. Perceptual measures (post-experiment questionnaire) recorded.Perceptual measures (post-experiment questionnaire) recorded.
Analysis:Analysis: Compatible with three-way ANOVA design.Compatible with three-way ANOVA design.
Within and between conditions.Within and between conditions. Within and between subjects.Within and between subjects. Within and between tasks.Within and between tasks.
Paper-Based Condition (PB – 1)Paper-Based Condition (PB – 1)
Similar to “traditional paper-based” document review.Similar to “traditional paper-based” document review. Subjects may browse paper-based plots in any order.Subjects may browse paper-based plots in any order. Uses A1 sized plotted drawing set (plan and section).Uses A1 sized plotted drawing set (plan and section). Details are on Standard (8Details are on Standard (8½×11") paper.½×11") paper.
Computer-Supported Condition (CS – 2)Computer-Supported Condition (CS – 2)
A mix of “paper-based” and digital references.A mix of “paper-based” and digital references. Uses the same A1 sized plotted drawing set (plan and Uses the same A1 sized plotted drawing set (plan and
section).section). Details are on a computer monitor.Details are on a computer monitor. Subjects may browse details by clicking on a detail Subjects may browse details by clicking on a detail
thumbnail to enlarge in a main browser window.thumbnail to enlarge in a main browser window.
CAPRI Condition (CP – 3)CAPRI Condition (CP – 3)
A mix of “paper-based” and digital references, but with A mix of “paper-based” and digital references, but with automated linking and detail sorting.automated linking and detail sorting.
Uses the same A1 sized plotted drawing set (plan and Uses the same A1 sized plotted drawing set (plan and section).section).
Relevant details are projected on Standard (8Relevant details are projected on Standard (8½×11") paper½×11") paper.. Subjects may browse details by overlying paper on detail Subjects may browse details by overlying paper on detail
bugs.bugs.
ProcedureProcedure
Consent formsConsent forms InstructionsInstructions Training phaseTraining phase First condition (randomized)First condition (randomized) InstructionsInstructions Training PhaseTraining Phase Second condition (randomized)Second condition (randomized) InstructionsInstructions Training PhaseTraining Phase Third condition (randomized)Third condition (randomized) Post-experiment QuestionnairePost-experiment Questionnaire Debriefing (optional)Debriefing (optional) Participants paidParticipants paid
Pre-experiment Consent FormPre-experiment Consent Form
PURPOSE OF THE STUDYPURPOSE OF THE STUDY The study is to test different methods of architectural document The study is to test different methods of architectural document
review. With the help of participants, we will assess the review. With the help of participants, we will assess the effectiveness and usability of three different tools of document effectiveness and usability of three different tools of document review. review.
STUDY PROCEDURESSTUDY PROCEDURES The study involves sitting, reading and executing certain tasks. The study involves sitting, reading and executing certain tasks.
Participant will be given a learning phase in which to understand Participant will be given a learning phase in which to understand the task, then 45 discrete questions related to architectural the task, then 45 discrete questions related to architectural drawings will be presented. drawings will be presented.
[…][…]
RISKS, STRESS, OR DISCOMFORTRISKS, STRESS, OR DISCOMFORT This study involves sitting for more than an hour, which might This study involves sitting for more than an hour, which might
cause discomfort to some subjects. There is also a possibility that cause discomfort to some subjects. There is also a possibility that a participant may experience mild eye-strain while looking at the a participant may experience mild eye-strain while looking at the computer screen. computer screen.
If any of the above symptoms occur, and the participant does not If any of the above symptoms occur, and the participant does not wish to continue, they may alert the attending researcher and wish to continue, they may alert the attending researcher and immediately stop the experiment without penalty.immediately stop the experiment without penalty.
WRITTEN PERMISSION MUST BE GAINED BEFORE EXPERIMENT.WRITTEN PERMISSION MUST BE GAINED BEFORE EXPERIMENT.
InstructionsInstructionsExample from CAPRI condition:Example from CAPRI condition:““In this segment, you are asked to find answers to a In this segment, you are asked to find answers to a
set of questions relating to an architectural drawing set of questions relating to an architectural drawing set. The relevant details necessary to answer each set. The relevant details necessary to answer each of the questions are contained in the drawing set. of the questions are contained in the drawing set. Your task is to read the questions and find the Your task is to read the questions and find the answer.” answer.”
““In this segment, you will use a standard paper In this segment, you will use a standard paper drawing and pieces of standard paper to query the drawing and pieces of standard paper to query the drawing for detail. When the upper right-hand drawing for detail. When the upper right-hand corner of a piece of paper is placed on a bug, the corner of a piece of paper is placed on a bug, the relevant detail is projected from above onto the relevant detail is projected from above onto the page…”page…”
““Please answer all the questions as quickly and Please answer all the questions as quickly and accurately as possible.”accurately as possible.”
Sample TasksSample Tasks
““What is the distance between the 1What is the distance between the 1stst step and the end of the hand-rail on step and the end of the hand-rail on stair 3B?”stair 3B?”
““What kind of sign is in between the What kind of sign is in between the two doors in the east side of the two doors in the east side of the lobby?”lobby?”
““What is the width of central aisle of What is the width of central aisle of the auditorium?”the auditorium?”
““On the southern wall of the On the southern wall of the vestibule seminar room, what is the vestibule seminar room, what is the width of the whiteboard?”width of the whiteboard?”
Controls - LearningControls - Learning COUNTERBALANCED CONDITION ORDERSCOUNTERBALANCED CONDITION ORDERS
First ConditionFirst Condition Second ConditionSecond Condition Third ConditionThird Condition
Subject 01Subject 01 22 33 11
Subject 02Subject 02 11 22 33
Subject 03Subject 03 11 33 22
Subject 04Subject 04 22 33 11
Subject 05Subject 05 33 11 22
Subject 06Subject 06 11 33 22
Subject 07Subject 07 33 11 22
Subject 08Subject 08 22 33 11
Subject 09Subject 09 33 22 11
Subject 10Subject 10 33 11 22
Subject 11Subject 11 11 22 33
Subject 12Subject 12 22 11 33
Subject 13Subject 13 11 33 22
Subject 14Subject 14 11 33 22
Subject 15Subject 15 11 22 33
KEYKEY
1 = Paper-Based (PB)1 = Paper-Based (PB)2 = Computer-2 = Computer-
Supported (CS)Supported (CS) 3 = CAPRI (CP)3 = CAPRI (CP)
Data Collected: PerformanceData Collected: PerformanceTwo performance measures:Two performance measures: Time (in milliseconds)Time (in milliseconds) Error (correct/incorrect)Error (correct/incorrect)
Measurement Systems:Measurement Systems: Time: Participants press a key to start the Time: Participants press a key to start the
time trial and again when finished. time trial and again when finished. Subjects may pause the experiment at Subjects may pause the experiment at any time, but not continue with tasks.any time, but not continue with tasks.
Error: Participants record their answers to Error: Participants record their answers to questions in pencil on a worksheet. All questions in pencil on a worksheet. All errors are hand-checked and recorded. errors are hand-checked and recorded.
Data Collected: PerceptualData Collected: Perceptual The perceptual measures were recorded in the form of a The perceptual measures were recorded in the form of a
subjective post-experiment questionnaire.subjective post-experiment questionnaire.
All questions are standard Likert-scale rankings.All questions are standard Likert-scale rankings.
Forced choice, no center (6-level scale)Forced choice, no center (6-level scale)
4.) “Locating a detail bug in the plan drawing is an easy task.”4.) “Locating a detail bug in the plan drawing is an easy task.” 1 21 2 3 3 4 4 5 5 6 6
1 = Completely disagree1 = Completely disagree
2 = Disagree2 = Disagree
3 = Somewhat disagree3 = Somewhat disagree
Unstructured comment space will be provided at the end of Unstructured comment space will be provided at the end of the post-experiment questionnaire to record any notes about the post-experiment questionnaire to record any notes about their experience or feedback about the interfaces. their experience or feedback about the interfaces.
4 = Somewhat agree4 = Somewhat agree5 = Agree5 = Agree6 = Completely agree6 = Completely agree
Analysis Format (Sample Data)Analysis Format (Sample Data)
Mean Response Time
0
5
10
15
20
25
Paper Based Computer Supported CAPRI
Experimental Condition
Res
po
nse
Tim
e (s
eco
nd
s)
Analysis Format (Sample Data)Analysis Format (Sample Data)
"Locating a Detail Bug in the Plan Drawing is an easy task."
0
1
2
3
4
5
6
7
CompletelyDisagree
Disagree SomewhatDisagree
SomewhatAgree
Agree CompletelyAgree
Participant Responses
Res
pons
es
ChallengesChallenges
Statistical Comparison Statistical Comparison Architectural data setsArchitectural data sets Research MethodologyResearch Methodology Human Subjects issuesHuman Subjects issues CAPRI interface issuesCAPRI interface issues Confound it!Confound it!