Human Computer Interaction Institute | Spring 2005 | ONR2 onr 2 Spring Semester Update 0 5 – 0 5 -...

Human Computer Interaction Institute | Spring 2005 | ONR2

onr2Spring Semester Update

0 5 – 0 5 - 0 5

Office of Naval Reseach - Team 2


P R O B L E M

Technical manuals that currently exist for reference:• Are oversized, awkward, hard to move• One manual can lead to many others

IETMs • Repair manuals scanned into PDF forms• Very few manuals in XML format

The Navy must maintain and repair a large array of complex on-board machinery.


V I S I O N

The system we are building will negate the need for paper manuals.

It will use speech and touch input on a Fujitsu Tablet PC, providing flexible access to maintenance information.

Annotation

Search for the chapters and easily navigate through the steps and diagrams

Collaboration

Placeholding

Leave notes to themselves or others

Keep track of where they have been and store temporary references

Work + communicate with non-collocated users

Navigation


L I T E R A T U R E R E V I E W


Understand how users manage multiple modalities and design an interface to support this use (Ayoob, Bodine, Bohus, Rudnicky, Siegel, 2004)

Accurately modeling users’ existing multimodal integration patterns and not forcing users to adopt a particular style (Oviatt, et al., 2003)

Detecting and adapting to a user’s dominant modality pattern (Oviatt, et al., 2003)

L I T R E V I E W : Challenges in multimodality


Mobile and portable

Positioned in delicate ways to support mutual access and collaboration

Ad hoc annotations

Representations on paper can be easily memorized, navigated, and scanned.

L I T R E V I E W : Affordances of Paper

Only Touching the Surface: Creating Affinities Between Digital Content and Paper [Luff, Heath, Norrie, Signer, Herdman, 2004]


• Conversations focus on:– identification of target objects– descriptions of actions on those objects– task completion confirmation

(Fussell et al., 2004)

• Pointing gestures– used to refer to task objects and locations

(Fussell et al., 2004)

• Representational gestures– represent the form of task objects, the

nature of actions to be performed on those objects(Bekker et al.)

L I T R E V I E W : Collaboration


[Ayoob, Bodine, Bohus, Rudnicky, Siegel, 2004]:

• Provide task monitoring and feedback

• Should be able to rotate the pictures

• Browse and navigate information from visual displays

L I T R E V I E W : Diagrams and Graphics


• Fluent human-human speech is disrupted by pauses as brief as 1 to 2 seconds(Oviatt, Cohen, et. al, 1992)

• Supports eyes-busy tasks(Ayoob, Bodine, Bohus, Rudnicky, Siegel, 2004)

• Provides poor visual orientation(Ayoob, Bodine, Bohus, Rudnicky, Siegel, 2004)

• Need to allow for control over pace(Ayoob, Bodine, Bohus, Rudnicky, Siegel, 2004)

• User must always initiate an action to hear an audio response (IETM Workshop, 1999)

L I T R E V I E W : Speech and Audio


• 10% faster task completion rate, 36% fewer errors (Oviat, et al, 2001)

• Mutual Disambiguation: Better error handling procedures because of increased redundancy (Oviatt, 2000)

• Navigating spatially-oriented diagrams can be done more precisely, rapidly and with less effort in pen + speech (Oviatt, 1996)

• Optimize speed through selection of a particular input mode (Oviatt, 1996)

+

L I T R E V I E W : Multimodality


• Presenting text descriptions and concurrent audio descriptions helps long term memory and learning (Moreno & Mayer, 2002)

• On the other hand, presenting concurrent text and audio speech did not assist with reading a technical diagram or chart (Kalyyuga & Chandler, 2004)

+

L I T R E V I E W : Multimedia Learning


F O C U S S E T T I N G

FOCI

How do documents and reference manuals aid or hinder the task? (productivity, process & repair)

Life + Flow of Information: How does information transfer between workers? How is new information documented?

OTHER ISSUES

Are manuals effective in supporting error/emergency situations?How are they presented to the users?How do users feel about the manuals?What other tools are being used?

1

2


Contextual in TRANSITION

C O N T E X T U A L I N Q U I R I E S


U S E R N E E D S : A Review

Annotation

Search for the chapter they are looking for and easily navigate through the steps and diagrams

Collaboration

Placeholding

Leave notes to themselves or others

Keep track of where they have been and store temporary references

Work with others on tasks where non-collocated users have to communicate with each other

Navigation

9 CIs with a total of 16 interviewees were conducted with the Navy and analogous work domains.

Here we highlight the breakdowns and how they map to our 4 user needs.


Annotation

• Manuals are complex and huge

• Navy maintenance workers did not realize they were doing something wrong

• Too many steps to find fault in manual, have to flip lots of pages to find where the problem is

Collaboration

Placeholding

S T R A T E G I E S

• Place manual on closest table with relevant page left open

• Communicate to split up tasks

• Communicate to know what the other is doing

• Environment is loud, cramped and small

Navigation

C I : N A V Y V I D E O S [Spring 2004]

B R E A K D O W N S


Annotation

• Have to switch CD’s

• Don’t depend on digital manual

• Rely on what “feels” right

• Older manuals are unmanageable (too many) and greasy/ dirty

Collaboration

Placeholding• Use for special cases, reinforcement, or printout

Navigation

C I : C A R M E C H A N I C S S H O P

S T R A T E G I E S

• Print out diagrams

• Rely on their experiences and use manual as a backup

B R E A K D O W N S


Annotation

• Student must take diagram out of binder in order to rotate diagram

Collaboration

Placeholding

• Students ask instructor for help before reading manual

• Don’t understand instructions

• Delayed response to question

• Has to wait for outside information before doing task

• Page ripped out of manual

• All 4 try to be on the same page (6:20) – flip through manuals

• “We don’t know what’s really going on” – U1 and U2 waiting for U3 and U4

Navigation

C I : P I A (Pittsburgh Institute of Aeronautics)

S T R A T E G I E S

• Bring manual with them to task

• Rotate/ orient diagram for task

B R E A K D O W N S


Annotation

• In the install, matched up the wrong polarities (after 10:00)

• Too many different types of trucks to remember exactly how the polarities align (12:00)

Collaboration

Placeholding

• Updates have to be made manually to each manual

• Redundant part # ordering

• Expert’s tips are warnings from the manual

• Battery instructions placed on vehicle as a sticker

• Others ask expert for help if unsure (15:00) – allowed to interrupt

• Mechanics must go into back computer room (Away from workspace) to find part #s to order parts

Navigation

C I : H U N T A R M O R Y

S T R A T E G I E S

• Novices ask expert for help before consulting manual

B R E A K D O W N S


Annotation

• While searching for a maintenance card in the white binder, could not find it

• Can't watch Oscope and tweak the system at the same time

• Have to lean over to read the Oscope

Collaboration

Placeholding

• Required to have a maintenance log

• Writes brief notes to himself

• Antenna is outside, but workers have to keep the manual inside. So if you want access to the manual you have to keep going inside and outside.

• Two workers have to shout back and forth over loud noise even if partner is less than 10 feet away

• Antenna is five levels up from where the DCC console is

Navigation

C I : N O R F O L K I N T E R V I E W S

S T R A T E G I E S

• Use pen or finger to mark a spot in the manual

• Reduce speaking to only what’s relevant (noisy)

• Write task on paper instead of taking manual if working outside

B R E A K D O W N S


C O N S O L I D A T E D M O D E L S


T O T A L B R E A K D O W N S

AnnotationCollaboration

PlaceholdingNavigation

5

5

10

13


Design

F R O M D A T A T O D E S I G N


D A T A to D E S I G N

1. Breakdowns User Needs

2. User Needs Extreme Design / Preliminary Prototypes

3. Extreme Design Norfolk Prototype

4. Norfolk Prototype Norfolk Iteration

5. Norfolk Iteration Final Spring Prototypes


N O R F O L K D E S I G N : 1


Users do not go step by step but skip around [Navy TA2, PIA1, PIA2]

Manuals and diagrams are too complex/ unreadable [Navy TA2, Car Mech]

R A T I O N A L E : Step Overview

C A

N P


Current step highlighted.

Felt short abbreviated text was not helpful, probably wouldn’t need it [U2]

No helpful information conveyed in the first few words of a step

F E E D B A C K : Step Overview


Keep one finger on the manual and another on diagram [PIA1, PIA2]

Manuals are too complex/ unreadable [Navy TA2, Car Mech]

R A T I O N A L E : Step + Diagram

C A

N P


Good to see steps and diagrams together

Diagram was too small. [U4]

F E E D B A C K : Step + Diagram


Complex and big diagrams [Navy TA2, Colouring Book, SLQ-32]

Need to consult parts of a diagram [Assumption (complexity of diagrams, limited screen space)]

R A T I O N A L E : Diagram Zoom


Tapping on a specific part of a diagram zooms in (negative comment)[U1]

Drill down/ more detailed diagrams would really help. [U5]

F E E D B A C K : Diagram Zoom


Find appropriate manual [Navy TA2, Car Mech, HA1]

Flips Through Manual for a long time [Navy TA2, PIA1, PIA2]

Information is haphzardly distributed in looseleaf papers [Car Mech, HA1, PIA1]

R A T I O N A L E : Navigation Bar


Felt like he could tap on the top to access all relevant parts of the IETM [U1]

Users Usually use Index to find PMS tasks.

Volume names not descriptive enough[U6]

F E E D B A C K : Navigation Bar


Hard to find fault number/ part number in index [Navy TA2, PIA1, PIA2]

Flips Through Manual for a long time [Navy TA2, PIA1, PIA2]

Finding right reference Index Number [Car Mech, PIA2]

R A T I O N A L E : Search


Fault Codes are starting points for troubleshooting, need to easily search for them instead of looking through the index

F E E D B A C K : Search


Users use fingers, pens, whatever they can get their hands on to mark pages in manuals as they are flipping between sections [PIA1, PIA2]

Users lose references if remove finger [PIA2]

Flips Through Manualfor a long time [Navy TA2, PIA1, PIA2]

R A T I O N A L E : Bookmarks

C A

N P


Uses screwdrivers and pens to stick between sheets of manual for placeholding [Norfolk Observation]

Bookmarks that can let you go back to where you were [U2]

Bookmarks only needed for current task [U4]

F E E D B A C K : Bookmarks


After the first day of evaluation at Norfolk, we continued on to build a consolidated prototype preserving wanted features and adding new ones. Another prototype focused on collaboration was also developed.


PRESERVED ELEMENTS FROM PREVIOUS ITERATIONS:

– Top Navigation Bar

– Search Bar

– Overall step overview from Proto2

– Diagram + Step from Proto1

– Bookmarks

C O N S O L I D A T E D P R O T O T Y P E


NEW ELEMENT: ANNOTATION

• Some people don’t like marking in the book, but U1 does.

• User uses other people’s notes that they previously wrote in the manual since they are often pertinent to his tasks

• The information in the notes from previous people often carry over to his current tasks.

• When looking at a diagram, user will circle things, write notes, trace signal on page, map where he is.

• If he has to leave task, often writes a short note with where he is at.

• Uses exterior tools such as sticky notes to notate and mark manuals.



NEW ELEMENT: LOGGING• Run back and forth to enter logs• Logging : "___ was broken, fixed it by ___” • Different types of logs: Checklists for maintenance tasks, Logging for troubleshooting• PMS log to makes sure that checks are going on.• Logs are usually there for reference• Date of log would not matter as much to user

NEW ELEMENT: DRILL-DOWN DIAGRAMS• Drill down diagrams helpful. [U5]• Diagrams should become more detailed instead of just bigger.



F E E D B A C K

• Validated comments from previous day• Liked the side overview• Logging feature clear to user and useful• Tabs were easy to use• Text of step inside diagram space helpful• Placeholder asterisk confusing



C O L L A B O R A T I O N

• 2 units on ship, 1 inside 1 outside

• One can’t work on equipment until other turns it off

• Sound Power phones don't always work, communication can be hard

• Never alone when working on the radar system

• Will not know what the other person is doing without direct communication

• “Personal relationships can make or break whether a fault is found while collaborating” [U2]

• Instant messenger would be "great"

C O L L A B O R A T I O N P R O T O T Y P E


Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks - Susan R. Fussell, Leslie D. Setlock, Jie Yang, Jiazhi Ou, Elizabeth Mauer, and Adam D. I. Kramer, Carnegie Mellon University



F E E D B A C K

• Users LOVED collaboration feature.

• Collaboration toolbox icons are ambiguous

• Did not need to see what step the other person was on, because usually are on the same step



– Must always carry manuals everywhere [HA1, PIA1, Norfolk CI]

– Must lug lots of equipment up to location, multiple trips, so tablet is really the same as an antelope. [Norfolk Observation]

– Computer systems are not portable [HA1, HA2, Norfolk CI]

– Lack of space while doing some tasks [Norfolk Observation]

– Awkward body positioning [Norfolk Observation]

– Need High –Resolution for Diagrams

F U J I T S U T A B L E T


After returning from Norfolk, we concentrated our efforts on developing two prototypes that centered around 2 of our 4 main needs: Navigation and Collaboration. We will now present the Navigation Prototype.


• Navigation

– Steps, Figures, and Tables in tabs

– Steps fully written out

– Steps are clearly separated

• Placeholding

– Bookmark the whole page or the whole diagram not individual steps

O V E R V I E W : The Navigation Prototype


N A V I G A T I O N P R O T O T Y P E


Can see multiple steps to skim ahead [Navy TA2, PIA1, PIA2]

Steps written out completely

Clear separation of steps and Highlight the current step know / keep track of where are in the task

R A T I O N A L E : The Navigation Prototype


Steps, Figures, and Tables in separate tabs Alex’s lit review that figures are better by themselves – less cognitive load with less text; workers would bookmark only diagrams



Placeholding a whole procedure or diagram not a single step [PIA1, PIA2, Norfolk]



Transition: Walkthrough

N A V I G A T I O N P R O T O T Y P E :end to end walkthrough .


C O L L A B O R A T I O N P R O T O T Y P E :end to end walkthrough .


O V E R V I E W : The Collaboration Prototype

C O L L A B O R A T I O N

Personal Space- Work space to import diagrams and steps you need to manage- Materials and “Backboard” can be manipulated/annotated- Items can be copied over to the “Shared Space”

Shared Space- Affords real time collaborative manipulation/annotation- Works in conjunction with VoIP- Work can then be saved back into “Personal Space”

N A V I G A T I O N- Placement in Task Overview- Current step enlarged


S U M M E R P L A N S


• Run Think Aloud Studies on Antenna Rotator Task

– Paper prototypes

– Flash prototypes

– Grammars

– Interaction Techniques

• Begin to integrate speech into prototype

• Prep for Norfolk Trip

– Protocol

– Design for task

M I D - M A Y t o E N D O F M A Y


• Norfolk Trip

– Include speech: test effectiveness of voice input

– Test with real users in Context

• Take user test data and iterate

• User test on Voyager, Hunt Armory, PIA

• Take user test data and iterate

J U N E t o M I D - J U L Y


• Final Presentation

• Final Paper

• Final Prototype

M I D - J U L Y t o E N D O F J U L Y


F E A T U R E C Y C L E:

- Two weeks per cycle

- Two teams work in parallel

- Integration of two features with system at the end of each feature cycle

- Example Features: Annotation tool functionality, step navigation, shared space view, drill down diagrams, collaborationt ool functionality, navigational drop down menus

- Mockup Paper prototype- Internal Testing : HE, CW- External Testing: TA - Iterate on Feature

- Implement Feature in Sphinx/Salt/JS- Internal QA: Bug Test- Integrate w/other features

- Create Mid-fi Flash Proto- External User studies - Iterate on Design

P A P E R

F L A S H

J A V A


TEAM A

MAY

TEAM B

JUNE

JULY

AUGUST


TEAM A TEAM B

Week 1 15-May System Architecture Design

Week 2 22-May

Feature 1 Feature 2

Week 3 29-May Jus tin Absent (1 week) Carol Absent (4 days)

Week 4 5-Jun Integration Checkpoint (ONR1)

Feature 3 Feature 4

Week 5 12-J un Norfolk

Week 6 19-J un

Feature 5 Feature 6

Week 7 26-J un

Week 8 3-Jul Integration Checkpoint (ONR1) Team absent (3 days)

Feature 7 Feature 8

Week 9 10-J ul

Week 10 17-J ul

Feature 9

Feature 10

Week 11 24-J ul

Prepare Final Deliverable

Week 12 31-J ul Final Presentation


Questions?

Human Computer Interaction Institute | Spring 2005 | ONR2 onr 2 Spring Semester Update 0 5 – 0 5 -...

Documents

Transcript of Human Computer Interaction Institute | Spring 2005 | ONR2 onr 2 Spring Semester Update 0 5 – 0 5 -...