R. AltermanBrandeis Unviersity

Review of Human FactorsDiscovery and Invention Projects

Code 342Cognitive and Neural Sciences

Office of Naval Research

5-6 September 2001

Cognitive Model of Computer-Mediated Joint Activity

Objective

• Develop a model of how participants in a computer-mediated cooperative activity coordinate and jointly make sense of the unfolding situation.

• Develop a methodology and an analyst’s tool that supports a development team in constructing secondary representations that improve the performance of the users in staying coordinating and maintaining a common viewpoint during a dynamic situation.

Problem/Deficiency Being Addressed

• The Navy continues to move into an era of distributed, computer-mediated planning, problem solving, and joint activity. The joint activity among ship, air, submarine, and shore personnel is largely mediated by computer.

• A significant task is to develop tools and interface technology that structures the activities of distributed planners and actors in a way that simplifies ongoing coordination and reduces the work involved in sharing meanings, understandings, and assessments, given the press of time in an ongoing crisis situation.

Technical Approach

• Coordinating representation is an external representation. It is structured and shared amongst a group of cooperating actor. Its designated purpose is to simplify the interactions among participants at the critical junctures of a joint cooperative activity.

• Use coordinating representation as a signaling system to improve coordination and communication during the performance of computer-mediated tasks.

• Develop a methodology and an analyst’s tool for building a set of effective coordinating representations for a given computer-mediated cooperative task.

Participants and Staffing

• Rick Alterman, PI (1 month)

• Mix of Ph.D., Masters, Undergraduates, Post Docs, and Consultant– Roughly equivalent to 3 Ph.d’s

• Jim Storer (1 month)

Current Progress

• Completed an experiment that confirmed the utility of the coordinating representation for improving performance. – A significant part of the analysis of the data was to determine the

methods used among the participants to jointly make sense of a shared dynamic situation.

• Developed a VCR-like device that allows an analyst to replay the activities of a set of users in a computer-mediated problem-solving session.

• Began to develop an analyst’s tool for constructing a set of effective coordinating representations for a given computer-mediated cooperative activity.

Accomplishments

• Completed initial experiment to confirm utility of coordinating representations.

• Also analyzed data to develop model of how participants maintain a consistent view of the world.– Coordination of Talk; Coordination of Action. To appear in special

issue of HCI Journal on “Talking about Things”

– Conference paper appeared in Proceedings of Twenty-Fourth Annual Meeting of the Cognitive Science Society, 2001

Future Directions/Payoffs

• Continue development of analyst’s tool– Model interaction among participants

• Apply methodology and use tool to develop coordinating representations for another domain.

• The methodology and the analyst’s tool we develop will have wide application and is likely to significantly impact the development of the next generation of groupware applications.

Outline

• Testbed (VesselWorld)

• Joint Sensemaking

• Secondary Structures

• Examples of Coodinating Representations

• Experiment

• Methodology & Tools

Joint Sensemaking

• Report• Review • Repair• Confirm

Review1. Crane1: center is clean – nothing in

the center 4 quads

2. Crane2: ok, we forgot the west, I didn’t get down that far

3. Tug1: 600-400 square is clean, I don’t have to worry about tit?

4. Crane2: correct, that’s where I started

5. Crane2 about 100 N of sb

6. Tug1: okay cool, grabbing the sb

Secondary Structures

• Created by participants to simplify talk at critical junctures of cooperative activity

• Indicate problem areas where the introduction of a coordinating representation could be productive

Close Coordination

1. Co-ordination of talk

a. Adjacency pairs to propose and confirm next action

b. Expectations that adjacency pairs will occur for each of the actions in an extended sequence of tightly coupled cooperative behaviors

2. Coordination of action

1. Crane1: sub lift

2. Crane2: LL

3. Crane2: k

4. Crane2: sub load

5. Tug1: the next XL needs nothing

6. Crane1: k

7. Crane2: ok, then XLD right?

8. Crane2: sub Lift

9. Tug1: yep

10. Crane1: k

11. Crane2: sub load

12. Crane1: k

13. Crane2: sub sep

14. Crane1: sep

Coordinating Representations (Two Examples)

• Reduce work in timing closely couple joint actions

• Maintain consistent information about shared domain objects

Timing of Joint Actions

Object List

Experiment

• Three groups with Coordinating Representations; Three groups without Coordinating Representations

• Training + 10 hours of problem-solving

– 49% improvement in clock time – 38% reduction in the number of events generated– 57% reduction in the amount of electronic chatting– 61% reduction in total errors

Tools & Methodology

• VCR-Tool– Replay online activities of participants in a

computer-mediated cooperative activity

• Analyst’s Tool– Segment and sort the conversational threads.– Profile and model interactions among

participants

Analysis tool, part 1

• Analyst can to tag and classify utterances– Mark up text with standard tags (errors, conventions, 2ndry structures) for

profiling later

– Allows note-taking on a per-utterance basis

SOFT ERROR

UTTERANCE

37 crane1: didn't work again?

38 crane2: spilled

39 crane1: drat

40 tug1: sX is killed?

41 crane1: killed sX at 500 275

42 tug1: yay

43 crane2: 500, 275?

44 crane1: yes

45 crane2: c1, grab that, and I'll get sX off barge?

46 crane1: oh, ok

47 crane1: grabbed it already

Annotated Chat

CONVENTION

CONVENTION

HARD ERROR

NOTES

failed b/c equip - see 33

1st use of "killed"

sw/dredge; see 25

(it=sX on barge)

TAG

Analysis tool, part 2

• Analyst can cluster utterances and mark conversation threads• Emergent categorization of utterances for flexibility in analysis

• Threaded view reveals conversational structure and allows tracking of errors due to causes like participants' mismatched understanding of conversational state

Chat Buckets

36 crane1: Dropping off

waste; I'll grab the sX at

500 275

40 tug1: sX is killed?

41 crane1: killed sX at

500 275

45 crane2: c1, grab that,

and I'll get sX off barge?

47 crane1: grabbed it

already

45 crane2: c1, grab

that, and I'll get sX off

barge?

47 crane1: grabbed it

already

secondary structures?

object references...

commitment errors sX at 500 275

(fail to get sense across?)

40 tug1: sX is killed?

41 crane1: killed sX at

500 275

42 tug1: yay

43 crane2: 500, 275?

44 crane1: yes

5 utterances 2 utterances 5 utterances

Average thread length: 4.2 lines

Average duration: 0:07:26.35

+ sX at 500 275 [6 lines: 36, 40-44]

36 crane1: Dropping off waste; I'll grab the sX at 500 275

40 tug1: sX is killed?

41 crane1: killed sX at 500 275

42 tug1: yay

+ crane1 verifies (subconversation) [2 lines: 43, 44]

+ Waste at 435, 400 [3 lines: 37-39]

37 crane1: didn't work again?

38 crane2: spilled

39 crane1: drat

- waste at 335, 235 [11 lines: 45-47, 49-51, 57, 67-70]

- search for lost sw/X [48, 52-56, 58-66]

Threaded Chat

Number of threads: 17

Max thread length: 6

Selected Publications

• Alterman, R., Feinman, A., Introne, J., and Landsman, S., Coordination of Talk; Coordination of Action. To appear in special issue of HCI Journal on “Talking about Things”.

• Alterman, R., Feinman, A., Introne, J., and Landsman, S., Coordinating Representations in Computer-Mediated Joint Activities. Proceedings of Twenty-Fourth Annual Meeting of the Cognitive Science Society, 2001.

• Alterman, R. and Garland, A. Convention in Joint Activity. Cognitive Science 25:4, 611-657, 2001.

• Garland, A. and Alterman, R. Learning Procedural Knowledge to Better Coordinate. International Joint Conference on Artificial Intelligence, 2001.