A Model Of Collaborative Search

A model of collaborative search

Gene Golovchinskyand

Jeremy Pickens

FX Palo Alto Laboratory, Inc.

NIST October 22, 2009Golovchinsky and Pickens

A Model of Collaborative Search

The elephant in the room• Many studies have show that info

seeking is not a solitary pursuit– Allen (1977)– O’Day and Jeffries (1993)– Twidale, Nichols and Paice (1997)– Talja (2002)– Hansen & Järvelin (2005)– Hyldegård (2006)– Hertzum (2008)– Morris (2008)– Evans and Chi (2008)– Reddy & Jansen (2008)

• Many tasks have been shown to benefit from collaboration

– Engineers collaborating on design– Reference librarians working with

patrons– Medical / drug research teams– Student project teams– Patent Search (multiple legal

clerks working on same case)– Patient care teams (multiple

doctors with differing expertise caring for the same individual)

– Families making important decisions (e.g. real estate, travel)

– Friends or colleagues planning activities together

Yet most systems for info seeking are designed for only one person



What about Social Search?

• Social feedback– Observe patterns of behavior or opinion– Identify useful documents– Infer similar information needs– Retrieve useful documents

• Social answering – Explicit use of social networks to help answer questions

• Answer Garden (Ackerman and McDonald, 1996)• Aardvark (www.vark.com)• Ad hoc: uses of Facebook, Twitter, other comm. channels to help

find information (Evans and Chi, 2008)

Taxonomy proposed by Chi (2009)



But…• Similarity of query terms does not imply similarity of info need

– Inference is imperfect; users may have differing needs– Indirect and noisy feedback

• Finding same documents as others may not be useful in some cases– Known-item search as the consumption of knowledge vs.– Exploratory search as the creation of knowledge

• Interaction with system – Is still often solitary– Is limited to communication with others

Social search is not truly collaborative



Collaborative Search vs. Social Search

• Social (feedback) search:– Infers similar information need– Leverages wisdom of crowds for known-item

search

• Collaborative search:– Assumes shared information need– Combines multiple perspectives to improve

search results



A history of modeling

• Single-user information seeking– Iterative process (The usual suspects)

• Collaboration on teams– Group cohesion (Hertzum, 2007)– Classes of collaborative activity (Järvelin and Hansen, 2005)

• Social answering– Patterns of information flow among people (Evans and Chi,

2008)

What’s missing?– Interaction, coordination, mechanics– Foundation for principled design for collaboration



A model for collaborative search

• User behavior– Intent

• Implicit vs. explicit

• System behavior– Depth of mediation

• Communication vs. UI vs. algorithmic mediation

– Synchronization• Asymmetric vs. Symmetric



Intent

• Implicit– User may be aware that others’ data are used to inform search– System infers similarity of information need– System recommends documents based on inference of similarity– Good for finding what others have already found, thus

• May not effective for exploratory search

• Explicit– Users explicitly declare shared information need– System combines contributions from collaborators to find new

information– Good for exploratory search



Depth of mediation• Communication

– People communicate about search tasks, about search results– Neither the interface nor the algorithms know that multiple people are involved

• UI– Each person uses system independently– Retrieval system is unaware of multiple people

• Algorithmic– Each person’s contributions are tracked separately by the retrieval system– Contributions may be combined to produce desired retrieval effects

Aspects are cumulative– UI may also include communication– Algorithmic mediation may also include UI and communication



Data synchronization

• Symmetric influence– Data generated by one person are available to all collaborators

for the same search task• SearchTogether• Cerchiamo

• Asymmetric influence– Some people do not see contributions of others

• Recommendations

Synchronization is not synchronicity– No implication when people search– No requirement of WUSIWIS– Describes availability of other peoples’ data with respect to an

information need



Examples

• Ariadne (Twidale, Nichols, Paice, 1997)– Explicit intent, communication-only mediation, no data

synchronization

• Recommendation systems (e.g., Linden et al, 2003)– Implicit intent, algorithmic mediation, asymmetric data

synchronization

• SearchTogether (Morris,& Horvitz, 2007)– Explicit intent, UI mediation, symmetric data synchronization

• Cerchiamo (Pickens, et al., 2008)– Explicit intent, algorithmic mediation, symmetric data

synchronization



Example: Ariadne

• Explicit intent– People are working toward a shared information need

• Communication-level mediation– Chat about search– System does not mediate

• No data synchronization– Each person is responsible for individual search

Twidale, Nichols and Paice (1997)



Example: Recommendation Systems and Social Search

• Implicit intent– Each person is pursuing an individual agenda

• Algorithmic mediation – System keeps track of each user’s actions– System aggregates information to generate recommendations

• Asymmetric data synchronization– Only “future” searchers benefit from recommendations

• Many implementations– Linden et al (2003)– Smyth et al (2004)



Example: SearchTogether


• UI-level mediation – System keeps track of each user’s documents, judgments– System does not differentiate among searchers with respect to

retrieval

• Symmetric data synchronization– Judgments of relevance are available to all collaborators

Morris and Horvitz (2007)



Example: Cerchiamo


• Algorithmic mediation – System keeps track of each user’s documents, judgments– System integrates inputs from both searchers to produce search

results, term suggestions

• Symmetric data synchronization– Terms and documents are a shared among all searchers– Different roles are assigned different interfaces

Pickens, Golovchinsky, Shah, Qvarfordt, and Back (2008)



Cerchiamo, some details…

• Two users in different roles– Prospector opens up new paths for exploration– Miner digs through promising results

• Bi-directional influence– Prospector issues queries– System shows shots for miner to judge– Miner makes relevance judgments– System suggests search terms to prospector

• System coordinates– Different inputs from each person– Different outputs to each person



AlgorithmicMediation Engine

InputCoordinator

User 1

OutputCoordinator

User 2

UserInterface

Regulator(Roles)

CollaborativeSearch



What’s the purpose of this complexity?

• Collaboration seems to help on difficult search tasks– TRECVid 2007 ad hoc task

• Cerchiamo vs. post-hoc pooling of results from two “prospector” users– 24 searchers– Paired standalone vs. paired collaborative

• Metric– Viewed precision = Fraction of viewed documents that is relevant

• Results– When finding documents is easy (plentiful topics)

• No advantage to collaboration– When finding documents is harder (sparse topics)

• Collaboration is more efficient (more relevant shots found despite examining fewer documents)



Efficiency: Large advantage for sparse topics

Average # examined @15 minutes

• Plentiful topics– Collaborative: 2,352

– Post hoc Merged: 2,168

• Sparse topics– Collaborative: 2,877

– Post hoc Merged: 3,787



Uniqueness

System A

System B



All Other Systems

Uniqueness

System ASystem B



Effectiveness: UniquenessUniqueness

-5

0

5

10

15

20

25

3.75 min 7.5 min 11.25 min 15 min

Time Elapsed (min)

% D

iffe

ren

ce i

n U

niq

uen

ess

Plentiful

Sparse



Summary of results

• For difficult information needs– Collaboration improves efficiency– Collaboration improves effectiveness

• Preliminary evidence– Need to perform more evaluations– Need to evaluate on text-only documents

• Conception of the task affects design– Social search improves known-item search– Collaboration improves exploratory search



Comments on Evaluation

• Need to invent new ways to assess performance– Dividing by number of searchers not always appropriate– Mythical “man month”

• Need outcome-centered measures– Interactive task– MAP not appropriate– Set-based metrics more appropriate– “Viewed” recall and precision capture human judgment, not just

system performance

• How to compare iterative processes?– Document-centric rather than query-centric metrics?– Need to account for relevant-but-redundant documents



References• Ackerman, M. S. and McDonald, D. W. (1996) Answer Garden 2: merging organizational memory with collaborative help. In Proceedings

of the 1996 ACM Conference on Computer Supported Cooperative Work (Boston, Massachusetts, United States, November 16 - 20, 1996). M. S. Ackerman, Ed. CSCW '96. ACM, New York, NY, 97-105.

• Allen, T. (1977) Managing the flow of technology: Technology transfer and the dissemination of technological information within the R&D organization. MIT Press

• Chi, Ed H. (2009) Information Seeking Can Be Social, Computer, vol. 42, no. 3, pp. 42-46, Mar. 2009 • Evans, B.; Chi, E. H. (2008) Towards a Model of Understanding Social Search. In Proc. of Computer-Supported Cooperative Work

(CSCW). ACM Press. San Diego, CA • Hansen, P. and Järvelin, K. (2005) Collaborative Information Retrieval in an information-intensive domain. Information

Processing&Management 41. pp. 1101–1119.• Hertzum (2008) Collaborative information seeking: The combined activity of information seeking and collaborative grounding. Information

Processing&Management 44 pp. 957–962.• Hyldegård, J. (2006) Collaborative information behaviour––exploring Kuhlthau’s Information Search Process model in a group-based

educational setting. Information Processing&Management 42. pp. 276–298• Linden, G., Smith, B. and York, J. (2003) Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Computing,

7 (1). 76-80. • Morris, M. R. and Horvitz, E. (2007) SearchTogether: an interface for collaborative web search. In Proceedings of UIST '07. ACM, New

York, NY, 3-12.• O'Day, V. & Jeffries, R. (1993) Information artisans: patterns of result sharing by information searchers. In: Proceedings of the ACM

Conference on Organizational Computing Systems, 98-107. ACM Press. • Pickens, J., Golovchinsky, G., Shah, C., Qvarfordt, P., and Back, M. (2008) Algorithmic Mediation for Collaborative Exploratory Search. In

Proceedings of SIGIR 2008, July 22-25. ACM Press. • Reddy & Jansen (2008) Learning about Potential Users of Collaborative Information Retrieval Systems. In Proc. JCDL Collab IR

Workshop• Smyth, B., Balfe, E., Freyne, J., Briggs, P., Coyle, M. and Boydell, O. (2004) Exploiting Query Repetition and Regularity in an Adaptive

Community-Based Web Search Engine. UMUAI 14, 5. 383-423.• Talja, S. (2002) Information sharing in academic communities: types and levels of collaboration in information seeking and use. New

Review of Information Behavior Research, 3(1), 143-159.• Twidale, M., Nichols, D. & Paice, C. (1997) Browsing is a collaborative process. Information Processing and Management, 33(6), 761-

783.

Questions?

CFP: 2nd International Workshop on Collaborative Information Seeking

Held in conjunction with CSCW 2010 in Savannah, GA

February 2010

http://www.fxpal.com/cscw2010cis

A Model Of Collaborative Search

Technology

Transcript of A Model Of Collaborative Search