From Exploration to Construction - How to Support the Complex Dynamics of Information Seeking

From Exploration to ConstructionHow to Support the Complex Dynamics of Information Seeking

Hugo C. Huurdeman PhD candidate University of Amsterdam

webarchiving.nl

Introduction: a paradox

• Models of information seeking describe fundamentally different macro-level stages in complex tasks

+ uncertainty -

uncertainty optimism confusion clarity confidence (dis)satisfaction doubt direction

FormulationInitiation Selection Exploration Collection Presentation

Introduction: a paradox

• Models of information seeking describe fundamentally different macro-level stages in complex tasks

+ uncertainty -



Search• However, current search systems usually provide a streamlined and static feature set

• To what extent do current search approaches support complex tasks?

Multistage Information Seeking Modelsmacro perspective1

Read more:

Huurdeman & Kamps (2014), From Multistage Information Seeking Models to Multistage Search Systems. Proc. IIiX 2014. http://dx.doi.org/10.1145/2637002.2637020

Huurdeman & Kamps (2015). Supporting the Process - Adapting Search Systems to Search Stages. Proc. ECIL 2015. http://dx.doi.org/10.1007/978-3-319-28197-1_40

http://dx.doi.org/10.1145/2637002.2637020

http://dx.doi.org/10.1007/978-3-319-28197-1_40

1.1 Information Seeking Models

• Information seeking modeled in a multitude of ways: • as behavioral patterns (Ellis) • as nonlinear activities (Foster) • as problem-solving (Wilson) • as temporal stages (Kuhlthau), ..

• Our main focus: • temporally based IS models

• Kuhlthau [1991] (Vakkari [2001]) • cognitively complex (work) tasks

• involving learning & constructioninformation

search

informationseeking

information behavior

[Wilson99]

1.2 Kuhlthau: Information Search Process [1991]

+ uncertainty -

feelings

thoughts

actions

vague focused

seeking relevant information (exploring)

seeking pertinent information (documenting)



1.2 Vakkari’s adaptation (in [Vakkari01])

Prefocus Focus formulation Postfocus

seeking general background information

seeking specific information

faceted backgr.information

relevance hard to judge relevance easier to judge

decrease of number of broader terms

informationsought

relevance

search terms increase of number of search terms, synonyms, narrower terms

1.3 Implications for design of search systems

• Observation: good general understanding of macro level inf. seeking stages, but hard to translate to concrete micro level system design choices.

macro

micro system design

inf. seeking stages

Search user interfaces supporting seekingmicro perspective2

3.1 Search user interfaces supporting seeking

• Search User Interface (SUI) design: • no straightforward task to design a UI with a high usability [Shneiderm05]

• A (limited) number of available frameworks, guidelines and design pattern libraries for SUIs • e.g. M. Wilson’s framework of SUI features [Wilson11]

Search

Input

Control

Informational

Personalizable

2.1 SUI approaches: traditional search

• Streamlined interfaces

• Focus on • query formulation • result list inspection

• Advantages: [Hearst09]

• lower cognitive load • more accessible • more understandable

Highly optimized for lookup tasks, less for open-ended queries

• Research-based tasks

• WebART project • new media researchers • action research setting, structured literature review

• Search systems allow for answering new research questions, but also have limitations • lack of transparency • lack of process support

2.2 When traditional search does not work well

researcher research activities

corpus creation

analysis

dissemination

webarchiving.nl

2.3 SUI approaches: exploratory search

• Supporting open-ended inf. seeking

• Support learning and investigation activities for complex information problems [Marchionini06]

• Many potential exploratory SUI features [White09], e.g. • rapid query refinement, facets (input, control) • leveraging context, visualizations (informational) • histories/workspaces/task management (personalizable)

2.4 SUI approaches: sensemaking & analytics

• Support analysis & synthesis in interface

• Potential functions facilitating notetaking, hypothesis formulation & collaborative search [Hearst09] • some overlap with exploratory search

2.5 Implications for search stage support (2/2)

• Observation: good understanding of search system features at the micro level, but fragmented understanding of how they can support information seeking stages at the macro level

macro

microsearchsystem features

inf. seeking stages

Reconciling macro and micro views

• Would it be possible to reconcile the macro level and micro level views?

macro

microsearchsystem features

inf. seeking stages

“The Utility of SUI Features”Our study: investigating the utility of various SUI features at different macro-level stages

From: Huurdeman, Wilson & Kamps (2016), Active and Passive Utility of Search Interface Features in Different Information Seeking Task Stages.

Proc. ACM CHIIR 2016. http://dx.doi.org/10.1145/2854946.2854957

http://dx.doi.org/10.1145/2854946.2854957

3. Setup

• User study (26 participants; 24 analyzed) • Undergrads Univ. of Nottingham (6 F, 12 M, 18-25y)

• Experimental SUI resembling common Search Engine

• Within-participants• Task stage independent variable

• Task design: explicit multistage approach

3. Setup: Multistage Task Design

sim. work task: writing essay

subtask subtask subtaskprepare list of

3 topicschoose topic;

formulate specificquestion

find and select additional

pages to cite

15 minutes 15 minutes 15 minutes

initiationtopic selectionexploration

focus formulationcollecting

presenting

3. Setup: Multistage Task Design

sim. work task: writing essay

subtask subtask subtaskprepare list of

3 topicschoose topic;

formulate specificquestion

find and select additional

pages to cite

15 minutes 15 minutes 15 minutes

initiationtopic selectionexploration

focus formulationcollecting

presenting

General Assigned Topics (b/o discussions teaching staff) • Autonomous Vehicles (AV) • Virtual Reality (VR)

3. Setup: Protocol

Training task

Pre-Questionnaire

Topic Assignment

Introduction system

Task

Post-task Questionnaire

3x

Post-experiment questionnaire

Debriefing interview

• Experimental system: SearchAssist• Results, Query Corrections, Query

Suggestions: Bing Web API• Category Filters: DMOZ

• Categorization and analysis: • Max Wilson’s framework of SUI features

[Wilson11]

Control

Control

Input

Control

Input

Informational

Control

Input

PersonalizableInformational

3. Setup: Logging

eyetribe.com

http://eyetribe.com

3. Setup: Data / Task details

• AV & VR topics invoked comparable behaviours: • analysed as one topic set

• Total duration main tasks• Total task time: 32:56

• 36.8% SUI, 33% Task screen, 30.2% Webpages

Stage 1: 11:32 Stage 2: 8:24 Stage 3: 12:59

Findings: Active Behaviourbehaviour directly and indirectly derivable from logs4

4.1 Active Behaviour: Clicks

0

4

8Sig. clicks on interface

features over time

Stage 1 Stage 2 Stage 3


0

4


features over time

Category filters ⬇



0

4


features over time

Category filters ⬇Tag Cloud ⬇



0

4


features over time

Category filters ⬇Tag Cloud ⬇Search button ⬇



0

4


features over time

Category filters ⬇Tag Cloud ⬇Search button ⬇Saved Results ⬆


4.2 Active Behaviour: Queries

•Mean number of queries** (unique): • Stage 1: 9.5 (8.1) ➡ Stage 2: 5.5 (5.1) ➡ Stage 3: 5.9 (5.3)

0

2,5

5

7,5

10

Stage 1 Stage 2 Stage 3Search BoxQuery SuggestionsRecent Queries

4.3 Active Behaviour: Query words

•Mean number of query words**:

“virtual reality” (P.02) “impact of virtual reality on society art and culture“

“autonomous vehicles” (P.06) “autonomous vehicles costsinsurance industry”

01,25

2,53,75

5

Stage 1 Stage 2 Stage 3Mean Number of Query words

4.4 Active Behaviour: Visited pages

• Visited pages (unique)**:

• Stage 1: 8.0 (7.3)

• Stage 2: 6.4 (5.9)

• dwell time highest

• Stage 3: 14.2 (10.8)

• Mean rank visited pages

• from 3.1 to 6.4

0

4

8

12

16


Results ListSaved Results

4.5 Active Behaviour: Wrapup

• Clicks:• decreasing for Query Box (input), Category Filters & Tag

Cloud (control)

• increasing for Saved Results (personalizable)

• Queries:

• decreasing over time, but more complex

• Popularity of certain features and impopularity of others:

•Some features used in passive instead of active ways?

Findings: Passive Behaviourbehaviour not typically caught in interaction logs5

eyetribe.com

Passive behaviour: mouse hovers

• Mouse movements:• movements to reach a feature, also to aid processing contents [Rodden08]

• Focus here on mouse movements not leading to click • Tendencies mostly overlap with active interaction measure

0%

25%

50%

75%

100%

1 2 3


Category filters** ⬇



0%

25%

50%

75%

100%

1 2 3


Category filters** ⬇Tag Cloud* ⬇



0%

25%

50%

75%

100%

1 2 3


Category filters** ⬇Tag Cloud* ⬇Query Box** ⬇



0%

25%

50%

75%

100%

1 2 3


Category filters** ⬇Tag Cloud* ⬇Query Box** ⬇Results List* ⤻



0%

25%

50%

75%

100%

1 2 3

5.2 Passive Behaviour: eye fixations

Stage 1 (exploration) Stage 2 (focus formulation)Stage 3 (postfocus, collection)

• Overview of eye movement via heatmaps:

Passive behaviour: eye tracking

eye tracking fixations 0

25

50

75

100

1 2 3

• Further insights via eye tracking fixation counts • fixations > 80 ms, similar to e.g. [Buscher08]



25

50

75

100

1 2 3


Query Suggestions* ⬇



25

50

75

100

1 2 3

Tag Cloud* ⬇





25

50

75

100

1 2 3






25

50

75

100

1 2 3


Query Box** ⬇





25

50

75

100

1 2 3


Query Box** ⬇Results List* ⤻



3.4 Passive Behaviour: Active vs. Passive

0%

2%

4%

6%

8%


Subtle differences between passive and active use:


0%

2%

4%

6%

8%


Tag Cloud [5.8% fixations ⬌ 3.1% clicks]



0%

2%

4%

6%

8%


Query Suggestions [3.6% fix. ⬌ 1.9% clicks]Tag Cloud [5.8% fixations ⬌ 3.1% clicks]



0%

2%

4%

6%

8%



Recent Queries [3% fix. ⬌ 2% clicks]



0%

2%

4%

6%

8%



Recent Queries [3% fix. ⬌ 2% clicks]


Opposite for Category Filters [5% ⬌ 3.8%]

5.4 Passive Behaviour: Wrapup

•Fixations & mouse moves • validating active behaviour

• subtle differences active and passive use

• Could subjective ratings and qualitative feedback provide more insights?

Findings: Perceived Feature Utilityperceived usefulness (post-stage & experiment)6

6.2 Perceived Usefulness: post-experiment

• Post-experiment questionnaire: • In which stage or stages were SUI features most useful?

• Pronounced differences• significant differences for all features

0%

25%

50%

75%

100%

Query Box / Results List

Category Filters

Tag Cloud

Query Suggestions

Recent Queries

Saved Results

6.3 Perceived Usefulness: Category Filters

• “good at the start (…) but later I wanted something more specific” (P.11)

• common remarks in 2nd and 3rd stage:

• “… could be more specific in its categories”

• “…hard to find the category I want” (P.27)

6.3 Perceived Usefulness: Tag Cloud

• at the start:

• “…aids exploring the topic” (P.06);

• “came up with words that I hadn’t thought of”

• later stages:

• “doesn’t help to narrow the search much” (P.18)

• “in the end seemed to be too general” (P.07)

6.3 Perceived Usefulness: Tag Cloud

• at the start:

• “…aids exploring the topic” (P.06);

• “came up with words that I hadn’t thought of”

• later stages:

• “doesn’t help to narrow the search much” (P.18)

• “in the end seemed to be too general” (P.07)

• Post-experiment comments:• “…was good at the beginning, because when you

are not exactly sure what you are looking for, it can give inspiration” (P.12)

• “… nice to look at what other kinds of ideas [exist] that maybe you didn’t think of. Then one word may spark your interest” (P.15)

6.3 Perceived Utility: Query Suggestions

• “…was good at the start but as soon as I got more specific into my topic, that went down” (P.11)

• “clicked [it] .. a couple of times .. it gave me sort of serendipitous results, which are useful” (P.24)

6.3 Perceived Utility: Recent Queries

• Naturally: “…most useful in the end because I had more searches from before” (P.26)

• “The previous searches became more useful ‘as I made them’ because they were there and I could see what I searched before. I was sucking myself in and could work by looking at those.” (P.23)

• May aid searchers in their information journey..

6.3 Perceived Utility: Saved Results

• “most useful in the end” (P.12)

• “At the start [I was] saving a lot of general things about different topics. Later on I went back to the saved ones for the topic I chose and then sort of went on from that and see what else I should search” (P.26)

• “I just felt I was organizing my research a little bit” (P.18)

• It “helps me to lay out the plans of my research”.

Conclusiontowards more dynamic support7

0%!

20%!

40%!

60%!

80%!

100%!

Stage 1! Stage 2! Stage 3!

Perc

enta

ge o

f par

ticip

ants!

input / informational!

control!

personalisable!

0%!

20%!

40%!

60%!

80%!

100%!

Stage 1! Stage 2! Stage 3!

Perc

enta

ge o

f par

ticip

ants!

input / informational!

control!

personalisable!

Conclusion: Findings Summary

• Informational features highly useful in most stages

• Decreasing use of input features

• Control features decreasingly useful

• likely caused by a user’s evolving domain knowledge

• Personalizable features increasingly useful

• ‘growing’ with a user’s understanding, task management support

SUI features perceived as most useful, per stage

7. Conclusion: theoretical roundup

complex information seeking task

pre-focus stage: • vague understanding • limited domain knowledge • trouble expressing

information need • large amount of new

information

• explaining prominent role of control features

• explore information• filter result set

using [Kuhlthau04,Vakkari&Hakkala00,Vakkari01]





information



focus formulation stage: • more directed search • better understanding • seeking more relevant

information, using differentiated criteria

• control features become less essential

• “not specific enough”• personalizable feat’s more

important: may “grow” with emerging understanding






information



focus formulation stage: • more directed search • better understanding • seeking more relevant

information, using differentiated criteria

• control features become less essential

• “not specific enough”• personalizable feat’s more

important: may “grow” with emerging understanding

postfocus stage • specific searches • re-checks additional

information • precise expression • low uniqueness, high

redundancy of info

• long, precise, queries• further decline of control

features • frequent use of

personalizable features• “see what else to search”


7. Conclusion: Future Work

•Our study: essay writing simulated work task • Extension to other types of complex tasks, user

populations

•Further research into task-aware search systems• additional features may be useful at different stages

• e.g. user hints, assistance

• improvement of current features

Towards “stage-aware” Systems

prefocus

focus formulation

postfocus

searchersearchsystem

searchinterface

search stage

stage-independent functionalityranking stage-dependent

functionalitymanual or automatic

detection

• INEX/CLEF Interactive Social Book Search Lab

• http://social-book-search.humanities.uva.nl/#/interactive

• Gaede, Hall, Huurdeman, Kamps, Koolen, Skov, Toms, Walsh (2015)

• Aim: support different stages in the search process: browse, search & review

• Joint study across universities, 192 participants

Multistage interface: search

Multistage interface: browse

Multistage interface: review

Example: multistage interface

http://social-book-search.humanities.uva.nl/#/interactive

7. Conclusion: towards dynamic SUIs

•Most Web search systems converged over static and familiar designs • trialled features often struggled to provide value for

searchers • perhaps impeding search [Diriye10] if introduced in simple

tasks, or at the wrong moment

•Our work provides insights into when SUI features are useful during search episodes

• potential responsive and adaptive SUIs for complex tasks

References (1/2)

[Ahlberg&Shneiderman94] C. Ahlberg and B. Shneiderman. Visual information seeking: Tight coupling of dynamic query filters with starfield displays. In CHI, pages 313–317. ACM, 1994. [Buscher08] G. Buscher, A. Dengel, and L. van Elst. Eye movements as implicit relevance feedback. In CHI’08 extended abstracts on Human factors in computing systems, pages 2991–2996. ACM, 2008. [Diriye10] A. Diriye, A. Blandford, and A. Tombros. When is system support effective? In Proc. IIiX, pages 55–64. ACM, 2010. [Diriye13] A. Diriye, A. Blandford, A. Tombros, and P. Vakkari. The role of search interface features during information seeking. In TPDL, volume 8092 of LNCS, pages 235–240. Springer, 2013.[Donato10] D. Donato, F. Bonchi, T. Chi, and Y. Maarek. Do You Want to Take Notes?: Identifying Research Missions in Yahoo! Search Pad. In Proc. WWW’10, pages 321–330, 2010. ACM. [GaedeEtAl15] Maria Gäde, Mark Hall, Hugo Huurdeman, Jaap Kamps, Marijn Koolen, Mette Skov, Elaine Toms, and David Walsh. Overview of the SBS 2015 interactive track. In CLEF’15 Working Notes. CEUR-WS, 2015. [Hearst09] M. A. Hearst. Search user interfaces. Cambridge University Press, 2009. [Hearst13] M. A. Hearst and D. Degler. Sewing the seams of sensemaking: A practical interface for tagging and organizing saved search results. In HCIR. ACM, 2013. [Huurdeman&Kamps15] Hugo C. Huurdeman and Jaap Kamps (2015). Supporting the Process: Adapting Search Systems to Search Stages. In: S. Kurbanoğlu, S. Špiranec, J. Boustany, E. Grassian, D. Mizrachi, & L. Roy (Eds.), Information Literacy: Moving towards sustainability, Communication in Computer and Information Science series (Vol. 552, pp. 394-404).[Huurdeman&Kamps14] H. C. Huurdeman and J. Kamps. From Multistage Information-seeking Models to Multistage Search Systems. In Proc. IIiX’14, pages 145–154, 2014. ACM [Kuhlthau91] C. C. Kuhlthau. Inside the search process: Information seeking from the user’s perspective. JASIS, 42:361–371, 1991. [Kuhlthau04] C. C. Kuhlthau. Seeking Meaning: A Process Approach to Library and Information Services. Libraries Unlimited, 2004. [Kules12] B. Kules and R. Capra. Influence of training and stage of search on gaze behavior in a library catalog faceted search interface. JASIST, 63:114–138, 2012. [LiuBelkin15] J. Liu and N. J. Belkin. Personalizing information retrieval for multi-session tasks. JASIST, 66(1):58–81, Jan. 2015.[Marchionini06] G. Marchionini. Exploratory search: from finding to understanding. CACM, 49(4):41–46, 2006. [Niu14] X. Niu and D. Kelly. The use of query suggestions during information search. IPM, 50:218–234, 2014. [Proulx06] P. Proulx, S. Tandon, A. Bodnar, D. Schroh, W. Wright, D. Schroh, R. Harper, and W. Wright. Avian Flu Case Study with nSpace and GeoTime. In Proceedings of the IEEE Symposium on Visual Analytics Science and Technology (VAST'06). IEEE, 2006.

http://link.springer.com/chapter/10.1007/978-3-319-28197-1_40

References (2/2)

[Toms11] E. G. Toms. Task-based information searching and retrieval. In Interactive Information Seeking, Behaviour and Retrieval. Facet, 2011.[Rodden08] K. Rodden, X. Fu, A. Aula, and I. Spiro. Eye-mouse coordination patterns on web search results pages. In CHI’08 Extended Abstracts, pages 2997–3002. ACM, 2008. [Shneiderman05] B. Shneiderman and C. Pleasant. Designing the user interface: strategies for effective human-computer interaction. Pearson Education, 2005. [Tunkelang09] D. Tunkelang. Faceted search. Synthesis lectures on information concepts, retrieval, and services, 1(1):1–80, 2009. [Vakkari01] P. Vakkari. A theory of the task-based information retrieval process: a summary and generalisation of a longitudinal study. Journal of Documentation, 57:44–60, 2001. [White05] R. W. White, I. Ruthven, and J. M. Jose. A study of factors affecting the utility of implicit relevance feedback. In SIGIR, pages 35–42. ACM, 2005. [White09] R. W. White and R. A. Roth. Exploratory search: Beyond the query-response paradigm. Synthesis Lectures on Information Concepts, Retrieval, and Services, 1:1–98, 2009. [Wilson&schraefel08] M. L. Wilson and m. c. schraefel. A longitudinal study of exploratory and keyword search. In In Proc. JCDL’08, pages 52–56. ACM, 2008. [Wilson99] T. D. Wilson. Models in information behaviour research. Journal of Documentation, 55:249–270, 1999. [Wilson11] M. L. Wilson. Interfaces for information retrieval. In I. Ruthven and D. Kelly, editors. Interactive Information Seeking, Behaviour and Retrieval. Facet, 2011.

From Exploration to ConstructionHow to Support the Complex Dynamics of Information Seeking

Hugo C. Huurdeman University of Amsterdam @TimelessFuture

webarchiving.nl

From Exploration to Construction - How to Support the Complex Dynamics of Information Seeking

Education

Transcript of From Exploration to Construction - How to Support the Complex Dynamics of Information Seeking