The HCI Perspective on IR (DIR2016 Keynote)

Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw

The HCI Perspective on IRMax L. Wilson

DIR2016 Keynote

http://cs.nott.ac.uk/~pszmw


My Talk

• A brief view of how our fields differ

• A brief history of IR in HCI

• What CHI wants from IR

• The Challenges of giving CHI what it wants from IR

• Examples of getting there



I admire the fundamentals of batch-processing IR

Queries Right Answers



Interesting Differences between HCI & IR

• IR is build on top of prior work - HCI is explore new ideas (not necessarily on top)

• IR is identify a rule to improve system performance - HCI is to extract a principle of Human Behaviour - or User Interface Design rule

• IR Research is temporally close to IR implementation - HCI is 10-15y gap

All the papers in between wrestle with these tensions



Microsoft Surface Dialimage: mashable.com

Fitzmaurice, Ishii, and Buxton. "Bricks: laying the foundations for graspable user interfaces."

CHI’95. ACM


http://mashable.com


A History of IR in HCI(not comprehensive)

(or rigorous)(actually: IR papers in CHI)

(based on the ACM DL’s ranking)

http://bit.ly/IR-HCI-Papers



IR papers at CHI• “Information Retrieval” in CHI in ACM DL

- 5000+ papers sorted* by ‘relevance’ - of 450,000+ IR papers in whole DL

• Export gives me most relevant 1018 - 373 Full Papers, 65 Short Papers, 580 Extended Abstracts - of Full Papers, only first 25 tag themselves with IR

• Looked at Peer-Reviewed papers in 200 most relevant (N=84) - Rejected a further 12 as not really being about IR

*Order keeps changing when press ‘next page’ :-/




author 'tle yearJoW.TombaughandSco1A.McEwen

ComparisonofTwoInforma>onRetrievalMethodsonVideotex:Tree-structureVersusAlphabe>calDirectory 1982

G.FischerandH.Nieper-Lemke Helgon:ExtendingtheRetrievalbyReformula>onParadigm 1989

StuartK.CardandGeorgeG.RobertsonandJockD.Mackinlay TheInforma>onVisualizer,anInforma>onWorkspace 1991

MichaelMillsandJonathanCohenandYinYinWong AMagnifierToolforVideoData 1992

H.UlrichHoppeandFranzSchiele TowardsTaskModelsforEmbeddedInforma>onRetrieval 1992

BenShneidermanandChristopherWilliamsonandChristopherAhlberg

DynamicQueries:DatabaseSearchingbyDirectManipula>on 1992

DanielM.RussellandMarkJ.StefikandPeterPirolliandStuartK.Card TheCostStructureofSensemaking 1993

VickiL.O'DayandRobinJeffries OrienteeringinanInforma>onLandscape:HowInforma>onSeekersGetfromHeretoThere 1993

GeneGolovchinskyandMarkChignell Queries-R-Links:GraphicalMarkupforTextNaviga>on 1993

The oldest IR papers in CHIhttp://bit.ly/IR-HCI-Papers



In general, little evidence for the superiority of one method can be

found. There were no significant differences in search times, or in the number of pages accessed for the

two methods. [BUT]

Users switched twice as often from the tree to the directory than fromthe directory to the tree. Thus, while

users started with the tree and directory an equal number of times,

the answer was more often obtained with the directory

method.



The concept goes beyond the usual notion of an information retrieval system to encompass the cost structure of information

from secondary storage to immediate use.



The direct manipulative qualities of incrementability, reversibility, and smooth graphical feedback encourage the user

to explore the database, freely manipulating the properties without fear of syntax errors or of getting lost.



author 'tle year Cita'ons

KerryRoddenandKennethR.Wood HowDoPeopleManageTheirDigitalPhotographs? 2003 4261

KevinLarsonandMaryCzerwinski WebPageDesign:Implica>onsofMemory,StructureandScentforInforma>onRetrieval 1998 3884

ElizabethSillenceandPamBriggsandLesleyFishwickandPeterHarris TrustandMistrustofOnlineHealthSites 2004 3572

EdH.ChiandPeterPirolliandKimChenandJamesPitkow

UsingInforma>onScenttoModelUserInforma>onNeedsandAc>onsandtheWeb 2001 3188

StuartK.CardandGeorgeG.RobertsonandJockD.Mackinlay

TheInforma>onVisualizer,anInforma>onWorkspace 1991 2789

BillN.SchilitandGeneGolovchinskyandMorganN.Price

BeyondPaper:Suppor>ngAc>veReadingwithFreeFormDigitalInkAnnota>ons

1998 2570

JurgenKoenemannandNicholasJ.Belkin ACaseforInterac>on:AStudyofInterac>veInforma>onRetrievalBehaviorandEffec>veness 1996 2537

Joseph'Jofish'KayeandJanetVertesiandShariAveryandAllanDafoeandShay

ToHaveandtoHold:ExploringthePersonalArchive 2006 2262

DanielM.RussellandMarkJ.StefikandPeterPirolliandStuartK.Card TheCostStructureofSensemaking 1993 2188

The Most Cited IR Papers in CHIhttp://bit.ly/IR-HCI-Papers



Subjects really "liked" the penetrable version that

allowed them to manipulate the list of

suggested terms.



Most Recent IR Papers at CHIauthor 'tle yearMartaE.CecchinatoandAbigailSellenandMiladShokouhiandGavinSmyth FindingEmailinaMul>-Account,Mul>-DeviceWorld 2016

DeokgunParkandSimranjitSacharandNicholasDiakopoulosandNiklasElmqvist

Suppor>ngCommentModeratorsinIden>fyingHighQualityOnlineNewsComments 2016

HaiziYuandBiplabDekaandJerryO.TaltonandRanjithaKumar

Accoun>ngforTaste:RankingCuratorsandContentinSocialNetworks 2016

SaraschandraKaranamandHerrevanOostendorp

Age-relatedDifferencesintheContentofSearchQuerieswhenReformula>ng 2016

BenediktLoeppandKatjaHerrmannyandJürgenZiegler

BlendedRecommending:Integra>ngInterac>veInforma>onFilteringandAlgorithmicRecommender 2015

KhalilKloucheandTuukkaRuotsaloandDiogoCabralandSalvatoreAndolinaand DesigningforExploratorySearchonTouchDevices 2015

JaimieY.ParkandNeilO'HareandRossanoSchifanellaandAlejandroJaimesandChin-

ALarge-ScaleStudyofUserImageSearchBehaviorontheWeb 2015

YanirKleimanandJoelLanirandDovDanonandYasminFelberbaumandDanielCohen-Or

DynamicMaps:Similarity-basedBrowsingThroughaMassiveSetofImages 2015

BenSteichenandLuanneFreund Suppor>ngtheModernPolyglot:AComparisonofMul>lingualSearchInterfaces 2015




A few more IR papers at CHI




Not all these ideas workall the time

we know humans do not ‘do’ relevance feedback explicitly - even though it would help them



Image: Yahoo Research Blog



add donato paper?



Google’s Wonder Wheel

image: blogoscoped.com


http://blogoscoped.com


What does CHI want from IR?

A principle of human behaviour and when it holds trueor

A user interface idea and when it will work

A clear generalisable problem and context being studiedand



Challenges in HCIR

• Effect of Document Collection on Recommendations - WWW, Collections, Personal Data, Social Media, Multimedia

• Effect of Task Design - Exploratory is extremely vague - subjective, comprehensive, comparative, investigative, etc

• >> When should we provide support

• What to Measure about it



https://ils.unc.edu/searchtasks



Task Taxonomy

• Goal: Learn

• Topic: Conceptually General

• Target: Open ended

• Target: Multiple items

• Target: Uncertainty

• Goal: Ambiguous

• Process: dynamic

• Process: Long

• Goal: multi-faceted

• Goal: not too easy



Challenges in HCIR

• Focus of Searching - Options - WWW, Collections, Personal Data, Social Media, Multimedia

• Task Design - Exploratory is extremely vague - subjective, comprehensive, comparative, investigative, etc





IIiX 2010



Our findings provide further empirical validation to Vakkari’s task-based theory of IR [26], [..] when search interfaces provide

support to the search activities associated with the search task, effective interaction ensues helping the user address it

efficiently. Otherwise, excessive and unrelated search support can hamper progress and distract the searcher.



Teevan Thumbnails

only when refinding



Challenges in HCIR







Challenges in HCIR



• When should we provide support

• >> What to Measure about it



tag cloud papersCHI2011



CHI2012



Wilson & Wilson



Challenges in HCIR







What is evident is that thereis a cost-benefit in

Search User Interface Design



BCI Evaluation for IR

• 2009 - Concluded that we needed to understand the Cognitive Load created by Search User Interfaces

• 2011 - EuroHCIR paper on it

• 2012 - EuroHCIR paper using EEG

• 2012 onwards - focusing on fNIRS



Neuro IR


Dr Max L. Wilson http://cs.nott.ac.uk/~mlw/

SUI Design + Brain ResponseCognitive Load Theory

Total Mental Capacity

Simple UI

Easy Task

http://red.cs.nott.ac.uk/~mlw/




Simple UI

Hard Task





Complex UI

Hard Task



Three Related Theories

• Semantic vs Syntactic Load (HCI / CompSci)

• Cognitive Load Theory (Education)

• Mental Workload (Human Factors & Psychology)



Mental Workload

system takes place in working memory and consists of awide variety of the mental activities. In relation to IR, itis interesting to observe how elements of cognition, such asrehearsal of information, planning the search strategy anddeciding on the search keywords interconnect.

Multiple Resource Model. One model of mental work-load that has been widely accepted in Human Factors isWickens Multiple Resource Model [20] (Figure 2). The ele-ments of this model overlap with the needs and considera-tions of evaluating complex tasks (such as IR). He describesthe aspects of human cognition and the multiple resourcetheory in four dimensions:

Figure 2: The 4-D multiple resource model [20]

• The STAGES dimension refers to the three main stagesof information processing system (Wickens, 2004 [21]).

• The MODALITIES dimension indicating that audi-tory and visual perception have di↵erent sources.

• The CODES dimension refers to the types of memoryencodings which can be spatial or verbal.

• The VISUAL PROCESSING dimension refers to a nesteddimension within visual resources distinguishing be-tween focal vision (reading text) and ambient vision(orientation and movement).

Our aim is to understand how these elements link togetherand compose more complex components/tasks. Additionallywe want to consider how complex tasks (such as a searchtask) can be divided into primary components according tothe models described. This will help identify possible prob-lems in SUI design as well as indicating a possible solutionto the problem (suggested implications by Wickens [21]):

• Minimize working memory load of the SUI system andconsider working memory limits in instructions;

• Provide more visual echoes (cues) of di↵erent typesduring IR (verbal vs spatial);

• Exploit chunking (Miller, 1956 [14]) in various ways:physical size, meaningful size, superiority of lettersover numbers, etc;

• Minimize confusability;

• Avoid unnecessary zeros in codes to be remembered;

• Encourage regular use of information to increase fre-quency and redundancy;

• Encourage verbalization or reproduction of informa-tion that needs to be reproduced in the future;

• Carefully design information to be remembered;

Resource vs Demands. One other model that is of inter-est is the limited resource model [22] describing the relation-ship between the demands of a task, the resources allocatedto the task and the impact on performance.

Figure 3: Resources available vs task demands !impact on performance [22]

The graph from Figure 3 is used to represent the lim-ited resource model. The X-axes represent the resourcesdemanded by the primary task and as we move to the rightof the axes, the resources demanded by the primary taskincrease. The axes on the left indicate the resources beingused, but also the maximum available resources point (if wethink of working memory that is limited in capacity). Theright axes indicate the performance of the primary task (thedotted line on the graph). The key element of this model isthe concept of a limited set of resources which, if exceeded,has a negative impact on performance. However, it does notdistinguish between resource modality, therefore we proposeto use both the limited and multiple resources models toinform our work.

5. PATH 2: SUI EVALUATIONRelating quantitative data from brain sensing devices into

feedback about SUI designs is one of our ultimate goals inconducting this research. SUIs are inherently informationrich and thus a↵ect both visual (results page layout) andverbal (text based results) memory. Detecting a change in ei-ther verbal or spatial working memory would help determineif a workload di↵erence was caused by SUI design (spatial)or the amount of information the design provides (verbal).Our first in-progress study has stimulated each memory typein di↵erent tasks - Verbal memory was tested by performingan n-back [13] number memory task, whereas spatial mem-ory was tested using an n-back visual block matrix task.Other studies have also looked at each type of memory andconfirmed fNIRS ability to detect changes in heamodynamicresponses accordingly [9].In addition to developing an understanding of the ex-

tent to which we can monitor di↵erent memory, our ini-tial study also sought to measure the e↵ect of artefacts onthe fNIRS data. Controlling the environment and humanderived sources of noise is a potentially di�cult factor tocontrol without e↵ecting the ecological validity of a study.

Megaw, T. (2005) The definition and measurement of mental workload. Evaluation of human work, 525-551.



Erin’s paper



Measuring Mental Workload

Nasa-TLX fMRI

EEG

fNIRS



Challenges in HCIR







Questions?http://bit.ly/IR-HCI-Papers


The HCI Perspective on IR (DIR2016 Keynote)

Technology

Transcript of The HCI Perspective on IR (DIR2016 Keynote)