Click Models for Web Search - Lecture 4 · First assignmentSummaryAdvanced click modelsAdvanced...

First assignment Summary Advanced click models Advanced click models (self-study) Second assignment

Click Models for Web SearchLecture 4

Aleksandr Chuklin§,¶ Ilya Markov§ Maarten de Rijke§

[email protected] [email protected] [email protected]

§University of Amsterdam¶Google Research Europe

AC–IM–MdR Click Models for Web Search 1


Course overview

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research



This lecture

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research



Lecture outline

1 The first assignment

2 A quick summary

3 Advanced click models

4 Advanced click models (self-study)

5 The second assignment



Up next

Practical Session 1



Lecture outline


2 A quick summary






What have seen so far?

Basic click models

Methods for parameter estimation

Methods for evaluating click models

Main evaluation outcomes

CTR-based and cascade model outperformed by others interms of log-likelihood and perplexityEM-based models (PBM, UBM, DBN) outperform MLE-basedmodelsUBM tends to have highest log-likelihood, lowest perplexity



Lecture outline


2 A quick summary

3 Advanced click modelsAggregated searchBeyond clicksBeyond a single SERP





So many models . . .

Discussed here

Aggregated search

Beyond clicks

Beyond a single SERP

Skipped in the lecture but discussed in Das Buch:

User and query diversity

Using editorial judgments

Non-linear SERP examination

Using features



Lecture outline




Aggregated search



Federated click model (FCM)

P(A = 1) = χc

P(Er = 1 | A = 0) = εr

P(Er = 1 | A = 1) = εr + (1− εr )βdist

χc can depend on

vertical rank: χrvert

vertical content: χvert

dist = ru − rvertcan be negative



Vertical-aware click model (VCM)

1 Attraction bias

2 Global bias

3 First place bias

4 Sequence bias



Intent-aware click models

P(C1, . . . ,Cn) =∑i

P(I = i) · P(C1, . . . ,Cn | I = i)



Lecture outline




Beyond clicks



Mousing and scrolling

P(Er = 1 | ∃h ∈ H : r ≤ rh) = 1

P(Er = 1 | r ∈ V ) = 1

H – set of hovered documents

V – set of shown documents

Including hover information leds to improved perplexity

J. Huang, R.W. White, G. Buscher, K. Wang. Improving searcher models using mouse cursor activity. In SIGIR,2012. ACM Press.



From skimming to reading

Do mouse movements or eye positions obtained with an eyetracker indicate that a user is reading a snippet?



From skimming to reading

reading. Reading a result requires that the result be first skimmed. Similarly, not all results read by the user are clicked by users while almost all clicked results are read by users. This observation motivates us to consider an examination as a two-stage process. In the first "skimming to reading" stage (Stage 1) which is featured by sometimes unconscious eye fixations, users quickly look through results and decide whether one result should be ignored or paid further attention to. In Figure 1, the user's attention on the first two results corresponds to Stage 1. In this example, the user chose to ignore these results. In the second "reading to clicking" stage (Stage 2) which is usually remembered by users, they carefully read and comprehend the results selected from Stage 1 and based on the reading, decide whether to click on it or not. In Figure 1, the user's examination on the third result might come into Stage 2, and the user remembers that it has been read.

Figure 1. A user's eye fixation sequence and the corresponding explicit feedback on result reading for top results in a search session of query “学雷锋作文" (Essays on learning from Lei

Feng in Chinese). Radius of circle means fixation length.

Our proposed two-stage examination model and the choice of two stages are inspired by the attention selection mechanism [33] which is widely accepted in cognitive psychology studies. It says that human attention consists of two functionally independent, hierarchical stages: An early, pre-attentive stage (similar to Stage 1) that operates without capacity limitation and in parallel across the entire visual field, followed by a later, attentive limited-capacity stage (similar to Stage 2) that can deal with only one item (or at most a few items) at a time. Attention selection is one of the basic cognitive mechanisms of human beings and the two-stage examiantion model can be regarded as an attempt to explain how the mechanism works in Web search environment. What we propose in this paper is as follows: z [Two-stage Examination Model] With analysis of user’s

search interaction process, we show that users may examine SERPs with a two-stage strategy. This two-stage examination model reveals the relationship among eye fixation, result reading and click-through behaviors. It also helps us to understand the mechanism with which search users allocate their attention selectively.

z [Behavior Biases in Two-stage Examination] While revisiting the search behavior biases including position bias [5], domain bias [13] and attractiveness bias [1, 22], we found that these biases have different impacts on user behavior in different examination stages. It means that users may rely on different signals to make decisions in different stages. These findings also reaffirm the necessity of the proposed two-stage model.

z [Two-stage Examination and Relevance Prediction] A prediction model is constructed to identify result examination in different stages with mouse movement information that could be collected at large scale. After that, a learning method is proposed to estimate the relevance of a result based on the two-stage examination model. The two-stage model is found to significantly outperform the orginal single-stage model.

The remainder of this paper is organized as follows. In Section 2 we review some related studies on user interactions in Web search. Section 3 describes the framework of the experimental system and the interaction data collected in our study. Section 4 analyzes the relationship between fixation, reading and click-through behaviors and proposes the two-stage examining model. Section 5 focuses on the behavior biases in the two-stage model. In Section 6 we attempt to predict two-stage examination behavior using mouse movement information and then use this information to estimate result relevance. In Section 7 we discuss the extension of our work before some concluding remarks.

2. RELATED WORK Two lines of research are related to this work. One focuses on current endeavors to infer user intention and examination directly from the user’s gaze movements on a SERP. Our work explores further in this line by looking into user’s result reading process and we propose a two-stage examination model. The second line focuses on the relationship between gaze and mouse movement, and exploits mouse movement information for relevance prediction. We follow this line by utilizing mouse movement (rather than gaze) for relevance estimation using our two-stage examination model.

2.1 Eye-tracking Studies in Web Search The application of eye-tracking devices to Web search has received a considerable amount of attention from both academia and industry. Eye-tracking devices allows researchers to record users' real-time eye movement information, which helps better understand how users examine results on SERPs. Granka et al. [10], Richardson et al. [27] and Joachims et al. [18] use eye-tracking devices to analyze user’s basic eye movements and sequence patterns throughout search tasks. Guan et al. [11] found that the decrease of user’s attention in search sessions is closely related to query intents. Cutrell et al. [6] further investigated into how user's eye movement behavior varies for different query intents. Wang et al. [35] and Diaz et al. [7] found that different result appearances might create different biases on eye movement behavior for both vertical and other results on SERPs. Navalpakkam et al. [23] found that the flow of user attention on nonlinear page layouts is different from the widely believed top-down linear examination order of search results. Cole et al. [39] identify different user behavior patterns while performing different Web search tasks. Based on these findings, a number of generative click models [4, 5, 8, 35] have been constructed to model users' behavior during the search process. Most of these studies follow the strong eye-mind hypothesis [19] and regard eye fixation sequences to be the same as user’s examination sequences. However, cognitive processes may be more complex than what a simple eye fixation sequence can describe. Theeuwes et al. [34] showed that eyes will move to new objects unconsciously without the mind's control due to the selective attention mechanism [33]. Shiffrin et al. [30] pointed out that although overt attention (with eye fixation) is a significant part in cognitive processes, covert attention (usually without fixation) also helps to direct the gaze toward objects of interest. More importantly, Just et al. [33] found that there are no mapping rules between what is being fixated and what is being internally processed if the visual display is not relevant to the user’s current task. Considering the many distracting factors on SERPs (e.g. ads, multimedia components and results that are not so relevant), it is difficult for us to assume that users always have full attention to all results. Therefore, whether strong eye-mind hypothesis holds in Web search remains to be further investigated.

850

P(Cu = 1) = P(Cu = 1 | Ru = 1) · P(Ru = 1 | Fu = 1) · P(Fu = 1)

Ru – reading (examining) a document

Fu – fixing eyes on a document

Richer model leads to better relevance prediction

Picture taken from Y. Liu, C. Wang, K. Zhou, J. Nie, M. Zhang, and S. Ma. From skimming to reading: Atwo-stage examination model for web search. In CIKM, 2014. ACM Press.



Predicting mouse movements

2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

1.0

position

MTMT (image@4)RBP (p=0.7)DCG

Figure 4: Position discount weights compared to examina-tion inferred from mouse-tracking data for two layouts.

There has been some work on relaxing the linear scan usermodel. Wang et al. propose the application of partially ob-servable Markov (POM) models to address non-sequentialuser models [35, 15]. This work is conceptually very close toour own with one important di↵erence. The POM model as-sumes a fixed topology and cannot generalize to novel pagearrangements (topologies). Punera and Meguru present amethod for modeling nonmonotonic scan data but limit anal-ysis to click data with an underlying ranking [28].

Modeling searcher behavior on SERPs is a fundamentalpart of the feedback used in web search engine design. Log-ging simple click and skip statistics can be exploited to im-prove ranking performance [19]. More sophisticated modelsof user interaction also based on click information include as-sumptions about user satisfaction [10], result attractiveness[8], and document utility [11]. Since most existing SERPmodels leverage data only from user click information, ourwork can be seen as an extension of these models to incor-porate mousing data.

Outside of web search, there exist many models of visualattention. Models include those based on the visual salience[27], and biology [4]. Our work is most closely related tomachine learning models of visual attention [6, 20]. In thesecases, the authors attempt to predict the eye-tracking datafrom a small eye-tracking study using image-based signals.Practical issues prevent these experiments from being con-ducted on larger populations. These issues include the ex-pense of storing heavy image data for each SERP and re-liably transferring and/or rendering a user’s precise layout.Our work can be seen as an e�cient extension of these mod-els to large data sets with mouse-tracking data. The addi-tion of such data provides a more complete representationof search activity, which may be useful in developing moreaccurate behavior models.

3. MOTIVATING ANALYSISSeveral o✏ine and online evaluation metrics make assump-

tions about the relationship between document rank positionand examination. In this section, we will investigate the sup-port for these assumptions in mouse-tracking data.

The models underlying metrics such as discounted cumu-lative gain (DCG) and rank biased precision (RBP) assumethat the probability of examination of an item is conditioned

2 4 6 8 10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

position

MTMT (image@4)

Figure 5: Probability of starting at di↵erent rank positionsbased on mouse-tracking data.

a

h

r

s

10

9

8

7

6

5

4

3

2

1

n

n 1 2 3 4 5 6 7 8 9 10 s r h a

(a) Linear Scan

a

h

r

s

10

9

8

7

6

5

4

3

2

1

n

n 1 2 3 4 5 6 7 8 9 10 s r h a

(b) Actual Mousing

Figure 6: Hinton diagrams representing the probabilityof transitioning between pairs page modules, including theten algorithmic results as well as navigational modules (n),query suggestion (s), related searches (r), search history (h),and advertisements (a). Each row shows the conditional dis-tribution of the second position (column id) given the firstposition (row id). Figure 6(a) reflects the transition prob-abilities assumed under the linear scan assumption. Figure6(b) displaying the empirical transition probabilities frommouse-tracking data for pages including only ‘ten blue links’.

only on its position. In Figure 4, we compare position dis-count weights based on DCG and RBP to probability ofexamination based on mouse-tracking data for two layouts,a standard ‘10 blue links’ layout and a layout including animage vertical at position 4. We consider a module exam-ined if the user moused over it during a page view. We thennormalize, per position, by the number of page views, con-sidering page views which include a mouse-over on at leastone result. There are two noteworthy findings from this plot.First, neither of the two empirically-derived probabilities ob-serve probability 1 at the first position. This implies that,for roughly 20% of the page views, users never moused overthe first result, even though they moused over others. Sec-ond, page views with an image have significantly di↵erentprobabilities of examination for each position compared tothe ‘standard’ SERP. Compared to the position models, theempirical probabilities are flatter, suggesting more examina-tion than is suggested by those models. We will explore whythis is later in this section.

1453

2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

1.0

position

MTMT (image@4)RBP (p=0.7)DCG

Figure 4: Position discount weights compared to examina-tion inferred from mouse-tracking data for two layouts.

There has been some work on relaxing the linear scan usermodel. Wang et al. propose the application of partially ob-servable Markov (POM) models to address non-sequentialuser models [35, 15]. This work is conceptually very close toour own with one important di↵erence. The POM model as-sumes a fixed topology and cannot generalize to novel pagearrangements (topologies). Punera and Meguru present amethod for modeling nonmonotonic scan data but limit anal-ysis to click data with an underlying ranking [28].

Modeling searcher behavior on SERPs is a fundamentalpart of the feedback used in web search engine design. Log-ging simple click and skip statistics can be exploited to im-prove ranking performance [19]. More sophisticated modelsof user interaction also based on click information include as-sumptions about user satisfaction [10], result attractiveness[8], and document utility [11]. Since most existing SERPmodels leverage data only from user click information, ourwork can be seen as an extension of these models to incor-porate mousing data.

Outside of web search, there exist many models of visualattention. Models include those based on the visual salience[27], and biology [4]. Our work is most closely related tomachine learning models of visual attention [6, 20]. In thesecases, the authors attempt to predict the eye-tracking datafrom a small eye-tracking study using image-based signals.Practical issues prevent these experiments from being con-ducted on larger populations. These issues include the ex-pense of storing heavy image data for each SERP and re-liably transferring and/or rendering a user’s precise layout.Our work can be seen as an e�cient extension of these mod-els to large data sets with mouse-tracking data. The addi-tion of such data provides a more complete representationof search activity, which may be useful in developing moreaccurate behavior models.

3. MOTIVATING ANALYSISSeveral o✏ine and online evaluation metrics make assump-

tions about the relationship between document rank positionand examination. In this section, we will investigate the sup-port for these assumptions in mouse-tracking data.

The models underlying metrics such as discounted cumu-lative gain (DCG) and rank biased precision (RBP) assumethat the probability of examination of an item is conditioned

2 4 6 8 10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

position

MTMT (image@4)

Figure 5: Probability of starting at di↵erent rank positionsbased on mouse-tracking data.

a

h

r

s

10

9

8

7

6

5

4

3

2

1

n

n 1 2 3 4 5 6 7 8 9 10 s r h a

(a) Linear Scan

a

h

r

s

10

9

8

7

6

5

4

3

2

1

n

n 1 2 3 4 5 6 7 8 9 10 s r h a

(b) Actual Mousing

Figure 6: Hinton diagrams representing the probabilityof transitioning between pairs page modules, including theten algorithmic results as well as navigational modules (n),query suggestion (s), related searches (r), search history (h),and advertisements (a). Each row shows the conditional dis-tribution of the second position (column id) given the firstposition (row id). Figure 6(a) reflects the transition prob-abilities assumed under the linear scan assumption. Figure6(b) displaying the empirical transition probabilities frommouse-tracking data for pages including only ‘ten blue links’.

only on its position. In Figure 4, we compare position dis-count weights based on DCG and RBP to probability ofexamination based on mouse-tracking data for two layouts,a standard ‘10 blue links’ layout and a layout including animage vertical at position 4. We consider a module exam-ined if the user moused over it during a page view. We thennormalize, per position, by the number of page views, con-sidering page views which include a mouse-over on at leastone result. There are two noteworthy findings from this plot.First, neither of the two empirically-derived probabilities ob-serve probability 1 at the first position. This implies that,for roughly 20% of the page views, users never moused overthe first result, even though they moused over others. Sec-ond, page views with an image have significantly di↵erentprobabilities of examination for each position compared tothe ‘standard’ SERP. Compared to the position models, theempirical probabilities are flatter, suggesting more examina-tion than is suggested by those models. We will explore whythis is later in this section.

1453

Estimate the probability of mousing element j after element i

Use Maximum Likelihood Estimation and Farley-Ring Model

Picture taken from F. Diaz, R.W. White, G. Buscher, and D. Liebling. Robust models of mouse movement ondynamic web search results pages. In CIKM, 2013. ACM Press



Lecture outline




Pagination

Pagination is clicked 5–10%of times

Should be considered asa separate clickable object



Task-centric click model (TCM)

Cr – click at rank r

Ar – attractiveness at rank r

Er – examination at rank r

M – whether a query matchesa user’s need

Fr – whether a documentat rank r is “fresh”

WSDM 2016

San Francisco

San Francisco hotels

Cr = 1⇔ M = 1,Fr = 1,Er = 1,Ar = 1



Task-centric click model (TCM)

P(M = 1)︸︷︷︸match user need

= µ

P(N = 1 | M = 0) = 1

P(N = 1 | M = 1)︸︷︷︸submit new query

= ν

Hr = 1︸︷︷︸history

⇐⇒ ∃ previous occurence of ur

P(Fr = 1 | Hr = 0) = 1

P(Fr = 1 | Hr = 1)︸︷︷︸document freshness

= φ

P(Er = 1 | E<r ,S<r ,C<r ) = εr

P(Ar = 1) = αurq

WSDM 2016

San Francisco

San Francisco hotels



EM update rules for PBM parameters

Cr = 1⇔ Er = 1,Ar = 1

Attractiveness

α(t+1)uq =

1

|Suq|∑s∈Suq

(c(s)u +

(1− c

(s)u

) (1− γr )αuq

1− γrαuq

)

Examination

γ(t+1)r =

1

|Sr |∑s∈Sr

(c(s)u +

(1− c

(s)u

) (1− αuq) γr1− γrαuq

)



EM update rules for TCM parameters

Cr = 1⇔ Er = 1,Ar = 1,M = 1,Fr = 1

Attractiveness

α(t+1)uq =

1

|Suq|∑s∈Suq

(c(s)u +

(1− c

(s)u

) (1− εrµφ)αuq

1− εrαuqµφ

)

Examination

ε(t+1)r =

1

|Sr |∑s∈Sr

(c(s)u +

(1− c

(s)u

) (1− αuqµφ) εr1− εrαuqµφ

)



EM updating rules for TCM parameters (cont’d)

Freshness

φ(t+1) =1

|S|∑s∈S

1

|s|∑

u∈s:H(s)u

(c(s)u +

(1− c

(s)u

) (1− εrαuqµ)φ

1− εrαuqµφ

)



Lecture 4 summary

Aggregated search

Beyond clicks




Lecture outline


2 A quick summary


4 Advanced click models (self-study)User diversityNon-linear examinationUsing features




Lecture outline




User diversity

Picture taken from J. Huang, R. White, S. Dumais. No Clicks, No Problem: Using Cursor Movements toUnderstand and Improve Search. In CHI, 2011, ACM Press.



Matrix factorization click model (MFCM)

Tensor

Qu

erie

sDocuments

Users

Queries(Q)

Documents (U)

Users (V)

αuqv

P(Au = 1) = αuqv

αuqv ∼ N(Uu ◦ Qq ◦ Vv , σ

2)



Other models with users

UBM-userP(Au = 1) = αuqεv

LOG-rd-user (logistic model)

P(Er = 1 | Er−1 = 1,C<r) = σ (γrr ′ + εv )

P(Ar = 1) = σ (αurq + εv )

σ(x) =1

1 + e−x

DIL-user (dilution model)

P(Er = 1 | Er−1 = 1,C<r) = (βv )r−1(λv )k(µv )r−r′

P(Ar = 1) = αurqεv



Lecture outline




Non-linear examination

Hello world program - Wikipedia, the free encyclopediaA " Hello world" program is a computer program that prints out " Hello world" on a display device. It pre-dates the age of the World Wide Web where posted messages ...

Hello-World: World Languages for Children of all agesGames, songs and activities make learning any language fun. Use hello-world by itself or as an enhancement to any language program!

CMT : Videos : Lady Antebellum : Hello WorldWatch Lady Antebellum's music video Hello World for free on CMT.com

hello world

Image Results

Search

Hello WorldHello World Approach and Methodology The idea of Hello World was first conceived by scholars teaching English in Asia in the early 1990's. Their basic belief was that ...

Hello World SoftwareBuy hello world.

FiltersAnytimePast DayPast WeekPast Month

hello world Search

(a) Presentation

9

1

3

5

6

7

8

42

(b) Arrangement

9

1

3

5

6

7

8

42

(c) Mouse Data

Figure 1: Module-level representation of mouse-tracking data. The session sequence for this data would be[1, 3, 5, 6, 7, 6, 5, 3, 5].

Figure 2: Distribution of unique page arrangements forSERPs from two large scale web search engines. The hor-izontal axis indicates the rank of the arrangement whensorted by frequency. The vertical axis indicates the fre-quency of that arrangement.

In addition, we propose a user model which allows us togeneralize to arbitrary page arrangements. This is impor-tant because previous user models based on click logs allassume a single topology across all queries. That is, by ig-noring non-web modules, the graph structure in Figure 3(a)is shared across all queries. In our case, the topology in Fig-ure 3(b) might be di↵erent for two arbitrary queries. There-fore, the edge weights learned for one query will be uselessof a novel arrangement (topology).

In order to estimate the parameters of our user model,we exploit user mouse behavior associated with a SERP ar-rangement. We adopt this strategy because of the high cor-relation in general between eye fixation and mouse position[9]. Previous work has confirmed this correlation for SERPs[30, 16].

The focus of our study will be on the problem of construct-ing robust models able to make predictions about mouse be-havior on arrangements for which we have little or no dataavailable. Having such models provide a tool which can beused when manually designing new pages [31]. At a largerscale, mouse-tracking models could be useful for retrospec-

m1

m2

m3

m4

m5

m0

m6

(a) linear

m1

m2

m3

m4

m5

m0

m6

(b) relaxed

Figure 3: The linear scan model and its relaxation.

tively detecting ‘good abandonments’, cases where the userwas satisfied without clicking a link [21].

In this paper, we make the following contributions,

• a generalization of the linear scan model.

• an e�cient and e↵ective method for estimating thegeneralized model.

• an e�cient and e↵ective method for estimating param-eters of unobserved arrangements (topologies).

• experiments reproduced on data sets from two largecommercial search engines.

2. RELATED WORKThe motivation for capturing mouse movement at scale

originates from results demonstrating a strong correlationbetween eye and mouse position [9]. In the context of websearch, this correlation has been reproduced on SERPs [30],suggesting that, with some care [16], we can use loggedmouse data as a ‘big data’ complement to eye-tracking stud-ies [3]. Such studies have found that mouse-tracking is usefulfor click prediction [17] and advertisement interest predic-tion [14]. In fact, mouse movement analysis has been sug-gested as useful for web site usability analysis in general [2,3]. Even without assuming a relationship between eye andmouse, important search signals such as query intent [13]and document relevance [18] can be detected.

1452




Click models for aggregated search



Temporal click models

POM-based click models

User actions as a Markov chainTemporal information in search logs

Temporal Hidden Click Model(THCM)

P(Er+1 = 1|Er = 1) = αP(Er−1 = 1|Er = 1) = γ

Partially Sequential Click Model(PSCM)

Between adjacent clicks usersexamine results in a single directionUsers can skip results withoutexamining them



Whole-page click model (WPC)

Transitions between blocks (macro-model)are modeled as a Markov chain

Behavior within a block (micro-model)is modeled using standard click models

Hello world program - Wikipedia, the free encyclopediaA " Hello world" program is a computer program that prints out " Hello world" on a display device. It pre-dates the age of the World Wide Web where posted messages ...

Hello-World: World Languages for Children of all agesGames, songs and activities make learning any language fun. Use hello-world by itself or as an enhancement to any language program!

CMT : Videos : Lady Antebellum : Hello WorldWatch Lady Antebellum's music video Hello World for free on CMT.com

hello world

Image Results

Search

Hello WorldHello World Approach and Methodology The idea of Hello World was first conceived by scholars teaching English in Asia in the early 1990's. Their basic belief was that ...

Hello World SoftwareBuy hello world.

FiltersAnytimePast DayPast WeekPast Month

hello world Search

(a) Presentation

9

1

3

5

6

7

8

42

(b) Arrangement

9

1

3

5

6

7

8

42

(c) Mouse Data

Figure 1: Module-level representation of mouse-tracking data. The session sequence for this data would be[1, 3, 5, 6, 7, 6, 5, 3, 5].

Figure 2: Distribution of unique page arrangements forSERPs from two large scale web search engines. The hor-izontal axis indicates the rank of the arrangement whensorted by frequency. The vertical axis indicates the fre-quency of that arrangement.

In addition, we propose a user model which allows us togeneralize to arbitrary page arrangements. This is impor-tant because previous user models based on click logs allassume a single topology across all queries. That is, by ig-noring non-web modules, the graph structure in Figure 3(a)is shared across all queries. In our case, the topology in Fig-ure 3(b) might be di↵erent for two arbitrary queries. There-fore, the edge weights learned for one query will be uselessof a novel arrangement (topology).

In order to estimate the parameters of our user model,we exploit user mouse behavior associated with a SERP ar-rangement. We adopt this strategy because of the high cor-relation in general between eye fixation and mouse position[9]. Previous work has confirmed this correlation for SERPs[30, 16].

The focus of our study will be on the problem of construct-ing robust models able to make predictions about mouse be-havior on arrangements for which we have little or no dataavailable. Having such models provide a tool which can beused when manually designing new pages [31]. At a largerscale, mouse-tracking models could be useful for retrospec-

m1

m2

m3

m4

m5

m0

m6

(a) linear

m1

m2

m3

m4

m5

m0

m6

(b) relaxed

Figure 3: The linear scan model and its relaxation.

tively detecting ‘good abandonments’, cases where the userwas satisfied without clicking a link [21].

In this paper, we make the following contributions,

• a generalization of the linear scan model.

• an e�cient and e↵ective method for estimating thegeneralized model.

• an e�cient and e↵ective method for estimating param-eters of unobserved arrangements (topologies).

• experiments reproduced on data sets from two largecommercial search engines.

2. RELATED WORKThe motivation for capturing mouse movement at scale

originates from results demonstrating a strong correlationbetween eye and mouse position [9]. In the context of websearch, this correlation has been reproduced on SERPs [30],suggesting that, with some care [16], we can use loggedmouse data as a ‘big data’ complement to eye-tracking stud-ies [3]. Such studies have found that mouse-tracking is usefulfor click prediction [17] and advertisement interest predic-tion [14]. In fact, mouse movement analysis has been sug-gested as useful for web site usability analysis in general [2,3]. Even without assuming a relationship between eye andmouse, important search signals such as query intent [13]and document relevance [18] can be detected.

1452




Lecture outline




Using features290 Benchmarking Learning-to-Rank Algorithms

For the “Gov” corpus, 64 features were extracted for each query–

document pair, as shown in Table 6.2.

For the OHSUMED corpus, 40 features were extracted in total, as

shown in Table 6.3.

Table 6.2 Learning features of TREC.

ID Feature description

1 Term frequency (TF) of body

2 TF of anchor

3 TF of title

4 TF of URL

5 TF of whole document

6 Inverse document frequency (IDF) of body

7 IDF of anchor

8 IDF of title

9 IDF of URL

10 IDF of whole document

11 TF*IDF of body

12 TF*IDF of anchor

13 TF*IDF of title

14 TF*IDF of URL

15 TF*IDF of whole document

16 Document length (DL) of body

17 DL of anchor

18 DL of title

19 DL of URL

20 DL of whole document

21 BM25 of body

22 BM25 of anchor

23 BM25 of title

24 BM25 of URL

25 BM25 of whole document

26 LMIR.ABS of body

27 LMIR.ABS of anchor

28 LMIR.ABS of title

29 LMIR.ABS of URL

30 LMIR.ABS of whole document

31 LMIR.DIR of body

32 LMIR.DIR of anchor

33 LMIR.DIR of title

34 LMIR.DIR of URL

35 LMIR.DIR of whole document

36 LMIR.JM of body

37 LMIR.JM of anchor

38 LMIR.JM of title

39 LMIR.JM of URL

(Continued)



Regression-based click models

P(Cu = 1) =1

1 + e−Z

Z =∑

wi fi

Factorization machines

Neural networks (for clickprediction in sponsored search)



General click model (GCM)

Click model parameters can depend on other attributes(e.g., time of the day, user browser, length of a URL, etc.)

P(Cr = 1 | Er = 1) = P

∑i

θf useri+∑j

θf urlj+ ε > 0

CM, DCM, CCM and DBN can be considered as special casesof GCM



Advanced models summary

User diversity

Non-linear SERP examination

Using features



Lecture outline


2 A quick summary






Up next

Practical Session 2



Acknowledgments

All content represents the opinion of the authors which is not necessarily shared orendorsed by their respective employers and/or sponsors.


Click Models for Web Search - Lecture 4 · First assignmentSummaryAdvanced click modelsAdvanced...

Documents

Transcript of Click Models for Web Search - Lecture 4 · First assignmentSummaryAdvanced click modelsAdvanced...