User engagement in the digital world

User Engagement in the Digital World

Mounia Lalmas Yahoo! Labs

Barcelona

[email protected]

A bit about myself   1999-2008: Lecturer (assistant professor) to Professor at

Queen Mary, University of London   2008-2010 Microsoft Research/RAEng Research

Professor at the University of Glasgow   2011- Visiting Principal Scientist at Yahoo! Labs

Barcelona

 Research topics  XML/structured retrieval and evaluation (INEX)  Quantum theory to model interactive information retrieval  Aggregated search  Bridging the digital divide  Models and measures of user engagement

Outline

 Characteristics and measurement

 (my) Vision and focus

 Some results … and ideas

 What next?

User Engagement – connecting three sides   User engagement is a quality of the user experience

that emphasizes the positive aspects of interaction – in particular the fact of being captivated by the technology.

  Successful technologies are not just used, they are engaged with.

user feelings: happy, sad, excited, bored, …

The emotional, cognitive and behavioural connection that exists, at any point in time and over time, between a user and a technological resource

user interactions: click, read comment, recommend, buy, …

user mental states: concentrated, lost, involved, …

S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.

Would a user engage with this web site?

http://www.nhm.ac.uk/

Would a user engage with this web site? (content)

http://www.amazingthings.org/ (art event calendar)

Would a user engage with this web site? (aesthetics)

http://www.lowpriceskates.com/ (e-commerce – skating)

Would a user engage with this web site? (navigation)

http://chiptune.com/ (music repository)

Would a user engage with this web site? (navigation)

http://www.theosbrinkagency.com/ (photographer)

Characteristics of user engagement (I) • Users must be focused to be engaged • Distortions in the subjective perception of time used to

measure it Focused attention

• Emotions experienced by user are intrinsically motivating •  Initial affective hook can induce a desire for exploration, active

discovery or participation Positive Affect

• Sensory, visual appeal of interface stimulates, promote focused attention

• Linked to design principles (e.g. symmetry, balance, saliency) Aesthetics

• People remember enjoyable, useful, engaging experiences and want to repeat them

• Reflected in e.g. the propensity of users to recommend an experience/a site/a product

Endurability

Characteristics of user engagement (II)

•  Novelty, surprise, unfamiliarity and unexpected •  Appeal to user curiosity, encourages inquisitive

behavior and promotes repeated engagement Novelty

•  Richness captures the growth potential of an activity •  Control captures the extent to which a person is able

to achieve this growth potential Richness and control

•  Trust is a necessary condition for user engagement •  Implicit contract among people and entities which is

more than technological

Reputation, trust and expectation

•  Difficulties in setting up “laboratory” style experiments •  Why should user engage?

Motivation, interests, incentives, and

benefits

Forrester Research – The four I’s •  Presence of a user •  Measured by e.g. number of visitors, time spent Involvement

•  Action of a user •  Measured by e.g. CTR, online transaction, uploaded

photos or videos Interaction

•  Affection or aversion of a user •  Measured by e.g. satisfaction rating, sentiment

analysis in blogs, comments, surveys, questionnaires Intimacy

•  Likelihood a user advocates •  Measured by e.g. forwarded content, invitation to join Influence

Measuring Engagement, Forrester Research, June 2008.

Measuring user engagement Measures Characteristics

Self-reported engagement

Questionnaire, interview, report, product reaction cards

Subjective, user study (lab/online)

Mostly qualita,ve

Cognitive engagement

Task-based methods (time spent, follow-on task)

Neurological measures (e.g. EEG)

Physiological measures (e.g. eye tracking, mouse-tracking)

Objective, user study (lab/(online))

Mostly quan,ta,ve

Scalability an issue?

Interaction engagement

Web analytics + “data science”

Information retrieval metrics + user models

Objective, data study

Quan,ta,ve Large scale

Objective measures – Online activities Proxy of user engagement

Online measures as proxy of user engagement!

measuring user engagement and interpreting metrics is hard!!

Multimedia search activities often driven by entertainment needs, not by information needs

Online measures as proxy of user engagement!

M. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011.

Outline




 What next?

What affect user (& site) engagement?

Web page & site style?

Web page & site content?

+ layout + links + saliency + content + sentimentality, …

user engagement within and across site

Measurements and methodologies + online analytics metrics (dwell time, CTR, …) + complex networks metrics + new metrics + survival analysis

+ questionnaires, surveys, … + crowd-sourcing

+ biometrics (eye tracking, mouse tracking, …)

Goals + Models of user engagement + Metrics of user engagement

The three sides

+ emotional

+ cognitive

+ behavioral

User engagement… connecting three sides

self-reported engagement

interaction engagement

User Engagement – connecting three measurement approaches

Diagnostic and what we have done

Diagnostic: work exists, but fragmented. In particular: o What and how to measure depend on services and goals o  Lack of understanding of how to relate subjective and

objective measures

The rest of this talk: 1.  Models of user engagement 2.  Attention & affect & saliency 3.  Attention & affect & gaze & sentimentality 4.  Attention & affect & mouse tracking (results pending)

Outline




 What next?

Models of user engagement Online sites differ concerning their engagement!

Games Users spend much time per visit

Search Users come frequently and do not stay long

Social media Users come frequently and stay long

Special Users come on average once

News Users come periodically

Service Users visit site, when needed

Is it possible to model these differences?

Data and Metrics Interaction data, 2M users, July 2011, 80 US sites

Popularity #Users Number of distinct users

#Visits Number of visits

#Clicks Number of clicks

Activity ClickDepth Average number of page views per visit.

DwellTimeA Average time per visit

Loyalty ActiveDays Number of days a user visited the site

ReturnRate Number of times a user visited the site

DwellTimeL Average time a user spend on the site.

Methodology General models Time-based models

Dimensions

8 metrics weekdays, weekend

8 metrics per time span #Dimensions 8 16

Kernel k-means with Kendall tau rank correlation kernel

Nb of clusters based on eigenvalue distribution of kernel matrix Significant metric values with Kruskal-Wallis/Bonferonni

#Clusters (Models) 6 5

Analysing cluster centroids = models

Models of user engagement [6 general]

•  Popularity, activity and loyalty are independent from each other •  Popularity and loyalty are influenced by external and internal factors

  e.g. frequency of publishing new information, events, personal interests

•  Activity depends on the structure of the site

interest-specific

media (daily) search

periodic media

e-commerce

models based on engagement metrics only

Time-based [5 models] Models based on engagement over weekdays and weekend

hobbies, interest-specific weather

daily news work-related

time-based models ≠ general models

next put all and more together! let machine learning tell you more!

Models of user engagement – Recap & Next  User engagement is complex and standard

metrics capture only a part of it  User engagement depends on time (and users)  First step towards a taxonomy of models of user

engagement … and associated metrics

 Next  More sites, more models?  Interaction between sites (online multi-tasking)  User demographics, time of the day, geo-location, etc

J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.

Online multi-tasking

users spend more and more of their online session multi-tasking, e.g. emailing, reading news, searching for information ONLINE MULTI-TASKING navigating between sites, using browser tabs, bookmarks, etc seamless integration of social networks platforms into many services

leaving a site is not a “bad thing!”

July 2011, 25M sessions avg session length 26mn (sd 44)

1.7 Yahoo! sites and 4.9 external (sd 3.1 and 8.6)

(fictitious navigation between sites within an online session)

Saliency, attention and positive affect  How the visual catchiness (saliency) of

“relevant” information impacts user engagement metrics such as focused attention and emotion (affect)  focused attention refers to the exclusion of

other things  affect relates to the emotions experienced

during the interaction  Saliency model of visual attention

developed by Itti and Koch L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40, 2000.

Manipulating saliency

Web page screenshot Saliency maps

salie

nt c

ondi

tion

non-

salie

nt c

ondi

tion

Study design   8 tasks = finding latest news or headline on celebrity or

entertainment topic   Affect measured pre- and post- task using the Positive

e.g. “determined”, “attentive” and Negative e.g. “hostile”, “afraid” Affect Schedule (PANAS)

  Focused attention measured with 7-item focused attention subscale e.g. “I was so involved in my news tasks that I lost track of time”, “I blocked things out around me when I was completing the news tasks” and perceived time

  Interest level in topics (pre-task) and questionnaire (post-task) e.g. “I was interested in the content of the web pages”, “I wanted to find out more about the topics that I encountered on the web pages”

  189 (90+99) participants from Amazon Mechanical Turk

PANAS (10 positive items and 10 negative items)  You feel this way right now, that is, at the

present moment [1 = very slightly or not at all; 2 = a little; 3 = moderately;

4 = quite a bit; 5 = extremely] [randomize items]

distressed, upset, guilty, scared, hostile, irritable, ashamed, nervous, jittery, afraid

interested, excited, strong, enthusiastic, proud, alert, inspired, determined, attentive, active

D. Watson, L.A. Clark, A. Tellegen. Development and validation of brief measures of positive and negative affect: The PANAS Scales. Journal of Personality and Social Psychology, 47, 1988.

7-item focused attention subscale (part of the 31-item user engagement scale)

5-point scale (strong disagree to strong agree) 1.  I lost myself in this news tasks experience 2.  I was so involved in my news tasks that I lost track of

time 3.  I blocked things out around me when I was completing

the news tasks 4.  When I was performing these news tasks, I lost track of

the world around me 5.  The time I spent performing these news tasks just

slipped away 6.  I was absorbed in my news tasks 7.  During the news tasks experience I let myself go

H.L. O'Brien. Defining and Measuring Engagement in User Experiences with Technology. PhD Thesis, 2008.

Saliency and positive affect  When headlines are visually non-salient

 users are slow at finding them, report more distraction due to web page features, and show a drop in affect

 When headlines are visually catchy or salient  user find them faster, report that it is easy to focus,

and maintain positive affect

 Saliency is helpful in task performance, focusing/avoiding distraction and in maintaining positive affect

Saliency and focused attention  Adapted focused attention subscale from the online

shopping domain to entertainment news domain

 Users reported “easier to focus in the salient condition” BUT no significant improvement in the focused attention subscale or differences in perceived time spent on tasks

 User interest in web page content is a good predictor of focused attention, which in turn is a good predictor of positive affect

Saliency and user engagement – Recap & Next  Interaction of saliency, focused attention, and

affect, together with user interest, is complex  Next:

 include web page content as a quality of user engagement in focused attention scale

 more “realistic” user (interactive) reading experience

 bio-metrics (mouse-tracking, eye-tracking, facial expression, etc)

L. McCay-Peet, M. Lalmas, V. Navalpakkam. On saliency, affect and focused attention, CHI 2012

Gaze, sentimentality, interest … and user engagement

  News + comments   Sentiment, interest   57 users (lab-based)   Reading task (114)

  Questionnaire (qualitative data)   Record mouse tracking, eye tracking, facial

expression, EEG signal (quantitative data)

Three metrics: gaze, focus attention and positive affect

Interesting content promote users engagement metrics  All three metrics:

 focus attention, positive affect & gaze

 What is the right trade-off?  news is news

 Can we predict?  provider, editor, writter, category, genre, visual aids,

…, sentimentality, …  Role of user-generated content (comments)

 As measure of engagement?  To promote engagement?

Lots of sentiments but with negative connotations!  Positive effect (and interest, enjoyment and wanted

to know more) correlates  Positively () with sentimentality (lots of emotions)  Negatively () with positive polarity (happy news)

SentiStrenght (from -5 to 5 per word)

sentimentality: sum of absolute values (amount of sentiments) polairity: sum of values (direction of the sentiments: positive vs negative)

M. Thelwall, K. Buckley, G. Paltoglou, Sentiment strength detection for the social web. JASIST, 63,1, 2012.

Effect of comments on user engagement

 6 ranking of comments:  most replied, most popular, newest  sentimentality high, sentimentality low  polarity plus, polarity minus

 Longer gaze on  newest and most popular for interesting news  most replied and high sentimentality for non-interesting

news

 Can we leverage this to prolonge user attention?

Gaze, sentimentality, interest – Recap and Next  Interesting and “attractive” content!  Sentiment as a proxy of focus attention, positive

affect and gaze?

 Next  Larger-scale study  Other domains (beyond daily news!)

 Role of social signals (e.g. Facebook, Twitter)  Lots more data: mouse tracking, EEG, facial

expression I. Arapakis, M. Lalmas, B. Cambazoglu, M.-C. Marcos, J. Jose. Examining User Engagement through the Prism of Interest, Sentiment and Gaze, Submitted for Publication, 2012.

Mouse tracking … and user engagement   400 users from Amazon Mechanical Turk   Two domains (BBC and Wikipedia)   Two tasks (reading and quiz)   “Normal vs Horrible” interface

  Questionnaires (qualitative data)   Mouse tracking (quantitative data)   Interaction data (page view, dwell time)

  Results pending! … Hawthorne Effect!!!!!!!!!!

Mouse tracking … and user engagement (Taxonomy? Correlation vs Causation? Measurement? … )

Outline




 What next?


interaction

engagement

Outline

digital libraries

user engagement

digital world

mobile & tablet

Thank you

Ioannis Arapakis Ricardo Baeza-Yates Georges Dupret Janette Lehmann Lori McCay-Peet Vidhya Navalpakkam David Warnock Elad Yom-Tov

and many others at Yahoo! Labs

Collaborators

User engagement in the digital world

Technology

Transcript of User engagement in the digital world