Understanding real life website adaptations by investigating the relations
-
Upload
mark-graus -
Category
Science
-
view
100 -
download
0
Transcript of Understanding real life website adaptations by investigating the relations
Understanding Real-Life
Website Adaptations by
Investigating the Relations
between User Behavior and
User Experience
Mark P. Graus
Martijn C. Willemsen
Kevin Swelsen
• Three countries
– Netherlands
– Belgium
– UK
• Aimed at IT/CE-enthusiasts
• Second Biggest IT website in the
Netherlands: 8+ mln
pageviews/month
• Editorial Board + Price Comparison
• 1.500 products tested and reviewed
each year
• Active Community
3
Hardware.Info Online
Can we provide visitors with better content based
on an educated guess of their interests?
4
5
Hardware.info
Introducing a User-Centric Evaluation Framework
6
7
User-Centric Evaluation Framework
Predicted Segment
Shown Content
PerceivedAccuracy
Perceived Effectiveness
NavigationBehavior
Knijnenburg, B. et al. 2012. Explaining the user experience of recommender systems. UMUAI.
• When looking at what behavior is relevant, intuition might point us in
the wrong direction
Number of videos watched in a video browsing system correlated with
lower system satisfaction.
Knijnenburg et al., 2010, Receiving recommendations and providing
feedback: The user-experience of a recommender system
8
Why User Evaluation?
• During 2 weeks on hardware.info we ran an online experiment
• Collected 2 weeks worth of data
– 100k unique visitors
– 3k completed surveys
9
Process
Enter5
PageviewsSidebar Element
Pageviews Survey Pageviews
Analysis:
Predictive Modeling (Post-Hoc)
Evaluation of Adaptation
Evaluation of Adaptation10
Data
5 pageviews
Rest of VisitContentEUPHCMix
Survey Data
To what extent are you interested in HC/EUP?
11
Predictive Modeling
Model (Behavioral
Labels)
Labels (Behavioral Data)Labels
(Survey Data)
Model (Survey Labels)
12
Comparison of Predictions
Model (Survey Data)
Model (Behavioral Data)
PredictionsPredictions
Shown Content
PredictedSegment
Predicted Segment
Rest of Visit
HC HC HC • Clicks on Sidebar Element
EUP EUP EUP • Pageviews
Mix • Sessions
Explain
AIC
Labels Clicks on Sidebar Clicks on Sidebar (Boolean)
Pageviews Sessions
Behavior 834,821.3 26,910.6 23,362.0 517,453.3
Survey 832,555.5 26,832.5 23,270.2 514,761.0
13
Regression Model Fit
• A model based on Survey Data provides predictions that better
describe response to the Sidebar Element than models based on
Behavioral Data
• Despite less information (3k vs 100k)
• We are predicting segments for 100.000 visitors while using data
from only 3,000
14
Conclusion
Effects of the Adaptation
Evaluation of Adaptation15
16
Predicted Segment
Shown Content
(Congruency)
Perceived Accuracy
PerceivedEffectiveness
NavigationBehavior
17
Predicted Segment
Shown Segment
(Congruency)
Perceived Accuracy
PerceivedEffectiveness
NavigationBehavior
“The Items in the Sidebar matched my preferences.”
18
Predicted Segment
Shown Segment
(Congruency)
Perceived Accuracy
PerceivedEffectiveness
NavigationBehavior
“The sidebar helps me in finding new and interesting articles.”
19
Predicted Segment
Shown Segment
(Congruency)
Perceived Accuracy
PerceivedEffectiveness
NavigationBehavior
0
0.02
0.04
0.06
0.08
0.1
0.12
Non-congruent congruent
Content
Proportion of Visitors that Click Sidebar Element
HC
EUP
• Congruent Content Leads to • More Clicks on the Sidebar element• More Pageviews• More Visits
• Stronger effects for people interested in EUP
20
Putting it All Together: Path Model
Adaptation
ClickedINT
perceived
AccuracySSA
.212 (.09)
p<.05
.595 (.024)
p<.005
.103 (.03)
p<.005
-.138 (.056)
p<.05
perceived
EffectivenessEXP
HC Segment(versus EUP)
PC
Congruent(versus non-
congruent) OSA
HC:Congruent
OSA/PC
Take Home Message
21
• If you do Predictive Modeling of Latent User Characteristics
– Using a single question provides ground truth
• Reliable
• Cheap: 3,000 surveys cost us one SSD and one mobile
phone
• If you want to Ground Behavioral Changes in User Experience
– Use a full survey and analysis
• Answer the question: Why does behavior change?
22
Consider Surveys!
• Our thanks also goes out to
– hardware.info
– Bart Knijnenburg
• Questions/remarks?
Mark Graus – PhD Student
Human-Technology Interaction Group
https://twitter.com/newmarrk
https://linkedin.com/in/markgraus
http://www.marrk.nl
23
Thank You