Introducing a New UX Maturity Metric: Team Engagement...
Transcript of Introducing a New UX Maturity Metric: Team Engagement...
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Introducing a New UX Maturity Metric: Team Engagement Score
(TES) During Usability Testing
Bill Albert
Executive Director, Bentley University User Experience Center
Josh Rosenberg
User Experience Researcher, DonorsChoose.org
June 6, 2017
@BentleyUXC | @UXMetrics
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Two Short Stories
@BentleyUXC | @UXMetrics
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Why Team Engagement Matters
• Adoption of UX recommendations
• Efficiency in the design process and product development
• Culture of UX throughout the organization
• Sparking innovation
• Value to co-workers
@BentleyUXC | @UXMetrics
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Our Motivation
• Formalize our thinking through development of a team engagement metric
• Launch discussions internally – explore the value of a team engagement metric
• Share with others in the UX community
• Promote team engagement
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
TES: Team Engagement Score
TES = Observation + Collaboration + Behavior
• Observation = Observing usability sessions
• Collaboration = Being part of the team during testing
• Behavior = Being actively engaged when observing
@BentleyUXC | @UXMetrics
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 1: Observation
@BentleyUXC | @UXMetrics
# Actual Observed Sessions
# Total Observable Sessions
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
25 Actual Observed Sessions100 Total Observable Sessions
Component 1: Observation
@BentleyUXC | @UXMetrics
0.25=
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 2: Collaboration
# Feasible In-Person Observations
# In-Person Observations
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 2: Collaboration
2 In-Person Observers
5 Possible Observers
= 0.40
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 3: Behavior
@BentleyUXC | @UXMetrics
10
1-10 Rating
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 3: Behavior
@BentleyUXC | @UXMetrics
Took Notes
Follow-up Questions
Paid Attention
Brainstormed Solutions
Etc.
Rating: 8
✓
✓
✓
Rating: 8
10
= 0.80
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 3: Behavior
@BentleyUXC | @UXMetrics
Component 3: B
@BentleyUXC | @UXMetrics
• Attention (0 -2)• Low = meetings, multi-tasking; High = undivided attention
• Conversation (0-2)• Low = none; High = creative brainstorming based on insights
• Notes (0-2)• Low = no notes; High = took detailed notes
• Attitude (0-2)• Low = discredits participants; High = empathizes with participants
• Questions (0-2)• Low = no questions; High = a lot of questions
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Calculating TES
• TES = Observation + Collaboration + BehaviorTES = ((# actual observations/ # total possible observations) +
(# in-person observers / # feasible in-person) +
(engagement score/10)) / 3
• Average the three numbers• TES = (0.25 + 0.4 + 0.8) = 1.45, divided by 3 = 0.48
• TES runs from 0 – 1 (higher the better)
• All three dimensions weighted equally, but can apply different weights
• TES works for large or small teams
@BentleyUXC | @UXMetrics
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Scenario 1: High Engagement
@BentleyUXC | @UXMetrics
• Observation = 10 sessions (2 days), 8 attend all 10 sessions, 2 attend the first day only (5) = 90/100 = 0.90
• Collaboration = 5 people local, all attend in person = 5/5 = 1.0
• Behavior = highly engaged = 9/10 = 0.9
• TES = 0.9 + 1.0 + 0.9 = 2.8/3.0 = .93
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Scenario 2: Low Engagement
@BentleyUXC | @UXMetrics
• Observation = 10 sessions (2 days), 4 attend day 1 only, 6 don’t attend any sessions = 20/100 = 0.20
• Collaboration = 5 people local, 2 attend in person = 2/5 = 0.4
• Behavior = low engagement = 3/10 = 0.3
• TES = 0.2 + 0.4 + 0.3 = 0.9/3.0 = .30
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Scenario 3: Remote Only Studies
@BentleyUXC | @UXMetrics
• Observation = 10 sessions (2 days) with 10 invited - 3 attend both days, 5 attend day 1 only, 2 don’t attend any sessions = 45/100 = 0.45
• Collaboration = all remote, so this component is not considered
• Behavior = moderate engagement = 6/10 = 0.6
• TES = 0.45 + 0.6 = 1.05/2.0 = .525
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Changing Weights
@BentleyUXC | @UXMetrics
• Change weights if you value one component more than another
• Observation = 0.5 x 2 = 1.0 (weighted twice)
• Collaboration = 0.4 x 0.5 = 0.2 (weighted half)
• Behavior = moderate engagement = 0.6 (no change)
• TES = 1.0 + 0.2 + 0.6 = 1.8/3.0 = 0.6
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Interpreting TES
@BentleyUXC | @UXMetrics
0.1 0.2 0.3 0.4 0.60.5 0.7 0.90.8
Big Problem Fully EngagedOpportunity to ImproveWork to Do
Take it Seriously
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Improving TES
• Formalize the process
• Bake into design schedule
• Senior-level support
• Correlate metrics to individual/team performance reviews
• Free food is no longer viable!
@BentleyUXC | @UXMetrics
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
How Would You Measure Team Engagement?
@BentleyUXC | @UXMetrics
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Value of TES
• Easy to collect and measure
• Track team engagement over time
• Report with other UX metrics
• Roll into larger UX maturity score
• Is this something you would use?
@BentleyUXC | @UXMetrics
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Limitations of TES
• Not a validated metric, no normative data
• Engagement is more than participating in usability testing
• Not all observers are the same, not all sessions are the same
• Potential political ramifications
• Other dimensions of engagement are hard to measure
@BentleyUXC | @UXMetrics
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Delivering Value
@BentleyUXC | @UXMetrics
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Questions?
Bill Albert
Executive Director
Bentley University User Experience Center
www.bentley.edu/uxc
@BentleyUXC | @UXMetrics