Post on 19-Oct-2014
description
Lies, Damned Lies &Vanity Metrics
Bernard Leong
A Simple Guide to Feedback, Analytics & Iteration in Product Development
1
A good theoretical physicist is one who makes all the mistakes as quickly as possible.
entrepreneur
3
Feedback Generation• How do we collect and generate user feedback?
• Qualitative: Surveys, Talk to customers, Create channels via social media (twitter/facebook).
• Quantitative: Google Analytics, Mixpanel metrics.
• User interface & experience testing: User education and perception of how product works.
5
• Qualitative - Break it into the following buckets:
• Spam/Troll & Insults - Ignore
• “Blue Sky” or “N=1” comments in the form of “you should build this or that” - Think about it but don’t act
• “N>50” comments on a feature or users demanding for a feature - Ascertain whether people really want it and then act.
Feedback Maintenance
Obviously, you need to find metrics to do that ...
6
Avoid Vanity Metrics• Number of hits.
• Number of page views.
• Number of visits.
• Number of unique visitors.
• Number of followers/friends/likes
• Time on site/number of pages.
• Emails collected
• Number of downloads
7
Element Function Relevant Metrics
Acquisition Generate attention through various means
Traffic, Mentions, Cost per Clicks, Search Results, Cost of acquisition,
Open Rate
ActivationTurn the resulting drive-by visitors into users who are
enrolled
Enrollments, signups, completed onboarding process, used the service
at least once, subscriptions
RetentionConvince users to return
repeatedly or exhibit stickiness behavior
Engagement, time since last visit, daily & monthly active use, churns.
Revenue Business outcomes depending on business model
Customer lifetime value, conversion rate, shopping cart size & click thru
revenue
Referral Viral or word of mouth invitations to other users
Invites sent, viral coefficient, viral cycle time.
9
How to identify key metricsAcquisition
Activation
Retention
Revenue
Referral
How do users discover you?
Do users have a frictionless or seamless first experience?
Do users return?
Do users want to pay you money?
Do users spread the word?
10
Constructing a FunnelInternet Marketing - Google Ads
200K impressions = 20K click thrus (10%)
User Signups20K click thrus = 2K signups (10%)
User create activity2K signups = 200 users return
with 5 actions (10%)
User pay for premium service = 20 users
Users send invites to other users = 2 users
How do users discover you?
Do users have a frictionless or seamless first experience?
Do users return?
Do users want to pay you money?
Do users spread the word?
11
Figuring out the customer lifecycle and working out the metrics which
you want to measure.
Case Study: SaaS services
13
References• Alistair Croll & Benjamin Yoskovitz, “Lean
Analytics”
• Jon Yongfook, “21 Actionable Growth Hacking Tactics”, http://yongfook.com/actionable-growth-hacking-tactics.html
• Ash Maurya: 3 Rules to actionable metrics in a lean startup: http://www.ashmaurya.com/2010/07/3-rules-to-actionable-metrics/
14