Building Products Quantitatively
-
Upload
soren-harner -
Category
Technology
-
view
109 -
download
4
description
Transcript of Building Products Quantitatively
© 2003 – 2013 Bigcommerce Pty. Ltd.
Building Products Quantitatively in SaaS
Businesses
Soren Harner, Bigcommerce
Wednesday, 6 November 13
Stage 1 StartupBookstrapped
Stage 2 StartupStage 3 AcquisitionVenture-backed > $80MM
2000 Search & Analytics, Director Engineering
Stage 2 Startup2007 VP Engineering
2010 Founder & Datavore
Stage 2 StartupVenture-backed $35MM
VP Engineering
About me
2012
Wednesday, 6 November 13
Hill Climbing
Wednesday, 6 November 13
SaaS Business Levers• Cost to acquire customers (CAC)
• Average Revenue per User (ARPU)
• Churn
• Goal: Negative revenue retention (upgrades)
• Conversion (CVN) by channel
• Payback
• Life time value (LTV)
• LTV = ARPU/Churn
• LTV / CAC > 1 is success!
• Monthly Recurring Revenue (MRR)
• Magic Number http://blog.asmartbear.com/coc-cancellation-saas-metric.html
Wednesday, 6 November 13
LTV / CAC
0
0.3
0.6
0.9
1.2
Q1-10 Q2-10 Q3-10 Q4-10 Q1-11 Q2-11 Q3-11 Q4-11
Model“successful”
To IPO range is maybe 4 to 6 LTV / CAC today
Fine print: Like all metrics use to track, not as strategy
Wednesday, 6 November 13
Tested hundreds of versions
Wednesday, 6 November 13
Signup, upgrade, onboarding
• Invest in a great experience
• Test with usertesting.com
• Track the funnel:
• Number of trials
• Leads to Trials to Paid by weekly cohort
• Onboarding success
Wednesday, 6 November 13
Product UX Testing
Workshops(personas)
Concepting(sketches)
Prototypes Qualitative(Axure)
Quantitative(Loop11)
You can use an iPad at a Cafe and show keynote.Key point is to test mockups on people first.
Numbers don’t tell you how to improve.
Wednesday, 6 November 13
Feedback
Wednesday, 6 November 13
Measure Product Success
• Customer success
• Time to first sale
• Time to launch
• Trend monthly or weekly cohorts
• Engagement score during trial
• Churn and conversion predictive models
Wednesday, 6 November 13
Feature KPI• Before you build a
feature, define a success KPI
• e.g. Number of product reviews completed
• e.g. 30% fewer abandoned carts
• e.g. completing task more quickly
• Add instrumentation before and after
• Deploy to subset of customers to test
• Consider Mixpanel
Wednesday, 6 November 13
Split Test Important Things
• Persona analysis and customer interviews on new brand
• Route traffic p new, 1-p old (control)
• Measure effect on conversion with 95% confidence interval of a difference and ZScore
Wednesday, 6 November 13
https://developer.amazon.com/sdk/ab-testing/reference/ab-math.html
A/B Testing Report and significance
p +/- (1.65 x SE)
Wednesday, 6 November 13
Network Diffusion
Atlassian JIRA to OS communities.
SEO / SEM limitations — budget and effectiveness.
Wednesday, 6 November 13
Power law
http://www.hpl.hp.com/research/idl/papers/ranking/ranking.html
Think 80/20 rule.
Store revenue, traffic, disk usage, number products, etc.
Wednesday, 6 November 13
Continuous Delivery
• Feature flag everything and gently deploy
• Deploy daily — huge difference!
• Read 12factor.net (dev, staging, production)
• Monitor (i.e. NewRelic)
• Practice BDD & continuous integration
• Always have a recovery plan
Wednesday, 6 November 13
Ask permission, get feedback
Wednesday, 6 November 13
User Stories
Wednesday, 6 November 13
Commutative flow
Wednesday, 6 November 13
Startup Friendly Stack• Use Ruby Sinatra (or Clojure) +
CoffeeScript + Backbone + Handlebars on Heroku
• Nesta for CMS and learn textmate, Haml & Sass, see https://github.com/sharner/Workshop-Site
• HTML5 mobile first, responsive design
• Use Mailchimp, Wufoo, SurveyMonkey
• R for data (http://r-project.org) — treat as code, put in Github
• Log events to understand customer success (see Heroku Plugins)
Wednesday, 6 November 13
Peopleware
• Do learn to code simple website changes and growth hacking
• Learn some stats
• Do find a technical co-founder
• Don’t offshore
• Don’t churn people
• Do find designer who codes frontend and does user testing
Wednesday, 6 November 13