ITERATE. CALCULATE. COMMUNICATE.

CLOSING THE GAP BETWEEN MEASURE + LEARN

LinkedIn, Medium & Twitter: @laurex

LAURE PARSONS SENIOR PRODUCT MANAGER

“THE GOAL OF BUILD-MEASURE-LEARN IS NOT TO BUILD A FINAL PRODUCT TO SHIP OR EVEN TO BUILD A PROTOTYPE OF A PRODUCT, BUT TO MAXIMIZE LEARNING THROUGH INCREMENTAL AND ITERATIVE ENGINEERING”

Steve Blank, author of The Four Steps to the Epiphany

BE ACTIONABLE!

1. WHAT SHOULD YOU TRACK? 2. WHAT MAKES A METRIC “GOOD?”

3. HOW CAN YOU GET ALIGNMENT WITH YOUR TEAM AND LEADERSHIP?

4. WHAT SHOULD YOU DO TO ENSURE YOU LEARN?

‣ CASE STUDY: BUILDING A NEW FEATURE!

ACQUISITION ACTIVATION REFERRAL RETENTION REVENUE

AARRR: PIRATE METRICS

ACQUISITION ACTIVATION REFERRAL RETENTION REVENUE TEAM

AARRRT: TEAM METRICS

THE ONE METRIC

From Lean Analytics by Benjamin Yoskovitz & Alistair Croll

UX TESTS MVP A/B TESTSCLASSIC LEAN MEASUREMENT

WHO, WHAT, WHEN, WHERE, WHY & HOW

▸WHO CUSTOMERS ARE

▸WHAT CUSTOMERS DO AND/OR TRY TO DO

▸WHEN DO USERS DISENGAGE

▸WHERE CUSTOMERS COME FROM

▸WHY CUSTOMERS CONVERT

▸HOW MUCH YOU CAN BUILD

GOOD METRICS

GOOD METRICS

COMPARABLE

RATIOS & RATES

GOOD METRICS

UNDERSTANDABLE

GOOD METRICS

ACTIONABLE

BE ACTIONABLE!

TWO TYPES OF ACTIONABLE DATA▸“Accounting” metrics can make your

predictions more accurate

▸“Experimental” metrics help optimize product, pricing, or market. Agree on what will change before you collect the data.

“IF YOU MEASURE SOMETHING AND IT’S NOT ATTACHED TO A GOAL, IN TURN CHANGING YOUR BEHAVIOUR, YOU’RE WASTING YOUR TIME.”

LEAN ANALYTICS by Ben Horowitz and Alistair Kroll

BE ACTIONABLE!

ALIGNMENT & COMMUNICATION

OKRS

WHO OWNS THE DATA?

WHAT ACTIONS WILL YOU TAKE?

WHO NEEDS TO KNOW?

HOW WILL THEY KNOW?

THE DATA TOOL YOUR TEAM WILL ACTUALLY USE

‣A single source of truth

‣Company-wide common understanding and definitions

‣Appropriate permissions

‣Transparency

‣Collaboration via shared data and commenting

‣Automated integration from your tools

‣Sharing in Slack, where your team is already talking

‣Dashboards to track the important things for each team

‣One-click reports to share with management and stakeholders

CASE STUDY

CASE STUDY QUESTIONS

DISCOVERY & LEAN CUSTOMER DEVELOPMENT▸ Based on our problem

hypothesis, what percentage of the 6-10 interviews we conduct confirm what we think is the problem?

CASE STUDY QUESTIONS

MVP

▸ Solves a problem in the simplest way possible

▸ Could actually be better than the intended product, with “concierge onboarding"

▸ Asks a question, which should be clear and measurable.

CASE STUDY QUESTIONS

EFFORT TO BUILD▸ How much dev effort is

anticipated in points or cost per developer hour?

▸ Reduce scope to the most essential parts of the feature. Build “nice to haves” when you have signal

▸ Track velocity, throughput, cycle times to anticipate & understand building costs

EFFORT TO BUILD

CASE STUDY QUESTIONS

WHO CARES?▸ Beta test the feature with customers

who’ve requested it and/or with freemium or opt-in customers

▸ If you have a large enough customer base, survey your customers with a tool like Intercom or Qualaroo when they use the feature

▸ Measure feature adoption in Mixpanel or other event-based tools. See how often customers try the feature and how many times they re-use it.

CASE STUDY QUESTIONS

COSTS BEYOND DEVELOPMENT

▸ Track support requests tagged with the feature name

▸ Track marketing spend on launching the feature

▸ Track escaped bugs

CASE STUDY QUESTIONS

THE BOTTOM LINE

▸ Do a cohort analysis of customers before and after the feature was introduced.

▸ Track number of signups, churn, and referrals.

▸ If the feature is plan-gated, you can also track by pricing segments.

WHAT ACTION FOLLOWS?

COMMUNICATION CHECKLIST

ARE YOU COMMUNICATING?▸ Does your whole team and company know why

you are tracking these metrics?

▸ Do you have agreed-upon actions based on the results?

▸ Do you have to track down people to get the data?

▸ Does someone own the metric?

DON’T STOP NOW!

LEARN MORE

▸Lean Analytics by Benjamin Yoskovitz & Alistair Croll

▸Getting Started with OKRs & Metrics Guide

▸The School of Little Data

▸The AARRRT of Product Metrics

THANK YOU!