Product Testing and Refinement
40
Product Testing and Refinement Chris Cera, CTO @ Arcweb July 9, 2014
-
Upload
chris-cera -
Category
Business
-
view
833 -
download
4
description
This presentation was given to Techgirlz annual entrepreneurial summer camp for middle school girls. This presentation was part of a 2-hour workshop on product design, testing, and refinement.
Transcript of Product Testing and Refinement
What is the name of this cookie?
Samoas or Caramel deLites?
What should we name our product?
The Scientific Method
The Scientific Method
1. Ask a question2. Do background research3. Construct a hypothesis4. Test your hypothesis5. Analyze your data and draw a conclusion6. Report the results
“Guess then test”
Validate your hypothesis
Talk to prospective customers
Currency
Currency
Currency
Currency
Currency
What should we name our product?
Hypotheses
1. “Samoas”2. “Caramel deLites”3. “Caramel DeLites”4. …5. …6. ...7. …8. …9. …10. …
How to test our hypotheses?
What should we name our product?
Book Funnel Analysis
• English speakers on earth – 100%• Read a business book – 45%• Discover the product ad – 30%• Click the product ad – 15%• Click the “Add to cart” button – 10%• Start the checkout process – 8%• Enter their credit card information – 7%• Enter their shipping information – 6%• Buy the book – 5%
Optimize
Hypotheses
1. “The 4-Hour work week”2. “How to work less”3. “How to get rich quick”4. …5. …6. ...7. …8. …9. …10. …
Example Advertisement
Budget Challenge: $???
• Given– We have 10 names to test– We will test each one on 10 different people– It costs $1 for each test (i.e. to buy an ad)
• How much money do we need?
Budget Challenge: $100
The 4-Hour Workweek (of course)
How else could this have been tested?
A/B Testing Examples
A/B Testing Examples
How will users want to sign in?
Sign In Funnel Analysis
• English speakers on earth – 100%• Discover the product – 50%• Download the product – 25%• Launch the product – 20%• Create a username – 10%• Sign into their account – 9%• Launch the product monthly – 5%• Launch the product weekly – 3%• Launch the product daily – 2%• Launch the product hourly – 1%
Sign In Page: Example
Sign In Page: Borrowing
Sign In Page: Borrowing
Sign In Page: Borrowing 3x
Sample Test Results
• Twitter converts 2% of people• Facebook converts 10% of people
A/B Testing Execution
• Given 3,000 people• A/B test for the first 300 (10% of them)– 6 different campaigns with 50 tests each
• Then test final 2,700 people with best campaign.
• Results– Worst campaign (Twitter) converted 2%– Best campaign (Facebook) converted 10%– Difference between 54 and 270 people.– For a $50 product, that is $10,800!
Budget Challenge: Implementation
• Implement Twitter login: $10• Implement Facebook login: $15• Implement LinkedIn login: $20
• But we only have $10!?! What do we do?
Group Work
Group Work
• Assignment [50 minutes]– List 1-2 of your funnels, then “guess and test”– Identify a step to optimize– List your hypothesis and test(s)– Test them on other group members, collect currency
• Group Presentations [30 minutes]– Steps in the funnel(s)– Optimization step(s)– List your hypothesis and test(s)– Summarize the results
