Milion Dollar Impact Through Metrics, Analytics & A/B Testing
-
Upload
azhar-bandeali -
Category
Data & Analytics
-
view
79 -
download
1
Transcript of Milion Dollar Impact Through Metrics, Analytics & A/B Testing
![Page 1: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/1.jpg)
Million Dollar Impact Through Metrics, Analytics, & A/B Testing
![Page 2: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/2.jpg)
Hello!I am Azhar Bande-Ali
Software Engineer turned PM with a passion for people and data
You can find me at:@AzharBA
![Page 3: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/3.jpg)
In this talk
◎ Discuss web products at scale◎ Narrow approach due to time constraints◎ Principles can be adapted to your world
Feel free to ask questions
![Page 4: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/4.jpg)
![Page 5: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/5.jpg)
![Page 6: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/6.jpg)
![Page 7: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/7.jpg)
![Page 8: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/8.jpg)
![Page 9: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/9.jpg)
MetricsDefine your measurement criteria
![Page 10: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/10.jpg)
Top Down Metrics
Measure what your business cares about.
Everything else is vanity.
![Page 11: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/11.jpg)
Place your screenshot here
E-commerce WebsiteImprove yearly revenue from online sales
![Page 12: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/12.jpg)
Metrics for growing revenue
![Page 13: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/13.jpg)
Conversion Rate
Metrics for growing revenue
Drop-off Rate Return Rate
![Page 14: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/14.jpg)
Conversion Rate
% of unique sessions with a submitted order
Metrics for growing revenue
Drop-off Rate
For every page in the ordering flow: % of sessions that don’t graduate to next step
Return Rate
% of new users who repeat order
Compare each metric to your target/goal and industry average!
![Page 15: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/15.jpg)
AnalyticsMeasure your metrics
![Page 16: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/16.jpg)
OR
![Page 17: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/17.jpg)
Getting started with Analytics
◎ Needs code change - development cost◎ Measure:
○ Conversion rate of funnels and ○ Drop off rates on pages
◎ Event tracking for more intricate analysis
![Page 18: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/18.jpg)
Marry analytics to business goals
Conversion Rate
Drop off Rate
Return Rate
![Page 19: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/19.jpg)
Measure
Frameworks
◎ Funnels
◎ AIDA○ Awareness○ Interest○ Desire○ Action
![Page 20: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/20.jpg)
Measure
Frameworks
◎ Funnels
◎ AIDA○ Attention○ Interest○ Desire○ Action
Events
![Page 21: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/21.jpg)
Drop-offs
![Page 22: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/22.jpg)
ROI
![Page 23: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/23.jpg)
How to get buy in from your boss and partners
![Page 24: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/24.jpg)
$1,085,136How to get buy in from your boss and partners
![Page 25: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/25.jpg)
A/B TestValidate your hypothesis
![Page 26: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/26.jpg)
![Page 27: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/27.jpg)
HypothesisMaking “Continue” button more prominent will improve the checkout conversion rate.
![Page 28: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/28.jpg)
Variant 1Remove “Cancel Order” Button
![Page 29: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/29.jpg)
Variant 2Remove “Cancel Order” button and rename “Continue” to “Checkout”
![Page 30: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/30.jpg)
And the winner is..
![Page 31: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/31.jpg)
And the winner is....neither!
![Page 32: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/32.jpg)
Truth about A/B testing
Diminishing ReturnEven a successful A/B test variant will eventually start failing for a variety of reasons.
No WinnersIt is possible that none of your variants cause a big enough change to be declared winner
Success RateAccording to KissMetrics, only 1 out of 8 A/B tests have a valid winner
SignificanceSample being tested on should be big enough to represent population
Control GroupAlways show a ‘no change’ state to a control group to compare understand what would happen if you did nothing
RepresentationThe test should be conducted on the same group of users that are a sample of the actual user set
![Page 33: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/33.jpg)
Next Steps
Org SupportEnsure that your organization recognizes the value of the opportunity enough to prioritize it
IterateBe ready to come up with several hypothesis about which features create value for users
FailureEven the best teams have a 12.5% chance of success. Know that it isn’t easy and you’ll fail a lot
![Page 34: Milion Dollar Impact Through Metrics, Analytics & A/B Testing](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a648c217f8b9a76568b49f7/html5/thumbnails/34.jpg)