Dynamic A/B testing with AB/CD
-
Upload
memi-beltrame -
Category
Design
-
view
297 -
download
1
Transcript of Dynamic A/B testing with AB/CD
![Page 1: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/1.jpg)
Dynamic A/B Testing with AB/CD
Frontend Conference ZurichSeptember 2. 2016
Memi Beltrame
![Page 2: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/2.jpg)
A/B Tests
Buy Now! Buy for $1’000
1. Divide users in 2 groups
A B
![Page 3: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/3.jpg)
A/B Tests
Buy Now! Buy for $1’000
1. Divide users in 2 groups2. Each group gets a button
A B
![Page 4: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/4.jpg)
A/B Tests
Buy Now! Buy for $1’000
1. Divide users in 2 groups2. Each group gets a button3. See what works and what not
A B
![Page 5: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/5.jpg)
Example with 1000 users
Buy Now! Buy for $1’000500 500
![Page 6: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/6.jpg)
Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion
500 500
![Page 7: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/7.jpg)
Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion15 Sales 60 Sales
500 500
![Page 8: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/8.jpg)
Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion15 Sales 60 Sales$15’000 $60’000
500 500
![Page 9: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/9.jpg)
Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion15 Sales 60 Sales$15’000 $60’000
500 500
A difference of $45’000
![Page 10: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/10.jpg)
Finding out which button works
just cost you $45’000
![Page 11: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/11.jpg)
Is there a way to bring down that number?
![Page 12: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/12.jpg)
Dynamic A/B Tests
![Page 13: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/13.jpg)
2 Rules
Buy Now! Buy for $1’000
![Page 14: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/14.jpg)
2 Rules
Buy Now! Buy for $1’000
1. 10% get a button at random
![Page 15: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/15.jpg)
2 Rules
Buy Now! Buy for $1’000
1. 10% get a button at random2. 90% get the best button
![Page 16: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/16.jpg)
Find the best button
By tracking 3 values for each one
![Page 17: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/17.jpg)
Find the best button
1. Views
By tracking 3 values for each one
![Page 18: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/18.jpg)
Find the best button
1. Views2. Clicks
By tracking 3 values for each one
![Page 19: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/19.jpg)
Find the best button
1. Views2. Clicks3. Ratio of Clicks / Views
By tracking 3 values for each one
![Page 20: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/20.jpg)
How are Options Tracked?Start of Test
![Page 21: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/21.jpg)
How are Options Tracked?After first user
![Page 22: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/22.jpg)
How are Options Tracked?After 1000 users
Resulting in $111’000 for the test
![Page 23: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/23.jpg)
Finding out which button works
just cost you $9’000
![Page 24: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/24.jpg)
Finding out which button works
just cost you $9’000
That’s $36’000 less.
![Page 25: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/25.jpg)
AB/CD
![Page 26: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/26.jpg)
![Page 27: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/27.jpg)
A service to set up dynamic A/B tests that handles the math behind it.
![Page 28: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/28.jpg)
AB/CD
http://abcd.lessamess.ch
![Page 29: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/29.jpg)
Setup
![Page 30: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/30.jpg)
After Setup
1. Request an option via an API
![Page 31: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/31.jpg)
After Setup
1. Request an option via an API2. Store the option in a session
![Page 32: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/32.jpg)
After Setup
1. Request an option via an API2. Store the option in a session3. Report back to AB/CD when
an option performs
![Page 33: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/33.jpg)
API
![Page 34: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/34.jpg)
What you getResults in real time
![Page 35: Dynamic A/B testing with AB/CD](https://reader033.fdocuments.us/reader033/viewer/2022052418/58f065421a28aba37f8b45a9/html5/thumbnails/35.jpg)
Thanks!
Memi Beltrame@bratwurstkomet