Big Data sessie Maurits Kaptein
-
Upload
infonl -
Category
Technology
-
view
117 -
download
2
description
Transcript of Big Data sessie Maurits Kaptein
![Page 1: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/1.jpg)
Big Data @ PersuasionAPI
Maurits Kaptein
Co-founder / Chief Scientist Science Rockstars
www.persuasionapi.com
![Page 2: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/2.jpg)
Big Data?
Big data is not really defined.
“Datasets that are larger than ‘common’ machines can handle”
![Page 3: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/3.jpg)
What I will and won’t talk about
Yes: What are the challenges that are associated with big data
Yes: How did we solve them in PersuasionAPI (high level)
No: Algorithms
No: Infrastructure / Technical details
![Page 4: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/4.jpg)
3 Key Challenges
• Focus on meaningful data• So much data, but which is useful?
• Move from Analytics to Advice• No reports in hindsight but direct responses
• Inability to run analysis on all of the data• Need for summaries / online learning
![Page 5: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/5.jpg)
Challenge 1:What is meaningful?
![Page 6: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/6.jpg)
What is meaningful
Depends obviously on what your aim is as a company.
We help companies increase conversion (Click-through, sales, etc.)
![Page 7: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/7.jpg)
Persuasion plays a big role:
![Page 8: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/8.jpg)
8Beta Launch presentations Q2 2012
6 Principles of Persuasion
8
8
![Page 9: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/9.jpg)
9Beta Launch presentations Q2 2012
Persuasion Online
9
![Page 10: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/10.jpg)
Should we use all the strategies we can think off?
At the same time?For the same product?
![Page 11: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/11.jpg)
Comparing many strategies with single strategies
![Page 12: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/12.jpg)
Should we use all the strategies we can think of?
No, we are better of selecting a specific one.
![Page 13: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/13.jpg)
Should we use the same strategies for everyone?
Strategies not equally effective for everyone?
Large differences based on personality traits
![Page 14: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/14.jpg)
14Beta Launch presentations Q2 2012
2 Scenarios:
14
Effect of using a strategy
Avera
ge
Individuals
+-
Individuals
Effect of using a strategy
Avera
ge
+-
![Page 15: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/15.jpg)
Should we use the same strategies for everyone?
No, people are distinct in their reactions to different strategies.
![Page 16: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/16.jpg)
Challenge 1:Meaningful data
Identify Persuasive Strategies
Select distinct strategies
Adapt to individuals
Data:{ userId : “zcvx2312”, strategyId : 4, implementation: 32, estimatedSucces : 0.23, certainty : 0.013}
![Page 17: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/17.jpg)
Challenge 2:Moving from analysis to advice
![Page 18: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/18.jpg)
Choose not to produce reports after logging responses…
But rather summarize all the data to be available for direct recommendations.
![Page 19: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/19.jpg)
19Beta Launch presentations Q2 2012
Persuasion Profile:
•A persuasion profile is a collection of the estimates of the effect of persuasion principles for each individual user
Normal Page:
A1 (Scarcity):
A2 (Authority):
A3 (Consensus):
Effect
19
Normal Page:
A1 (Scarcity):
A2 (Authority):
A3 (Consensus):
Effect
![Page 20: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/20.jpg)
20Beta Launch presentations Q2 2012
We log the success of each attempt
• Based on the dynamic image and the link we can monitor the success of each page served to a user.
• We will keep updates of the average performance of your served page variations, and of the performance for each client.
Normal Page:
A1 (Scarcity):
A2 (Authority):
A3 (Consensus):
Effect
20
![Page 21: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/21.jpg)
21Beta Launch presentations Q2 2012
We improve the personal profile
• Based on the response of each client we will update our advice for that user
• The new advice is a combination of the response of that client, as well as that of other clients
Normal Page:
A1 (Scarcity):
A2 (Authority):
A3 (Consensus):
Effect
21
Normal Page:
A1 (Scarcity):
A2 (Authority):
A3 (Consensus):
Effect
![Page 22: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/22.jpg)
22Beta Launch presentations Q2 2012
User navigates, we improve
And so on, for each individual client...
Real time analytics is most effective in predicting behavior
Normal:
A1:
A2:
A3:
Effect
First page served:
Normal:
A1:
A2:
A3:
Effect
Second page served:
Normal:
A1:
A2:
A3:
Effect
Third page served:
22
![Page 23: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/23.jpg)
23Beta Launch presentations Q2 2012
Competing Principles
23
![Page 24: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/24.jpg)
24Beta Launch presentations Q2 2012
Example of adjusted page
1: Log Client ID (e.g. via dynamic image, cookie, etc)
2. Link(s) to log success of the Sales Strategy
3. Hooks to log non-responsiveness to a Sales Strategy
24
![Page 25: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/25.jpg)
Challenge 2:We provide “advice” stating which Strategy to Use for your current customer.
In between page views…
![Page 26: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/26.jpg)
Challenge 3:How do we deal with all the data?
![Page 27: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/27.jpg)
Problem 1: Impossible fitting to all of the data in memory
Move fully to “online” learning:Handle datapoint for datapoint
Do not focus on ( theta | data ) but rather on ( theta | prior(s) )• Summarize all meaningful info in the priors.
Find out what data you need and don’t need to make an impact on the bottom line.• E.g. no demographic data
Use M/R jobs for re-estimating
![Page 28: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/28.jpg)
Problem 2: Individual level estimates are needed fast
Use hierarchical models:Aggregated level => Input for new users
User level => Start model for known users
Apply shrinkage Link the two levels
Use user-level model in isolation if necessaryAnalytical updates thus very fast.
![Page 29: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/29.jpg)
Challenge 3:How do we deal with all the data:
Use online learning and split different levels of the model
![Page 30: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/30.jpg)
Slide with the towell example
30Beta Launch presentations Q2 2012
Results
30
Increase in email click through: >100%(at the 5th reminder)Increase in e-commerce revenue:
>25%
![Page 31: Big Data sessie Maurits Kaptein](https://reader034.fdocuments.us/reader034/viewer/2022051412/54c76b434a79597a648b461f/html5/thumbnails/31.jpg)
My Big Data considerations:
Focus on meaningful data: Persuasion at an individual level.
Move from analytics to real time response: Provide real-time advice
Inability to analyze all of the data: Use online learning and hierarchical models.