Delta Analytics Open Data Science Conference Presentation 2016
-
Upload
sara-hooker -
Category
Technology
-
view
64 -
download
1
Transcript of Delta Analytics Open Data Science Conference Presentation 2016
![Page 1: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/1.jpg)
Data Science For Good: Accelerating Mobile Learning in Kenya
![Page 2: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/2.jpg)
Delta Analytics | 2
About Us
Sara Hooker - Data Scientist at Udemy, Executive Director at Delta Analytics
Steven Troxler - Data Scientist at Stitch Fix, Project Lead at Delta Analytics
![Page 3: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/3.jpg)
Delta Analytics | 3
●Why data for good?●Who is Delta?●Technical Deep dive -
Eneza Education
![Page 4: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/4.jpg)
Delta Analytics | 4
Why data for good?
![Page 5: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/5.jpg)
Delta Analytics | 5
●Data revolution●Skills gap is larger than
ever
Why data for good?
![Page 6: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/6.jpg)
Delta Analytics | 6
New tools for data collection
Focus on accountability
Data Revolution
![Page 7: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/7.jpg)
Delta Analytics | 7
Spread of mobile phones across industries
Data Revolution
![Page 8: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/8.jpg)
Delta Analytics | 8
Public big data
Complex Problems Now Within Reach
![Page 9: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/9.jpg)
Delta Analytics | 9
Non-profits are pushing the hardest for more data:
●focus on accountability●desire to understand
impact
Data Revolution
![Page 10: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/10.jpg)
Delta Analytics | 10
Filling the data skills gap
Skills Gap is Larger Than Ever
![Page 11: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/11.jpg)
Delta Analytics | 11
Other organizations working in this space
Skills Gap is Larger Than Ever
![Page 12: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/12.jpg)
Delta Analytics | 12
And most importantly… you.
Skills Gap is Larger Than Ever
![Page 13: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/13.jpg)
Delta Analytics | 13
Who is Delta Analytics?
![Page 14: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/14.jpg)
Delta Analytics | 14
Delta Analytics collaborates with non-profits and other public service organizations to generate positive social impact through key data insights and management services. Driven by a passion for numbers and dedication to community engagement, we help public service organizations with all their data-driven needs.
Our mission, quite simply, is data for change.
Who is Delta Analytics?
![Page 15: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/15.jpg)
Delta Analytics | 15
Main constraints for Non-Profits● Resources (staff or specialized
skills) for data work● Infrastructure requirements● Longitudinal data collection ● Analyzing the data itself
Focus on helping other non-profits
![Page 16: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/16.jpg)
Delta Analytics | 16
19 projects with non-profits and social impact organizations
60 Fellows volunteering part-time over 3 years
$0.00 charged for services11 US and 8 International projects (Tanzania, UK, Kenya, and more)
Over 15,000 hours donated
Who is Delta Analytics?
![Page 17: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/17.jpg)
Delta Analytics | 17
Community
Engagement
Education
Economic Developm
ent
Environmental
Which sectors do we serve?
![Page 18: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/18.jpg)
Delta Analytics | 18
Where do Delta Fellows Work?
And many more!
![Page 19: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/19.jpg)
Delta Analytics | 19
What does the fellowship look like?
6 month engagement between non-profit and teams of 3 to 4 full-time data professionals
Monthly program-wide hackathons and ongoing social events
External speakers and trainings for ongoing technical growth and skill development
![Page 20: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/20.jpg)
Delta Analytics | 20
Technical Deep Dive -
![Page 21: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/21.jpg)
Delta Analytics | 21
Eneza Education enables access to education on a low cost mobile phone.
Eneza Education
![Page 22: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/22.jpg)
Delta Analytics | 22
Eneza Education
![Page 23: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/23.jpg)
Delta Analytics | 23
Incredible growth in number of students actively learning each month.
Eneza Education
![Page 24: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/24.jpg)
Delta Analytics | 24
simple text based model with instant feedback to questionspre smart phone portal =KES 10 for a weekly subscription ($0.098).
Eneza Education
![Page 25: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/25.jpg)
Delta Analytics | 25
Eneza Education
![Page 26: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/26.jpg)
Delta Analytics | 26
Gamification techniques to improve retention with students.
Eneza Education
![Page 27: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/27.jpg)
Delta Analytics | 27
Gamification techniques to improve retention with students.
Eneza Education
![Page 28: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/28.jpg)
Delta Analytics | 28
Eneza started in Kenya, branching out to Tanzania and Ghana.
Eneza Education
![Page 29: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/29.jpg)
Delta Analytics | 29
Eneza Education
Why is there a spike in activity towards the evening?
![Page 30: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/30.jpg)
Delta Analytics | 30
Students mainly use their parents’ cellphones to access Eneza.
Eneza Education
![Page 31: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/31.jpg)
Delta Analytics | 31
Eneza Education
Because students have to use parents phone, Eneza creates quizzes for parents to also stay engaged with the platform.
![Page 32: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/32.jpg)
Delta Analytics | 32
Eneza Education
The types of quizzes help us understand how Eneza is connected with the region of the world it solves. 81% of Kenyans are Christians.
![Page 33: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/33.jpg)
Delta Analytics | 33
Eneza Education
The types of quizzes help us understand how Eneza is connected with the region of the world it solves. 11% of Kenyans are Muslim.
![Page 34: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/34.jpg)
Delta Analytics | 34
Why are there big spikes in engagement towards the end
of the year?
Eneza Education
![Page 35: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/35.jpg)
Delta Analytics | 35
Activity peaks close to the Kenyan annual exams inOctober; it is at its lowest in January after Exams.
Eneza Education
![Page 36: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/36.jpg)
Delta Analytics | 36
Eneza Education
Kiswahili and English are the national languages in Kenya. English, Kiswahili and Mathematics are all obligatory
subjects for the secondary certification exam. This helps explain their popularity.
![Page 37: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/37.jpg)
Delta Analytics | 37
Lead you through one of the data products we built for Eneza.
Eneza Education
![Page 38: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/38.jpg)
Delta Analytics | 38
Student engagement at Eneza: a churn analysis
Eneza Education
![Page 39: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/39.jpg)
Delta Analytics | 39
Student engagement at Eneza: a churn analysis
Final product:
● We will build a model to help Eneza identify quizzes that are associated with high churn rates.– allows Eneza to quality-check the difficulty and
content of quizzes that look like they may not engage students
![Page 40: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/40.jpg)
Delta Analytics | 40
Student engagement at Eneza: a churn analysisQuestions we need to answer to build final product:
● How frequently do students “churn” (not come back to the site)?
● How does it relate to student’s passing quizzes?● What does the distribution of the number of quizzes
students take before churning look like?● Does “churn” behavior change as students take more
quizzes?● Does churn behavior vary as students take more
quizzes?
![Page 41: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/41.jpg)
Delta Analytics | 41
Student engagement at Eneza: a churn analysisFirstly let’s define churn:
If a user returns for another quiz, they are most likely to do so on the same day. Based upon this plot we can conservatively define churned students (students who never come back) as:
a student has churned if they not returned to the site within 30 days since the last quiz.
![Page 42: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/42.jpg)
Delta Analytics | 42
Student engagement at Eneza: a churn analysisLet’s understand the number of quizzes taken when students churned:
![Page 43: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/43.jpg)
Delta Analytics | 43
Student engagement at Eneza: a churn analysisLet’s understand how many quizzes students takebefore they churn: Most students who churn only
take a few quizzes before doing so.
This suggests the first few quizzes a student takes are very important for their experience, but also may be because Eneza is simply not a good fit for all students (natural dropoff).
![Page 44: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/44.jpg)
Delta Analytics | 44
Student engagement at Eneza: a churn analysisLet’s understand how many quizzes students takebefore they churn: Most students who churn only
take a few quizzes before doing so.
This suggests the first few quizzes a student takes are very important for their experience, but also may be because Eneza is simply not a good fit for all students (natural dropoff).
![Page 45: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/45.jpg)
Delta Analytics | 45
Student engagement at Eneza: a churn analysisWe can also think about this as a churn rate out of all students that take a number of quizzes.
![Page 46: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/46.jpg)
Delta Analytics | 46
Student engagement at Eneza: a churn analysisThe churn rate is highest at the first quizzes, but as students become more “sticky” and take more quizzes it falls to ~3%.
![Page 47: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/47.jpg)
Delta Analytics | 47
Student engagement at Eneza: a churn analysisHow is churn related to students passing quizzes?
![Page 48: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/48.jpg)
Delta Analytics | 48
Student engagement at Eneza: a churn analysisStudents who have churned are overwhelmingly more likely to have failed a quiz. This suggests difficulty of quiz is a big factor in understanding churn.
![Page 49: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/49.jpg)
Delta Analytics | 49
Student engagement at Eneza: a churn analysisConclusions based upon the finding that difficulty of quiz is linked to churn:
● Students may need positive feedback of a win (passing the quiz) to stay engaged
● One recommendation is that Eneza should think carefully about the first few quizzes a new student takes
![Page 50: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/50.jpg)
Delta Analytics | 50
Student engagement at Eneza: a churn analysis
Now, let’s isolate potentially problematic quizzes for Eneza. These are quizzes with very high churn rate.
We can start by just looking at the raw churn rate:
quiz churn rate=the number of students last quiz before churn/number of students who took quiz
![Page 51: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/51.jpg)
Delta Analytics | 51
Student engagement at Eneza: a churn analysisMost courses have very low high churn rate but there is a long tail which suggests some courses are problematic.
![Page 52: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/52.jpg)
Delta Analytics | 52
Student engagement at Eneza: a churn analysis
Problems with using raw churn rate:
● Simply reporting the quizzes with the highest churn rate is problematic because many quizzes have small sample size.
● We could address this sample size issue by setting a minimum sample size threshold before calculating churn rate. But this isn't what we want either, because the earlier Eneza knows that a quiz might have problems the earlier they can take action.
![Page 53: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/53.jpg)
Delta Analytics | 53
Student engagement at Eneza: a churn analysis
Our solution is to instead take an Empirical Bayesian approach:
● imagine that the churn rate for each quiz is some unknown probability, drawn from a distribution of probabilities
● then, fit a model to the data that estimates this distribution and uses it to smooth out the empirical churn rates based on how much data is available.
![Page 54: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/54.jpg)
Delta Analytics | 54
Student engagement at Eneza: a churn analysis
Applying this shrinkage technique we expect:
● the model predictions to look similar to the averages where there is good data
● we expect the predictions to be reasonably close to the overall average if the sample size is small (this is quite important, since the goal of the model is to be actionable, especially on new quizzes, which it will not be if the regularization is poor)
![Page 55: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/55.jpg)
Delta Analytics | 55
Student engagement at Eneza: a churn analysisWe can sanity check our results by looking at the churn prediction for small sample sizes:
If we just used the raw churn rate we would have predicted 100% churn. We have succeeded at reducing noise in the small sample size.
![Page 56: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/56.jpg)
Delta Analytics | 56
Student engagement at Eneza: a churn analysis
A look across a sample result shows model is performing well across different sample sizes.
![Page 57: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/57.jpg)
Delta Analytics | 57
Student engagement at Eneza: a churn analysis
So, given we are happy with our model. What are the most problematic quizzes?
![Page 58: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/58.jpg)
Delta Analytics | 58
Student engagement at Eneza: a churn analysis
Eneza will take a closer look at these problematic quizzes and try and understand why they are causing high churn with students.
![Page 59: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/59.jpg)
Delta Analytics | 59
Student engagement at Eneza: a churn analysis
This is a simple model which can be operationalized easily and helps Eneza take action as soon as quiz is live.
![Page 60: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/60.jpg)
Delta Analytics | 60
Thank you! Questions?
Fellowship Program 2017
![Page 61: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/61.jpg)
Delta Analytics | 61
We are working with Eneza to share all of our code publically. Watch our public repo for when it becomes available.
A tutorial of how to use the different libraries we used is available publically here: https://github.com/DeltaAnalytics/python_tutorials.
Eneza Education
![Page 62: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/62.jpg)
Delta Analytics | 62
Ways to be involved:
1. Fellow2. Mentor3. Guest Speaker
Fellowship Program 2017
Visit our website at deltanalytics.org for more information.
![Page 63: Delta Analytics Open Data Science Conference Presentation 2016](https://reader035.fdocuments.us/reader035/viewer/2022070515/587a33e21a28abdb1c8b5369/html5/thumbnails/63.jpg)
Delta Analytics | 63
Preview of some of the organizations currently in our application process
Fellowship Program 2017