Data Science at Atlassian: The transition towards a data-driven organisation
40
Data Science at Atlassian The transition towards a data-driven organisation Dr Ilias Flaounas Capital Markets CRC, Sydney 13 May 2015
-
Upload
ilias-flaounas -
Category
Data & Analytics
-
view
146 -
download
0
Transcript of Data Science at Atlassian: The transition towards a data-driven organisation
Data Science at Atlassian The transition towards a data-driven
organisation
Dr Ilias Flaounas Capital Markets CRC, Sydney
13 May 2015
Atlassian - Trivia• Atlassian is an Australian enterprise software company
specialised in products that support project management and collaboration.
• ~12 years old
• Latest valuation at $US3.3 billion in 2014, (x8 since 2010).
• >1.5K employees
• Headquarters in Sydney
• Best place to work in Australia (The Australian Financial Review, 2014)
• ~43,000 enterprise customers
• Tens of thousands more own $10 starter and free community licenses
JIRA
JIRA
Key Requirements:!
• Support from upper management!
• Some minimum critical mass of customers!
• Serious investmentin infrastructure, people,time
Product Growth Team• Team was formed ~2 years ago
• Main mission: growth!
• Discover value in heaps of behavioural data!
• Champion data-driven decision making
Analytics Infrastructure (v.2)
~3-4TB overall (highly compressed)!
800GB per day produced (uncompressed)
We analyse events, i.e., user actionsTimestamp Instance User Event Attributes
1356958862 mySuperCompany John Login1356968863 mySuperCompany Anne Login1356978862 mySuperCompany John dashboard view1356988862 mySuperCompany Anne dashboard view1357884442 anotherCompany John view issue {issueID = 1234}1357956862 mySuperCompany John create issue {issueID = 1000}1357957862 mySuperCompany Anne view issue {issueID = 1000}1357958862 mySuperCompany Anne comment issue {issueID = 1000}1357959862 mySuperCompany John create issue {issueID = 1001}
… … … … …
The Challenges• Build infrastructure!
- break silos!- replace existing “solutions”!- add instrumentation!- improve data quality!- deal with catch-22…
The Challenges
• Educate!- “analytics is counting”!- help people ask the
correct questions
• Build infrastructure!- break silos!- replace existing “solutions”!- add instrumentation!- improve data quality!- deal with catch-22…
JIRA Agile!Which Agile report is used the most?
However, counting is hard…
Report 1 Report 2 Report 3 Report 4 Report 5 Report 6 Report 7 Report 8 Report 9
No usage
No usage Low usage
No usage Low usage Medium usage High usage
… and beyond counting…
Predicting conversionsHow many variables can you measure?
Top customersnone, n/a, home, self, test,
personal, atlassian, “-”, na, private, abc, personal use, freelance,
my company, student, testing, company, no company,
private, mycompany, a, no, test company, noname, testcompany,
self employed, individual, tester, me, “.” , independent, trial, myself, nocompany, xyz
Predicting conversions• Baseline set at 50% • Using only the sign-up information we have an
accuracy of 65%
The Challenges
• Convince- challenge gut feelings- demonstrate value
• Educate!- “analytics is counting”!- help people ask the correct
questions
• Build infrastructure - replace existing “solutions” - data quality - catch-22…
The argument of common sense
How Common Sense Fails Duncan Watts, 2011
κοινὴ αἲσθησις Aristotle, 384–322 BCE
• Source: P. Chopra, The ultimate guide to A/B testing, http://www.smashingmagazine.com, 2010
A/B Tests
• A/B tests allow a constant improvement of products in a measurable and predictive way
• Last year we deployed ~118 experiments.
• The volume of customers, evaluators, visitors, has become a limited and valuable resource
Subject: %%first_name%%, assemble the bitbucketeers!
Subject: Grab the Bitbucketeers...4, 3, 2, 1
LAUNCH!
Do A/B tests slow down development?
“Move fast with stable infra” Mark Zuckerberg
Facebook F8, 2014
“Pauca sed matura” i.e., “Few, but mature” Carl Friedrich Gauss
1777-1855
“Move Fast and Break Things” “Move Fast and Break Things”
Instead of a Conclusion: We simply recycle the scientific method from other fields, e.g., medicine…
Patient User
Medical doctor Product manager
Illness Bad User Experience
Diagnosis Interviews, surveys…
Treatment UX modification
Medical researchers Growth team
Observations Data mining
Medical trials A/B tests
Ethics Privacy Policy
Evidence-basedmedicine
Data-drivendevelopment
Reed Johnson
Ilias Flaounas
Stephen Lee
Shaun Clowes
Graeme Smith
Tim Garnsey
Fardin Sarker
Theo Voronov
Herman Chow
Leo Balan
John Jiang
Rob Sangster
Andrew Wakeling
Martin Smyrk
Kostas Servis
Houda Chehab
Lakshan Perera
We want YOU!
