The story of the Product Growth team at Atlassian

Ilias Flaounas Sr. Data Scientist

!!!!

Web Analytics Meetup, Sydney 14 October 2015

Atlassian - Trivia• Atlassian is an Australian enterprise software

company specialised in products that support project management and collaboration.

• ~13 years old

• >1.5K employees

• Headquarters in Sydney

• Best place to work in Australia (2014 & 2015)(The Australian Financial Review)

Key Requirements:!

• Support from upper management!

• Some minimum critical mass of customers!

• Serious investmentin infrastructure, people,time

Product Growth Team• 18 members (from initially 6 members

on 2013)

• Main mission: Increase Active Users!

• How? * Discover value in behavioural data * Experiment* Suggest improvements!

• Champion data-driven decision making

Reed Johnson

Ilias Flaounas

Stephen Lee

Shaun Clowes

Graeme Smith

Tim Garnsey

Fardin Sarker

Theo Voronov

Herman Chow

Leo Balan

Rob Sangster

Andrew Wakeling

Martin Smyrk

Houda Chehab

Lakshan Perera

Isabelle Flores

Antony Russel

Benjamin Humphrey

Where do we fit in the org? A kind of stretched analogy…

Patient User

Illness Bad User Experience

Medical doctor Product manager

Diagnosis Interviews, surveys…

Treatment UX modification

Medical researchers Growth teamObservations Data mining

Medical trials A/B tests

Ethics Privacy PolicyEvidence-based

medicineData-driven

development

The Challenges• Build infrastructure!

- break data silos - replace existing “solutions” - improve data quality - add instrumentation

We analyse the behavioural dataTimestamp Instance User Event Attributes

1356958862 abcd2134abcd AB12FF89 Login1356968863 abcd2134abcd DA99A11 Login1356978862 abcd2134abcd AB12FF89 dashboard view1356988862 abcd2134abcd DA99A11 dashboard view1357884442 abcd2134abcd FF12FF89 view issue {issueID = 1234}1357956862 abcd2134abcd AB12FF89 create issue {issueID = 1000}1357957862 abcd2134abcd DA99A11 view issue {issueID = 1000}1357958862 abcd2134abcd DA99A11 comment issue {issueID = 1000}1357959862 abcd2134abcd AB12FF89 create issue {issueID = 1001}

… … … … …

Analytics Infrastructure (v.2)On 2014:!

~3-4TB overall (highly compressed)!

800GB per day produced (uncompressed)

• Build infrastructure!- break data silos - replace existing “solutions” - add instrumentation - improve data quality

• Cultural change & Education!- help people ask the correct

questions!- challenge gut feelings!- “analytics is counting”

The Challenges

Help people ask the correct questions…

Help people ask the correct questions…

Report 1 Report 2 Report 3 Report 4 Report 5 Report 6 Report 7 Report 8 Report 9

No usage

No usage Low usage

No usage Low usage Medium usage High usage

Deal with gut feelings & common sense…

How Common Sense Fails Duncan Watts, 2011

κοινὴ αἲσθησις Aristotle, 384–322 BCE

Correlation & Causality

• Source: P. Chopra, The ultimate guide to A/B testing, http://www.smashingmagazine.com, 2010

Voting time: Raise your Left or Right hand for the version that worked best; No hand if you believe there’s no difference

Subject: %%first_name%%, assemble the

bitbucketeers!

Subject: Grab the Bitbucketeers...4, 3, 2, 1

LAUNCH!

Voting time: Raise your Left or Right hand for the version that worked best; No hand if you believe there’s no difference

Voting time: Raise your Right hand for success; Left hand for failure; No hand if you believe there’s no difference

Voting time: Raise your Right for success; Left hand for failure; No hand if you believe there’s no difference

Voting time: Raise your Right hand for success; Left hand for failure; No hand if you believe there’s no difference

Deal with “Analytics is counting”…

X. Fu & H. Asorey, “Data driven product Innovation”, Tutorials, KDD 2015

We are completing a transition…

• Phase A: Data scientists as Owners • Phase B: Data scientists as a Service • Phase C: Data scientists as Partners

Take away message

• Becoming data-driven is hard

• Cultural change is the key challenge - More difficult than the engineering part

• The transition is a marathon not a sprint

• There’s no common sense, thus potential is high

Thank you!