Building a data driven business

Konstantin Savenkov, CEO, Intento

Konstantin Savenkov, PhDChief Research Officer, Zvooq COO, Bookmate and Dream Industries now Founder & CEO

Turned a number of companies data-driven

• B(2B)2C content services

• edutainment

• ad tech• commercial space management

• consulting: online2offline, UGC, biotech

Bookmate, Zvooq

Theory&Practice, Exchanges

Unisound

DI Telegraph

@ksavenkov

Without data, people take

decisions based on biases and

beliefs@ksavenkov

…data replaces beliefs and provides a competitive

advantage@ksavenkov

If you’re competitor-focused, you have to

wait until there’s a competitor doing something.

Being customer-focused allows you to be more

pioneering.Jeff Bezos Amazon

@ksavenkov

…data helps to be customer-centric

@ksavenkov

Information is the oil of the 21st century, and analytics is the combustion engine.

Peter Sondegaard Gartner

@ksavenkov

…check the gas quality and do the maintenance

@ksavenkov

Half of the money spent on advertising

is wasted; the trouble is I don’t know which half.

John Wanamaker or William Lever@ksavenkov

…channel attribution and ROI will tell you

@ksavenkov

The average conversion rate in the United States whether you are selling

elephants or iPods, will be 2%

(с) Авинаш Кошик@ksavenkov

…data get you above the average

@ksavenkov

MasteryBusiness Metamorphosis

Data AwareData Monetisation

Data-drivenBusiness Optimisation

AnalyticsBusiness Insights

Data-Driven Maturity Index

Collect DataBusiness

Monitoring

@ksavenkov

@ksavenkov

To be data driven, it’s not enough to

collect and mine data. You need the culture.

@ksavenkov

Don’t optimise for data! Optimise for learning!

@ksavenkov

All initiatives must improve KPI

Formulate hypotheses

Run experiments@ksavenkov

Data validate hypotheses

behind your own decisions.

Data does not make decisions for you!

@ksavenkov

DATA

CulturePeople

ProcessTechnology

Analytics Business model

PlanningIterations

KPI

Hypotheses

Experiments

Changes

@ksavenkov

ISetting Goals

and KPI

@ksavenkov

Looks simple

MRR MarginMAU

but there are issues

@ksavenkov

IT’S UNCLEARwho’s responsible?

how to imrove?

how does that relate with resources spent?

(besides CEO)

(besides “work better”)

(when most of the business processes are automatic)

@ksavenkov

KPI Trees

MAU

New Loyal Returned

Traffic Conversion Retention Reactivation

@ksavenkov

KPI TreesMRR

MRR stable net new MRR

list MRR

MRR upgrades

MRR downgrades

New clients

MRR expansion

new MRR

Reactivation

direct sales

@ksavenkov

KPI Trees

Margin

LTVCAC

COGSARPU

Lifetime

Commissions

@ksavenkov

ADVANCED MODE

Ratio Analysis

Cohort Analysis

@ksavenkov

HELPS TOIdentify bottlenecks

Define roles and responsibilities

Generate ideas

Measure impact of initiatives

@ksavenkov

IIMeasure Impact

of Initiatives

@ksavenkov

For planning, groups tasks in Epics that matter

Track results of launched Epics

A culture of data success learning

@ksavenkov

Case studyMarketing project were tracked until launched

Study of a half-year long cross-promo campaign discovered a significant loss via COGS.

We’e identified key mechanics that led to the loss.

We’e adjusted the mechanics, re-negotiated with partners on the ongoing campaigns.

PROFIT@ksavenkov

IIIHypotheses and

experiments

@ksavenkov

It doesn’t work this way:

marketing campaign

@ksavenkov

impact!

product feature launched

billing failure

pupils back from holidays

AppStore featuring

people back from vacations

@ksavenkov

It doesn’t work this way:

marketing campaign

impact!

product feature launched

billing failure

pupils back from holidays

AppStore featuring

people back from vacations

@ksavenkov

It doesn’t work this way:

marketing campaign

impact!

DIFFERENT CHANNELS CONVERT WITH A

DIFFERENCE IN ORDER OF MAGNITUDE

ALSO:A lot of Epics that shouldn’t have been launched at all

A culture of praising a random success and explaining failures by external causes

Impossibility to learn by mistakes

@ksavenkov

formulate measurable hypotheses

carefully plan experiments

define a condition of success/failure prior to implementation

data collection and attribution, split-testing, controlled variables,

statistical significance

demonstrate explicit risks use models built on past data

prioritisation aid

and how the failure affects the roadmap

ability to prove multiple hypotheses simultaneously

@ksavenkov

Case StudiesEstimating mechanics of marketing projects before they launched

Product improvement that results in 2.5x conversion, 2x lifetime

Split testing of targeting and creative materials for ad campaigns, resulting in conversion 2-3 times higher than organic

Immediate increase of “conversion” from all initiatives to successful ones

This approach is behind all conclusions in these slides

lots of failed experiments success despite of all external factors

NO OTHER WAY@ksavenkov

Case Study: Recommender system for Conversion

• Hypotheses to prove:1. There’re enough users who will use RS output2. Their conversion will be above average

• A/B testing is the only way:– different channels convert with up to 20x difference– current traffic mix is unpredictable and hard to control

in case of app installs

• Do pilots:– Run with limited resources, then extrapolate and decide

if run full-scale

Case Study: Recommender system for Conversion

Let’s look at the economics

• In case of using a third-party RS on a CPO basis, in this case the CPO is limited by $0.14 (actually, much less)

• In case of a flat fee of $1000**/month, this is feasible starting from 7143 new subscribers/month, or $35K of marketing budget.

* CAC and marketing budget are model data** some arbitrary number

IVLeading indicators

@ksavenkov

Make decisions on monthly KPI or

finished projects is like going backwards

@ksavenkov

Look ahead or at least watch your step, not

backwards

Make the data work for you

@ksavenkov

Daily indicators

Incremental indicators

Leading indicators

Predictive models

spot problems and anomalies just in time

example: baremetrics.io

a perfect input for inbound marketing

accurate goals and perfect financial planning

@ksavenkov

Case StudySome loyal subscribers churned away for 2-3 months, to come back later. It was historically attributed to holidays and other external factors.

The daily indicators have shown that all churned users subscribed on weekdays. What a riddle!

It turned out that on weekdays the code base is frequently deployed to the production server, flushing the message queue of subscription renewals.

The fix increased a lifetime of paying users by 20%

@ksavenkov

Case StudyPredicting a lifetime for users that registered right now (10% accuracy)

Accurate unit economics for contracts with B2B2C partners,

content providers, pricing@ksavenkov

Case Study

A probabilistic model for segmenting users

Input data for chained communication

(inbound marketing)

@ksavenkov

Case StudyAccurate goals

loyal

new

churned

guaranteed growth?

inevitable stagnation!*

*unless KPI are increased@ksavenkov

Case StudyAccurate financial planning

Operational model

Marketing plan-fact

Financialplan-fact

Marketing budget

CAC, conversion

organic forecast

deal terms

unit economics deals forecast revenue forecasts

@ksavenkov

VBusiness process

automatisation

@ksavenkov

Hire another employee

or train another model?

@ksavenkov

Affects all business processes that scale linearly with a headcount:

customer supporteditorial office

content managementmarketing

@ksavenkov

Case StudiesUser base grows from 1M to 2M, doubling the headcount in customer support?

Implemented auto-reply using our knowledge base and smart templates for support engineers

A number of markets increased, adding more editors?Created an algorithm to provide a short-lists

based on a user behavior

An amount of UGC explodes, more content-managers?

Improving reduplication and computer aid based on the collected data

@ksavenkov

VIImproving the

Company Itself

@ksavenkov

Operational analytics - improve the process of KPI improvement

(productivity)

@ksavenkov

ITERATION

Case StudyImproving the Agile process

the expectations:

@ksavenkov

ROADMAPS DESIGN DEVELOPMENT QA SHIPPED

PLANNING

ITERATION

Case StudyImproving the Agile process

the reality:

@ksavenkov

INFLATING EFFORT

ROADMAPS DESIGN DEVELOPMENT QA SHIPPED

BACKLOG

NEW STUFF

TECH. DEBT

PLANNING

BUGS

UX

SOFTWARE OPS

URGENT STUFF

UNCLEAR DESIGN

UNCLEAR TECH

ITERATING

Data as an Asset

@ksavenkov

Case Stuies• Compare B2B2C deals through unit economics

• Estimate traffic quality for partner ad networks

• Data partnerships

• Targeted user communication

• Personalisation and recommender systems (the next slide)

• Bonus track: Investigating a large number of purchase returns for an internet retailer

@ksavenkov

Using Recommender Systemsand personalisation

CAC

LTV

ContentCosts

Marketing Expenses

New Customers

ARPU

Lifetime

Consumed Content Mix

Conversion

Retention

Reactivation

ExposedContent Mix

÷

×

* the recommendation fairy

*

Innovative business experiments, as there’s no

recipes

To pioneer, you need to iterate quicker than competitors

@ksavenkov

Data, models built over the data and experimental results is the

main asset created and exploited

by the innovative business

@ksavenkov

“THE UNFAIR ADVANTAGE”

Q&A

Konstantin Savenkov ks@inten.to