Pivotal: Data Scientists on the Front Line: Examples of Data Science in Action

1 © Copyright 2013 EMC Corporation. All rights reserved.

Pivotal Data Scientists on the Front Line: Examples of Data Science in Action

Getting to Know Your Customer with Big & Fast Data

Pivotal Data Science Team


Welcome – It’s a Pleasure to Meet You • The Launch of Pivotal

• Pivotal Data Science Team

• Getting to Know Your Customer - Meet your customer: Build Models - Learn more about your customer: More Data - Adapt to your customer: Dynamic Models

• Let’s Get Started: Pivotal Data Science Labs

• Q & A

A NEW PLATFORM FOR A NEW ERA


Pivotal, The New EMC Spin-out

Private Cloud

Public Cloud

Pivotal is building a new platform for a new era This platform enables customers

to build a new class of applications

That leverage Big and Fast Data

All this with the power of cloud independence


Introducing the Pivotal Stack

Cloud Storage

Virtualization

Data & Analytics Platform

Cloud Application

Platform

Data-Driven Application

Development

Pivotal Data Science Labs


Pivotal Services: Rapid Time to Value

Pivotal Labs: Quickly create and deploy new applications • Proven methodology to

remove risk and accelerate results

Pivotal Data Science Labs: A proven data science practice to accelerate analytics projects • Drive business value

through data analytics

Open Source Support: Collaborative and customer-driven open source support, services and co-development


Pivotal Data Science


Tell Me About Data Science What it is:

– Data preparation – Data exploration and visualization – Feature creation based on data and domain knowledge – Quantitative modeling & model validation – Scoring data

What it is not: – A set of tools – Application development


Platform-Driven Data Science Paradigm Shift

1. Modeling on more data

2. Rapid ingestion of new data

3. Re-use of valuable data

4. Faster model building

5. Scalable advanced modeling

6. Faster model refreshing

7. Faster data scoring


Pivotal Data Science Knowledge Development


Data Science Strategy

Point Model Development

Multiple Model Development

Transformation to “Predictive Enterprise”

Pivotal Data Science Labs


Getting to Know Your Customer Deeper Insights With Data Science


More Data Science Deeper Insights

Meet Your Customer

Learn More About Your Customer

Adapt to Your

Customer

Build Models More Data Dynamic

Models


The New Normal: “An Audience of One” DATA DEVICES

Media

Banks

Delivery Services

Marketers

Government

Individuals Employers

Data Users/Buyers

Analytic Services

Advertising

Catalog Co-ops

List Brokers

Websites

Information Brokers

Credit Bureaus Media

Archives

Data Aggregators

CONTENT

GOVERNMENT

PHONE/ TV

INTERNET AD

AGENCY

RETAIL


Targeting & Retention

Social Media Analysis

Campaign optimization

Data-driven Customer Analytics

Transaction History

Purchases

Clickstream

Customer Data

Unified data supporting re-usable predictive models

GB

TB

PB

Data Size



Meet Your Customer


Adapt to Your

Customer


Models


Who Are Our Customers? • One way of learning about

customers is to divide them into characteristic groups

• This is called segmentation

• Let’s take a look at a segmentation exercise Pivotal did with a large medical insurance company…


What Did We Have to Work With?

Product Sales Claims Data

Provider Information

Consumer Data

Population Served


Before – Random Clusters After – Cohesive Clusters

So What Did We Do With this Data?


New Clinics Neighborhood Clinics

Pirate Clinics Established Clinics

What Was the Outcome?


Churn Models

Micro-segmentation

Marketing Mix Model

Cross-Sell/Up-Sell Optimization

Lifetime Value Calculation

Consumer-Provider

Recommendation Engine

Where to Next?

Segmentation as

Foundational Analytics


•Improve understanding of customer

Objective:

•Existing EDW sources •New big data sources that capture customer demographics, such as the publicly available US Census

Data:

•Segmentation via k-means clustering

Data Science Methodology:

•Dramatically increase familiarity with makeup and behavior of customer base •Drive targeted marketing efforts •Lay foundation for higher-quality future models

Business Impact & Improvement:

Summary: Get to Know Your Customer by Building Data-Driven Models



Meet Your Customer


Adapt to Your

Customer


Models


Churn Models for Telecom Industry Goal

– Identify and prevent customers who are likely to churn.

Challenges – Cost of acquiring new customers is high – Recouping cost of customer acquisition high if customer is not retained long enough – Lower barrier to switching subscribers – With mobile number portability, barrier to switching even lower

Good News – Cost of retaining existing customers is lower!


Structured Features for Churn Models The problem is extensively studied with a rich set of approaches in the literature

These features are great, but the models soon hit a plateau with structured features!

Device Texting Stats Call Stats Rate Plans Customer

Demographics


Blending the Unstructured with the Structured

What other sources of previously untapped data could we use ?

Are our customers happy ? Where ? What segments ? What are the common topics in their conversations online ?


Sentiment Analysis and Topic Models

Sentiment Analysis Engine

(Classifier)

Structured Data: EDW

BETTER PREDICT LIKELIHOOD TO CHURN

Topic Engine (LDA)

Topic Dashboard

Unstructured Data

External Internal


Topic Clouds from Twitter - An Example Baby shower & Coupons: 13%

Convenience: 26%

Store experience: 13%

Misc: 32% Promotions, deals: 17%


•Improve accuracy of churn models by blending structured features with unstructured text

Objective:

•Existing structured features (call data records, device type, rate plans etc.) •Call center memos

Data:

•Sentiment Analysis and Topic Modeling


•Achieved 16% improvement in ROC curve for Churn prediction •Topic Models automatically identified common themes in call center memos •Laid foundation for Text Analytics


Summary: More Data to Drive Additional Customer Insights



Meet Your Customer


Adapt to Your

Customer


Models


State of Data at Telco Company Customer Segments

New Data Sources

Multi-Gadget Families Affluent Matures

Thrifty Families High Tech Singles

Budget Singles Seniors

Internet Deep Packet Inspection TV Consumption (Linear)

Video On Demand Consumption


Understanding Subscriber Behavior Native Services

Video On Demand TV Internet

Internet Devices

OTT Services

What is the level of engagement with Client’s products (TV, VOD, Internet)?

What are the patterns of device usage behavior?

What is the level of OTT engagement, by segment, and by bandwidth?

http://images.search.yahoo.com/images/view;_ylt=A0PDoKrx_ZxQMlEARk6JzbkF;_ylu=X3oDMTBlMTQ4cGxyBHNlYwNzcgRzbGsDaW1n?back=http://images.search.yahoo.com/search/images?p=pandora&n=30&ei=utf-8&fr=my-myy&tab=organic&ri=13&w=512&h=512&imgurl=wp.appadvice.com/wp-content/uploads/2010/10/Pandora-RadioLarge.jpg&rurl=http://appadvice.com/appnn/2010/10/jailbreak-pandora-hack-kills-skip-limits&size=33.9+KB&name=Pandora+Radio+is+one+of+the+greatest+apps+out+there.+This+service+allows+you+to+listen+to+an+unlimited&p=pandora&oid=05c6d4baf523c968148384275a002c39&fr2=&fr=my-myy&tt=Pandora+Radio+is+one+of+the+greatest+apps+out+there.+This+service+allows+you+to+listen+to+an+unlimited&b=0&ni=96&no=13&ts=&tab=organic&sigr=12bfksgbn&sigb=130293gs1&sigi=122mjl9qr&.crumb=hxLAwDIjGQy


Newly Identified Behavior-Based Segments

Sub

scri

bers

Moderates

OTT & Data Heavyweights

Portable OTT Entertainment Seekers

iPhone Heavy

Android Heavy

iPad Heavy

In-Home OTT Entertainment Seekers

In-Home Native Content Seekers

VOD Heavy

TV Heavy




Going Further: Crossing Behavior-Based Segments on Existing Customer Segments

Moderates

OTT & Data Heavyweights

In-Home OTT Entertainment Seekers

Portable OTT Entertainment Seekers - iPhone Heavy

Portable OTT Entertainment Seekers - Android Heavy

Portable OTT Entertainment Seekers - iPad Heavy

In-Home Native Content Seekers - VOD Heavy

In-Home Native Content Seekers - TV Heavy

Existing Segments

Newly Discovered Usage-Based Segments

Multi-Gadget Families

Affluent Matures

Thrifty Families

Budget Singles

Seniors

High Tech Singles

Customized Micro-Segments!


Driving New Business Value by Leveraging Data Science

Upsell and Cross-Sell New Product Offerings Data Monetization


•Combine existing models with new models derived from big data sources

Objective:

•Existing EDW sources •New big data sources that capture subscriber behavior, including machine generated sources such as DPI & VOD set-top box data

Data:

•Micro-segmentation via clustering


•Reduce operational and financial dependence on survey data •Lay foundation for data monetization •Generate tailored upsell & cross-sell opportunities •Real, customer behavior driven guidance for product & app development


Summary: Adapt to Your Customer with More Data Science


Let’s Get Started Transforming Your Business with Data Science


Process in New World Order


Pivotal Data Science Labs: Packaged Services

LAB PRIMER (2-Week Roadmapping)

• Analytics Roadmap

• Prioritized Opportunities

• Architectural Recommendations

LAB 600 (6-Week Lab)

• Prof. services

• Data science model building

• Ready-to-deploy model(s)

LAB 1200 (12-Week Lab)

• Prof. services

• Data science model building

• Ready-to-deploy model(s)

LAB 100 (Analytics Bundle)

• On-site MPP Analytics Training

• Analytics tool-kit

• Quick insight (2 weeks)

*Pivotal platform priced separately


Thank You Do you have any questions?


Pivotal Sessions at EMC World Session Presenter Dates/Times The Pivotal Platform: A Purpose-Built Platform for Big-Data-Driven Applications

Josh Klahr Tue 5:30 - 6:30, Palazzo E Wed 11:30 - 12:30, Delfino 4005

Pivotal: Data Scientists on the Front Line: Examples of Data Science in Action

Noelle Sio Tue 10:00 - 11:00, Lando 4205 Thu 8:30 - 9:30, Palazzo F

Pivotal: Operationalizing 1000-node Hadoop Cluster – Analytics Workbench

Clinton Ooi Bhavin Modi

Tue 11:30 - 12:30, Palazzo L Thu 10:00- 11:00 am, Delfino 4001A

Pivotal: for Powerful Processing of Unstructured Data For Valuable Insights

SK Krishnamurthy

Mon 4:00 - 5:00, Lando 4201 A Tue 4:00 - 5:00, Palazzo M

Pivotal: Big & Fast data – merging real-time data and deep analytics

Michael Crutcher

Mon 1:00 - 2:00, Lando 4201 A Wed 10:00 - 11:00, Palazzo M

Pivotal: Virtualize Big Data to Make The Elephant Dance June Yang Dan Baskette

Mon 11:30 - 12:30, Marcello 4401A Wed 4:00 - 5:00, Palazzo E

Hadoop Design Patterns Don Miner Mon 2:30 - 3:30, Palazzo F Wed 8:30 - 9:30, Delfino 4005

Pivotal: Data Scientists on the Front Line: Examples of Data Science in Action

Technology

Transcript of Pivotal: Data Scientists on the Front Line: Examples of Data Science in Action