Monetizing Big Data at Telecom Service Providers

Juergen UrbanskiTech Alpha

Hadoop Makes Shareholders Happy

The World's Largest Telcos are Driving Business Performance with Hadoop at the Center of an Enterprise-Wide Modern Data Architecture

Juergen Urbanski CEO, Tech AlphaBoard Member Big Data & Analytics, BITKOM (German IT Industry Association)

Agenda

• Telco Data Management Challenges

• Hadoop Business Value

• Data Lake Business Value

• Data Lake Reference Architecture

• 21 Telco Use Cases for Hadoop

– Network Infrastructure

– Service and Security

– Sales and Marketing

– New and Adjacent Business

3

Enterprise Data Management Challenges

Limited Insight:• Schema On Write• Data In Silos

Limited Scale:• Not Designed to Scale• Not Affordable at

ScalePhysicalInfrastructure

Presentation &Application

Data Access

Data Management

EngineeredSystems

Shared Storage Systems

OLTP OLAPTraditionalAnalytics

=

=

– 4 –

Business Value of Hadoop

Data Access Layer

Data Management Layer

Hadoop Core Capabilities:

Broader Insights:• Allows simultaneous access by and

timely insights for all your users across all your data

• Irrespective of the processing engine, analytical application or presentation

• Enabled by schema on read and enterprise-wide pool of data

Unlimited Scale:• Allows to acquire all data in its original

format and store it in one place, cost effectively and for an unlimited time

• Affordable and performing well into the 100+ petabyte scale

=

=

– 5 –

A New Approach for Broader Insights

HADOOPIterate over structure

Transform and analyze

Hadoop Approach• Apply schema on read• Support range of access patterns to

data stored in HDFS: polymorphic access

Batch Interactive Real-time

Right Engine, Right Job

In-memory

Traditional Approach• Apply schema on write• Heavily dependent on IT

Determine list of questions

Design solution

Collect structured data

Ask questions from list

Detect additional questions

Single Query EngineSQL

– 6 –

Compelling Economics Allow Scale

0 5 10 15 20 25 30 35 40

SAN

EDW / MPP

Engineered System*

NAS

HADOOP

Cloud Storage

Min

Max

Fully Loaded Cost per Raw TB Deployed

US$ ‘000s

Hadoop Provides Highly Scalable Data Storage at 5% of the Cost of Alternatives

36 to 180

20 to 80

12 to 18

10 to 20

0.250 to 1

0.1 to 0.3

* E.g., Oracle Exadata

– 7 –

5 Capabilities of Hadoop 2.x Enable the Data Lake

– 8 –

Data Integration & Governance

Integrate with existing systems.

Move data into, within and out of the environment

Security

Provide layered approach to

security

Operations

Deploy and manage a

multi-tenant, environment easily, using existing tools

where possible

Environment and Deployment Model

Run anywhere

Data Lake Functional Requirements

1 32

4

Data Access = Insight

…ask questions later (or in the moment)

Data Management = Scale

Store first…

Presentation & ApplicationEnable existing and new

applications

5

Data Lake Reference Architecture

– 9 –

DeploymentModel

Environment

Data Integration & Governance

Data Access

Security Operations

Data Management

Storage: HDFS (Hadoop Distributed File System)

Multitenant Processing: YARN(Hadoop Operating System)

Online

HBaseAccumulo

Real-Time

Storm

Others

Commodity HW

Linux Windows

Appliance

On Premise Virtualize

Cloud/Hosted

AuthenticationAuthorizationAccountabilityData Protection

acrossStorage: HDFS

Resources: YARNAccess: Hive, … Pipeline: Falcon

Cluster: Knox

Provision, Manage & MonitorAmbari

SchedulingOozie

Data WorkflowData Lifecycle

Falcon

Real-time and Batch Ingest

FlumeSqoop

WebHDFSNFS

Batch

MapReduce

Script

Pig

SQL

Hive

In-memory

Spark

Metadata ManagementHCatalog

Presentation & Application

Multiple Use Cases and Tools Run on Hadoop as a Shared Service

– 10 –

Hadoop 2.x: Shared Service = Data Lake

Hadoop 1.x:Dedicated Project Silos = Data Ponds

BU2 BU3BU1

Customer Intimacy

HbaseOpera-tional

Excellence

Lucene New BusinessStorm

Risk Manage-

ment

Map-Reduce

BU4

Customer Intimacy

Hbase

Opera-tional

Excellence

Lucene

New Business

Storm

Risk Manage-

ment

Map-Reduce

Enterprise-wide

• Poor resource management

• Limited governance

• Batch processing, no streams

Shared service operational benefits similar to infrastructure cloud Speed of provisioning and de-provisioning for capacity and users Fast learning curve and reduced operational complexity Consistent enforcement of data security, privacy and governance Optimal capital efficiency driven by scale and load balancing 

Value grows exponentially as data from more applications lands in one Hadoop 2.x data lake Marginal cost of retaining data is less than marginal value Able to run a broader range of analyses More data in one place usually leads to better answers Results is order-of-magnitude better insights

Data Lake Business Rationale

– 11 –

Technical and Business Drivers

– 12 –

Foundation for a modern data architecture

New data types

Sensors

Machine Generated

Geolocation

Documents, Email,

Voice to Text

Social Networks

Web Logs,Click Streams

Operational excellence E.g., Network Maintenance

Compliance & Risk Mgt.E.g., Fraud Reduction

Customer Intimacy

E.g., 360o View of Customer

New BusinessE.g., Data as a

Product

Business drivers

Network capacity planning Network upgrades Network maintenance Network performance management Network traffic shaping

21 Telco Use Cases for Hadoop

– 13 –

Use Case

Network Infrastructure

Function

Customer experience analytics Contact center productivity Field service productivity Data protection and compliance End-user device security

Service and Security

360-degree view of customer value Personalized marketing campaigns Upselling and cross-selling Next-product-to-buy (NPTB) Churn reduction

Sales and Marketing

New product development Actionable intelligence serving:

Advertisers Merchants/retailers Payment processors Federal governments Local governments

New and Adjacent Business

NetworkCareSales

New Biz

Hadoop in Network Infrastructure

– 14 –

Business Problem

Network capacity planning Network upgrades Network maintenance Network performance management Network traffic shaping

Hadoop is used to optimize the rollout of 4G coverage in time and space to match the likely pick-up in service revenue, allowing an operator to defer more than 10% of capex for the same resulting revenue.

Hadoop helped detect that only a small number of congested cable network nodes were responsible for the majority of churn, and could thus be prioritized for maintenance and upgrades.

Network function virtualization, software defined networking and unified all IP networks vastly increase the amount of machine and log data relevant for trouble shooting. Hadoop helps with root cause analysis and may even be used to reason on the data in real-time.

Value Realized

NetworkCareSales

New Biz

Network Infrastructure – Network Capacity Planning

– 15 –

Business Problem

The consumption of services and resulting bandwidth in a particular neighborhood may be out of sync with a telco’s plans to build new towers or transmission lines in that same neighborhood.

This leads to a mismatch between expensive infrastructure investments and the actual revenue from those investments.

Examples:

4G (LTE)

FTTC (fiber to the curb)

FTTH (fiber to the home)

One European carrier used Hadoop to optimize the rollout of 4G coverage in time and space to match the likely pick-up in service revenue, based on detailed cell tower traffic data of the last few years.

With their prior, less informed approach, they would have had to spend 10% more capex for the same outcome.

Value Realized

NetworkCareSales

New Biz

Network Infrastructure – Network Upgrades

– 16 –

Business Problem

Hadoop is used for targeted network maintenance and upgrades by cable companies.

One large US cable MSO was unsure how cable network congestion affects churn, and where exactly network upgrades produce the most incremental revenue.

The result was that only a small number of nodes were responsible for the majority of the negative customer experience, and could therefore be prioritized for upgrades.

Value Realized

NetworkCareSales

New Biz

Hadoop in Network Infrastructure – Network Upgrades Improve the Customer Experience

– 17 –

• Correlate network congestion and customer experience

• 11 different data sources

• 4m subscriber records, 12m work orders, 9m calls, 42m IPDRs, 20m Tivoli NPMs

• Finding: Only a few nodes responsible for most of the negative customer experience

Network Node

TNMP CMTS

Performance

Network

Sensors

IPDR Cable

Modem Usage

CompetitiveSpendData

HouseholdHousehold

Master Subscriber

Record

Marketing Demo-

graphics

Caller Experience

Work Orders

Mobile Devices

Customer

Premise

Equipment

Online

Transactions

Social Media

Interactions

SOURCE DATA

NetworkCareSales

New Biz

Network Infrastructure – Network Maintenance

– 18 –

Business Problem

Radio access networks provide the air interface between a mobile provider and the end user mobile devices.

Maintenance and repair of radio access networks poses substantial logistical challenges. In most countries, mobile networks cover more than 95% of a country’s surface area.

Many transmission towers are in remote and difficult to access locations.

In high-density areas, pico- and femto-cells optimize local coverage, but in turn require coordination with the building owner for maintenance.

Hadoop improves a provider’s ability to service equipment proactively, which is always cheaper and less disruptive than the replacement of equipment that has already failed.

Value Realized

NetworkCareSales

New Biz

Network Infrastructure – Network Performance Management

– 19 –

Business Problem

Existing network management platform meant to diagnose poor cellular service such as dropped calls or poor audio quality.

Overwhelmed by data volume, ingesting 10 million messages per second

Each analysis was limited to a 24-hour time window and only one-fiftieth the surface area of the United States.

Same customer issue may generate multiple support calls, but the operator’s team cannot see relationships between multiple variables across time.

Is the problem with the customer’s device? Is it their neighborhood or proximity to a tower? Is it because of how they use their phone?

With more history, they are able to explore root causes that they have never been able to identify by reviewing just one day’s data, allowing them to to improve cell phone service.

Value Realized

NetworkCareSales

New Biz

Hadoop in Service and Security

– 21 –

Business Problem

Customer experience analytics based on call detail records (CDRs)

Contact center productivity Field service productivity Data protection and compliance End-user device security

With Hadoop, one operator detected that 25% of callers were contacting the call center merely to have their late fees on the monthly bill waived. Clearly a case for call deflection to interactive voice recognition and online self-service.

Contact center agents had insufficient ways of diagnosing what was wrong with customers, leading to many unnecessary truck rolls. Hadoop helped avoid these.

3% of smartphones account for 10-15% of traffic because of malware (notably on Android phones) and some fair use violations. Hadoop helps detect that so operators can take remedial action.

Value Realized

NetworkCareSales

New Biz

Service and Security – Customer Experience Analytics Based on Call Detail Records (CDRs)

– 22 –

Business Problem

A typical mobile service provider generates >1 billion CDRs per day, ingesting millions of CDRs per second.

System holds >100 billion records, half a petabyte added every month!

Due to the cost of existing solutions, the data expires after 60 days

CDRs need to be analyzed and archived for compliance, billing and congestion monitoring.

Example: forensics on dropped calls and poor sound quality.

High volume makes pattern recognition and root cause analysis difficult.

Often those need to happen in real-time, with a customer waiting for answers.

With Hadoop the carrier can to retain some data for up to three years

Hadoop provides both a cost advantage – Hadoop provides storage 20x cheaper than enterprise-grade storage – and better insights.

Better analysis to continuously improve call quality, customer satisfaction and servicing margins.

Value Realized

NetworkCareSales

New Biz

Service and Security –Contact Center Productivity

– 23 –

Business Problem

A US-based mobile provider struggled with a combination of high costs but low customer satisfaction related to customer care.

An increasing share of support cases are related to mobile data usage and associated charges.

Traditionally, contact center agents did not have granular insights into a particular customer’s data usage, hence were unable to provide effective call resolution.

With Hadoop, one operator detected that 25% of callers were contacting the call center merely to have their late fees on the monthly bill waived.

The provider was able to off-load these cases to online self-service and interactive voice recognition.

Frees up the agents to focus on more valuable customer interactions.

The provider is now extending this solution to focus on issue resolution.

Value Realized

NetworkCareSales

New Biz

Service and Security – Field Service Productivity

– 24 –

Business Problem

A provider’s contact center agents had insufficient ways of diagnosing what was wrong with customers, leading to many unnecessary truck rolls.

In particular, the agents were not able to triage network vs. home-based problems accurately enough.

Therefore, technicians were dispatched to the customer premises for problems that reside within the network.

The provider was able to avoid a large number of “false positive” truck rolls.

With each truck roll costing about $150 fully loaded, the provider was able to save several million dollars already in the first year.

Value Realized

NetworkCareSales

New Biz

Service and Security –End User Device Security

– 26 –

Business Problem

A mobile operator needed to identify real-time malware threats from non-trusted application stores and contain their impact on customers.

3% of smartphones account for 10-15% of traffic because of malware (notably on Android phones) and some fair use violations.

Hadoop helps detect that so operators can take remedial action, thus eliminating a disproportionate share of network tonnage.

Options ranged from notifying an affected customer all the way to blocking certain URLs for the whole network.

Value Realized

NetworkCareSales

New Biz

Hadoop in Sales and Marketing

– 27 –

Business Problem

360-degree view of customer value Personalized marketing campaigns Upselling and cross-selling Next-product-to-buy (NPTB) Churn reduction

Telesales revenue increase by 50% by tracking competitors web-sites visited and counter offers to products searched

+20% conversion rate increase by optimizing and personalizing the path-to-transaction

$1.65 ARPU increase for 1 million customers boosts topline by $20 million per year.

Reducing cable subscriber churn (“cord cutting”). Every 100,000 subscribers equates to customer lifetime value of $1 billion

Churn model quality increase Price related churn down by 40%

Value Realized

NetworkCareSales

New Biz

Sales and Marketing – 360 Degree View of Customer Value

– 28 –

Business Problem

Telcos and cable companies interact with customers across many channels and points in time.

Data about those interactions is stored in silos.

Difficult to correlate data about customer purchases, marketing campaign results, and online browsing behavior.

Problem is exacerbated by recent acquisitions and a proliferation in the volume and type of customer data.

Merging that data in a relational database structure is slow, expensive and technically difficult.

Enterprise-wide data lake of several petabytes

360-degree unified view of the customer (or household) life time value based on usages patterns across time, products and channels.

Value Realized

NetworkCareSales

New Biz

Sales and Marketing – Personalized Marketing Campaigns

– 29 –

Business Problem

Marketers have long sought ways to tailor their marketing campaigns to the needs of each individual customer.

Telcos are uniquely positioned to deliver on that goal because mobile phones not only follow their owners everywhere, but also reveal a lot about their owners’ interests through browsing behavior and the applications present on the phone.

Telcos are looking for ways to mine that information.

Provider risked losing substantial revenue as prepaid customers were starting to switch to a competitor as a result of a particularly effective marketing campaign.

The provider used Hadoop to pinpoint those individual customers most at risk of churning, and then built a highly targeted campaign to retain the remaining customers in that segment.

A churn alarm system was established and revenue leakage was minimized.

Telesales revenue increase by 50% by tracking competitors web-sites visited and counter offers to products searched

+20% conversion rate increase by optimizing and personalizing the path-to-transaction

$1.65 ARPU increase for 1 million customers boosts topline by $20 million per year.

Value Realized

NetworkCareSales

New Biz

Sales and Marketing – Up-selling and Cross-selling

– 30 –

Business Problem

The provider needed to find an approach to upsell smart phones into a user base that was still largely on legacy feature phones.

The operator converted many hundred thousand feature phone users to smart phones with associated data plans.

Value Realized

NetworkCareSales

New Biz

Sales and Marketing –Next Product to Buy (NPTB)

– 31 –

Business Problem

As telco product portfolios grow more complex, there are ever more opportunities to sell additional services to the same customer base.

Many sales reps however are overwhelmed with that complexity and struggle to translate the breadth of the product portfolio into incremental sales.

Confident NPTB recommendations, based on data from all its customers, empower sales associates and improve their interactions with customers pre-transaction.

Value Realized

NetworkCareSales

New Biz

Sales and Marketing – Churn Reduction

– 32 –

Business Problem

A North American provider faced the following challenge: 50% of new customers churned off within 6 months of acquisition.

The average customer life time in this segment was 13 months, well short of the 18 months needed to break even.

The provider increased the “right” customer acquisitions by 27% and decreased subsequent churn in this segment by 50%.

Price related churn down by 40% Reducing cable subscriber churn (“cord

cutting”). Every 100,000 subscribers equates to customer lifetime value of $1 billion

Value Realized

NetworkCareSales

New Biz

Hadoop in New and Over-the-Top / Adjacent Businesses

– 33 –

Business Problem

New product development Actionable intelligence serving:

Advertisers

Merchants/retailers

Payment processors

Federal governments

Local governments Hadoop-as-a-Service

Telcos are well positioned to provide big data as a service to retail, hospitality and logistics customers. This can generate $50-100m in annual revenue for each medium-sized country.

Value Realized

NetworkCareSales

New Biz

New and Adjacent Businesses –New Product Development

– 34 –

Business Problem

Mobile devices produce large amounts of data about where, when, how and why they are used.

This data is extremely valuable for product managers, yet much of it is out of reach. Either it is never captured or never converted into business insight. Its volume and variety make it difficult to ingest, store and analyze at scale.

One provider who logged 27m devices with more than 1bn events per month has developed more than 20 projects and pilots within 18 months after launch, leading to increased revenue and profitability.

Value Realized

NetworkCareSales

New Biz

New and Adjacent Businesses –Actionable Intelligence Serving Advertisers

– 35 –

Business Problem

Europe’s leading real estate marketplace Scout24 – a subsidiary of Deutsche Telekom – features more than one million properties for rent or sale at any given time, and has facilitated more than 20 million property transactions over the last few years.

The company wanted to drive more market share to Scout24 by offering advertisers – typically real estate agents and brokers – an even better service.

A small team consisting of a product manager, a data scientist and a few developers was able to make a meaningful contribution to revenue growth.

Value Realized

NetworkCareSales

New Biz

Big Data as a Product: ImmobilienScout (Deutsche Telekom)

– 36 –

NetworkCareSales

New Biz

New and Adjacent Businesses –Actionable Intelligence Serving Merchants

– 37 –

Business Problem

A French mobile service provider is a great example for how location information per customer segments can be used to optimize promotions and point-of-sale locations of bricks-and-mortar retailers.

The retailers were able to increase their reported same-store-sales through better campaign management and in-store optimizations. They also gained valuable insights to optimize their store network.

Value Realized

NetworkCareSales

New Biz

New and Adjacent Businesses –Actionable Intelligence Serving Payment Processors

– 38 –

Business Problem

Credit card issuers experience increasing fraud when their card members are travelling abroad.

95% of travelers opted into the SMS alerting service, resulting in a substantial decrease in fraud related to card use in foreign countries.

Value Realized

NetworkCareSales

New Biz

New and Adjacent Businesses –Actionable Intelligence Serving Federal Governments

– 39 –

Business Problem

The Eastward expansion of the European Union has resulted in a longer and more porous border to non-EU member states.

This has made it more difficult to protect the EU against a stream of illegal goods and refugees, which often travel over land from the EU’s Eastern and South-Eastern neighbors.

Law enforcement agencies are able to target their scarce resources much more effectively, for instance choosing to intercept suspicious cars traveling in certain directions at speeds above 130km/h.

This radically increases their hit rate per mission.

Value Realized

NetworkCareSales

New Biz

New and Adjacent Businesses –Actionable Intelligence ServingLocal Governments

– 40 –

Business Problem

In a large French city, traffic to large events regularly caused massive congestion on the city’s streets and highways.

The city identified and implemented dozens of specific traffic management measures, relieving congestion around major events.

They are also exploring how to use these insights for environmental impact studies, city planning and disaster management.

Value Realized

NetworkCareSales

New Biz

• Makes capital investments more efficient

• Leads to a better customer experience

• Lowers churn

• Increases conversions

• Strengthens security

• Opens up new markets

Hadoop Drives Business Outcomes for the World’s Telcos and Cable Companies!

– 41 –

Questions?

Email juergen@techalpha.com for a copy of the presentation.

LinkedIn: juergenurbanski

Download 200-page BITKOM / Forrester Guide to Big Data Technologies (in German):

http://www.bitkom.org/files/documents/BITKOM_Leitfaden_Big-Data-Technologien-Wissen_fuer_Entscheider_Febr_2014.pdf