Gene Villeneuve - Moving from descriptive to cognitive analytics

© 2013 IBM Corporation

A New Era of Smart

Moving from Descriptive to Cognitive Analytics on your Big Data Projects

Date: October 7, 2014

Gene VilleneuveDirector & European Sales Leader Predictive & Business Intelligence

© 2013 IBM Corporation2

A New Era of Smart

Agenda

Introduction and some clarification regarding terminology

The evolution of analyticsDescriptive Predictive Prescriptive Cognitive

Analytics in the Context of Big Data

Big Data & Analytics Reference Model

Sample projects and customer case studies illustrating the evolution of analytics

Current research & development areas


A New Era of Smart

INTRODUCTION & TERMINOLOGY


A New Era of Smart

Analytics: a Business Imperative across Industries

LOB buyers are driving new demand for industry solutions

The new era of computing enables new analytic methods

At the pointof impact

Big Data and

Analytics

Big Data and

Analytics

All perspectivesAll perspectives

All decisionsAll decisions

All informationAll information

All peopleAll people

Programmatic

SearchDeterministicEnterprise dataMachine languageSimple outputs

Cognitive

Discovery Probabilistic Big Data Natural language Intelligent options

* Source: IBM Market Development & Insight – GMV 1H2013


A New Era of Smart

The Evolution of Analytics

CognitiveAnalytics

PredictiveAnalytics

PrescriptiveAnalytics

DescriptiveAnalytics

Descriptive “After-the-facts”

analytics by analyzing historical data

Provides clarity as to where an enterprise or an organization stands related to defined business measures

Applied to all LoB for fact finding, visualization of success and failure

Cognitive Pertaining to the

mental processes of perception, memory, judgment, learning, and reasoning

Range of different analytical strategies that are used to learn about certain types of business related functions

Natural language processing

Predictive Leverages data

mining, statistics and ML algorithms, etc. to analyze current and historical data to predict future events and business outcome.

Discovers patterns derived from historical and transactional data to optimize business measures

Prescriptive Synthesizes big data,

mathematical and computational sciences, and business rules to suggest decision options

Takes advantage of a future opportunity or mitigate a future risk and shows the implication of each decision option


A New Era of Smart

What could happen in the future?

Information Layer

How is data managed and stored?

How can everyonebe more right…….more often?

How can everyonebe more right…….more often?

Descriptive

What has already happened?

Predictive

Prescriptive

How can we achieve the best outcome?

Cognitive

How can we learn dynamically?

Bus

ines

s V

alue

Bus

ines

s V

alue

Reasoning Learning Natural Language

Reasoning Learning Natural Language

Alerts & Drill Down Ad hoc Reports Standard Reports

Alerts & Drill Down Ad hoc Reports Standard Reports

Big Data Platforms Content Management RDBMS and Integration

Big Data Platforms Content Management RDBMS and Integration

Machine learning Forecasting Statistical Analysis

Machine learning Forecasting Statistical Analysis

Optimization Rules Constraints

Optimization Rules Constraints

IBM Big Data & AnalyticsIBM Big Data & Analytics

The Scope of Advanced Analytics

• IBM analytics breadth covers the full spectrum of decisions

• IBM is the undisputed leader in advanced analytics


A New Era of Smart

Accelerating the Client’s Journey to Cognitive

Natural, Intuitive or Automated Interaction

Co

nte

xt S

pec

ific

Usa

ge Opportunities to infuse cognition and

collaboration in existing solutions and products for differentiation

The Analytics

Contin

uum

INFORMATIO

N FOUNDATION

DESCRIPTIVE

PREDIC

TIVE

PRESCRIPTIVE

C

OGNITIVE Win on

Innovation

Compete on time to business value – through context specific data, methods, workflow.

Reasoning

Learning

Natural Language

Optimization

Rules

Predictive Modeling

Forecasting

Statistical Analysis

Alerts

Drilldown Query

Ad-hoc Reports

Standard Reports

Big Data Platforms

ECM

Information Integration

RDBMS


A New Era of Smart

Analytics: a Business Imperative across Industries

Clients realize value through solutions

* Source: IBM Market Development & Insight – GMV 1H2013

IBM Predictive Maintenance & QualityImproves productivity, prevents downtime and reduces costs

IBM Predictive Maintenance & QualityImproves productivity, prevents downtime and reduces costs

IBM Credit Risk ManagementDerive competitive advantage from risk management processes

IBM Credit Risk ManagementDerive competitive advantage from risk management processes

IBM Enterprise Marketing ManagementDiscover and react in real time to how consumers are interacting

IBM Enterprise Marketing ManagementDiscover and react in real time to how consumers are interacting

IBM Social Media AnalyticsUncover customer sentiment, predict behavior, improve marketing

IBM Social Media AnalyticsUncover customer sentiment, predict behavior, improve marketing

IBM Watson Engagement AdvisorTransforms client experience with deep personalized Q&A

IBM Watson Engagement AdvisorTransforms client experience with deep personalized Q&A


A New Era of Smart

IBM’s Portfolio delivers Business Value

Business value from automation of routine decisions, to transformative new usages of data

Line of Business Leaders

Industry Solutions Integrated by Design

Big Data & Analytics

Mobile Social

Cloud

Client-Driven Capabilities and

Platforms

Market-Growth Initiatives

Big Data Infrastructure

Predictive Prescriptive Cognitive

Sup

port

ed b

y IB

M e

xper

tise

thr

ough

BA

O s

ervi

ces

Smarter Commerce

Smarter Workforce

Smarter Cities

CPO CMO CIOCFOCHRO CRO Mayors

Smarter Analytics

Cloud


A New Era of Smart

1.3M IOPS Scalability99.997% Availability 0 Incidents, Vulnerability

Pow

er S

yste

ms

Des

ign

Pow

er S

yste

ms

Des

ign

Open & flexible infrastructure - Available on premise or through the CloudOpen & flexible infrastructure - Available on premise or through the Cloud

Industry SolutionsIndustry Solutions Cognitive ComputingCognitive ComputingBusiness & Predictive AnalyticsBusiness & Predictive Analytics

Pow

er S

olu

tion

sP

ower

Sol

utio

ns

IBM WatsonIBM Watson

1,000+ Concurrent Queries

Parallel processing

Large-scale memory processing

Massive IO bandwidth

Stream Computing

Real-time Analytics

Natural Language Learning

Continuous data load

Power Systems enables next Generation Big Data and Analytics Applications


A New Era of Smart

ANALYTICS IN THE CONTEXT OF BIG DATA


A New Era of Smart

Data Content>80%<20%

Data Content>80%<20%

Analytics in the Context of Big Data - The Big Data Analytics Challenge

Requiring overcoming the high volume, real-time, and unstructured nature of social media and Enterprise data streams

From noisy data to trustworthy insights

Understand jargon and acronyms, eliminate spam

Heterogeneous data

Combine, correlate information over 100’s of sources (sites, forums, message boards, newswires…)

Timely Decision making Make decisions in near real-time over 10K+ messages/second

Growing volume of data

Social media or other media source data

Extract concepts from several 100M messages/day

100M+ active users per source

VeracityVeracity

VarietyVariety

VelocityVelocity

VolumeVolume

Data Volume

360-degree Profiles• Micro-segmentation• Predict Behavior

360-degree Profiles• Micro-segmentation• Predict Behavior

Manual Interaction• Polling & ExtrapolationManual Interaction• Polling & Extrapolation

Listening and Monitoring• Sentiment, Buzz• Key influencers

Listening and Monitoring• Sentiment, Buzz• Key influencers

Ana

lytic

s C

ompl

exity

Learning, NLP, Discovery• Auditory & visual processing • Logic & reasoning • Improve interventions

Learning, NLP, Discovery• Auditory & visual processing • Logic & reasoning • Improve interventions


A New Era of Smart

Analytics in the Context of Big Data - Key Drivers for Cognitive Analytics

The need for cognitive analytics is driven by the confluence of SoLoMo (Social, Local, Mobile), Big Data, and Cloud

VeracityVeracity VarietyVariety

VelocityVelocity VolumeVolume

Cognitive Systems


A New Era of Smart

Analytics in the Context of Big Data - Veracity / Trust / Sentiment

Addressing the information trustworthiness of social media data

Some dimensions of trustworthiness /

Trustworthiness Sentiment

– Jokes– Prosody– Sarcasm– Seriousness– Emotion– Mood– Ambiguity– Humor– Dialect– Social factors …– Social media languages– Context– etc.

VeracityVeracity

InformationProvenance

AuthorClassification

IntegrityAssumption

ContentAnalysis

RelevanceDetermination

UsageIntention

InformationProvenance

AuthorClassification

IntegrityAssumption

ContentAnalysis

RelevanceDetermination

UsageIntention


A New Era of Smart

Analytics in the Context of Big Data DeepQA: The Architecture underlying Watson Generates many hypotheses, collects wide range of evidence, balances the combined

confidences of >100 different analytics that analyze the evidence from different dimensions

Answer Scoring

Models

Answer & Confidence

Evidence Sources

Models

Models

Models

Models

ModelsPrimarySearch

CandidateAnswer

Generation

HypothesisGeneration

Hypothesis and Evidence Scoring

Final Confidence Merging & Ranking

Synthesis

Answer Sources

Question & Topic Analysis

EvidenceRetrieval

Deep Evidence Scoring

Learned Modelshelp combine and

weigh the Evidence

HypothesisGeneration

Hypothesis and Evidence Scoring

QuestionDecomposition

Each year the EU selects capitals of culture; one of the 2010 cities was this Turkish “meeting place of cultures”


A New Era of Smart

99%60%10%

Understands natural language and human speech

Adapts and Learns from user selections and responses

Generates and evaluates

hypothesis for better outcomes

3

2

1

…built on a massively parallel probabilistic evidence-based

architecture optimized for Linux on POWER7+

Analytics in the Context of Big Data - Watson drives optimized outcomes


A New Era of Smart

BIG DATA ANALYTICS REFERENCE MODEL


A New Era of Smart

Big Data & Analytics Platform

An innovative, foundational big data platform can help tackle big data’s four V’s (volume, variety, velocity and veracity) with an integrated set of big data technologies to address the business pain, reduce time and cost, and provide quicker return on investment

More cost-effectively analyze petabytes of structured and unstructured formation

Analyze streaming data and large data bursts for near-real-time insights

Access deep insight with advanced in-database analytics and operational analytics

Big data platform

Data warehouseApache Hadoop system Stream computing

Data Media Content Machine Social

Systems management Application development Discovery

Information integration and governance


A New Era of Smart

Infrastructure Services

Dat

a Tr

ansf

orm

ation

& In

tegr

ation

Lay

er

Het

erog

eneo

us D

ata

Sour

ces

Visu

aliz

ation

&Re

porti

ng L

ayer

Data Persistency Layer

Business Analytics &Applications Layer

Infrastructure Services

Dat

a Tr

ansf

orm

ation

& In

tegr

ation

Lay

er

Het

erog

eneo

us D

ata

Sour

ces

Visu

aliz

ation

&Re

porti

ng L

ayer

Data Persistency Layer

Business Analytics &Applications Layer

Big Data Analytics Reference Model - Key Capabilities

Components to build a trusted information integration layer with ETL, data quality, real-time data processing, federation, metadata mgmt, …

Comprehensive Big Data advanced analytics layer with applications & research assets on heterogeneous source data

Traditional reporting and BI analytics, with visualization & exploration of heterogeneous data

Traditional DW system (SOR, ODS, marts) with MDM system, DW appliances, and augmented with Hadoop platform

Common infrastructure services, such as systems management, security, backup, information governance, …

Heterogeneous data landscape including existing data stored in BSS systems, from the network, external, customer touch points


A New Era of Smart

SAMPLE PROJECTS AND CUSTOMER CASE STUDIES ILLUSTRATING THE EVOLUTION OF ANALYTICS

(IN THE CONTEXT OF BIG DATA)

CognitiveAnalytics

PredictiveAnalytics

PrescriptiveAnalytics

DescriptiveAnalytics


A New Era of Smart

Predictive AnalyticsDemographics Enrichment for unknown Subscribers

Gain analytical insight for pre-paid demographics Understand post-paid subscribers

– Using post-paid demographics data (age, gender, income, …)– Gaining insight: propensity/predictive modeling, micro-segmentation,

clustering, sentiment analytics, … from appl usage data, web browsing, CDR, social media

Understand pre-paid subscribers– Gaining insight: propensity/predictive modeling, micro-segmentation,

clustering, sentiment analytics, …– Demographics data isn't available or not sufficiently trustworthy

Correlate post- with pre-paid subscribers and map demographics– Correlate post- with pre-paid segments, clusters, behavior, interest, …– Map known demographics for post-paid to corresponding pre-paid

subscribers

Required Data Sources Voice & data CDR (MSISDN & Usage) Behavioral data:

– Web browsing & search (internal and external), user agent: browser, appl and/or device that made request, content type: type of data sent/downloaded

Public sources (will be used, not required from CSP):– Wikipedia– IMDB http://www.imdb.com/– Open Directory Project (ODP)

Subscriber reference data (e.g. from CRM or EDW)

PredictiveAnalytics

PredictiveAnalytics


A New Era of Smart


CSP

DATA SOURCESCSP & other

Voice & data CDR (MSISDN & Usage)

MSP (MSISDN & URL) Behavioral data (e.g. blogs, use of

mobile apps, Web browsing & Web search )

Public sources (e.g. ODP) Metadata, e.g. time, size, … CRM or EDW

Data understandingData transformationData preparation

IBM BigInsights Admin Customer Modeler Admin(Predictive Analytics)

IBM Singapore

Data anonymization Data provisioning

CorrelationPredictive modelingPropensity modelingMicro-segmentationClusteringSentiment

Analytical insightVisualizationConsumption by Advertisement

PRODUCTS & Tools

BigInsights (incl. BigSheets, SystemT, HDFS, Jaql, …)

Customer Modeler SPSS Modeler NLP DB2

SaaS

PredictiveAnalytics

PredictiveAnalytics


A New Era of Smart

HDFSAnalytical Model

(pre-paid)

DB2Predictive Model

(for pre-paid)

HDFSAnalytical Model

(post-paid)


Post-paid CSP

Data Sources:Voice/Data CDRs Behavioral Data

• InfoSphere BigInsights• Customer Modeler• SPSS / DB2 / NLP

GTS SmartCloud Enterprise

Analysis/InsightPre-paid:

• Age• Gender• Income

TransformationAnonymization

(to be validated)

Source DataTransformation

Pre-paid CSP

Data Sources:Voice/Data CDRs Behavioral Data

Public Sources(not from CSP):

WikipediaIMDBODP

Used for gainingAnalytical Insight

Post-paid CSP

Data Sources:Subscriber

Demographics

Visualization

Used for buildingPredictive Model

PredictiveAnalytics

PredictiveAnalytics


A New Era of Smart

XO Communications takes control of customer satisfaction

142 percent reductionin revenue erosion for customers at most risk of churning

$10 million+ savings/yearfrom increased retention and reduced customer service costs

5 months to achieve full return on investment

Solution components

The transformation: XO Communications had already taken the first steps in identifying customer retention risks through analytics; now it wanted to seize the opportunity to put these insights into action more effectively. By using IBM® SPSS® solutions to hone its predictive models, the company built a richer, more up-to-date picture of its client base and began delivering this data to a greater range of employees.

“We are only just starting to realize the true potential that IBM analytics holds across the business.”

— Bill Helmrath, Director of Business Intelligence, XO Communications • IBM® SPSS® Analytics Catalyst• IBM SPSS Modeler• IBM SPSS Modeler Server• IBM SPSS Statistics• IBM InfoSphere® BigInsights™

YTP03235-USEN-00


A New Era of Smart

IBM® AIX® IBM Cognos® Business Intelligence IBM DB2® IBM InfoSphere® Warehouse IBM PowerHA® IBM PowerVM® IBM SPSS® IBM Tivoli® Storage Manager and

System Automation for Multi-Platforms

IBM WebSphere® Application Server IBM Power® 770

Fiserv cuts IT costs while enhancing analytics capabilities with software and infrastructure from IBM

$8 million savedin IT costs over a five-year period

90% reductionin the number of midrange servers under management

Boosts availabilityand improves the agility of service delivery

Solution Components

Business Challenge: Fiserv was seeking new ways to attract, retain and grow profitable customer relationships while helping its clients compete with newer and larger banks. Leveraging predictive analytics applications proved key to this goal, but Fiserv realised that it also needed a more agile, available and scalable IT infrastructure to support its new capabilities.

The Solution: IBM information management and predictive analytic solutions enable Fiserv to transform billions of raw transactions into actionable insights that help small and midsize banks better target offers and maximize their marketing dollars. The use of cloud technologies to consolidate and virtualize servers helps reduce costs and accelerate time-to-market.

“We have estimated a five-year-cumulative run rate reduction of about $8 million with the server consolidation and virtualization project.”

—Leroy Hill, Manager, Midrange Engineering, Fiserv


A New Era of Smart

Cognitive AnalyticsHalalan 2013 Social Media Tracking

BUZZ – candidates, topics, personalities, broadcasters

– How much / What is being said about the candidates (ongoing and for key “events” like debates, advertisements, etc.), different shows, news anchors.

– How does this change over time, what is trending.

SENTIMENT – popular opinion– What do voters like or dislike about the candidates,

the parties, campaigns, constituents, etc.– How does this sentiment break down by the

different groups (voters, political affiliation, news professionals, demographics, affinity groups, etc.)

– Understand brand sentiment, i.e., whether ABS-CBN is being perceived as unbiased and trusted. How are the different news personalities being perceived: credible, neutral, fair?

INTENT – action– What is the intent to act (support / vote) for each

candidate.– What election outcomes can be predicted (shifts in

candidate sentiment, voter intent, etc.)

CognitiveAnalytics

CognitiveAnalytics


A New Era of Smart

CURRENT RESEARCH & DEVELOPMENT AREAS

(just a few examples)


A New Era of Smart

Cognitive Analytics: Technical Capabilities requiredWatson Solutions – Build on repeatable Assets

Watson forHealthcare

Watson forFinancial Services

Watson forClient Engagement

Watson for Industry

Sol

utio

ns Sample Advisor Solutions Sample Advisor Solutions Sample Advisor Solutions

Utilization

Oncology

Research

Care Mgt.

Banking

Financial Markets

Insurance Call Center

Knowledge

Help Desk

Technical

NLP & MachineLearning

Data Analytics Cloud Mobile Workload OptimizedSystems

100111001

10010010010

1000101100101

10001010010

00110101

Cap

abili

ties

ASK Services DISCOVER Services DECISION Services

Pla

tfor

m

Content Tooling Methods Algorithms APIs

Ready Build Teach RunFull Lifecycle


A New Era of Smart

Massive Scale SNA (X-RIME) over BigInsightsCurrent Research Area Project Overview

– X-RIME is a library that consists of MapReduce programs, which are used to do raw data pre-processing, transformation, SNA metrics and structures calculation, and graph / network visualization

– Based on IBM InfoSphere BigInsights (Hadoop) – Goes beyond SPSS SNA for churn propensity modeling

Reference– Commercial Solution: China Mobile enterprise blog analysis solution– ARL MSA on Power Benchmarking: Pageranking 390 millions of nodes

on 10-nodes power7 cluster (2 hours per iteration)– Integrated to SystemG as GraphBase– Open Source X-RIME on SourceForge

Selected X-RIME SNA Algorithms

– Vertex degrees (in/out/both/average/max)

– Weekly connected components

– Bi-connected components– Breadth first search (BFS) – K-core– Maximal clique– Community detection

based on label propagation

– Community detection based on scored label propagation

– Community detection based on propinquity

– Modularity evaluation– Hyperlink induced topic

search (HITS)– Pagerank– Minimal spanning tree

(MST)– Ego-centric network– Vertex clustering

coefficient– Edge clustering coefficient

HDFSHDFS

Graph Data Model (Object)Graph Data Model (Object)

Message Passing FrameworkMessage Passing Framework

MapReduceMapReduce

SNA library SNA library

X-RIME Architecture

Gene Villeneuve - Moving from descriptive to cognitive analytics

Data & Analytics

Transcript of Gene Villeneuve - Moving from descriptive to cognitive analytics