Big Data Analytics with Oracle Advanced Analytics 12c and Big … · implemented as SQL functions...

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |

Big Data Analytics with Oracle Advanced Analytics 12c and Big Data SQL

Charlie Berger, MS Engineering, MBA Sr. Director Product Management, Data Mining and Advanced Analytics [email protected] www.twitter.com/CharlieDataMine

Faster than a Mouse: Turn Data Mining Strategy into Action

Miguel Barrera, Director of Risk Analytics, Fiserv Inc. Julia Minkowski, Risk Analytics Manager, Fiserv Inc.

Make Big Data + Analytics Simple

http://www.twitter.com/CharlieDataMine


Data, data everywhere

Oracle Confidential – Internal/Restricted/Highly Restricted

Data Analysis platforms requirements:

• Be extremely powerful and handle large data volumes

• Be easy to learn

• Be highly automated & enable deployment

Growth of Data Exponentially Greater than Growth of Data Analysts!

http://www.delphianalytics.net/more-data-than-analysts-the-real-big-data-problem/ http://uk.emc.com/collateral/analyst-reports/ar-the-economist-data-data-everywhere.pdf


Analytics + Data Warehouse + Hadoop

• Platform Sprawl

– More Duplicated Data

– More Data Movement Latency

– More Security challenges

– More Duplicated Storage

– More Duplicated Backups

– More Duplicated Systems

– More Space and Power


Vision

• Big Data + Analytic Platform for the Era of Big Data and Cloud

–Make Big Data + Analytics Simple

• Any data size, on any computer infrastructure

• Any variety of data, in any combination

–Make Big Data + Analytics Deployment Simple

• As a service, as a platform, as an application

5


Scalable in-database data mining algorithms and R integration

Powerful predictive analytics and deployment platform

Drag and drop workflow, R and SQL APIs

Data analysts, data scientists & developers

Enables enterprise predictive analytics applications

Key Features

Oracle Advanced Analytics Database Option Fastest Way to Deliver Scalable Enterprise-wide Predictive Analytics


OBIEE

Oracle Database Enterprise Edition

Oracle Advanced Analytics Database Architecture Multi-lingual Component of Oracle Database—SQL, SQL Dev/ODMr GUI, R

Oracle Advanced Analytics - Database Option SQL Data Mining & Analytic Functions + R Integration

for Scalable, Distributed, Parallel in-Database ML Execution

SQL Developer

Applications

R Client

Data & Business Analysts R programmers Business Analysts/Mgrs Domain End Users Users

Platform

Oracle Database 12c


Data remains in the Database

Scalable machine learning algorithms (e.g. clustering, regression, decision Trees, SVM, PCA, NB, text, AR, etc.) implemented as SQL functions

Parallelized SQL data mining functions, data preparation and execution of R open-source packages

High-performance parallel scoring of SQL dm functions and R models

Key Features

Oracle Advanced Analytics Database Option Fastest way to deliver enterprise-wide predictive analytics

avings

Model “Scoring” Embedded Data Prep

Data Preparation

Model Building

Oracle Advanced Analytics

Secs, Mins or Hours

Traditional Analytics

Hours, Days or Weeks

Data Extraction

Data Prep & Transformation

Data Mining Model Building

Data Mining Model “Scoring”

Data Prep. & Transformation

Data Import


avings


Data Preparation

Model Building


Secs, Mins or Hours



Data Extraction





Data Import

Lowest Total Cost of Ownership

Leverage investment in Oracle tech stack

Eliminate duplicate data & ETL

Eliminate separate analytical servers

Fastest way to deliver enterprise-wide predictive analytics

Analytical Database Applications

Key Features

Oracle Advanced Analytics Database Option Fastest way to deliver enterprise-wide predictive analytics


Function Algorithms Applicability

Classification

Logistic Regression (GLM) Decision Trees Naïve Bayes Support Vector Machines (SVM)

Classical statistical technique Popular / Rules / transparency Embedded app Wide / narrow data / text

Regression Linear Regression (GLM) Support Vector Machine (SVM)

Classical statistical technique

Wide / narrow data / text

Anomaly Detection

One Class SVM Unknown fraud cases or anomalies

Attribute Importance

Minimum Description Length (MDL) Principal Components Analysis (PCA)

Attribute reduction, Reduce data noise

Association Rules

Apriori Market basket analysis / Next Best Offer

Clustering Hierarchical k-Means Hierarchical O-Cluster Expectation-Maximization Clustering (EM)

Product grouping / Text mining Gene and protein analysis

Feature Extraction

Nonnegative Matrix Factorization (NMF) Singular Value Decomposition (SVD)

Text analysis / Feature reduction

Oracle Advanced Analytics In-Database Data Mining Algorithms—SQL & & GUI Access

A1 A2 A3 A4 A5 A6 A7

F1 F2 F3 F4



• R language for interaction with the database

• R-SQL Transparency Framework overloads R functions for scalable in-database execution

• Function overload for data selection, manipulation and transforms

• Interactive display of graphical results and flow control as in standard R

• Submit user-defined R functions for execution at database server under control of Oracle Database

• 15+ Powerful data mining algorithms (regression, clustering, AR, DT, etc._

• Run Oracle Data Mining SQL data mining functioning (ORE.odmSVM, ORE.odmDT, etc.)

• Speak “R” but executes as proprietary in-database SQL functions—machine learning algorithms and statistical functions

• Leverage database strengths: SQL parallelism, scale to large datasets, security

• Access big data in Database and Hadoop via SQL, R, and Big Data SQL

Other R packages

Oracle R Enterprise (ORE) packages

R-> SQL Transparency “Push-Down”

• R Engine(s) spawned by Oracle DB for database-managed parallelism

• ore.groupApply high performance scoring

• Efficient data transfer to spawned R engines

• Emulate map-reduce style algorithms and applications

• Enables production deployment and automated execution of R scripts

Oracle Database 12c R-> SQL

Results

In-Database Adv Analytical SQL Functions

R Engine Other R packages

Oracle R Enterprise packages

Embedded R Package Callouts

R

Results

How Oracle R Enterprise Compute Engines Work

1 2 3


You Can Think of Oracle Advanced Analytics Like This… Traditional SQL

– “Human-driven” queries

– Domain expertise

– Any “rules” must be defined and managed

SQL Queries – SELECT

– DISTINCT

– AGGREGATE

– WHERE

– AND OR

– GROUP BY

– ORDER BY

– RANK

Oracle Advanced Analytics - SQL & – Automated knowledge discovery, model

building and deployment

– Domain expertise to assemble the “right” data to mine/analyze

Analytical SQL “Verbs” – PREDICT

– DETECT

– CLUSTER

– CLASSIFY

– REGRESS

– PROFILE

– IDENTIFY FACTORS

– ASSOCIATE

+


Predicting Behavior Identify “Likely Behavior” and their Profiles

Consider: • Demographics • Past purchases • Recent purchases • Customer comments & tweets Unstructured data

also mined by algorithms

Transactional POS data

Generates SQL scripts for deployment

Inline predictive model to augment input data

SQL Joins and arbitrary SQL transforms & queries – power of SQL


SQL Developer/Oracle Data Miner 4.0 New Features SQL Script Generation

– Deploy entire methodology as a SQL script

– Immediate deployment of data analyst’s methodologies

R

http://search.oraclecorp.com/search/search


Fraud Prediction Demo

drop table CLAIMS_SET; exec dbms_data_mining.drop_model('CLAIMSMODEL'); create table CLAIMS_SET (setting_name varchar2(30), setting_value varchar2(4000)); insert into CLAIMS_SET values ('ALGO_NAME','ALGO_SUPPORT_VECTOR_MACHINES'); insert into CLAIMS_SET values ('PREP_AUTO','ON'); commit; begin dbms_data_mining.create_model('CLAIMSMODEL', 'CLASSIFICATION', 'CLAIMS', 'POLICYNUMBER', null, 'CLAIMS_SET'); end; / -- Top 5 most suspicious fraud policy holder claims select * from (select POLICYNUMBER, round(prob_fraud*100,2) percent_fraud, rank() over (order by prob_fraud desc) rnk from (select POLICYNUMBER, prediction_probability(CLAIMSMODEL, '0' using *) prob_fraud from CLAIMS where PASTNUMBEROFCLAIMS in ('2to4', 'morethan4'))) where rnk <= 5 order by percent_fraud desc;

Automated In-DB Analytical Methodology

POLICYNUMBER PERCENT_FRAUD RNK ------------ ------------- ---------- 6532 64.78 1 2749 64.17 2 3440 63.22 3 654 63.1 4 12650 62.36 5

Automated Monthly “Application”! Just

add:

Create

View CLAIMS2_30

As

Select * from CLAIMS2

Where mydate > SYSDATE – 30

Time measure: set timing on;



• On-the-fly, single record apply with new data (e.g. from call center)

More Details

Call Center Get Advice

Web Mobile

Branch Office

Social Media

Email

R

Select prediction_probability(CLAS_DT_1_2, 'Yes'

USING 7800 as bank_funds, 125 as checking_amount, 20 as credit_balance, 55 as age, 'Married' as marital_status, 250 as MONEY_MONTLY_OVERDRAWN, 1 as house_ownership)

from dual;

Likelihood to respond:

http://search.oraclecorp.com/search/search


Introducing Oracle Big Data SQL

32

Massively Parallel SQL Query across Oracle, Hadoop and NoSQL

Oracle Database 12c

Offload Query to Exadata Storage Servers

Small data subset quickly returned

Hadoop & NoSQL

Offload Query to Data Nodes

SQL

data subset

SQL


Manage and Analyze All Data—SQL & Oracle Big Data SQL

33

Store JSON data unconverted in Hadoop

JSON

Oracle Database 12c Oracle Big Data Appliance

SQL

Data analyzed via SQL or R Store business-critical data in Oracle

SQL


More Data Variety—Better Predictive Models

• Increasing sources of relevant data can boost model accuracy

Naïve Guess or Random

100%

0% Population Size

Res

po

nd

ers

Model with 20 variables



Model with “Big Data” and hundreds -- thousands of input variables including: • Demographic data • Purchase POS transactional

data • “Unstructured data”, text &

comments • Spatial location data • Long term vs. recent historical

behavior • Web visits • Sensor data • etc.

100%


Oracle R Advanced Analytics for Hadoop

35


• Fastest Way to Deliver Scalable Enterprise-wide Predictive Analytics

• OAA’s clustering and predictions available in-DB for OBIEE

• Automatic Customer Segmentation, Churn Predictions, and Sentiment Analysis

Pre-Built Predictive Models

Oracle Communications Industry Data Model Example Predictive Analytics Application


New Features


Oracle Data Miner 4.1 New Features

• JSON Query node

JSON Query node extracts BDA data via External Tables and parses out JSON data type and assembles data for data mining


Oracle Data Miner 4.1 New Features

• Oracle Data Miner Workflow API to Manage, Schedule and Run Workflows – PL/SQL APIs to enable

applications to execute workflows immediately or schedule them

– Oracle Scheduler for scheduling functionality

– ODMr repository views can be queried for project and workflow information

– Applications can monitor workflow execution and query generated results

CONNECT DMUSER/DMUSER SET SERVEROUTPUT ON DECLARE v_jobId VARCHAR2(30) := NULL; v_status VARCHAR2(30) := NULL; v_projectName VARCHAR2(30) := 'Project'; v_workflow_name VARCHAR2(30) := 'build_workflow'; v_node VARCHAR2(30) := 'MODEL_COEFFCIENTS'; v_run_mode VARCHAR2(30) := ODMRSYS.ODMR_WORKFLOW.RERUN_NODE_PARENTS; v_failure NUMBER := 0; v_nodes ODMRSYS.ODMR_OBJECT_NAMES := ODMRSYS.ODMR_OBJECT_NAMES(); BEGIN v_nodes.extend(); v_nodes(v_nodes.count) := v_node; v_jobId := ODMRSYS.ODMR_WORKFLOW.WF_RUN(p_project_name => v_projectName, p_workflow_name => v_workflow_name, p_node_names => v_nodes, p_run_mode => v_run_mode, p_start_date => '31-DEC-14 12.00.00 AM AMERICA/NEW_YORK', p_repeat_interval => 'FREQ=MONTHLY;BYMONTHDAY=-1', p_end_date => '31-DEC-15 12.00.00 AM AMERICA/NEW_YORK'); DBMS_OUTPUT.PUT_LINE('Job: '||v_jobId);


Oracle R Advanced Analytics for Hadoop

Oracle Internal - Proprietary 40

Algorithms and Functions


OAA Links and Resources • Oracle Advanced Analytics Overview:

– OAA presentation— Big Data Analytics in Oracle Database 12c With Oracle Advanced Analytics & Big Data SQL

– Big Data Analytics with Oracle Advanced Analytics: Making Big Data and Analytics Simple white paper on OTN

– Oracle Internal OAA Product Management Wiki and Workspace

• YouTube recorded OAA Presentations and Demos:

– Oracle Advanced Analytics and Data Mining at the YouTube Movies (6 + OAA “live” Demos on ODM’r 4.0 New Features, Retail, Fraud, Loyalty, Overview, etc.)

• Getting Started:

– Link to Getting Started w/ ODM blog entry

– Link to New OAA/Oracle Data Mining 2-Day Instructor Led Oracle University course.

– Link to OAA/Oracle Data Mining 4.1 Oracle by Examples (free) Tutorials on OTN

– Take a Free Test Drive of Oracle Advanced Analytics (Oracle Data Miner GUI) on the Amazon Cloud

– Link to OAA/Oracle R Enterprise (free) Tutorial Series on OTN

• Additional Resources:

– Oracle Advanced Analytics Option on OTN page

– OAA/Oracle Data Mining on OTN page, ODM Documentation & ODM Blog

– OAA/Oracle R Enterprise page on OTN page, ORE Documentation & ORE Blog

– Oracle SQL based Basic Statistical functions on OTN

– BIWA Summit’16, Jan 26-28, 2016 – Oracle Big Data & Analytics User Conference @ Oracle HQ Conference Center

https://drive.google.com/file/d/0ByYMyOPYt3TRYnBPZWl4cGdNbm8/view?usp=sharing




http://www.oracle.com/technetwork/database/options/advanced-analytics/oaa-12c-whitepaperv6-2618427.pdf





https://apex.oracle.com/pls/apex/f?p=44785:24:::NO:24:P24_CONTENT_ID,P24_PREV_PAGE:8297,1

https://stbeehive.oracle.com/teamcollab/wiki/Oracle+Advanced+Analytics:Home

https://blogs.oracle.com/datamining/entry/oracle_advanced_analytics_and_data

https://blogs.oracle.com/datamining/entry/evaluating_oracle_data_mining_has

http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=609&p_org_id=1001&lang=US&get_params=dc:D73528GC10,p_preview:N



https://apexapps.oracle.com/pls/apex/f?p=44785:24:13318499153035::NO::P24_CONTENT_ID,P24_PREV_PAGE:11925,1

https://apexapps.oracle.com/pls/apex/f?p=44785:24:13318499153035::NO::P24_CONTENT_ID,P24_PREV_PAGE:11925,1

http://www.vlamis.com/testdrive-registration/

https://apexapps.oracle.com/pls/apex/f?p=44785:24:4786379229847:PRODUCT:NO::P24_CONTENT_ID,P24_PREV_PAGE,P24_PROD_SECTION_GRP_ID:8984,141,

https://apexapps.oracle.com/pls/apex/f?p=44785:24:4786379229847:PRODUCT:NO::P24_CONTENT_ID,P24_PREV_PAGE,P24_PROD_SECTION_GRP_ID:8984,141,

http://www.oracle.com/technetwork/database/options/advanced-analytics/index.html

http://www.oracle.com/technetwork/database/options/advanced-analytics/odm/index.html

http://www.oracle.com/technetwork/database/options/advanced-analytics/odm/odm-documentation-170506.html

https://blogs.oracle.com/datamining/

http://www.oracle.com/technetwork/database/options/advanced-analytics/r-enterprise/index.html

http://www.oracle.com/technetwork/database/options/advanced-analytics/r-enterprise/r-enterprise-doc-1510100.html

https://blogs.oracle.com/R/

http://www.oracle.com/technetwork/middleware/bi-foundation/index-092760.html

http://www.biwasummit.com/





• Hands-on-Labs • Customer stories, told by the customers • Educational sessions by Practitioners and Direct from Developers • Oracle Keynote presentations • Presentations covering: Advanced Analytics, Big Data, Business Intelligence,

Cloud, Data Warehousing and Integration, Spatial and Graph, SQL • Networking with product management and development professionals

Faster than a Mouse: Turn Data Mining Strategy into Action

Miguel Barrera, Director of Risk Analytics, Fiserv Inc.

Julia Minkowski, Risk Analytics Manager, Fiserv Inc.

© 2014 Fiserv, Inc. or its affiliates.

Risk Analytics @ Fiserv Electronic Payments

• We prevent $200M in losses every year using data to monitor, understand and anticipate fraud

• We manage risk for $30BB in transfers, servicing 2,500+ financial institutions, including the 27 of the top 30 banks in the US

• A department of 6 people, we operated in start-up mode until we were acquired in 2011 by Fiserv

• We build our risk models, supervise their installation & develop the next-generation of strategies for risk mitigation


What is special about Fraud Prevention?

• Fraud is performed by organized criminal groups using sophisticated technologies and logistics

• Hard to detect: target has low frequency (2 in 10,000)

• Misclassification is expensive

$ Losses if you fail to detect fraud

60% increase in customer attrition if you miss-classify

• The environment changes fast, so you need to adapt quickly

Fraud prevention

is a great field for

the application of

predictive

analytics


Evolution of Model Deployment

3 months to run & deploy Logistic Regression

(using SAS) 1 month to estimate and deploy Trees and GLM

1 week to estimate, 1 week to install rules in online application

1 day to estimate and deploy Trees + GLM models (using Oracle Advanced Analytics)

2008 2010 2013 2015


Lessons we learned for our business

• Complex methods were hard to deploy

because they required large investments in

infrastructure and translation time

• Once we created good predictive attributes,

simple methods (Trees, GLM) were almost as

good as complex estimates (ensemble,

gradient descent)


What to Do?

• Option 1: Get a bigger door

(Netezza/ Hadoop / Pivotal )

• Option 2: Shrink the Elephant

Install Simpler Algorithms

+


Analytics Software + Infrastructure Looking for a cost-effective way to migrate our modeling requirements from other vendors to a scalable infrastructure: IBM/ Spark

• We stopped replicating

data into other software

tables

• We integrated all

preprocessing for the

models into the DB

• We installed OAA

analytics for model

development and

integrated R during 2014

Co

st


Why we like Oracle Advanced Analytics:

Accuracy

Agility

Scalability

• The loss reduction from timely deployment (hours) compensates for

the small increase of highly complex models

• No data transfer needed (in-database)

• New opportunities to combine structured data with unstructured data

• The integration with our DB replication makes re-fit inexpensive

• The same algorithm can scale-up for all other clients


Data Miner Survey 2015 by Rexer Analytics

While 6 out 10 data miners report the data is available for analysis within days of

capture, the time to deploy the models takes substantially longer. For 60% of the

respondents the deployment time will range between 3 weeks and 1year. Everyone

forgets about

deployment –

but is most

important

component!


In Fraud-Mitigation Speed is the Key How long can you wait to deploy a solution?


The Value of Time

© 2015, Oracle Corporation

avings


Data Preparation

Model Building


Secs, Mins or Hours



Data Extraction





Data Import


Accuracy / Agility vs. Cost to Deploy

• Pick the best combination of: • Less days to deployment

• High model accuracy

• Lower Cost

Application Deploy (Days) Accuracy Total Cost

SAS Server 3 0.92 x5

ODM 1 0.90 1

SAS Base 15 0.83 30%

Angoss 12 0.85 10%


Time to Fit & Deploy


How we have leveraged the Oracle-R interface

Expand Data Exploration and Visualization 1. Chart and store the results of existing procedures in the DB 2. Run R matrices for visualization of variable densities Model Fitting 1. Transport R models and legacy code from desktops to the DB 2. All the model documentation stays in the DB and access is public

Deployment 1. Run process directly from PL/SQL and visualize in the application 2. Call R models from stored procedures for scoring and direct action


OAA+ R Improves Exploration and Visualization

1. We process routines in R and save

images in DB

2. We removed the memory size constraint that made R-methods and plots impractical

3. Allowed to integrate leverage our DBAs and add them to the modeling team

Chart and store the results of existing

procedures in the DB


Fit from SQL Developer and Store in DB

• Wrap the script around your existing R code, just like you would with an existing R function

• The server stores the script in a table that you can call and just assemble the code dynamically to fit your models

• The SQL access allow you to preprocess information and sub-set your data as part of SQL calls in screen or procedures, so you can seamlessly integrate into your exploration or production scoring processes.


Store the results directly in the Database

• ggplot creates a chart, prints it to the output, then the

query runs the script and retrieves the output as an

image

• You can wrap a procedure and just run the analysis and

store the output directly into a table in the DB


Store models, data and performance in the DB

Turning Data Mining Strategy into Action


Stakeholders: Everyone has different incentives

Business Manager

Data Scientist IT Manager

• Preserve Service Level

Agreement

• Reduce Operational Risk

• Preserve Budget


Conflict of Interests?

Cannot agree on success factors?

Wonder why…?


Managing the Quants

• Define clearly the objective and constraints

• Implement SMART* goal setting

• Get familiar with basic analytics concepts

• Establish a time-line for delivery then multiply x 2

• Make sure you understand enough to explain to other executives… you will champion this initiative and negotiate the budgets

Source: Davenport, Tom (2013), “Keep up with your quants", Harvard Business Review, Issue July-August 2013


Take Care of Business (tips for Data Scientists)

• Communicate clearly business level information

• When and what is the expected result

• Present the key concept in 2 phrases

• Avoid technical language for communication

• If asked for more details, then present the “How”

• Provide a Business Dashboard

• Provide the $$ metrics profit/loss reduction

• Show the impact of algorithms deployed / provided

• Current vs. Historical

• Pick the right model - the model that maximizes the ROI

Source: Davenport, Tom (2013), “Keep up with your quants", Harvard Business Review, Issue July-August 2013


Tracking Performance: Dashboard

Our dashboards tracked the key performance metrics:

• Historical Trends for Fraud Rates and Losses (Business KPI)

• Percentage of Transfers affected by Risk Mitigation (Business KPI)

• % of population affected by policy and % of fraud prevented (KPI for Analytics)

• Fraud detection rates for rules installed (KPI for Analytics)


Key Takeaways On Fraud Modeling

• When choosing the tools for fraud management, speed is a critical factor

• Oracle Advance Analytics provided a fast and flexible solution for model building, visualization and integration with production processes

• The additional interface with R has allowed the group to leverage the skillset for both the analytics and data management team, accelerating adoption across our group

Turning Strategy into Action

• Involving the key stakeholders early in the process maximizes your chance for success. Once you have aligned the incentives for the team, selecting the appropriate techniques, tools and infrastructure becomes much simpler

• For data scientists it is important to select their models and projects based on the expected business impact and to translate their findings into the relevant metrics

© 2014 Fiserv, Inc. or its affiliates. 69

If you have further questions or comments, please contact:

[email protected]

[email protected]

Big Data Analytics with Oracle Advanced Analytics 12c and Big … · implemented as SQL functions...

Documents

Transcript of Big Data Analytics with Oracle Advanced Analytics 12c and Big … · implemented as SQL functions...