TIBCO Advanced Analytics Meetup (TAAM) November 2015
-
Upload
bipin-singh -
Category
Data & Analytics
-
view
1.290 -
download
1
Transcript of TIBCO Advanced Analytics Meetup (TAAM) November 2015
TIBCO Advanced Analytics Meetup : Q4 2015Michael O’ConnellChief Data [email protected]@MichOConnell
November 2015
• TIBCO Analytics & Data Science (Michael)• Data Analysis Pipeline
• Visual Analytics & Dashboards (Catalina Herrera)• Building Dashboards on the fly
• Predictive Analytics (Peter Shaw, Anna Nowakowska)• Simple Data Functions (R and TERR)
• Customer Analytics (Churn & Value)
• Data Access & APIs (Andrew Berridge, Mathew Lee)• Iron Python
• Advanced GeoAnalytics
• Wrap-Up / Questions (Michael)• Resources, Training, Links
Increase Productivity
Grow Revenue
Value
Reduce Risk
ROI
TIBCO Advanced Analytics Meetup (TAAM) Agenda
© Copyright 2000-2015 TIBCO Software Inc.
Data Access & Prep
Exploratory Data Analysis Features Visual
DashboardModel & Predict
Deploy Champion
Model
Test & Learn
Channel
Social
Loyalty
Campaign
Filter
Map
Merge
Shape
Propensity
Affinity
Improve Guided -------- Deploy -------- In-LineExplore Data
Aggregate
Prepare DataBusiness Case
Increase Productivity
Grow Revenue
EnsembleForest
RegressionAdditive ModelsSegment
Visualize
Pricing
Promotion
ChallengerModels
At Rest
In Motion
Value Theses
Reduce Risk
ROI
Value
Dashboard Updates
Data a Insight a Action
© Copyright 2000-2015 TIBCO Software Inc.
Spotfire Platform
© Copyright 2000-2015 TIBCO Software Inc.
SpotfireDesktop
© Copyright 2000-2015 TIBCO Software Inc.
Spotfire Platform
Immediate Long-Term Competitive AdvantageValue to the Organization
TIBCO is the only analytics platform that provides business value across this Analytics Spectrum
Self-service Dashboards API and CEP
Predictive and Prescriptive Analytics
Measure Diagnose Predict Optimize Operationalize Automate
Analytics Maturity
Insight to Action
Immediate Long-Term Competitive AdvantageValue to the Organization
TIBCO is the only analytics platform that provides business value across this Analytics Spectrum
Self-service Dashboards
Measure Diagnose Predict Optimize Operationalize Automate
Analytics Maturity
Predictive and Prescriptive Analytics API and CEP
Insight to Action
Immediate Long-Term Competitive AdvantageValue to the Organization
TIBCO is the only analytics platform that provides business value across this Analytics Spectrum
Self-service Dashboards
Measure Diagnose Predict Optimize Operationalize Automate
Analytics Maturity
API and CEP Predictive and
Prescriptive Analytics
Insight to Action
Immediate Long-Term Competitive AdvantageValue to the Organization
TIBCO is the only analytics platform that provides business value across this Analytics Spectrum
Self-service Dashboards API and CEP
Predictive and Prescriptive Analytics
Measure Diagnose Predict Optimize Operationalize Automate
Analytics Maturity
Insight to Action
Spotfire Visual Analytics - Interactive Brush-Linked
© Copyright 2000-2015 TIBCO Software Inc.
Spotfire Visual Analytics + R Extensions
3D rotate SurfacePolar
Contour Network Funnel
© Copyright 2000-2015 TIBCO Software Inc.
Spotfire + JS Extensions
Sankey Venn
ChordDonut
Gantt
WaterfallDials
© Copyright 2000-2015 TIBCO Software Inc.
Dashboards and Themes – KPIs
Dashboards and Themes – Company Specific
Mock-Up Only
Fictitious data
Dashboards and Themes – Color
Dashboards and Themes – Color
Map Charts
Map Layer
Marker Layer Feature Layer
Image LayerWMS Layer
Map Layers
© Copyright 2000-2015 TIBCO Software Inc.
Roads
Labels
Borders
Standard map
Standard map - light
Basic map
Basic map - light
Base Map Options
© Copyright 2000-2015 TIBCO Software Inc.
• Color• Shape • Size
• Relative amounts • Size
Marker Layer Feature Layer
• Color
Marker or Feature Layer
• Tooltip
• Labels
Map Elements
© Copyright 2000-2015 TIBCO Software Inc.
Building Dashboards
Catalina Herrera
Jaspersoft Pixel-Perfect Embedded Reports
Analytic Workspaces & Analytic Fabric
APIs
Sear
ch, S
harin
g et
c.
Business Analysts Report Developers
Data Discovery Analytics Dashboards Reports Embedding
Analytic WorkspacesAnalytic environments
Analytic FabricConnectivity layer
Data Professionals
25%
Business Professionals
75%
BusinessApplications
Communication& Collaboration
Platforms
© Copyright 2000-2015 TIBCO Software Inc.
Immediate Long-Term Competitive AdvantageValue to the Organization
TIBCO is the only analytics platform that provides business value across this Analytics Spectrum
Self-service Dashboards
Measure Diagnose Predict Optimize Operationalize Automate
Analytics Maturity
Predictive and Prescriptive Analytics APIs and CEP
From Dashboards to Value
Big Data Analytics
© Copyright 2000-2015 TIBCO Software Inc.
Data a Insight a Action
Data Access & Prep
Exploratory Data Analysis Features Visual
DashboardModel & Predict
Deploy Model
Test & Learn
Channel
Churn
Loyalty
Campaign
Filter
Map
Merge
Shape
Propensity
Affinity
Improve Guided -------- Deploy -------- In-LineExplore Data
Aggregate
Prepare DataBusiness Case
Increase Productivity
Grow Revenue
EnsembleForest
RegressionAdditive ModelsSegment
Visualize
Pricing
Promotion
ChallengerModels
At Rest
In Motion
Value Theses
Reduce Risk
ROI
Purchase
Dashboard Updates
Fast DataBig Data© Copyright 2000-2015 TIBCO Software Inc.
TIBCO’s Enterprise Runtime for R (TERR)
• TIBCO has rewritten R as a Commercial Compute Engine • Latest statistics scripting engine: S a S-PLUS® a R a TERR• Runs R code including CRAN packages
• Engine internals rebuilt from scratch at low-level• Redesigned data objects, memory management• High performance + Big Data
• TERR is licensed from TIBCO• TERR Installs with Spotfire Analyst / Desktop and other TIBCO products (CEP, Stat Services)• Spotfire Server can manage all TERR / R scripts, artifacts for reuse • Standalone Developer Edition: www.TIBCOmmunity.com • Supported by TIBCO
© Copyright 2000-2015 TIBCO Software Inc.
Spotfire and TERR local TERR on server
Spotfire-TERR – Local and Server
• Build models on data using local TERR engine embedded in Spotfire
• Build models on big data directly in TERR on server and display results in Spotfire
• Run TERR as parallel sessions on Hadoop cluster, controlled and visualized in Spotfire
Data Source TERR TSSS
Spotfire
Results
ODBCJDBCSDCFile
DataFunction
Larger Data
Modeling
Spotfire
LocalTERR
ODBCJDBCSDCFile
Data
Data Source
Both Spotfire and TERR can load data from any ODBC or JDBC compliant source or from Spotfire Data Connections (SDC) or Spotfire Information Links stored in the Spotfire library. © Copyright 2000-2015 TIBCO Software Inc.
Date
R P
acka
ges
1/1/2002 1/1/2003 1/1/2004 1/1/2005 1/1/2006 1/1/2007 1/1/2008 1/1/2009 1/1/2010 1/1/2011 1/1/2012 1/1/2013
5000
4500
4000
3500
3000
2500
2000
1500
1000
500
0
Number of R- or SAS posts to Stack Overflow by week. (copyright by r4stats.com)
Number of contributed packages on CRAN(http://cran.r-project.org/)
> 6,000 Packages !
R Community
© Copyright 2000-2015 TIBCO Software Inc.
126 R User Groups in US and EMEA 6,000 User-Contributed Packages Very Active R Community
Contextual Analytics- Right click- Forecasting
Contextual Analytics- Menu - Machine Learning
Simple Predictive Analytics in Spotfire
© Copyright 2000-2015 TIBCO Software Inc.
Extensible Predictive Analytics – Expressions
Lines by TERR( contourLines(x,y) )
Color by TERR( kmeans(x1,x2) )
Interactive Spotfire Analytics with R/TERR- Expression – on the fly R code- Points: kmeans(), …- Lines: contourLines(), spline(), …- Expression Functions – sharing
© Copyright 2000-2015 TIBCO Software Inc.
Analysis Workflows – Data Functions
Interactive Spotfire Analytics with R- Data Function- Robust Cluster Analysis- Any Analysis in R / CRAN
Variables driving segments- Random Forest
Revenue by product- Color by segment
Analysis Workflows – Route Optimization
Analysis Workflows – Route Optimization
Predictive Analytics Dashboards
Mock-Up Only
Fictitious data
Predictive Analytics
Customer Analytics Workflows
Anna NowakowskaPeter Shaw
Immediate Long-Term Competitive AdvantageValue to the Organization
TIBCO is the only analytics platform that provides business value across this Analytics Spectrum
Self-service Dashboards
Predictive and Prescriptive Analytics
Measure Diagnose Predict Optimize Operationalize Automate
Analytics Maturity
Event Processing
From Dashboards to Value
BIG DATAAT REST
FAST DATAIN MOTION
Insight to Action
Spotfire Data Access
DATASOURCES
XMLRDBMSFlatFiles CubesSpread-
sheetsHadoop & Big Data
stores
AnalyticalDWs e.g.Exadata
Event Data Streams
ActiveSpaces
In-MemoryLoad data from source in to memory
In-DatabaseLeave data in DBDynamically load and discard data to visualize
On-DemandDynamically swap data in and out of memory.
SQLMDX
10100110
Custom GUI-driven data access via SDK
SiebeleBusiness
Local data sources
AccessExcel STDF
Drag-and-drop
MySQL
SQL ServerOracle
Information Services(join, transform, reusable,
parameterized, dynamic query for in-memory use)
Databases
JDBC/ODBC
HadoopSFDC
PostgreSQL
TeradataNetezza
Etc.XML
RDBMS
FlatFiles
Spread-sheets
WebServices
OracleE-Business
RDBMSRDBMSRDBMS
SAP BWSAP R/3 DATA
FABRIC
Salesforce
ODBCOLE DB
SqlClient
Direct connection
OracleTeradataAsterMS SSAS
Teradata
Direct Query(dynamically query and retrieve data
for visualization and analysis)Databases
MySQLEtc.
OBIEE
NetezzaHadoop
© Copyright 2000-2015 TIBCO Software Inc.
Spotfire Data Access
Winner of 2014 Strata Cloudera AwardFor Best Advanced Analytics Application
Hadoop & Spark
TERR on the nodes of Hadoop Cluster
TERR in Action
• Hadoop cluster compute• TIBCO Cloud Compute Grid• TIBCO Streambase• TIBCO Business Events• KNIME• Lavastorm• Rstudio• Teradata• TIBCO Statistics Services• TIBCO Spotfire
Hadoop & Spark
Non Geographic Data – Map Chart Layers
© Copyright 2000-2015 TIBCO Software Inc.
Non Geographic Data – Interactive Spotfire Charts
Airplane Seating Chart
Football Field Seating Chart
© Copyright 2000-2015 TIBCO Software Inc.
Spotfire Interactive Charts created by TERR script
Spotfire API’s
Spotfire MapChart & GeoAnalytics
Andrew BerridgeMathew Lee
Optimize pricing Check for
fraud
Make offer to customer
Restock inventory
Reroute transport
Give customer service
Proactively maintain machines
Fast Data Analytics – Interventions
© Copyright 2000-2015 TIBCO Software Inc.
Spotfire – Getting Started
Spotfire Education
Spotfire Education
Contact – Follow me on Twitter
Thank you!Michael O’Connell, PhDChief Data ScientistTIBCO [email protected]@MichOConnellhttp://about.me/moconnell+1-919-7401560
© Copyright 2000-2015 TIBCO Software Inc.
Creating a TERR data function in
Spotfire
Peter Shaw
© Copyright 2000-2014 TIBCO Software Inc.
© Copyright 2000-2015 TIBCO Software Inc.
• Example shown here: Customer Churn
• Quick look at data
• Create data function in RStudio to draw curve on scatter plot
• Linear
• Quadratic
• Apply model to new data: Current customers.
• Create new column: Probability of churn
• Use this to drill-down to find at-risk customers
Outline
© Copyright 2000-2015 TIBCO Software Inc.
R code
For this demo, the data is read into the R environment directly.Here Rstudio is configured to use the TERR engine as will be used in the data function.Alternatively, the data function can use a save() command to save any incoming objects to a disk file, and then these can be loaded into the interactive R environment.
© Copyright 2000-2015 TIBCO Software Inc.
Data function development: linear model
© Copyright 2000-2015 TIBCO Software Inc.
Quadratic model
© Copyright 2000-2015 TIBCO Software Inc.
Streamlined data function for prediction
© Copyright 2000-2015 TIBCO Software Inc.
• Thanks to Ujval Kamath and Ian Cook for developing early form of this demo
Thanks
TIBCO Advanced Analytics Meetup
Product Update Including APIs
Andrew Berridge, Michael O’ConnellTIBCO Data Science
© Copyright 2000-2014 TIBCO Software Inc.
© Copyright 2000-2014 TIBCO Software Inc.
65
Product Updates and APIs
• IronPython and APIs• Why Use IronPython with the Spotfire APIs?• Custom Sort Order API Example• IronPython Authoring Improvements in Recent Versions of Spotfire
• Spotfire Updates• Styling and Themes• Annotation and Collaboration
Agenda
© Copyright 2000-2014 TIBCO Software Inc.
66
Two Books of Interest
A Picture is Worth a Thousand Tables• Graphics in Life-Sciences• Many useful lessons and analysis
examples for other industries
TIBCO Spotfire – A Comprehensive Primer• Covers how to use Spotfire• Practical real-world examples• A big section on IronPython, including documentation, examples
and code snippets• Covers predictive, geoanalytics and many other areas of Spotfire
© Copyright 2000-2014 TIBCO Software Inc.
67
Why Use IronPython with the Spotfire APIs?Apply Customizations and Automate Multiple Tasks With a Single Click!• Apply customizations:
• Change axes
• Create new visualizations
• Set colorings, update color rules
• Produce analyses customized for users
• Extend features of Web Player (Consumer)• Roll up multiple user actions into a single click
• Work with data:• Set custom sort orders
• Set filters, mark data with “where” clauses
• Load data, refresh data
Using IronPython Automation Services extension, automate IronPython scripts with Automation Services
© Copyright 2000-2014 TIBCO Software Inc.
68
Custom Sort Order API
Introduced in Spotfire 7.0• Important and not well known!
• In the past, it was not possible to programmatically set a custom sort order
• This could lead to unexpected sorting:• If not all values are known during analysis authoring;
• Then the data is refreshed, any new values will not be sorted accordingly
• API now allows sort order to be set (e.g. via an algorithm or external lookup
table)
• Can automate this via Automation Services Run Script extension; or via an
OnLoad IronPython script (triggered from JavaScript)
© Copyright 2000-2014 TIBCO Software Inc.
69
Significant Improvements Since Spotfire 6.0
• Improved debugging
• Line numbers are shown when exceptions are thrown (when using the script
editor)
• New dialog for managing scripts, accessed from Edit->Document
Properties menu
• One click to “Trust All” scripts (IronPython or JavaScript)
IronPython Authoring Improvements
© Copyright 2000-2014 TIBCO Software Inc.
70
Demo!
IronPython Authoring Improvements
© Copyright 2000-2014 TIBCO Software Inc.
71
Custom Sort Order API
Spotfire Desktop Client only GUI for altering the sort order…
Here we see fruits sorted in size order
Refresh the data, for example, to add “Plum”, then it will automatically appear at the bottom of the sort order
New API allows us to set a custom sort order
© Copyright 2000-2014 TIBCO Software Inc.
72
Custom Sort Order API
Demo!
© Copyright 2000-2015 TIBCO Software Inc.
73
Annotate Visualizations; Comment on and Discuss Visualizations
• New ability to attach annotations to visualizations
• Annotations can be styled and placed anywhere on the visualization
• Collaboration allows conversations between users
• Conversations are recorded alongside the visualizations
Spotfire 7.5 Updates – Collaboration and Annotation
© Copyright 2000-2014 TIBCO Software Inc.
74
Spotfire 7.5 Updates – Collaboration and Annotation
Demo!
© Copyright 2000-2015 TIBCO Software Inc.
Spotfire 7.0 Updates – Theming and StylingSpotfire can now be themed and styled to suit YOUR design!
• Every aspect of the visual styling is customizable
• Set colors and fonts
• Background images, styles and colors
• Borders, visual layout
• Apply the style of one DXP file to another
© Copyright 2000-2015 TIBCO Software Inc.
Spotfire 7.0 Updates – Theming and StylingExamples
© Copyright 2000-2014 TIBCO Software Inc.
77
Demo!
Spotfire 7.0 Updates – Theming and Styling
© Copyright 2000-2014 TIBCO Software Inc.
78
Spotfire 7.5 Updates – Infographics
79
© Copyright 2000-2014 TIBCO Software Inc.
Thank you!