The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data...
Transcript of The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data...
![Page 1: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/1.jpg)
1
An overview of some of the common terminology and usage concerning analytics and business intelligence.
Prepared by Justin Simonds for the IIBA Orange County Chapter
![Page 2: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/2.jpg)
The world is fundamentally changing
Market DisruptionsHealthcare
Fitbit
HumanResources
Insurance
Progressive
Tele-communication
Retail Transportation
Amazon Uber
Media and Entertainment
Netflix
FinancialServices
Apple Pay
![Page 3: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/3.jpg)
The Marketplace:
3
1/3rdof the top 20 market leaders will be disrupted by competitors by
2018(IDC Predictions)
68%of businesses view time to market as a critical differentiator
(CSO Magazine)
50+%of IT Budget going to LOB leaders vs. the CIO
(PWC Study)
5-20% increase in business value to organization via fast time to
market (CSO Insights)
1 Trillion sensors deployed globally by 2020(IBM estimate. IDC Estimate: 17.6 Billion)
![Page 4: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/4.jpg)
2 petabytesWalmart’s entire
transaction database
4 petabytesPer-day posting to Facebook across 1.1 active billion users
in May 2016
40,000 petabytes per day10m self-driving cars by 2020
Front camera20mb / sec
Front ultrasonic sensors
10kb / secInfrared camera
20mb / sec
Side ultrasonic sensors
100kb / sec
Front, rear and top-view cameras
40mb / sec
Rear ultrasonic cameras100kb / sec
Rear radar sensors100kb / sec
Crash sensors100kb / sec
Front radar sensors100kb / sec
4
The amount of data we process is about to change — again
![Page 5: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/5.jpg)
5
![Page 6: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/6.jpg)
6
![Page 7: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/7.jpg)
Data Warehousing up to the apocalypse
7
![Page 8: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/8.jpg)
The Opportunity – Unstructured Data
• Information and knowledge management is - the management of both structured data (15% of information) and unstructured data (85% of information), according to the Butler Group.
• 80 percent of business is conducted on unstructured information (Gartner Group).
• Over 43 percent of organizations have more than six content stores. (Forrester Research).
• Billions of dollars have been invested - using technologies such as data mining, business intelligence and on-line analytical processing tools to extract value from databases.
• The unstructured data market is a significant opportunity -estimated at least 4 times the size of the structured data market and growing exponentially
85 %
15 %
Unstructured
Information
Structured Data
Corporate Information Management
![Page 9: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/9.jpg)
Need for external data – structured, semi-structured and unstructured
9
![Page 10: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/10.jpg)
More data structured, hyper-structured and who knows…
10
![Page 11: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/11.jpg)
11
![Page 12: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/12.jpg)
Houston we have a problem…
• Traditional RDMS/EDW cannot handle
– New data types (Variety)
– Extended analytical and stream processing (Value)
– TB/hour loading with immediate Query access (Velocity)
• RDBS/EDW storage constraints (Volume)
12
![Page 13: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/13.jpg)
But not the end of the Data Warehouse!!!
Where the Data Warehouse ends…
![Page 14: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/14.jpg)
Hadoop versus EDW
• Hadoop is an open source distributed storage and processing framework
• Architectural differences
14
Database - Tables
System Tables
SQL Query Engine
HDFS Files (raw to highly structured)
Hcatalog (multiple possible descriptions)
Multiple Engines (SQL, NoSQL and non SQL)
STORAGE
METADATA
QUERY
All three layers provided as proprietary bundle
All three layers separate and independent allowing
multiple access
And
![Page 15: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/15.jpg)
Columnar
Streams to LakeETL
CRM
ERP
MES
DATALAKE
EDW
Ingest Decide Passthru
Real-time Analytics
FAST
ODS
Stock Feed
Weather
SQL
NoSQL
DATASTREAM
![Page 16: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/16.jpg)
16
![Page 17: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/17.jpg)
17
![Page 18: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/18.jpg)
18
ACID
•Atomicity
•Consistency
•Isolation
•Durability
BaSE
•Basic availability
•Soft state
•Eventually consistent
![Page 19: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/19.jpg)
19
OrganizedKey-Value
Node/relationship
![Page 20: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/20.jpg)
20
![Page 21: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/21.jpg)
Columnar
Streams to LakeETL
CRM
ERP
MES
DATALAKE
EDW
Ingest Decide Passthru
Real-time Analytics
FAST
ODS
Stock Feed
Weather
SQL
NoSQL
DATASTREAM
![Page 22: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/22.jpg)
Can we continue to move all this data?
22
![Page 23: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/23.jpg)
23EDGE
A new processing idea is processing at the edge where information is created. If we can handle it at the source perhaps we don’t need to move it?
![Page 24: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/24.jpg)
The need for speed
So initially we may require all data from shop floor/supply chain systems to be moved to the data lake where analytics are run against it. The analytics might uncover patterns indicating the requirement for maintenance or possible outage.
![Page 25: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/25.jpg)
Reducing Big Data movement and latency
Those patterns can be exported to local, smaller systems and the information processed locally without the need to move all the data just the information that cannot be processed locally.
![Page 26: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/26.jpg)
26
![Page 27: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/27.jpg)
27
![Page 28: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/28.jpg)
28
Heavy nodeEdge history computingMesh analytics
Heavy nodeEdge history computingMesh analytics
![Page 29: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/29.jpg)
![Page 30: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/30.jpg)
“Machine learning is basically a way for a computer to find the nuggets of information that a human can’t,” explains Fausto Ibarra, director of global product management for Google Cloud Platform. “Once you have your data and train and deploy your models, the machine can go through terabytes of data and get smarter and smarter—basically train itself—and ultimately make predictions for you.”
30
![Page 31: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/31.jpg)
Machine Learning – Deep Learning
• IBM Project Deep Blue
• Chess playing computer system
• May 1997 defeats Garry Kasparov – current world champion
31
![Page 32: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/32.jpg)
Machine Learning – Deep Learning
• IBM Project Deep QA aka “Watson”
• Developed NLP (Natural Language Processing) capabilities
• 2011 defeats the top Jeopardy champions Brad Rutter and Ken Jennings
32
![Page 33: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/33.jpg)
Machine Learning – Deep Learning
• Google DeepMind aka “AlphaGo”
• Developed to play Go, more variations than Chess
• March 2016 AlphaGo defeats Lee Sedol a 9th Dan Go champion
• Jan 2017 AlphaGo defeats Ke Jie the #1 ranked Go player in the world in 3 straight games
33
![Page 34: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/34.jpg)
Machine Learning – Deep Learning
• Carnegie Mellon University “Libratus”
• No-limit Texas Hold’em Poker
• January 2017 beats Jason Les, Dong Kim, Daniel McAulay and Jimmy Chou
34
![Page 35: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/35.jpg)
Placeholder for Watson Video
https://www.youtube.com/watch?v=HkEOJnn_zlg
![Page 36: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/36.jpg)
"There's a 60-year-old woman in Tokyo. She was at the University of Tokyo. She had been diagnosed with leukemia six years ago. She was living, but not healthy. So the University of Tokyo ran her genomic sequence through Watson and it was able to ascertain that they were off by one thing. Actually, she had two strains of leukemia. They did treat her and she is healthy."
David Kenny, General Manager of IBM Watson 12/7/2016
36
![Page 37: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/37.jpg)
Cataloging pictures
![Page 38: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/38.jpg)
38
Benefits
![Page 39: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/39.jpg)
Budget
39
![Page 40: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/40.jpg)
40
C-Suite involved
![Page 41: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/41.jpg)
Business Analysts and Business Intelligence
• Analytics have become strategic to the business (Competing on Analytics by Tom Davenport)
• BI transitioning from IT-led system of record reporting to Business-led self-service analytics
– Accessibility (democratize data)
– Agility
– Deeper analytical insights
• Diverse array of data sources
• Expanded range of users
41
![Page 42: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/42.jpg)
Seven areas where business analysts can help drive BI&A
• BI&A Platform modernization
– Non-technical users to autonomously create content
• Business value
– Driving efficiencies and innovation across business domains
• Tool Selection
– Necessary to maximize adoption of trusted analytics that impact the business
• Organizational change requirements
– Redefining roles and responsibilities of IT and business users
42
![Page 43: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/43.jpg)
Seven areas where business analysts can help drive BI&A
• Develop user competence
– New methods for training and support
– New rules for governing, managing, promoting and sharing content
• Agile development model
– Pace of business and exploratory nature of analytics requires agile methodology
• Key technologies and market trends
– The data lake and the EDW
– Cloud deployment options
– Benefits and compromises of open source
– Large, diverse and real-time streaming data
– Data Discovery
– Machine learning
– Deploying self-service at scale
43
![Page 44: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/44.jpg)
“What the industry is seeing is that more than 50 percent of big data projects fail, and fail miserably,” Poutonnet says. “Businesses are spending a lot of time with the data-gathering and data-preparation phases, as well as trying to figure out the data architecture, and they don’t have time to play with the data itself.”
Philippe Poutonnet, global product marketing lead for Google Cloud Platform
44
![Page 45: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/45.jpg)
ThankYou
![Page 46: The world is fundamentally changing Orange County Simonds...The Opportunity –Unstructured Data •Information and knowledge management is - the management of both structured data](https://reader036.fdocuments.us/reader036/viewer/2022071119/60195cb1e1a7bf6f09394824/html5/thumbnails/46.jpg)
If there’s no questions (unlikely)another Watson Video
https://www.youtube.com/watch?v=ymUFadN_MO4