Hitachi Solution for Databases - Oracle RAC Database 19c ...
Hitachi Data Systems Hadoop Solution
-
date post
19-Oct-2014 -
Category
Technology
-
view
892 -
download
5
description
Transcript of Hitachi Data Systems Hadoop Solution
HITACHI DATA SYSTEMS HADOOP SOLUTION
JUNE 12, 2012
Customers are seeing exponential growth of unstructured data from their social media
websites to operational sources. Their enterprise data warehouses are not designed to
handle such high volumes and varieties of data.
Hadoop, the latest software platform that scales to process massive volumes of
unstructured and semi-structured data by distributing the workload through clusters of
servers, is giving customers new option to tackle data growth and deploy big data analysis
to help better understand their business.
Hitachi Data Systems is launching its latest Hadoop reference architecture, which is pre-
tested with Cloudera Hadoop distribution to provide a faster time to market for customers
deploying Hadoop applications. HDS, Cloudera and Hitachi Consulting will present together
and explain how to get you there.
Attend this WebTech and learn how to
• Solve big-data problems with Hadoop.
• Deploy Hadoop in your data warehouse environment to better manage your
unstructured and structured data.
• Implement Hadoop using HDS Hadoop reference architecture.
HITACHI DATA SYSTEMS HADOOP SOLUTION
WEBTECH EDUCATIONAL SERIES
PRESENTERS
Shankar Radhakrishnan, Solutions Manager, Hitachi Data Systems
Sai Saiprabhu Director, Specialized Services, Hitachi Consulting
Art Vancil Big Data Senior Manager, Hitachi Consulting
Daniel Templeton, Partner Manager, Cloudera
4
ASK BIGGER QUESTIONS DANIEL TEMPLETON, PROGRAM MANAGER AT CLOUDERA
Enterprise Data Evolution A
MO
UN
T O
F D
ATA
• Data collection & reporting
• Process data faster
• Store data more cost-effectively
• Simplify infrastructure
• Combine data from across the business
• Ask new questions immediately
• Enable new real-time applications
CREATE COMPETITIVE ADVANTAGE
IMPROVE OPERATIONAL EFFICIENCY
Data Has Changed in the Last 30 Years D
ATA
GR
OW
TH
END-USER APPLICATIONS
THE INTERNET
MOBILE DEVICES
SOPHISTICATED MACHINES
STRUCTURED DATA – 10%
1980 2012
UNSTRUCTURED DATA – 90%
Data Management Strategies Have Stayed the Same
• Raw data on SAN, NAS and tape
• Data moved from storage to compute
• Relational models with predesigned schemas
Too Much Data, Too Many Sources
• Can’t ingest fast enough
Too Much Data, Too Many Sources
$
!
$ $
$
• Can’t ingest fast enough
• Costs too much to store
Too Much Data, Too Many Sources
1
2 3 4
5
• Can’t ingest fast enough
• Costs too much to store
• Exists in different places
Too Much Data, Too Many Sources
• Can’t ingest fast enough
• Costs too much to store
• Exists in different places
• Archived data is lost
Can’t Use It The Way You Want To
• Analysis and processing takes too long
Can’t Use It The Way You Want To
1
2 3 4
5 • Analysis and processing
takes too long
• Data exists in silos
Can’t Use It The Way You Want To
? ? ? • Analysis and processing
takes too long
• Data exists in silos
• Can’t ask new questions
Can’t Use It The Way You Want To
• Analysis and processing takes too long
• Data exists in silos
• Can’t ask new questions
• Can’t analyze unstructured data
16
Transform The Way You Think About Data
Cloudera
SIMPLIFIED, UNIFIED, EFFICIENT
• Bulk of data stored on scalable low cost platform
• Perform end-to-end workflows
• Specialized systems reserved for specialized workloads
• Provides data access across departments or LOB
COMPLEX, FRAGMENTED, COSTLY
•Data silos by department or LOB
• Lots of data stored in expensive specialized systems • Analysts pull select data into EDW
• No one has a complete view
The Cloudera Approach
17
Meet enterprise demands with a new way to think about data.
THE CLOUDERA WAY THE OLD WAY
Single data platform to support BI, Reporting &
App Serving
Multiple platforms for multiple workloads
Hadoop complements the Data Warehouse
18
OLTP
Enterprise Applications
Business Intelligence
Data Warehouse
Query (High $/Byte)
CLOUDERA
Store
Query Transform
ETL
Math
Load Archive
Operational BI
Archival Data, Exploration, Analytics
INGEST STORE EXPLORE PROCESS ANALYZE SERVE
CDH CLOUDERA MANAGER
CLOUDERA SUPPORT
Cloudera Enterprise: The Platform for Big Data
19
BRINGS STORAGE & COMPUTE TOGETHER
WORKS WITH EVERY TYPE OF DATA
CHANGES THE ECONOMICS OF DATA
MANGAGEMENT
A Revolutionary Solution Built on Apache Hadoop
CLOUDERA NAVIGATOR
CDH4
20
Big Data Storage, Processing & Analytics Based on Apache Hadoop
Store Land structured and unstructured data in a scalable, cost-effective repository
1
Process & Analyze Transform data in parallel and query at the speed of thought
2
Integrate Interoperate with existing platforms, systems and applications
3
Cloudera Manager
21
End-to-End Administration for CDH
Deploy Install, configure & start your cluster in 3 simple steps
1
Configure & Optimize Ensure optimal settings for all hosts & services 2
Monitor, Diagnose & Report Find & fix problems quickly, view current & historical activity & resource usage
3
Cloudera Navigator
22
Data Management Layer for Cloudera Enterprise
Audit & Access Control (AVAILABLE NOW)
Ensuring appropriate permissions and reporting on data access for compliance
1
Exploration & Lineage (COMING SOON)
Finding out what data is available, what it looks like and where it came from
2
Lifecycle Management (COMING SOON)
Migration of data based on policies 3
Cloudera Support
23
Our Team of Experts on Call to Help You Meet Your SLAs
Extend Your Team Get a dedicated team at your disposal to help you solve problems quickly
1
Leverage the Experts Take advantage of our expertise to make sure your cluster operates at its best
2
Influence Roadmaps Get advocacy with the open source community to build the features and functionality you need
3
Cloudera Manager
Management for the complete Hadoop system The most mature & functionally advanced The easiest to use w/built-in intelligence Integration w/enterprise monitoring tools
Cloudera Enterprise
24
CDH4
The only solution with real time query (Impala) The only solution with HDFS high availability The most widely deployed & proven The broadest ecosystem of certified partners 100% open source & built for the enterprise
The Best Hadoop-Based Platform
Cloudera Navigator
The only data management tool for Hadoop Cloudera Navigator 1.0: Data audit & access
control
Cloudera Support
Dedicated team with a global presence Contributors and committers for every part of CDH Tens of thousands of nodes under management
across industries
A Complete Solution
25
CLOUDERA UNIVERSITY
DEVELOPER TRAINING
ADMINISTRATOR TRAINING
DATA SCIENCE TRAINING
CERTIFICATION PROGRAMS
INGEST STORE EXPLORE PROCESS ANALYZE SERVE
CDH CLOUDERA MANAGER
CLOUDERA SUPPORT
CLOUDERA NAVIGATOR
ALTERNATE TITLE SLIDE PRESENTER NAME DATE
TITLE SLIDES
Additional title slide options
can be found in the HDS
Icon and Slide Library. (View in slideshow mode to activate link.)
NOTE
CHOOSING THE RIGHT INFRASTRUCTURE FOR HADOOP SHANKAR RADHAKRISHNAN, SOLUTIONS PRODUCT MANAGER – ORACLE, SAP HANA AND BIG DATA SOLUTIONS
© Hitachi Data Systems Corporation 2013. All Rights Reserved.
HADOOP APPLICATION EXAMPLE: GENOME ANALYSIS
National Institute of Genomics
– Japan
Challenge: Accelerate the
speed of analysis for genome
data from next-generation
sequencers
4 PB of data
Solution
‒ 115-node Hadoop cluster using Hitachi Compute Rack servers
‒ Reliable and scalable solution
PROACTIVE MAINTENANCE AT HITACHI SERVER DIVISION
28
User Inquiry
Hardware Auditing Log
Callcenter Log Maintenance
Report CRM Customer Data
Sales/Financial Data
Distribution/Stock Data
Location Information
Server Log
Operation History
BOM data Production Data Of Business System
・Proactive hardware maintenance from logs, call center data, and product
information
・Leverage historical data for future product development
Challenge
Solution: Hadoop + SAP HANA + SAP Visual Intelligence
• Cost-effective for low-fidelity data
• Increase efficiency and utilization of resources and meet
required service levels
• Hardware less prone to failures
• Easy to manage
• Scale out to handle petabytes of unstructured and semi-
structured data
• Keep data closer to CPU
DATA
GROWTH
COST
COMPLEXITY
INFRASTRUCTURE REQUIREMENTS FOR HADOOP
HADOOP IN THE ENTERPRISE: ARCHITECTURE
Data Warehouse
Hadoop Real Time
Computer
(Streaming)
Real Time
Computer
(Streaming)
Outside
Services
(Connect to
Facebook for
CRM, etc.)
One Platform for All Data, All Applications
Other Big Data Sources (Email,
Audio, Documents, etc.)
Business Apps
RDB
Real-Time
Computer
(Streaming)
Data Connector
CxOs Data Scientist Business Users /
Customers
Business Intelligence Dashboard
Hitachi Strength and Focus
INTRODUCING HITACHI REFERENCE ARCHITECTURE FOR HADOOP
Pretested and validated for interoperability, performance, and scalability
Flexible − customize to fit application
Pre-validated using Cloudera, leading Hadoop distribution (certification in progress)
Complementary to existing Hitachi platforms for block, file, and object
Seamless management integration with other Hitachi solutions
D
A
T
A
N
O
D
E
-
H
D
F
S
T
A
S
K
T
R
A
C
K
E
R
Name Node + Job Tracker
Secondary Name Node
Management
LAN
ENTERPRISE-READY INFRASTRUCTURE FOR HADOOP
D
A
T
A
N
O
D
E
-
H
D
F
S
T
A
S
K
T
R
A
C
K
E
R
LAN
REFERENCE ARCHITECTURE: HARDWARE COMPONENTS
Qty Form factor Component Description
1 1U Management node Hitachi server CR 210H
- 2 x quad-core E2600 series
- 64GB main memory
- 2 x GigE (onboard)
- 5 x 3.5-inch 3TB NL-SAS 7200 RPM
1 2U HDFS master name node
- Name node
- Job tracker
Hitachi server CR 220S
- 2 x quad-core E2600 series
- 64GB main memory
- 2 x GigE (onboard)
- 12 x 3.5-inch 3TB NL-SAS 7200 RPM
1 2U Secondary name node
Hitachi server CR 220S
- 2 x quad-core E2600 Series
- 64GB main memory
- 2 x GigE (onboard)
- 12 x 3.5-inch 3TB NL-SAS 7200 RPM
As needed 2U Data nodes
- Data node
- Task tracker
Hitachi server CR 220S
- 2 x quad-core E2600 series
- 64GB main memory
- 2 x GigE (onboard)
- 12 x 3.5-inch 3TB NL-SAS 7200 RPM
2 1U or 2U Ethernet switches
(10 GbE network)
Cisco Nexus 5548
- 48 x GigE / 10GigE or
Brocade VDX 6720-60
- 40 x GigE / 10GigE – form factor = 2U
1U
2U
CR220S
Switch-2
42U
Internal
HDD
Switch-1 1U
• High density (2U), high processing power (2 CPU sockets), large data storage (12 HDD)
• Redundant power supplies
• Eco-friendly power saving capabilities
Why Compute Rack Servers?
Component Version Description
Operating System 6.3 Redhat or CentOS 64-bit Linux distribution
Hadoop distribution CDH4 Cloudera Hadoop distribution
Hadoop
management
4.0.1 Cloudera Manager
Management
framework
n/a Hitachi Compute Systems Manager
REFERENCE ARCHITECTURE: SOFTWARE COMPONENTS
Tested Software
D
A
T
A
N
O
D
E
-
H
D
F
S
T
A
S
K
T
R
A
C
K
E
R
Name Node + Job Tracker
HA Name Node
Management
LAN
Reference Architecture White Paper Targeted
for June 2013
WHY HITACHI FOR HADOOP INFRASTRUCTURE
Enterprise-ready (RAS) for Hadoop
‒ Less worry about hardware failure, more focus on business value
Seamless management integration with Hitachi solutions
‒ Lower opex
Competitive pricing with commodity hardware
‒ Lower capex
One platform solution for all your data volumes, velocity
and types
‒ Lower TCO, faster ROI for your big data initiatives
35
HITACHI CONSULTING
SAI SAIPRABHU, DIRECTOR, SPECIALIZED SERVICES
ART VANCIL, BIG DATA SENIOR MANAGER
HITACHI CONSULTING
As the global consulting company of Hitachi, Ltd., Hitachi Consulting brings
business visions to life through in-depth industry expertise combined with
innovative technology solutions and services
From articulating strategy through deploying
and maintaining applications, Hitachi
Consulting helps clients quickly realize
measurable business value and achieve
sustainable ROI
The Hitachi Consulting client base includes 35
percent of the Fortune 100 and 25 percent of the
Fortune Global 100, along with many mid-market
leaders. With offices in North America, Europe,
the Middle East, and Asia, the company employs
more than 5,000 professionals, with delivery
centers in India and China for global delivery
scale
WHAT DO WE SEE WITH OUR CLIENTS?
Business Objectives
Refinement
Technology Adoption
without disruption
Data Science
Practice Adoption
Business
Intelligence Jump
Start With Big Data
Technologies
Emerging
Businesses
Business Intelligence
Practice Adoption
DO YOU NEED AN EXECUTIVE SPONSOR?
The Internet has driven most businesses to demand better information much faster than
ever before across almost every industry
Examples: Retailers can influence the next shopping visit based on analytics; Amazon
can tailor a shopping visit on a variety of dimensions (personalization, price incentives,
product combinations, etc.). How will similar dynamics impact your company?
Perhaps your company has not yet started using
Hadoop for big data initiatives. Or, perhaps you are
stuck in "discovery mode" trying to find
that golden nugget big idea from big data. If your
company is like mine, you will not be given permission
to simply play with Hadoop for months on end
In most companies your time spent on a project needs
to be backed by someone with a budget who wants to
get something done. Let's look at successful methods to
secure your big data executive sponsorship.
HOW DO I GET STARTED?
3
9
Award-winning luck #1
1. Your executive brings to you the
justification for big data
Award-winning luck #2
2. Your subject matter expert and your
data scientist pour over the data until
they find the “golden nugget” of
justification
If you have no budget for big data, then perhaps you are waiting for a stroke of luck?
Stop waiting, and begin now to collaborate with your business consultant to discover
the data value and the “essence” of your big data business opportunity
THE NITTY-GRITTY DETAILS
4
0
CEO/ CSO
• Predict the Future
COO
• Optimize the Business Process
CMO
CFO/ CTO
• Deliver Faster and Cheaper
Hitachi helps you to choose your big data solution
by targeting the message to your sponsor’s role
and asking the BIG QUESTIONS
• Nurture the Customer Relationship
FOR EXAMPLE
4
1
A high-end disk storage manufacturer collects daily performance data
from its customers’ storage devices, but cannot effectively analyze it
BECAUSE OF THE VOLUME
The big questions to ask: If we stored the data in Hadoop, then
Could we detect operational patterns that predict device failure worldwide?
Could we anticipate the failure AND suggest a replacement without downtime?
Could we sell the data analysis back to the customer for a fee?
Could we reduce the support effort by delivering proactive notifications?
How much revenue would we gain/costs would we eliminate?
SOLUTION SELECTION FRAMEWORK
The solution discovery and evaluation process is a top-down
survey of organizational leadership followed by a prioritization
and ranking, based upon business value and organizational
priorities All Possible Solutions and Purposes
Solution Solution
Solution
Solution
Solution
Solution
Solution
Solution
Prioritized
Big Data Solution Selection
Feasible Solutions
Solution
Solution
SPONSOR CONVERSATIONS: ESTABLISHED BUSINESS INTELLIGENCE ENVIRONMENT
Specific use cases that address chosen pain
points to be tackled using big data
capabilities
Measures that show how the use cases
alleviate current pain points
External expertise needed to augment your
big data jump start
Action plan to implement prioritized use
cases and evaluate larger adoption of big
data capabilities
Executive sponsor buy-in
Executive sponsor oversight
Funding
LEVERAGE BIG DATA CAPABILITIES
Extend Historical
Transactions
Availability
Extend Data Staging,
Volume Processing
and Complex Data
Processing
Extend Complex
Data Processing
Ability to Process
Large Volumes
Flexibility and
Complexity
Management
Leverage Emerging
Capabilities
Extends Existing Data
Management Environment
Introduces New Analytic
Capabilities
BIG DATA TECHNOLOGIES: ADOPTION STRATEGY
Protect Existing Investments That are Already in the Right Place. Introduce Big
Data Technologies to Enable new and Evolving Business Needs
Big Data Appliance
Existing
Transactional
Sources
Social Media
Sources
Existing
Analytic
CapabilitiesStructured Data Management and Existing Data Management
Batch or Stream
Current Augmentation to Structured Data Management (Limited)
Stream and Organize
Stream and Organize
Stream and Organize
Sporadic Analytic
Capabilities
Big Volume Data
Analyses
High Velocity
Data Analyses
Unstructured
Data Analyses
Protect Investments as Needed
Streamline as the Environment Matures
Expand as
Demand grows
Introduce New
Capabilities
Introduce,
Consolidate and
Expand New
Capabilities
Enterprise Analytics
1
2
4
3
SPONSOR CONVERSATIONS: EMERGING BUSINESS INTELLIGENCE ENVIRONMENT
Business intelligence competencies needed
to attain and sustain competitive edge
Measures that help monitor business
operations alignment with business
strategies
External expertise needed to augment your
Big data and business intelligence jump
start
Action plan to implement and evaluate larger
adoption of big data business intelligence
capabilities
Executive sponsor buy-in
Executive sponsor oversight
Funding
NEXT STEPS
• Hitachi Unified Compute Platform for Business Analytics web page • http://www.hds.com/products/hitachi-unified-compute-platform/business-analytics.html
• Contact your HDS sales rep for more information
QUESTIONS AND DISCUSSION
UPCOMING WEBTECHS
WebTechs
‒ Take SAP HANA From Proof of Value Through Production Deployment, June 20, 9 a.m. PT, noon ET
‒ A Cloud You Can Trust–Improve Datacenter Efficiency and Agility, June 26, 9 a.m. PT, noon ET
Check www.hds.com/webtech for
Links to the recording, the presentation, and Q&A (available next
week)
Schedule and registration for upcoming WebTech sessions
THANK YOU