Implementing a Scale-out Dynamic Workload Utility (DWU ...
36
© 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice Technology for better business outcomes Implementing a Scale-out Dynamic Workload Utility (DWU) using Moab and HP software Susanne M. Balle Cloud Architect Scalable Computing and Infrastructure
Transcript of Implementing a Scale-out Dynamic Workload Utility (DWU ...
© 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
Technology for better business outcomes
Implementing a Scale-out Dynamic Workload Utility (DWU) using Moab and HP software
Susanne M. Balle
Cloud ArchitectScalable Computing and Infrastructure
2
•
Scale-out challenges•
Challenges of dynamic IT environments•
DWU overview and capabilities•
DWU architecture•
Usage Scenarios•
Success story•
More information
3
Bridge the gap between scale, performance and results
•
Increase competitiveness–
Scale performance with resources–
Rapidly integrate new capacity–
Deploy globally
•
Data center constraints–
Power–
Cooling–
Space
•
Control IT spend–
Reduce TCO–
Reduce software costs–
Ease of management and automation
Scale-out challenges
4
Examples of HP’s
capabilities in cloud and scale-out infrastructure
New business models for delivering infrastructure
as a serviceHP Flexible Computing Services
HP AI as a Service
Critical Facilities Services: EYP MCF
Automate the environment for dynamic
workloadsHP Insight Software
HP Business Service Automation
Designed for massive scale-out, cloud computing & Web 2.0ProLiant
and BladeSystem
Servers
StorageWorks
New baselines for energy, floor space & coolingModular Cooling System
HP POD
Software
Infrastructure
Servers
& Storage
Data Center
Services
Data Center
Infrastructure
5
•
Scale-out challenges•
Challenges of dynamic IT environments•
DWU overview and capabilities•
DWU architecture•
Usage Scenarios•
Success story•
More information
6
Challenges of dynamic IT environments
•
Mismatch of static resources with dynamic workloads•
Underutilization of resources•
IT staff errors and delays while managing and provisioning systems
•
Weakened revenue and productivity due to slow response times
•
Scalability and growth•
Support for heterogeneous hardware and operating systems
•
Suitability for “Infrastructure as a Service”
7
Data Center workloads: more dynamic and diverse
•
More and new web-based services with diverse users and workloads
•
Broader data center consolidation with fluid mix of applications and workloads
•
More and fluctuating demands for real time resources
•
Competitive and economic pressures driving workload velocity and higher SLA demands
8
Dynamic Workload Utility Summary•
Rapid, scalable and automated allocation and delivery of system resources−
Provisioning of systems based on service demands and policy−
Deployable on a shared heterogeneous pool of distributed data centers
•
Enables real time infrastructures for:−
Delivering services−
Infrastructure as a service −
Global data center consolidation
•
Results:−
Lower hardware and IT admin costs−
Improved service levels for critical users and activities−
Flexibility: grow as needed, offer new services quickly
8
9
Dynamic Workload Utility Summary•
Joint Solution by HP and Cluster Resources, Inc. −Dynamic Workload Utility
•
Cluster Resources’
Moab product−Dynamically allocates resources to match specific
workload needs −Automates the virtualization and provisioning
technologies− Provides flexible management policies− Enforces Quality of Service and SLAs
to makes certain
company objectives are achieved•
Integration of Cluster Resources’
Moab with: −HP BSA (SA + OO) −CMU− iLO
10
Results with DWU •
Move from static silos to dynamic, shared environment
•
Optimize utilization of resources•
Automate management and minimize provisioning errors
•
Improve revenue and productivity•
Achieve scalability and growth•
Support heterogeneous environments•
Offer Infrastructure as a Service
11
•
Scale-out challenges•
Challenges of dynamic IT environments•
DWU overview and capabilities•
DWU architecture•
Usage Scenarios•
Success story•
More information
12
Dynamic Workload Utility Capabilities•
Automated close-loop environment incorporating:−Capacity management −Auto-provisioning − Resources flexing−Monitoring−Virtualization − Reporting and charge-back−Workload scheduling and load balancing of batch and
service jobs
•
Comprehensive event-driven automation capability, including integration with ILO for system events
•
Remote management access to the resources
13
DWU Capabilities (Cont.)•
Management of Service and Batch Jobs−
SLA Driven Dynamic Allocation of Resources to Workload−
Multiple applications share the resources−
Provision capacity on demand i.e. add hardware automatically to server farm based on thresholds
−
Use of Physical and Virtual Servers−
Green Computing•
Provision capacity on demand i.e. add hardware automatically to server farm based on thresholds
•
APIs for integration with 3rd
party software e.g. monitoring tool, etc.
•
Support for Heterogeneous Hardware and OS (incl. non-
HP vendors)•
Scalable to 10,000+ nodes•
IT infrastructure/component for Cloud Computing
14
HP Business Service Automation (BSA)Automate management of client, server, software, network, storage & process
Help desk/ change requests
ITSM
Monitoring events &
alerts
BSM
Service Automation Reporter
Universal CMDB
Operations Orchestration
IT Teams IT Systems
•
Servers & network devices
•
Software
•
Process-enabled storage provisioning
•
Audit
•
Remediation
•
Reporting
•
All applications & components
•
Server, network, storage
•
Applications
•
Configurations
•
Patching
•
Software distribution
•
Discover servers, software network devices, & storage
•
Interdependencies
•
Configurations
Discovery Provisioning & Scaling
Business Service Automation
Server Automation
Network
Automation
Service Automation Visualizer
Client Automation
Storage Automation
Live Network
15
HP Business Service Automation
Comprehensive infrastructure automationIntegrated solutions for client, applications, servers, networks, storage, physical and virtual
Enterprise-class IT service automation
Service-aware automationModel and integrate accurate service views into all automation solutions
Process-powered automationIntegrated process workflows for coordinating operations across team and tools
Proven enterprise-wide scale and deployment architecture
16
Resource management Lifecycle Operations Orchestration
Operations Orchestration is an innovative run book automation platform for IT Operations.
Cost EfficiencyEnables IT departments to eliminate manual and error-prone procedures by automating end-to-end IT processes
Quality and ConsistencyReduces server and application downtime by eliminating the inconsistency and misconfigurations
that are the root cause of most problems
Agility and ResponsivenessAllows IT departments to react more quickly to changing business needs by reducing the time to deploy new infrastructure
17
Server Automation
18
Operating System
Resources
Resource Manager
Meta-Orchestrator
Compute License Network Storage Other
Workload & Application
Batch Workload (Parallel & Serial)
Transaction / Workflows
Other Workload (DB Farm,
Web Farm, Etc.)
Home Grown Application
HPC RM: •
TORQUE• LSF, SGE• PBS Pro• SLURM
License Manager• FlexLM• Etc.
Provisioning Manager•CMU1•xCat2
Virtualization Manager• VMware• Xen
Hardware Monitor:• IPMI1, ILO1
•Nagios2
•Ganglia2
SUSE RedHat Other Scyld AIX Solaris IRIX Other Mac WindowsLinux Linux Linux BProc Unix OS X
Moab in the Software Stack
1HP Products 2Open source software with extensive HP expertise
StorageNetworkServer
Moab Adaptive Computing Suite: Integration Platform, Resource Orchestration, Event Orchestration, Scheduler, & Policy Engine
HP OOHP BSA1
19
•
Scale-out challenges•
Challenges of dynamic IT environments•
DWU overview and capabilities•
DWU architecture•
Usage Scenarios•
Success story•
More information
20
DWU Architecture
Workload Management
Adaptive Resource Orchestration
Load Balancing Optimization
Policy Enforcement
Fault Tolerance
Orchestrator (Moab)
Workload
Administrator Portals (Moab Cloud Manager, HP OO and
HP SA Web Client and Launcher) User Access Portal
(Moab Cloud Portal)
Inventory/Monitoring Hardware Config, Health,
Workload, etc.
Resource Manager Plug-Ins
(iLO, BareMetal, batch, etc)
IT Process Automation (HP OO)Virtual + Physical
Server Provisioning (HP SA)
Dynamic Compute Environment
Network and Storage provisioning
(HP BSA Suite, …)
21
Solution Components
Workload Management
Adaptive Resource Orchestration
Load Balancing Optimization
Policy Enforcement
Fault Tolerance
Orchestrator (Moab)Workload
Administrator Portals (e.g.,Moab
Cloud Manager, BSA portals)
User Access Portal (e.g.,Moab
Cloud Portal)
Inventory/Monitoring Hardware Config, Health,
Workload, etc.
Resource Manager Plug-Ins
(iLO, BareMetal, batch, etc)
ProvisioningVirtual + Physical
Server Provisioning (HP SA)
Dynamic Compute Environment
Network and Storage provisioning
(HP BSA Suite, …)
(4) Interfaces (User, Mgmt, Admin, etc.)
MAP, MCM, HP OVO Dashboard, etc.
Use Interfaces to Simplify Experience
Use Intelligence EngineTo Optimize Workload Placement and
Resource Adaptation DecisionsMoab
(3) Orchestration Intelligence(1) Information Gathering
HP SIMSupermonHP OVO
HP Sitescopeetc.
Gather Informationto Understand State
of Resources
Provision/Adapt Resources usingInformation Gathered
HP SA and HP OO
(2) Resource Provisioning/Adaptation Tools
22
•
Scale-out challenges•
Challenges of dynamic IT environments•
DWU overview and capabilities•
DWU architecture•
Usage Scenarios•
Success story•
More information
23
Usage Scenarios•
Business Continuity & Capacity on Demand: −Dynamic Multi-OS Hybrid Environments−SLA Driven Resource Escalation
•
Infrastructure as a Service: Instantly Create Project and Test Environments
24
Dynamic Multi-OS Hybrid EnvironmentsDynamically
monitor & then re-provision the operating system and/or VM and other environmental factors to meet the needs of current and upcoming workload. Automatically grow and shrink the allocated resources in order to meet QoS
targets.
Distributed ResourcesLinux Workload
Upcoming Workload
Windows Workload
Jobs
Dynamic Workload Utility
Provision(HP BSA, ..)
Allocation
decisions MonitorAllocations
MOAB
Linux
Windows
Adapted Node Allocation over Time
25
Local System
MoabMoab MoabMoab
Work
ScheduledWorkload
SurgeWork
2. Escalation Command Sent 3. Resource Reservation Activated4. Resource Released Until Next Reservation
Workload
LocallySatisfiedWorkload
Surge Workload
Workload Histogram: Re-occurring Surges-
Product Release Cycles, Daily Spikes, etc.
LocallySatisfiedWorkload
Surge Workload
Workload Histogram: Planned Surge-Financial Promotion, Oil Discovery,
Film, Final Product Testing, etc.
1. Reservation Period Begins
Provisioning Provisioning ..e.ge.g HP SAHP SA
Internal, Hosted or Cloud
Resources
SLA Driven Resource Escalation
HP SAHP SA
26
Adaptive IT Infrastructure for Cloud Computing
•
Virtualized Environment –
Fast access to computing resources•
Rapid provisioning & repurposing across multiple data centers•
Self-service web site•
Multi-vendor, multi OS•
Policy-driven Task Automation•
Single point of management and user access•
Dynamic inventory•
Billing and chargeback for use•
Platform for sharing IT services across the enterprise (multiple
data centers)
•
Reduce space & environment requirements•
Manage power and cooling•
Audit and verification
Mission Value•
Reduce costs / Consolidate / Shorten time to delivery / Simplify•
From cost centre to IT business shared services (chargeback)
27
Net
wor
k
MoabMoab Provisioning Orchestration“HP BSA”
StorageN
etw
ork
Request New Resources
Workload & Capacity Management
Adaptive Env. Orchestration
Cloud Access PortalEnd UsersMonitor, Diagnose,
Manage
Admin
Allocation Managementand Billing Integration
Monitor / Configure“HP SIM/HP OM/etc.”
Dynamically
Provision
ManagerReview Service Levels, Capacity
Planning ,etc.
Report Current State & Provision Resources
Compute Resources
Update Allocation, Usage, Charges
Monitor, Maintain
Report State/Key EventsMaintain Usage
Credits & ChargesProvisioning, Updates, etc.
Intelligent Policy Broker
“Adaptive Computing Portal”
Schedule Service Levels
OS, Middleware , Applications
Server Automation
Storage
Interfaces
Information Monitoring Adaptation Tools
Moab Cloud Manager + HP
Portals
Infrastructure as a Service: Instantly Create Project and Test Environments
Storage Automation
Network Automation
AuthenticationServer
WebService
VM Management tools
28
Generic Moab Cloud portal
29
•
Scale-out challenges•
Challenges of dynamic IT environments•
DWU overview and capabilities•
DWU architecture•
Usage Scenarios•
Success story•
More information
30
Defense Information Systems Agency (DISA)
3
0 Customer Presentation9/17/2009
Objective Approach Results•
Rapid access to computing resources
•
Eliminate need to procure physical infrastructure
•
Self service portal through a single, secure interface
•
User configurable server environments
•
Automated provisioning•
Flexible billing options•
Meet DoD
security requirements
•
HP C&I development of shared services utility for Rapid Access Computing Environment (RACE)
•
HP Server Automation and Operations Orchestration for provisioning and configuration management
•
HP Operations Manager for monitoring and control
•
HP Service Manager to automate incident & problem management
•
HP Systems Insight Manager and HP ProLiant
Essentials
•
Cluster Resources Moab for intelligent orchestration and Gold for billing
Business outcomes•
Reduced costs•
Consolidated simplified processes
•
Shortened time to deliveryIT improvements•
Flexible development platforms for Web, Application or Database
•
User can allocate own resources through Web interface
•
Can provision a server in a few minutes
•
CPU, memory, storage, virtual environment provided in one simple solution
31
•
Scale-out challenges•
Challenges of dynamic IT environments•
DWU overview and capabilities•
DWU architecture•
Usage Scenarios•
Success story•
More information
32
Highlights in HP’s
Scale-Out portfolio
ProLiant
BladeSystem
•
New G6 with latest multicore
options
•
Double compute density with BL2x220c
•
Maximum power & cooling efficiency
Performance Optimized Datacenter (POD)
Massive Scale Storage•
StorageWorks
ExDS9100•
MSA200 w/Lustre-based file system support
Top choice in TOP500
Available in 20 and 40 foot containers
New: Ibrix
acquisition
HP Cluster Management Utility•
Provisioning and mgmt for Linux scale
Qualified with Moab
33
ProLiant
SL Extreme Scale-Out (ExSO) server family
ProLiant
z6000 chassisShared infrastructure
–
fans, chassis, power
Save on cost and energy --
per node, rack and data
center
Mix and match configurations
Deploy with confidence
ProLiant
SL160z G6Large memory
-memory-cache apps
ProLiant
SL170z G6Large storage-Web search and database apps
ProLiant
SL2x170z G6Highly dense -
HPC compute
and web front-end apps
34
35
For more information
www.hp.com/go/enablethecloudwww.hp.com/go/cloudwww.hpl.hp.com/research/cloud.html
Send email to [email protected]
Thank you
