Utilizing Public AND Private Clouds with Bright Cluster Manager
-
Upload
ian-lumb -
Category
Technology
-
view
228 -
download
1
Transcript of Utilizing Public AND Private Clouds with Bright Cluster Manager
![Page 1: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/1.jpg)
Utilizing Public AND Private Clouds
with Bright Cluster Manager
Ian Lumb
Product Marketing Manager
Bright Topics Webinar RECORDING
March 25, 2015
![Page 2: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/2.jpg)
2
How Bright Cluster Manager allows you to seamlessly
make use public clouds like Amazon Web Services
(AWS)
How Bright Cluster Manager allows you to rapidly
deploy a private cloud based on OpenStack
Key Takeaways
WEBINAR RECORDING
![Page 3: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/3.jpg)
3
Public cloud
• Off-premise IT capabilities or applications, provided by others
Private cloud
• On-premise enablement of cloud capabilities with existing IT
Hybrid cloud
• Some combination of public and private clouds
Definitions
http://www.wired.com/2014/10/public-vs-private-cloud/
![Page 4: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/4.jpg)
4
How Bright Cluster Manager allows you to
seamlessly make use public clouds like Amazon
Web Services (AWS)
How Bright Cluster Manager allows you to rapidly
deploy a private cloud based on OpenStack
Key Takeaways
![Page 5: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/5.jpg)
“Both public and private clouds have management implications. However, by choosing a public cloud solution, an organization can offload much of the management responsibility to its cloud vendor. In a private cloud scenario, there is significant demand on resources to specify, purchase, house, update, maintain, and safeguard the physical infrastructure. Financially, deploying a private cloud can also create a large initial capital expense, with subsequent investment required as new equipment and capacity is added.”
http://www.akamai.com/html/solutions/public_private_cloud.html
![Page 6: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/6.jpg)
Scenario I — “Cluster on Demand”
node001
head nodenode002
node003
Cloud Utilization
![Page 7: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/7.jpg)
Scenario II — “Cluster Extension”
head node
node001 node002 node003
node004
node005
node006
node007
Cloud Utilization
![Page 8: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/8.jpg)
node004
node005
node006
node007
head node
node001 node002 node003
Cloud Utilization
data
1. User submits job to queue
2. Bright creates “data-transfer” job
3. Bright runs compute job when data-transfer job is complete
4. Bright transfers output data back after completion
job
Data-Aware Scheduling to the Cloud
![Page 9: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/9.jpg)
Case Study: Life Sciences
Customer: Major pharmaceutical company
Existing clusters at four locations
Regulatory and compliance concerns
Challenge
Integrate AWS into existing IT infrastructure • … with minimal administrative overhead
Solution
Bright Cluster Manager + PBS Professional + AWS• Secure AWS instance added as an on-demand location
Access to licensed software
Persistent resources/services– Bright Cloud Director
– AWS S3 and Glacier
Results
AWS seamlessly integrated into existing IT infrastructure with minimal administrative overhead
Case Study
![Page 10: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/10.jpg)
10
How Bright Cluster Manager allows you to seamlessly
make use public clouds like Amazon Web Services
(AWS)
How Bright Cluster Manager allows you to rapidly
deploy a private cloud based on OpenStack
Key Takeaways
![Page 11: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/11.jpg)
11
1. Cost effectiveness
• Trade CAPEX for OPEX
Open source is OPEX neutral
2. Data security
• IT security better aligned with business needs
3. Better control
4. True flexibility
• Customize as needed, not limited to a generic offering
5. Sync speed
• Internal networks don’t need to hide latency
10 reasons you should build your own cloud
Wallen, TechRepublic, 3/2013http://www.techrepublic.com/blog/10-things/10-reasons-you-should-build-your-own-cloud/
![Page 12: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/12.jpg)
12
6. Company integration
• Leverage existing in-house services – e.g., storage, ID
7. Unlimited size
• Trade CAPEX for OPEX
8. User management
• Customize as needed, not limited to a generic offering
9. Backup control
• IT backups better aligned with business needs
10.HIPPA compliance
• Compliance challenges simplified
10 reasons you should build your own cloud
Wallen, TechRepublic, 3/2013http://www.techrepublic.com/blog/10-things/10-reasons-you-should-build-your-own-cloud/
![Page 13: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/13.jpg)
http://docs.openstack.org/admin-guide-cloud/content/figures/2/figures/openstack-arch-havana-logical-v1.jpg
OpenStack Architecture (Havana)
![Page 14: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/14.jpg)
Role-Based Service Management
• OpenStack consists of services
• Services accomplish specific tasks
• Dependencies, interactions, …
• Bright role
• A task that can be assigned to a node
• Assigning/un-assigning role will
• Write out config file based on role parameters
• Start/stop/monitor relevant services
• Add/remove entries to Keystone service registry
• Bright roles can be composed
• Addresses dependencies and interactions
• e.g., introduction of HA capabilities
![Page 15: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/15.jpg)
OpenStack Configuration
OpenStack configuration through roles
• OpenStack consists of several services to accomplish specific tasks
• Tasks can be assigned by assigning roles to nodes
• Example roles:
Compute (Nova) Image (Glance)
Block Storage (Cinder) Dashboard (Horizon)
Auth (Keystone) Networking (Neutron)
• Assigning/unassigning role will:
• Write out config file based on role parameters
• Start/stop/monitor relevant services
• Add/remove entries to Keystone service registry
• OpenStack configuration is dynamically updated with cluster changes (e.g. hostname, network settings)
• Instance migration works out of the box
![Page 16: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/16.jpg)
Managing OpenStack Clusters
• Managing OpenStack Clusters just as difficult as other types of clusters
• Without proper infrastructure, OpenStack will not be able to run
• Setting up OpenStack manually is often extremely complex for production setups
• Bright Cluster Manager provides single-pane-of-glass to manage and monitor all aspects of OpenStack cluster
• Includes:
• Hardware (set up, configuration, monitoring)
• Operating system (provisioning, updates)
• OpenStack installation
• OpenStack configuration
• Users
• Bright Cluster Manager provides perfect environment for OpenStack to run on
![Page 17: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/17.jpg)
17
![Page 18: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/18.jpg)
OpenStack Metrics & Healthchecks
OpenStack related metrics. For example:
• Per-tenant quotas
• Swift: Object size, number of containers, API request rate, data transfer rates
• Cinder: Volume size
• Nova: CPU time used, disk I/O, network I/O
• Neutron: remaining number of floating IPs
OpenStack related healthchecks
• Verify that OpenStack related services are running properly
![Page 19: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/19.jpg)
Storage & Networking Management
Ceph is used by default for all storage related to OpenStack
• Bright integrates with Ceph to make it easy to set up, manage and monitor
• Ceph used for S3 & Swift compatible object storage
• Ceph used for block storage of VMs and storing OpenStackimages, volumes and snapshots
Bright uses VLAN pools or VxLAN to allow private networking to be set up between VMs.
• Several types of network setups supported
• No headaches
![Page 20: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/20.jpg)
Provision VMs using Bright
Allow VMs to be provisioned using Bright software images
• In plain OpenStack VMs are instantiated using static image
• After initial deployment managing software on VMs will become challenging
• Bright adds value by booting VMs into Bright node-installer and provisioning Bright software image onto VM
• VMs can then be managed the same way as physical nodes
• VMs will show up in Bright management tools
• VMs can be updated by updating the Bright software image and using Bright’s provisioning system to propagate updates
![Page 21: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/21.jpg)
Virtualized Sub-clusters
• Virtualized Bright clusters can be spun up with a few clicks within minutes using OpenStack Heat orchestration
• Allows tenants to have full control over their own Bright clusters
• Clusters can be easily be resized to meet tenant demands
• OpenStack Ironic also allows non-virtualized sub-clusters to be created
• Single CMGUI instance can manage both physical cluster as well as sub-clusters in single view
• SR-IOV allows HPC resources (e.g. InfiniBand and GPU) to be made available directly to VMs at near-native performance (work in progress)
![Page 22: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/22.jpg)
Product Differentiation
• Installs on bare metal through to the OpenStack distro
• Single-pane-of-glass management interface for HPC and Big Data Analytics clusters/clouds
Addresses the physical cluster as well as OpenStack
• Fully manages OpenStack services
• Customized monitoring and health checks
• Multiple instances of OpenStack
Architected specifically for OpenStack
• Simultaneous, independent instances on dedicated hardware
• Time-sliced instances on shared hardware
Supports various OpenStack usage models
![Page 23: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/23.jpg)
Slide courtesy Adaptive Computinghttps://vimeo.com/115659153
![Page 24: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/24.jpg)
24
How Bright Cluster Manager allows you to seamlessly
make use public clouds like Amazon Web Services
(AWS)
How Bright Cluster Manager allows you to rapidly
deploy a private cloud based on OpenStack
Key Takeaways
![Page 26: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/26.jpg)
Additional Slides
![Page 27: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/27.jpg)
server
farmsbig data
clusters
HPC
clusters
private
clouds
storage
clusters
database
clusters
Trends
- Centralization
- Consolidation
- Standardization
- Cloud
- General purpose
clusters
- Private clouds
- Across on-premise
and cloud
![Page 28: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/28.jpg)
28
The Problem — “clusters are difficult”
All clusters may need
• deployment
• provisioning
• updating
• managing
• monitoring
• security
• health checking
• alerting/alarming
• virtualization
• cloud
• workload management
What makes it hard
• everything affects everything
else
• everything changes all the
time
• hardware breaks all the time
• cloud, virtualization increase
complexity
IT Pain
• Downtime
• Delays
• Personnel cost
• Inability to react
![Page 29: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/29.jpg)
29
The Problem According to IDC
Clusters are still hard to
setup, operate, use and
manage
New buyers require “ease-of-
everything”
The shortage of skilled people is
a major hindrance
Software is still the #1 roadblock
Better management software is
needed
“
”
![Page 30: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/30.jpg)
30
The problem with “Toolkits”
Toolkits — “A patchwork of disparate tools”• Tools typically used: Ganglia, Nagios, Cfengine, System Imager,
Puppet, Chef, Cobbler, Hobbit, Big Brother, Zabbix, etc.
• Scripts
Issues with the “toolkit” approach:• Scripts poorly documented and hard to maintain
• Tools not designed to work together
• Each tool has its own user interface (CLI/GUI)
• Each tool has its own agent and database
• Tools rarely designed for scale & high performance
• Accelerators (GPUs, Phi) often not supported
Making a collection of unrelated tools work together• Requires a lot of expertise and scripting
• Rarely leads to a really easy-to-use and scalable solution
![Page 31: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/31.jpg)
31
A Better Solution
Bright Cluster Manager takes a much more fundamental & integrated approach• Designed and written from the ground up
• Single cluster management agent provides all functionality
• Single, central database for configuration and monitoring data
• Single UI for ALL cluster management functionality
Which makes Bright Cluster Manager …• Extremely easy to use
• Extremely scalable
• Secure & reliable
• Complete
• Flexible
• Maintainable
![Page 32: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/32.jpg)
32
CMDaemon
Bright Architecture
SOAP/JSO
NAPI+SSL
Cluster
Management
GUI
Cluster
Management
Shell
Web-Based
User Portal
Third-Party
Applications
SQL database
head node
node002
node004
node003
node001
node005
![Page 33: Utilizing Public AND Private Clouds with Bright Cluster Manager](https://reader034.fdocuments.us/reader034/viewer/2022051213/55acd0841a28ab23798b4591/html5/thumbnails/33.jpg)
GPUs in the Cloud? The Top Four Reasons
1. You can realize possibilities using the cloud
• You can scale up and scale out
2. You still realize the promise of GPU programmability
• … via HPC in the cloud
3. Your use of the cloud is transparent
• You’ve found ways to `hide’ latency
• Constraints apply for MPI apps
4. Your go-to apps still work in the cloud
http://info.brightcomputing.com/Blog/bid/196290/The-Top-4-Reasons-You-Should-Try-Cloud-Based-GPUs-for-HPC