Harness the Power of the Cloud for Grid Computing and Batch Processing Applications
-
Upload
rightscale -
Category
Technology
-
view
442 -
download
0
Transcript of Harness the Power of the Cloud for Grid Computing and Batch Processing Applications
![Page 1: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/1.jpg)
1
Harness the Power of theCloud for Grid Computing
June 29, 2010
![Page 2: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/2.jpg)
2
Your Panel TodayPresenting:• Michael Crandell, CEO• Josh Fraser, Business Development• Dave Welch, Sales Engineer
Q&A:• Matthew Small, Account Manager
Please use the questions window to ask questions anytime!
![Page 3: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/3.jpg)
3
Agenda• Welcome• RightScale Customer Use Cases• Grid Computing Challenges• Grid Computing in the cloud with RightScale• Demo #1• Automated Management• Demo #2• Quantifying the Benefits• Q&A
![Page 4: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/4.jpg)
4
• Cloud Management• Over 20,000 users• 1.2M+ servers
launched• Scaling events:
- 50-3500 servers 3 days- 8000 servers 1 co.- 0 to 35M users/day
• Largest production deployments in the cloud to date
![Page 5: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/5.jpg)
5
RightScale Cloud Management Easily Deploy and Control
![Page 6: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/6.jpg)
6
Grid WorkerGrid WorkerGrid WorkerGrid Worker
Grid WorkerGrid Worker
Customers on the CloudCustomers from Web 2.0 to Enterprise
From to
RightScale Grid Solution Pack
Grid Worker
![Page 7: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/7.jpg)
7
Cloud-based Grid Computing Use Cases
Pharmaceutical Analysis – Researchers expected a protein analysis comparing 2.5 million compounds to take a week of processing on internal servers
• Using hundreds of servers, the job was completed in one day
Web 2.0 – Transcoding images to render video on demand
• Processing time was reduced from hours on internal resources to minutes
Insurance Claims Loss Control – Systems for detecting fraudulent or duplicate claims in batches of millions would have required months of processing time to run and millions of dollars to build in the data center
• Batch runs finished in a few days at significantly lower cost
Financial Data Processing – Back testing environments that analyze data to test new trading strategies
• Trading strategies analyzed faster and more cost-effectively by scaling out servers
![Page 8: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/8.jpg)
8
Grid Computing & Batch Processing Challenges
For IT• High capital investment, typically low capacity utilization• Scheduling conflicts, constant provisioning• Specialized architectural and operational skills• Specialized software applications and unique datasets
For End Users• Wait for resources• Limited to resources available in the datacenter
For Operations For Users01
-Jan
04-Ja
n
07-Ja
n
10-Ja
n
13-Ja
n
16-Ja
n
19-Ja
n
22-Ja
n
25-Ja
n
28-Ja
n
31-Ja
n
03-F
eb
06-F
eb
09-F
eb
12-F
eb
15-F
eb
18-F
eb
21-F
eb
24-F
eb
27-F
eb
02-M
ar
05-M
ar
08-M
ar
11-M
ar
14-M
ar
17-M
ar
20-M
ar
23-M
ar
26-M
ar
29-M
ar0%
50%
100%
Resource Utilization
![Page 9: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/9.jpg)
9
Grid Computing in the Cloud
Cloud Computing Model
Resources on Demand
Virtually Infinite Resources
Pay as You Go
Grid Computing in the Cloud
Resolves Scheduling Issues
Supports Faster Processing
Matches Costs to Demand
![Page 10: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/10.jpg)
10
Grid Computing managed by RightScale
RightScale
Preconfigured components and solutions
Automation and systems management
User and cost control
Grid ComputingManaged by RightScale
Fast and easy provisioning
Reduced administration
Control and visibility
![Page 11: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/11.jpg)
11
Grid Computing Solution Pack
Preconfigured Framework for Grid ComputingProven, Best Practices Architecture on Amazon Web Services• Leverages AWS EC2, SQS, and S3• Multiple AWS Regions and Availability Zones
RightScale Professional Services and Support
“Lilly is leveraging RightScale's infrastructure management interface and services against appliance/application stacks in a ‘vending machine’ concept that allows self-service to infrastructure.”
- Dave Powers, Assoc. Information Consultant, Eli Lilly & Co. Top 25 Information Managers 2010
![Page 12: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/12.jpg)
12
Highlights
Complements existing internal grid engines (Sun Grid, etc.)TBDTBD
![Page 13: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/13.jpg)
13
RightScale Grid ArchitectureAutomated server scaling, operational remediation, server cost optimization
SQS Input Queue SQS Error Queue
SQS Output Queue
Amazon S3Amazon S3
Batch jobs from Your job producer
application
Scalable cloud servers using RightScale Server Templates
Your application or next batch process
job consumer
Worker Daemon
Your code
RightScale Amazon Cloud InfrastructureCustomer code
RightScale Management Interface
![Page 14: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/14.jpg)
14
Job CoordinatorInput file Creation and message Queuing
Work Queue
S3 Buckets
IN OUT LOG
{Work_Unit }
Job Producer CodeInputFile
1. Job Producer creates input file(s) 2. Job Producer constructs a work Unit3. Input files are placed in an S3 bucket4. Work_unit is packed into a message5. Message is placed in the SQS work queue.
{MSG}Encoder
(Pack Message)
Data Structure{ }LEGEND
• Easily schedule with your current scheduler (Tivoli, Condor, other?)
![Page 15: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/15.jpg)
15
Daemon / Worker CodeWork Queue and Work _Unit Processing
Work Queue
Worker daemon
Workercode
AWS Instance
MSGEnv{ }MSG{ }
Result{ }
1. RightScale instantiates server2. Worker Daemon copies a
message3. Unpacks the msg and
downloads files4. Worker code is called with both
the msg_env and original unpacked message
5. Worker does work6. Output files placed in the /out
directory7. Logs are created8. Results returned back to the
Daemon
/local
/tmp /in /out /log
S3 Buckets
IN OUT LOG
{Work_Unit }
Data Structure{ }LEGEND
![Page 16: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/16.jpg)
16
Daemon Results Handling
Elasticity daemon
Workercode
AWS Instance
Result{ }
Data Structure{ }LEGEND
1. Upload Output Files2. Upload log files3. Delete /tmp and /in files4. If worker returns success
A. Queue Results in results_queue
B. Post results to http server5. Else queue results to the error
queue6. Queue statistics in Audit Queue7. Delete Message from input
queue
/local
/tmp /in /out /log
S3 Buckets
IN OUT LOG
Results Queue {Results} {Results} Consumer
Error Queue {Results}
Audit Queue {Audit} {Audit}Consumer
{Results}Consumer
![Page 17: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/17.jpg)
17
Demo #1• Pre-configured Grid Architecture
• Job Coordinator Plug-ins
• Graceful Provisioning and De-provisioning
• Automated Queue Handling
![Page 18: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/18.jpg)
18
Automated Management
Server Array ScalingYour grid
processing code
Worker Daemon
1. ALERTS or ScheduleA. Scale up/down based on monitoring system metricsB. Scale up/down based on set schedule – TOD, DOW
2. QUEUES STATISTICSA. # of backlogged msgs per instanceB. Time msgs have stayed in queue
1) Avg. of the last 10 msgs2) Max length of the last 10 msgs
C. If idle after 55 mins from launch
![Page 19: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/19.jpg)
19
Automated Management Key Benefits
![Page 20: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/20.jpg)
20
Demo #2• Configuring array automation
• Analyze your results
![Page 21: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/21.jpg)
21
Key Economic Benefits
Cost Savings
No CAPEX
Pay only for what is used
SaaS-based management model
Agility
Your job is always first in line
Unlimited resources
Pre-configured, proven architectures
![Page 22: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/22.jpg)
22
Amazon’s Public Data Sets
Centralized repository of public data sets AWS is hosting the public data sets at no charge for the community, users pay only for the compute and storage they use for their own applications.Examples:• Economics data, e.g. the 2000 US Census • Life Sciences data, e.g GenBank
http://aws.amazon.com/publicdatasets/
![Page 23: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/23.jpg)
23
Agility Example: Cost-neutral Equation
This graphic compares running the same 10,000 jobs on 2 servers versus 1000 servers. The cost is the same for either scenario in RightScale using
AWS, but the difference in elapsed time is 499 hours.(assuming each server can process 10 jobs/hour)
10,000jobs
10,000jobs
2 server cloud
Total processing time: 500 hours
Total processing time: 1 hour
1000 server cloud
Output data
Output data
![Page 24: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/24.jpg)
24
Example Cost Savings – 1/4 The Price
Item Annual Cost
30 dedicated HP ProLiant DL360 G6 servers at $3,879 each, amortized over three years. Each server has two quad-core processors, for a total of 240 cores.
$38,790
Commercial grid middleware at $399/processor with basic support $95,760
Open source Resource Manager software at $199/processor with support contract $47,760
Systems administration*: 1.5 FTE at $150,937/year** $ 226,406 Hosting fees at $1,000/month for 30 machines *** $ 360,000
Total $768,716
Item Annual Cost
10 extra-large cloud servers running for a total of 6,570 hours per month at $.68 per hour. $53,611
RightScale subscription at $1,500/month plus $5,000 annual fee $23,000
Systems administration*: 0.6 FTE at $150,937/year** $90,562
Total $167,173
In-house Grid – with commercial grid software
Cloud-based Grid Computing
![Page 25: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/25.jpg)
25
Contact us:• [email protected]• (866) 720-0208• Twitter: @rightscale
Grid Computing Whitepaper: www.rightscale.com/gridsolution
Webinar Archive: www.rightscale.com/webinars
Q&A - Getting Started
![Page 26: Harness the Power of the Cloud for Grid Computing and Batch Processing Applications](https://reader034.fdocuments.us/reader034/viewer/2022042707/58f18d791a28ab56018b459f/html5/thumbnails/26.jpg)
26
Thank You!