Post on 11-Jan-2016
April 21, 20231
Performance Analysis of Cloud ComputingServices for Many-Tasks Scientific Computing
Berkeley, CA, USA
Alexandru Iosup, Nezih Yigitbasi, Dick Epema
Parallel and Distributed Systems Group,Delft University of Technology,The Netherlands
Simon Ostermann, Radu Prodan, Thomas Fahringer
Distributed and Parallel Systems,University of Innsbruck,
Austria
About the Team
• Team’s Recent Work in Performance• The Grid Workloads Archive (Nov 2006)• The Failure Trace Archive (Nov 2009)• The Peer-to-Peer Trace Archive (Apr 2010)• Tools: GrenchMark workload-based grid benchmarking, other
Monitoring and Perf. Eval. tools
• Speaker: Alexandru Iosup• Systems work: Tribler (P2P file sharing), Koala (grid scheduling),
POGGI and CAMEO (massively multiplayer online gaming)• Grid and Peer-to-Peer workload characterization and modeling
April 21, 20232
Many-Tasks Scientific Computing
• Jobs comprising Many Tasks (1,000s) necessary to achieve some meaningful scientific goal• Jobs submitted as bags-of-tasks or over short
periods of time• High-volume users over long periods of time
• Common in grid workloads [Ios06][Ios08]
• No practical definition (from “many” to “10,000/h”)
April 21, 20233
Cloud Futures Workshop 2010 – Cloud Computing Support for Massively Social Gaming 4
The Real Cloud
• “The path to abundance”• On-demand capacity• Cheap for short-term
tasks• Great for web apps (EIP,
web crawl, DB ops, I/O)
• “The killer cyclone”• Not so great
performance for scientific applications1 (compute- or data-intensive)
• Long-term perf. variability2
http://www.flickr.com/photos/dimitrisotiropoulos/4204766418/ Tropical Cyclone Nargis (NASA, ISSS, 04/29/08)
1- Iosup et al., Performance Analysis of Cloud Computing Services for Many Tasks
Scientific Computing, (under submission).
2- Iosup et al., On the Performance Variability of Production Cloud Services,
Technical Report PDS-2010-002, [Online] Available:
http://pds.twi.tudelft.nl/reports/2010/PDS-2010-002.pdf
VS
Research Question and Previous WorkDo clouds and Many-Tasks Scientific Computing
fit well, performance-wise?• Virtualization Overhead
• Loss below 5% for computation [Barham03] [Clark04]• Loss below 15% for networking [Barham03] [Menon05]• Loss below 30% for parallel I/O [Vetter08] • Negligible for compute-intensive HPC kernels [You06] [Panda06]
• Cloud Performance Evaluation• Performance and cost of executing a sci. workflows [Dee08]• Study of Amazon S3 [Palankar08]• Amazon EC2 for the NPB benchmark suite [Walker08] or
selected HPC benchmarks [Hill08]
Theory: just use virtualization overhead results. Practice?
April 21, 20235
April 21, 20236
Agenda
1. Introduction & Motivation2. Proto-Many Task Users3. Performance Evaluation of Four Clouds4. Clouds vs Other Environments5. Take Home Message
Proto-Many Task Users
MTC user• At least J jobs in
B bags-of-tasks
Trace-based analysis• 6 grid traces,
4 parallel productionenvironment traces
• Various criteria (combinations of values for J and B)
Results• “number of BoTs submitted 1,000 & number of tasks submitted
10,000”• Easy to grasp + Dominate most traces (jobs and CPUTime) + 1-CPU jobs
April 21, 20237
April 21, 20238
Agenda
1. Introduction & Motivation2. Proto-Many Task Users3. Performance Evaluation of Four Clouds
1. Experimental Setup2. Selected Results
4. Clouds vs Other Environments5. Take Home Message
Experimental SetupEnvironments
Four commercial IaaS clouds (NIST definitions)• Amazon EC2• GoGrid• Elastic Hosts• Mosso
No Cluster instances(not releasedin Dec’08-Jan’09)
April 21, 20239
Experimental SetupExperiment Design
Principles• Use complete test suites• Repeat 10 times• Use defaults, not tuning• Use common benchmarks
Compare results with results for other systems
Types of experiments• Resource acquisition and release• Single-Instance (SI) benchmarking• Multiple-Instance (MI) benchmarking
April 21, 202310
Resource Acquisition: Can Matter
• Can be significant• For single instances (GoGrid)• For multiple instances (all)
• Short-term variability can be high (GoGrid)
• Slow long-term growth
April 21, 202311
Single Instances: ComputePerformance Lower Than Expected• ECU = 4.4 GFLOPS (at 100% efficient code)
= 1.1GHz 2007 Opteron x 4 FLOPS/cycle (full pipeline)
• In our tests: 0.6-0.8 GFLOPS• Sharing of the same physical machines (working set)• Lack of code optimizations beyond –O3 –funroll-loops• Metering requires more clarification
• Instances with excellent float/double addition perf. may have poor multiplication perf. (c1.medium, c1.xlarge)
April 21, 202312
Multi-Instance: Low Efficiency in HPL
Peak Performance• 2 x c1.xlarge (16 cores) @ 176 GFLOPS,
HPCC-227 (Cisco, 16c) @ 102, HPCC-286 (Intel, 16c) @ 180
• 16 x c1.xlarge (128 cores) @ 1,408 GFLOPS, HPCC-224 (Cisco, 128c) @ 819, HPCC-289 (Intel, 128c) @ 1,433
Efficiency• Cloud: 15-50% even
for small (<128)instance counts
• HPC: 60-70%
April 21, 202313
Cloud Futures Workshop 2010 – Cloud Computing Support for Massively Social Gaming 14
Cloud Performance Variability
• Performance variability of production cloud services• Infrastructure:
Amazon Web Services• Platform:
Google App Engine
• Year-long performance information for nine services• Finding: about half of the cloud services
investigated in this work exhibits yearly and daily patterns; impact of performance variability depends on application.A. Iosup, N. Yigitbasi, and D. Epema, On the Performance Variability of Production Cloud Services, (under submission).
Amazon S3: GET US HI operations
April 21, 202315
Agenda
1. Introduction & Motivation2. Proto-Many Task Users3. Performance Evaluation of Four Clouds4. Clouds vs Other Environments5. Take Home Message
Clouds vs Other Environments
• Trace-based simulation, DGSim (grid) simulator• Compute-intensive, no data IO
• Source Env v Cloud w/ source-like performance v Cloud w/ real (measured) performance• Slowdown for Sequential: 7 times, Parallel: 1-10 times
• Results• Response time 4-10 times higher in real clouds• Good for short-term, deadline-driven projects
April 21, 202316
April 21, 202317
Take Home MessageTake Home Message
• Many-Tasks Scientific Computing• Quantitative definition: J jobs and B bags-of-tasks• Extracted proto-MT users from grid and parallel prod. envs.
• Performance Evaluation of Four Commercial Clouds• Amazon EC2, GoGrid, Elastic Hosts, Mosso• Resource acquisition, Single- and Multi-Instance
benchmarking• Low compute and networking performance
• Clouds vs Other Environments• An order of magnitude better performance needed for clouds• Clouds already good for short-term, deadline-driven sci.
comp.
April 21, 202318
Potential for CollaborationPotential for Collaboration
• Other performance evaluation studies of clouds• The new Amazon EC2 instance—Cluster Compute• Other clouds?
• Data-intensive benchmarks
• General logs• Failure Trace Archive• Grid Workloads Archive
• …
April 21, 202319
Thank you! Questions? Observations?
More Information:• The Grid Workloads Archive: gwa.ewi.tudelft.nl
• The Failure Trace Archive: fta.inria.fr
• The GrenchMark perf. eval. tool: grenchmark.st.ewi.tudelft.nl
• Cloud research: www.st.ewi.tudelft.nl/~iosup/research_cloud.html
• see PDS publication database at: www.pds.twi.tudelft.nl/
email: A.Iosup@tudelft.nl
Big thanks to our collaborators: U. Wisc.-Madison, U Chicago, U Dortmund, U Innsbruck, LRI/INRIA Paris, INRIA Grenoble, U Leiden, Politehnica University of Bucharest, Technion, …