Grid is Dead ? Nimrod on the Cloud
-
Upload
adianto-wibisono -
Category
Technology
-
view
1.101 -
download
0
Transcript of Grid is Dead ? Nimrod on the Cloud
![Page 1: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/1.jpg)
Mixing the Grid and Clouds: High-throughput Science using Nimrod
orIs the Grid Dead?
David Abramson
Monash e-Science and Grid Engineering Lab (MESSAGE Lab)Faculty of Information Technology
Science Director: Monash e-Research Centre
ARC Professorial Fellow
1
![Page 2: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/2.jpg)
Instructions ..• To highlight their science and its success' so
far and how their work utilizes advanced cyberinfrastructure
• Identify potential ITC barriers to ongoing success
• Paint a picture of the future for their research in the next 3 to 5 years and what demands it may create for cyberinfrastrcture
• Identify concrete experiments or demonstrations which will utilize and/or stress the infrastructure within 12-24 months
![Page 3: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/3.jpg)
What have we been doing over the Pacific?
![Page 4: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/4.jpg)
Strengthen Existing and Establish New Collaborations
Work with Science Teams to Advance Grid Technologies and
Improve the Underlying Infrastructure
In the Pacific Rim and Globally
PRAGMA
http://www.pragma-grid.net
A Practical Collaborative Framework
IOIT-VN
![Page 5: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/5.jpg)
PRIME @ Monash
• Engaged in PRIMEsince 2004
• Projects range from bio-engineering, theoretical chemistry to computer science
• Has underpinned long lasting academic collaborations– Publications
– Presentations at conferences
• Undergraduate students without research experience!
![Page 6: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/6.jpg)
MURPA Seminars
The massively increased
bandwidth was transformational.
Quantity begat quality.
Alan Finkel,
Chancellor, Monash Univ
I’ve participated in numerous
video conferences to date but
nothing like this. The quality
was so high that the
experience was almost as if we
were all in the same room.
![Page 7: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/7.jpg)
Students give seminars
![Page 8: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/8.jpg)
A bit about hype …Clouds and Grids and ….
![Page 9: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/9.jpg)
Gartner Hype Cycle 2000
![Page 10: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/10.jpg)
Gartner Hype Cycle 2005
![Page 11: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/11.jpg)
Gartner Hype Cycle 2007
![Page 12: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/12.jpg)
Gartner Hype Cycle 2008
![Page 13: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/13.jpg)
Gartner Hype Cycle 2009
![Page 14: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/14.jpg)
Background and motivation
![Page 15: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/15.jpg)
Introduction
• University research groups have used varying sources of infrastructure to perform computational science– Rarely been provided on a strict commercial basis.
– Access controlled by the users
– High end facilities peer re-viewed grant, usually made in terms of CPU hours
• Cloud computing is a major shift in – provisioning
– delivery of computing infrastructure and services.
• Shift from – distributed, unmanaged resources to
– scalable centralised services managed in professional data centres, with rapid elasticity of resource and service provisioning to users.
– Commercial cloud services
![Page 16: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/16.jpg)
Policy and technical challenges
• Free resources will not disappear!
• Commercial clouds could provide an overflow capability
• Potential – perform base-load computations on “free”
resources,
– pay-as-you-go ser-vices to meet user demand.
• To date, very few tools can support both styles of resource provisioning.
![Page 17: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/17.jpg)
Grid Enabled Elasticity
• Resources maintained by home organisation
• Distinct administrative domains• Unified compute, instruments and data• Middleware layer• Never solved deployment
– See Goscinski, W. and Abramson, D. “An Infrastructure for the Deployment of e-Science Applications”, in “High Performance Computing (HPC) and Grids in Action”, Volume 16 Advances in Parallel Computing, Editor: L. Grandinetti, March 2008, approx. 540 pp., hardcover, ISBN: 978-1-58603-839-7.
• Standards exploded this vision!– Plus a whole load of useless computer
scientists!
17
![Page 18: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/18.jpg)
Cloud Enabled Elasticity
• Home resource expands elastically
• Cloud providers “join” home resource
• Virtual machines deployed on demand
• Scalable infrastructure– Compute– Doesn’t address instruments
and data• Do we still have a whole load
of useless computer scientists?
18
![Page 19: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/19.jpg)
• Grid (Wide Area)– Wide area computing
– Instruments, data
– Security
– File transport
• Cloud (Local Area) – Elastic resources
– Virtual machines (deployment)
• Underpinned by a computational economy!– Abramson, D., Giddy, J. and Kotler, L. “High Performance Parametric Modeling
with Nimrod/G: Killer Application for the Global Grid?”, International Parallel and Distributed Processing Symposium (IPDPS), pp 520- 528, Cancun, Mexico, May 2000
Hybrid solutions
![Page 20: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/20.jpg)
High throughput science with Nimrod
![Page 21: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/21.jpg)
Nimrod supporting “real” science
• A full parameter sweep is the cross product of all the parameters (Nimrod/G)
• An optimization run minimizes some output metric and returns parameter combinations that do this (Nimrod/O)
• Design of experiments limits number of combinations (Nimrod/E)
• Workflows (Nimrod/K)
ResultsResultsNimrod/O Results
![Page 22: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/22.jpg)
22
Antenna DesignDrug Docking
Aerofoil DesignAerofoil Design
![Page 23: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/23.jpg)
Nimrod/K Workflows• Nimrod/K integrates Kepler with
– Massivly parallel execution mechanism
– Special purpose function of Nimrod/G/O/E
– General purpose workflows from Kepler
– Flexible IO model: Streams to files
Authentication
GUI
Vergil
SMS
Kepler
Core
Extensions
Ptolemy
…Kepler GUI Extensions…
Actor&Data
SEARCH
Type
System
Ext
Provenance
Framework
Kepler
Object
Manager
Documentation
Smart
Re-run /
Failure
Recovery
![Page 24: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/24.jpg)
Parameter Sweep Actors
• Using a MATLAB actor provided by
Kepler
• Local spawn
• Multiple thread ran concurrently on
a computer with 8 cores (2 x quads)
• Workflow execution was just under
8 times faster
• Remote Spawn
• 100’s of remote processes
![Page 25: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/25.jpg)
Nimrod/EK Actors
• Actors for generating and analyzing designs
• Leverage concurrent infrastructure
![Page 26: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/26.jpg)
Nimrod/OK Workflows
26
• Nimrod/K supports parallel execution
• General template for search– Built from key
components
• Can mix and match optimization algorithms
![Page 27: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/27.jpg)
A recent experimentResource #jobs completed Total job time
(h:m:s)
μ / σ Job runtime
(mins)
East 818 1245:37:23 91/5.7
EC2 613 683:34:05 67/14.2
![Page 28: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/28.jpg)
A Grid exemplar
![Page 29: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/29.jpg)
Microscopes
ClustersStorage
Visualization
Grid
Middleware
Grid Architecture for Microscopy
![Page 30: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/30.jpg)
ARC Linkage Grant with LeicaRemote control of Leica Microscope from Kepler
Nov 2008
First OptiPortal/Kepler link Feb 2009
First remote control of Leica Microscope in
Germany to Opti-portal in Australia using Kepler
March 2009.
![Page 31: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/31.jpg)
Bird’s eye capture and display
![Page 32: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/32.jpg)
Zooming into area of interest
![Page 33: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/33.jpg)
Image cleanup and rendering
![Page 34: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/34.jpg)
Image cleanup and renderingParallelism for free!
![Page 35: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/35.jpg)
Strawman Project:
Grid Enabled Microscopy Across the
Pacific (GEMAP)?
• Remote microscopes
– Currently Leica
• Mix of Compute Clusters
– University Clusters (Monash)
– NCRIS (APAC grid)
– Rocks Virtual Clusters (UCSD)
– Commercial services
(Amazon)
• Distributed display devices
– OptIPortals
• Cloud time
– Which cloud?
– Who pays?
• Network
– Reservation?
– Who pays?
• Project funding
– Who pays?
![Page 36: Grid is Dead ? Nimrod on the Cloud](https://reader035.fdocuments.us/reader035/viewer/2022081401/55a22a441a28aba8718b46d9/html5/thumbnails/36.jpg)
• Faculty Members– Jeff Tan
– Maria Indrawan
• Research Fellows– Blair Bethwaite
– Slavisa Garic
– Donny Kurniawan
– Tom Peachy
• Admin– Rob Gray
• Current PhD Students– Shahaan Ayyub
– Philip Chan
– Colin Enticott
– ABM Russell
– Steve Quinette
– Ngoc Dinh (Minh)
• Completed PhD Students– Greg Watson
– Rajkumar Buyya
– Andrew Lewis
– Nam Tran
– Wojtek Goscinski
– Aaron Searle
– Tim Ho
– Donny Kurniawan
• Funding & Support– Axceleon
– Australian Partnership for Advanced Computing (APAC)
– Australian Research Council
– Cray Inc
– CRC for Enterprise Distributed Systems (DSTC)
– GrangeNet (DCITA)
– Hewlett Packard
– IBM
– Microsoft
– Sun Microsystems
– US Department of Energy
37