The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf ·...
Transcript of The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf ·...
![Page 1: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/1.jpg)
The University of WashingtoneScience Institute
Ed LazowskaBill & Melinda Gates Chair in
Computer Science & EngineeringUniversity of Washington
DirectorUniversity of Washington
eScience Institute
Cloud Futures 2010April 2010
http://lazowska.cs.washington.edu/cloud2010.pdf
![Page 2: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/2.jpg)
This morning
The nature of eScienceA bit of historyThe University of Washington eScience InstituteSome example activitiesA few observationsA plug for computing research
![Page 3: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/3.jpg)
eScience: Sensor-driven (data-driven) science and engineering
Transforming science (again!)
Jim Gray
![Page 4: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/4.jpg)
Dan Reed
![Page 5: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/5.jpg)
![Page 6: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/6.jpg)
TheoryExperiment
Observation
![Page 7: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/7.jpg)
TheoryExperiment
Observation
![Page 8: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/8.jpg)
TheoryExperimentObservation
[John Delaney, University of Washington]
![Page 9: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/9.jpg)
TheoryExperiment
ObservationComputational
Science
![Page 10: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/10.jpg)
TheoryExperiment
ObservationComputational
ScienceeScience
![Page 11: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/11.jpg)
eScience is driven by data more than by cycles
Massive volumes of data from sensors and networks of sensors
Apache Point telescope, SDSS
80TB of raw image data (80,000,000,000,000 bytes)
over a 7 year period
![Page 12: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/12.jpg)
Large Synoptic Survey Telescope (LSST)
40TB/day(an SDSS every two days),
100+PB in its 10-year lifetime
400mbps sustained data rate between
Chile and NCSA
![Page 13: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/13.jpg)
Large Hadron Collider
700MB of dataper second,
60TB/day, 20PB/year
![Page 14: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/14.jpg)
IlluminaHiSeq 2000 Sequencer
~1TB/day
Major labs have 25-100
of these machines
![Page 15: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/15.jpg)
Regional Scale Nodes of the NSF
Ocean Observatories Initiative
1000 km of fiber optic cable on the
seafloor, connecting thousands of
chemical, physical, and biological
sensors
![Page 16: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/16.jpg)
The Web
20+ billion web pages x 20KB = 400+TB
One computer can read 30-35 MB/sec
from disk => 4 months just to read the web
![Page 17: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/17.jpg)
eScience is about the analysis of data
The automated or semi-automated extraction of knowledge from massive volumes of data
There’s simply too much of it to look atIt’s not just a matter of volume
VolumeRateComplexity / dimensionality
![Page 18: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/18.jpg)
eScience utilizes a spectrum of computer science techniques and technologies
Sensors and sensor networksBackbone networksDatabasesData miningMachine learningData visualizationCluster computing at enormous scale
![Page 19: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/19.jpg)
eScience is married to the Cloud: Scalable computing and storage for everyone
![Page 20: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/20.jpg)
eScience will be pervasive
Simulation-oriented computational science has been transformational, but it has been a niche
As an institution (e.g., a university), you didn’t need to excel in order to be competitive
eScience capabilities must be broadly available in any institution
If not, the institution will simply cease to be competitive
![Page 21: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/21.jpg)
Some history, from astronomy
Apache Point telescope, SDSS
80TB of raw image data (80,000,000,000,000 bytes)
over a 7 year period
![Page 22: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/22.jpg)
Project planSDSS was budgeted as a $16 million projectThe software was to be written by astronomy faculty during the summers, when they weren’t teachingUse Objectivity as the data store
Developed by Motorola for the Iridium satellite project
![Page 23: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/23.jpg)
Project reality$80 million30% spent on software, plus Microsoft’s enormous contributions through Jim Gray and his collaboratorsResearch impact: “If it weren’t for Jim Gray’s contributions, SDSS would have been more likely to yield 100 research papers than the 5,000 that actually resulted.”
- Andy Connolly, University of Washington
![Page 24: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/24.jpg)
How’d it come to be?
![Page 25: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/25.jpg)
Large Synoptic Survey Telescope (LSST)
40TB/day(an SDSS every two days),
100+PB in its 10-year lifetime
400mbps sustained data rate between
Chile and NCSA
![Page 26: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/26.jpg)
Why?
[Andy Connolly, University of Washington]SDSS LSST
![Page 27: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/27.jpg)
NSF ReviewDecember 15-17, 2009
Tucson, AZ
NSF ReviewDecember 15-17, 2009Tucson, AZ
LSST Data Management System is widely distributed
Base SiteBase CenterCo-locatedData Access Center (DAC)
Archive CenterCo-locatedData Access Center (DAC)
Archive SiteHeadquarters SiteSystems OperationsCenter (SOC)
Education and PublicOutreach Center (EPOC)
• Site• A physical
location/space that hosts DM centers
• Connected via dedicated, protected fiber optic circuits
• Center• A DM functional
capability hosted at a Site
[Andy Connolly, University of Washington, and LSST]
![Page 28: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/28.jpg)
LSST Data Management System relies on large-scale parallelism
• With few exceptions, LSST pipeline processing is “embarrassingly parallel”
– 3024 parallel image readouts
– O(108) sky tiles– O(109) objects
• Computational clusters are well matched to the available parallelism
– 5000 cores at Base
– 12000 (yr1) –33000 (yr10) cores at Archive
• Middleware implements flexible pipeline/ production model of parallelism
[Andy Connolly, University of Washington, and LSST]
![Page 29: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/29.jpg)
Project planFully 30% of project budget is allocated to software
![Page 30: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/30.jpg)
Data management in computational astrophysicsfopen()fread()fwrite()fclose()scp
– Jeff Gardner, UW eScience Institute
Each simulation generates a sequence of snapshots; each snapshot is a single flat file; analysis is via C or Fortran programs
But astronomy is substantially ahead of most other fields
![Page 31: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/31.jpg)
Data management in biology
90% of all business data is maintained in spreadsheets– Enrique Godreau, Voyager Capital
![Page 32: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/32.jpg)
Top faculty across all disciplines understand and fear the coming data tsunami
Survey of 125 top investigators
“Data, data, data”Flat files and Excel are the most common data management tools
Great for Microsoft …lousy for science!
Typical science workflow:2 years ago: 1/2 day/weekNow: 1 FTEIn 2 years: 10 FTE
Need tools, tools, tools!
![Page 33: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/33.jpg)
The University of Washington eScience Institute
Motivating observationsLike simulation-oriented computational science, data-intensive science will be transformationalUnlike simulation-oriented computational science, data-intensive science will be pervasiveEven more broadly than simulation-oriented computational science, data-intensive science draws on new techniques and technologies from computer science, statistics, and other fieldsCloud services are essential – “get computing out of the closet”If we don’t lead in the development and application of these techniques and technologies, we’re going to lose
![Page 34: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/34.jpg)
MissionHelp position the University of Washington at the forefront of research both in modern eScience techniques and technologies, and in the fields that depend upon these techniques and technologies
StrategyBootstrap a cadre of Research ScientistsHelp leading faculty become exemplars and advocatesBroaden impact by aggressive community-building and sharing of expertise and facilitiesAdd faculty in key fields
Launched in July 2008 with $1 million in permanent funding from the Washington State Legislature
Many grants received since then
![Page 35: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/35.jpg)
Technical staff
David Beck
Jeff Gardner Bill Howe
Erik Lundberg Chance Reschke
![Page 36: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/36.jpg)
Environmental metagenomics / metatranscriptomics / metaproteomics
Ginger Armbrust
![Page 37: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/37.jpg)
Study microbial populations sampled from the environment instead of individual organisms
Who is there? Which organisms make up the population?
What are they doing?Which metabolic pathways are present and active (and who is doing what)?
Compare datasetsAcross a transect (nearshore vs. deep ocean)Before/after some event (e.g., Spring flooding)Across salinity/temperature gradientsDiurnal cycles (day/night)
![Page 38: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/38.jpg)
Environmental Sampling
Pubic annotation DBs
Sequencing
metadata
search hitscorrelate diversity w/environment?
correlate diversity w/nutrients?
find new genes?find new taxa and their distributions? compare meta*omes?
Pfams, TIGRfams,COGs, FIGfams
Phylogeneticanalysis
taxonomic info
![Page 39: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/39.jpg)
Environmental Sampling
Pubic annotation DBs
Sequencing
Phylogeneticanalysis
metadata
search hits
taxonomic info
correlate diversity w/environment?
correlate diversity w/nutrients?
find new genes?find new taxa and their distributions? compare meta*omes?
Pfams, TIGRfams,COGs, FIGfams
SQL
“That took me a week with Excel!”“I can do science again!”
SQLShare
![Page 40: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/40.jpg)
![Page 41: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/41.jpg)
![Page 42: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/42.jpg)
Protein structure prediction and design
David Baker
![Page 43: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/43.jpg)
![Page 44: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/44.jpg)
![Page 45: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/45.jpg)
![Page 46: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/46.jpg)
![Page 47: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/47.jpg)
IP licenses with UW
Arzeda CorporationNew enzymes to drive the industrial biotech revolution
Spin-out from UW research group of David Baker from the Dept. of Biochemistry
At the convergence of digital biology and green chemistry
World leader in the computational design and commercialization of novel, proprietary enzymes
Computational enzyme design methodology in Rosetta
Incorporated in Seattle, WA
Seed investment, WRF Capital
2y research partnership w/ Dupont
March 08 Apr. 08 Apr. 09 June 09
NSF and USDA SBIR awards
Feb. 10
![Page 48: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/48.jpg)
Arzeda’s Platform: The Infrastructure LayerAchieving Scalability through Cloud Computing
Condor Scheduler
Condor Router
1000s of on-demand cores1Gb to 4Gb RAM each100Mb+ connectivity
1000s of on-demand cores1Gb to 4Gb RAM each100Mb+ connectivity
(integration with Condor will be operational late 2010)
Arzeda’s analysis cluster:
32 High-Perf. cores4GB RAM each
10Gb Ethernet connectivity
ARZETTA™(Arzeda’s design software)
Scalability: immediate scaling to 1000s of cores; only OpEx.Price performance: currently $0.08 per hour, going downMore info on condor: http://www.cs.wisc.edu/condor/
![Page 49: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/49.jpg)
Arzeda’s Cloud Computing WorkflowA Unified Interface to the Cloud based upon Open-Source Tools
User defines input files for her/his enzyme design task
Arzetta™ Source Code (C++)Cross-compiled
(Linux/MacOS/Windows HPC server)
Prepare Condor submission script on Linux Condor server
mark the submission as ‘EC2’ or ‘Azure’request a number of cores
Submit Condor script
Condor ‘startup’ hook scripts :start-up instancestransfer executable to instancestransfer input files
Condor ‘finalize’ hook scripts :copy all the output data onto local filesystemterminate instances
User analyzes the results (filters computational designs)
Uses Azure’s cloud filesystem (shared by all
instances)
Each instance uses only its local filesystem
(not S3)
![Page 50: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/50.jpg)
Azure Ocean:Azure Ocean:Visualization and Workflow for Ocean ScienceVisualization and Workflow for Ocean Science
![Page 51: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/51.jpg)
[John Delaney, University of Washington]
![Page 52: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/52.jpg)
![Page 53: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/53.jpg)
John Delaney
![Page 54: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/54.jpg)
Azure OceanAzure Ocean
COVE for Visualization
Trident for Processing
Azure for Data+ +
![Page 55: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/55.jpg)
COVECOVE
Research into new interfaces for crossResearch into new interfaces for cross--disciplinary ocean sciencedisciplinary ocean science
Extensive instrument and cable layout for creating experimentsExtensive instrument and cable layout for creating experiments
Flexible terrain and image engine for visualizing siteFlexible terrain and image engine for visualizing site
True 3D/4D science dataset visualizationTrue 3D/4D science dataset visualization
Field tested in RSN observatory layout and on ocean expeditionsField tested in RSN observatory layout and on ocean expeditions
Cross platform and extensible with python and workflow systemsCross platform and extensible with python and workflow systems
![Page 56: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/56.jpg)
TridentTrident
Microsoft Research scientific workflow systemMicrosoft Research scientific workflow system
Visual programming environment for connecting tasksVisual programming environment for connecting tasks
ScienceScience--specific task libraries including one for ocean sciencesspecific task libraries including one for ocean sciences
Automated provenance capture, monitoring, and fault toleranceAutomated provenance capture, monitoring, and fault tolerance
Runs on local system, Windows server, or HPC ClusterRuns on local system, Windows server, or HPC Cluster
Cross platform with Cross platform with SilverlightSilverlight and web service interfaceand web service interface
![Page 57: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/57.jpg)
AzureAzure
MicrosoftMicrosoft’’s cloud computing platforms cloud computing platform
Provides storage and computing as payProvides storage and computing as pay--asas--youyou--go servicesgo services
From development standpoint, system looks like provisioned From development standpoint, system looks like provisioned VMVM’’ss
SQL, table, and blob (file system) storage models are includedSQL, table, and blob (file system) storage models are included
Access to storage via Access to storage via RESTfulRESTful HTTP interfaceHTTP interface
![Page 58: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/58.jpg)
Azure OceanAzure Ocean
COVE + Trident + Azure provides visual analytics to scientistsCOVE + Trident + Azure provides visual analytics to scientists
Any component Any component –– VisualizationVisualization, , ComputingComputing, or , or DataData –– can be can be provisioned locally, on a server, or in the cloudprovisioned locally, on a server, or in the cloud
When on same machine, system APIs are leveraged for speedWhen on same machine, system APIs are leveraged for speed
When distributed, communication is through HTTP and When distributed, communication is through HTTP and RESTfulRESTful APIsAPIs
Flexible platform for the diverse ocean science needsFlexible platform for the diverse ocean science needs
![Page 59: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/59.jpg)
Modeling protein interactions in striated muscles
Tom Daniel
![Page 60: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/60.jpg)
Model the lever arm with multiple springs
Myosin’s lever-arm generates force
Power
Stroke
Incorporate into a multi-filament model
(an embarrassingly parallel Monte Carlo
simulation)
![Page 61: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/61.jpg)
Simple Python scripts automate the management of 1000s of simultaneous
experiments using EC2 API
EC2
![Page 62: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/62.jpg)
J. J. Rehr & R.C. AlbersRev. Mod. Phys. 72, 621 (2000)
A “cluster to cloud” story:
Naturally parallel
Each CPU calculates a fewpoints in the energy grid
Loosely coupled
Very little communicationbetween processes
http://leonardo.phys.washington.edu/feff/
FEFF: Real-space Green’s function code for electronic structure, x-ray spectra, …
John Rehr
![Page 63: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/63.jpg)
63
• Is Cloud Computing feasible for on-demand, High-Performance Computing (HPC) for scientific research in the face of declining budgets?
• Who is interested?• Is it for everybody?• What kind of code could benefit from it?• How do we make it possible?
Challenge of NSF Grant
Disadvantages of Current HPC Approach
• Expensive infrastructure:Big clusters = ~1000$/node + capital costs +
power + cooling + …• Expensive HPC staff & maintenance• Need expertise in HPC to use efficiently
![Page 64: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/64.jpg)
Advantages of CC for Scientific Computing
• For “casual” HPC users:– On-demand access without the need to purchase,
maintain, or even understand HPCs– Lease vs. buy: lease as many as needed at ~10¢/cpu-hr– Plug & Play HPC scientific codes
• For developers:– Scientific codes can be optimized and pre-installed
• For administrators & funding agencies:– HPC access to a wider class of scientists at lower costs
![Page 65: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/65.jpg)
Development Strategy
1. Develop AMI (Amazon Machine Image) customized for HPC scientific applications
2. Test single-instance performance
3. Develop shell-scripts that make the EC2 look and run like a local HPC cluster (”virtual supercomputer on a laptop”)
4. Test parallel performance
![Page 66: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/66.jpg)
FEFFMPI EC2 AMI
• Standard Linux AMI:Fedora 8 32-bit distribution with Gnu FORTRAN compilers (gfortran and g77)
• AWS tools for the EC2: AMI, API and S3 tools• LAM 7.1.4 for parallel MPI codes• Java Runtime Environment 6• Java Development Kit 1.6• EC2 Cluster tools• FEFF8.4 serial and parallel versions• JFEFF graphical interface for FEFF8.4
Custom Linux distribution replicated oneach instance in cluster
![Page 67: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/67.jpg)
Serial Performance of FEFF on EC2
Virtual machine performance similar to “real”
![Page 68: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/68.jpg)
Xeon E5345 (2.33 GHz)
½ Opteron 2218HE(“1.3” GHz)
Xeon E5345 (2.33 GHz)
Xeon E5345 (2.33 GHz)
Serial Performance of Gasoline on EC2
Virtual
Virtual
Virtual
Real
No penalty from virtualization
![Page 69: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/69.jpg)
1. Start cluster2. Configure nodes
Current MPI Scenario
EC2 Compute Instances
User interacts with control workstation
MPI Master
startd
MPI Slave
startd
MPI Slave
startd
Control Workstation
ec2_clust_*
![Page 70: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/70.jpg)
UW EC2 Cluster Tools
Tools in the local control machine
Name Function Analogec2_clust_launch N Launches cluster with N instances bootec2_clust_connect Connect to a cluster sshec2_clust_put Transfer data to EC2 cluster scpec2_clust_get Transfer data from EC2 cluster scpec2_clust_list List running clustersec2_clust_terminate Terminate a running cluster shutdown
The tools hide a lot of the “ugliness”:
ec2_clust_connect
ssh -i /home/fer/.ec2_clust/.ec2_clust_info.7729.r-de70cdb7/key_pair _fdv.pem [email protected]
![Page 71: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/71.jpg)
FEFFMPI on EC2
EC2 works well for highly parallelized applications like FEFF
EC2
UW
![Page 72: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/72.jpg)
SkyScraper: Scalable Image Registration and Query in the Cloud with MapReduce
Andy Connolly
M1
M2
M3
R1
M1
R1 R2
M2 M3 M4
![Page 73: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/73.jpg)
Horizon: Where the Ocean meets the Cloud
• Client + Cloud: VisTrails, GridFields, 400-node Hadoop Cluster (NSF CluE program)
• Need interactive “climatologies”: Decade-scale averages under different assumptions
• Must manipulate 40 terabytes the same way you manipulate 40 megabytes: efficiently, interactively, visually
Bill Howe Claudio Silva Juliana Freire
http://clue.cs.washington.edu/
![Page 74: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/74.jpg)
“EC2 is Google Docs for developers”
The cloud is the ultimate collaborative development environment
A shared environment outside of the jurisdiction of over-protective (or otherwise non-responsive) sysadminsNo bugs closed as “can’t replicate”
Example: New software for serving oceanographic model results, requiring collaboration between UW, OPeNDAP.org, and OOI
Bill Howe
Azure BPOS“EC2 is Google Docs for developers”
![Page 75: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/75.jpg)
Waited two weeks for credentials to be establishedGave up, spun up an EC2 instance, were rolling within an hour
Similarly, Seattle’s Institute for Systems Biology uses EC2/S3 for sharing computational pipelines
![Page 76: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/76.jpg)
Observations
Flat files and Excel spreadsheets are the most common data management tools for scientists
Data management workflows are choking scienceEven superb scientists are doing things you wouldn’t believe
Such as manual joins on huge spreadsheets, exemplified by Ginger Armbrust’s environmental metagenomics lab
Simple tools can change their livesE.g., the spreadsheet->SQLShare and web SQL query interface for Armbrust’s lab
Many of these tools have broad applicabilityE.g., the above, and the Condor-to-cloud interface designed for Arzeda
![Page 77: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/77.jpg)
Workflow management is hugely important; building on commercial workflow engines is the smart approach
Trident has been widely adoptedFlexible client+cloud architectures are winners –there is no “one size fits all”
COVE + Trident + Azure, HorizonA huge proportion of interesting science is, or can be made, embarrassingly parallel – many “HPC”researchers can thrive in the cloud
Tom Daniel’s Monte Carlo muscle simulationsJohn Rehr’s FEFF and Gasoline
![Page 78: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/78.jpg)
Many science apps lend themselves to MapReduce / Dryad – style computation
Andy Connolly’s SkyScraperBill Howe’s Horizon
“EC2 is Google Docs for developers”UW / OPeNDAP.org / OOIInstitute for Systems Biology
![Page 79: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/79.jpg)
[Werner Vogels, Amazon.com]
![Page 80: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/80.jpg)
[Werner Vogels, Amazon.com]
![Page 81: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/81.jpg)
[Werner Vogels, Amazon.com]
![Page 82: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/82.jpg)
[Werner Vogels, Amazon.com]
![Page 83: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/83.jpg)
Advances in computing change the way we live, work, learn, and communicateAdvances in computing drive advances in nearly all other fieldsAdvances in computing power our economy
Not just through the growth of the IT industry – through productivity growth across the entire economy
Computer science: Changing the world
![Page 84: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/84.jpg)
Forty years ago …
![Page 85: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/85.jpg)
![Page 86: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/86.jpg)
[Peter Lee, DARPA, and Pat Lincoln, SRI]
![Page 87: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/87.jpg)
![Page 88: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/88.jpg)
With forty years hindsight, which had the greatest impact?
Unless you’re big into Tang and Velcro (or sex and drugs), the answer is clear …
And so is the reason …
EXPONENTIALS US
![Page 89: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/89.jpg)
The past thirty years …
![Page 90: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/90.jpg)
![Page 91: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/91.jpg)
![Page 92: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/92.jpg)
![Page 93: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/93.jpg)
![Page 94: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/94.jpg)
The most recent ten years …
SearchScalabilityDigital mediaMobilityeCommerceThe CloudSocial networking and crowd-sourcing
![Page 95: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/95.jpg)
The cloud: A triumph of computing research
Enormous volumes of dataExtreme parallelismThe cheapest imaginable components
Failures occur all the timeYou couldn’t afford to prevent this in hardware
Software makes itFault-TolerantHighly AvailableRecoverableConsistentScalablePredictableSecure
![Page 96: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/96.jpg)
![Page 97: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/97.jpg)
Predominant CS component
Significant CS component
![Page 98: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/98.jpg)
We put the “smarts” in …
Smart homesSmart carsSmart bodiesSmart robotsThe data deluge (smart science)Virtual and augmented realitySmart crowds and human-computer systems
![Page 99: The University of Washington eScience Institutelazowska.cs.washington.edu/cloud2010.pdf · Achieving Scalability through Cloud Computing Condor Scheduler Condor Router 1000s of on-demand](https://reader034.fdocuments.us/reader034/viewer/2022042321/5f0b472e7e708231d42fb9ae/html5/thumbnails/99.jpg)
Is this a great time, or what?!?!