Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our...
Transcript of Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our...
![Page 1: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/1.jpg)
DataIntensive
ScalableComputing
http://www.cs.cmu.edu/~bryant
Randal E. BryantCarnegie Mellon University
Finding the Right Programming Models
![Page 2: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/2.jpg)
– 2 –
Outline
Data-Intensive Scalable ComputingFocus on petabytes, not petaflopsRethinking machine design
Map/Reduce Programming ModelSuitability for wide class of computing tasks
Strengths & LimitationsScalabilityPerformance
Beyond Map/ReduceSmall variationsOther programming models
![Page 3: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/3.jpg)
– 3 –
Our Data-Driven World
ScienceData bases from astronomy, genomics, natural languages, seismic modeling, …
HumanitiesScanned books, historic documents, …
CommerceCorporate sales, stock market transactions, census, airline traffic, …
EntertainmentInternet images, Hollywood movies, MP3 files, …
MedicineMRI & CT scans, patient records, …
![Page 4: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/4.jpg)
– 4 –
Why So Much Data?We Can Get It
Automation + Internet
We Can Keep ItSeagate Barracuda1.5 TB @ $76
5¢ / GB(vs. 40¢ in 2007)
We Can Use ItScientific computationBusiness applicationsHarvesting Internet data
![Page 5: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/5.jpg)
– 5 –
Oceans of Data, Skinny Pipes
1 TerabyteEasy to storeHard to move
Disks MB / s TimeSeagate Barracuda 115 2.3 hoursSeagate Cheetah 125 2.2 hours
Networks MB / s TimeHome Internet < 0.625 > 18.5 days
Gigabit Ethernet < 125 > 2.2 hours
PSC Teragrid Connection
< 3,750 > 4.4 minutes
![Page 6: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/6.jpg)
– 6 –
Data-Intensive System Challenge
For Computation That Accesses 1 TB in 5 minutesData distributed over 100+ disks
Assuming uniform data partitioning
Compute using 100+ processorsConnected by gigabit Ethernet (or equivalent)
System RequirementsLots of disksLots of processorsLocated in close proximity
Within reach of fast, local-area network
![Page 7: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/7.jpg)
– 7 –
Google Data Centers
Dalles, OregonHydroelectric power @ 2¢ / KW Hr50 MegawattsEnough to power 6,000 homes
Engineered for maximum modularity & power efficiencyContainer: 1160 servers, 250KWServer: 2 disks, 2 processors
![Page 8: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/8.jpg)
– 8 –
High-Performance Distributed Computing: Two VersionsGrid ComputingConnect small number of
big machinesAllow resource sharing among supercomputers
IssuesPrograms must usually be specialized for machineHard to get cooperation from multiple organizations
DISC SystemBuild big system from
many small machinesUse distributed systems principles to construct large-scale machineFile system provides distribution, reliability, recoveryDynamically scheduled task as basic processing unit
![Page 9: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/9.jpg)
– 9 –
Programming Model Comparison
Bulk SynchronousCommonly used for compute-intensive applications
Map/ReduceCommonly used for data-intensive applications
IssuesRaw performanceScalability
Cost required to increase machine size by K» ≥ K
Performance achieved by increasing machine size by K» ≤ K
Frequency and impact of failures
![Page 10: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/10.jpg)
– 10 –
Bulk Synchronous Programming
Solving Problem Over GridE.g., finite-element computation
Partition into Regionsp regions for p processors
Map Region per ProcessorLocal computation sequentialPeriodically communicate boundary values with neighbors
![Page 11: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/11.jpg)
– 11 –
Typical HPC OperationCharacteristics
Long-lived processesMake use of spatial localityHold all program data in memory (no disk access)High bandwidth communication
StrengthsHigh utilization of resourcesEffective for many scientific applications
WeaknessesRequires careful tuning of application to resourcesIntolerant of any variability
P1 P2 P3 P4 P5 MemoryShared Memory
P1 P2 P3 P4 P5
Message Passing
![Page 12: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/12.jpg)
– 12 –
Map/Reduce Programming Model
Map computation across many objectsE.g., 1010 Internet web pages
Aggregate results in many different waysSystem deals with issues of resource allocation & reliability
M
x1
M
x2
M
x3
M
xn
k1
Map
Reducek1
kr
• • •
• • •
Key-ValuePairs
Dean & Ghemawat: “MapReduce: Simplified Data Processing on Large Clusters”, OSDI 2004
![Page 13: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/13.jpg)
– 13 –
Map/Reduce Example
Create an word index of set of documentsMap: generate ⟨word, count⟩ pairs for all words in documentReduce: sum word counts across documents
Come,Dick
Come and see.
Come and see
Spot.
Come and see.
Come, come.
M Extract
Word-CountPairs
⟨dick, 1⟩
⟨see, 1⟩
⟨come, 1⟩
⟨and, 1⟩
⟨come, 1⟩
⟨come, 1⟩
⟨come, 1⟩
M M M M
⟨come, 2⟩
⟨see, 1⟩
⟨and, 1⟩⟨and, 1⟩
⟨spot, 1⟩
Sumdick∑
1
and∑
3
come∑
6
see∑
3
spot∑
1
![Page 14: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/14.jpg)
– 14 –
Map/Reduce OperationCharacteristics
Computation broken into many, short-lived tasks
Mapping, reducing
Use disk storage to hold intermediate results
StrengthsGreat flexibility in placement, scheduling, and load balancingCan access large data sets
WeaknessesHigher overheadLower raw performance
MapReduce
MapReduce
MapReduce
MapReduce
Map/Reduce
![Page 15: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/15.jpg)
– 15 –
System Comparison:Programming Models
Programs described at very low level
Specify detailed control of processing & communications
Rely on small number of software packages
Written by specialistsLimits classes of problems & solution methods
Application programs written in terms of high-level operations on dataRuntime system controls scheduling, load balancing, …
Conventional HPC
Hardware
Machine-DependentProgramming Model
SoftwarePackages
ApplicationPrograms
Hardware
Machine-IndependentProgramming Model
RuntimeSystem
ApplicationPrograms
DISC
![Page 16: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/16.jpg)
– 16 –
HPC Fault Tolerance
CheckpointPeriodically store state of all processesSignificant I/O traffic
RestoreWhen failure occursReset state to that of last checkpointAll intervening computation wasted
Performance ScalingVery sensitive to number of failing components
P1 P2 P3 P4 P5
Checkpoint
Checkpoint
Restore
WastedComputation
![Page 17: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/17.jpg)
– 17 –
Map/Reduce Fault ToleranceData Integrity
Store multiple copies of each fileIncluding intermediate results of each Map / Reduce
Continuous checkpointing
Recovering from FailureSimply recompute lost result
Localized effect
Dynamic scheduler keeps all processors busy
MapReduce
MapReduce
MapReduce
MapReduce
Map/Reduce
![Page 18: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/18.jpg)
– 18 –
DISC Scalability Advantages
Distributed system design principles lead to scalable designDynamically scheduled tasks with state held in replicated files
Provisioning AdvantagesCan use consumer-grade components
maximizes cost-peformance
Can have heterogenous nodesMore efficient technology refresh
Operational AdvantagesMinimal staffingNo downtime
![Page 19: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/19.jpg)
– 19 –
Getting StartedGoal
Get people involved in DISC
SoftwareHadoop Project
Open source project providing file system and Map/ReduceSupported and used by YahooRapidly expanding user/developer basePrototype on single machine, map onto cluster
![Page 20: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/20.jpg)
– 20 –
Exploring Parallel Computation Models
DISC + Map/Reduce Provides Coarse-Grained ParallelismComputation done by independent processesFile-based communication
ObservationsRelatively “natural” programming modelResearch issue to explore full potential and limits
Low CommunicationCoarse-Grained
High CommunicationFine-Grained
SETI@home PRAMThreads
Map/Reduce
MPI
![Page 21: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/21.jpg)
– 21 –
Example: Sparse Matrices with Map/Reduce
Task: Compute product C = A·BAssume most matrix entries are 0
MotivationCore problem in scientific computingChallenging for parallel executionDemonstrate expressiveness of Map/Reduce
10 20
30 40
50 60 70
A-1
-2 -3
-4
B-10 -80
-60 -250
-170 -460
C
X =
![Page 22: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/22.jpg)
– 22 –
Computing Sparse Matrix Product
Represent matrix as list of nonzero entries⟨row, col, value, matrixID⟩
StrategyPhase 1: Compute all products ai,k · bk,j
Phase 2: Sum products for each entry i,jEach phase involves a Map/Reduce
10 20
30 40
50 60 70
A-1
-2 -3
-4
B1 110
A
1 320A
2 230A
2 340A
3 150A
3 260A
3 370A
1 1-1B
2 1-2B
2 2-3B
3 2-4B
![Page 23: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/23.jpg)
– 23 –
Phase 1 Map of Matrix Multiply
Group values ai,k and bk,j according to key k
1 1-1B
2 1-2B
2 2-3B
3 2-4B
Key = row
1 110A
1 320A
2 230A
2 340A
3 150A
3 260A
3 370A
Key = 2
Key = 3
Key = 11 110
A
3 150A
2 230A
3 260A
1 320A
2 340A
3 370A
1 1-1B
2 1-2B
2 2-3B
3 2-4B
Key = col
![Page 24: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/24.jpg)
– 24 –
Phase 1 “Reduce” of Matrix Multiply
Generate all products ai,k · bk,j
1 1-10C
3 1-50A
2 1-60C
2 2-90C
3 1-120C
3 2-180C
1 2-80C
2 2-160C
3 2-280C
Key = 2
Key = 3
Key = 11 110
A
3 150A
2 230A
3 260A
1 320A
2 340A
3 370A
1 1-1B
2 1-2B
2 2-3B
3 2-4B
X
X
X
![Page 25: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/25.jpg)
– 25 –
Phase 2 Map of Matrix Multiply
Group products ai,k · bk,j with matching values of i and j
1 1-10C
3 1-50A
2 1-60C
2 2-90C
3 1-120C
3 2-180C
1 2-80C
2 2-160C
3 2-280C
Key = 1,2
Key = 1,1
Key = 2,1
Key = 2,2
Key = 3,1
Key = 3,2
1 1-10C
3 1-50A
2 1-60C
2 2-90C
3 1-120C
3 2-180C
1 2-80C
2 2-160C
3 2-280C
Key = row,col
![Page 26: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/26.jpg)
– 26 –
Phase 2 Reduce of Matrix Multiply
Sum products to get final entries
1 1-10C
2 1-60C
2 2-250C
3 1-170C
1 2-80C
3 2-460C
-10 -80
-60 -250
-170 -460
C
Key = 1,2
Key = 1,1
Key = 2,1
Key = 2,2
Key = 3,1
Key = 3,2
1 1-10C
3 1-50A
2 1-60C
2 2-90C
3 1-120C
3 2-180C
1 2-80C
2 2-160C
3 2-280C
![Page 27: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/27.jpg)
– 27 –
Lessons from Sparse Matrix Example
Associative Matching is Powerful Communication Primitive
Intermediate step in Map/Reduce
Similar Strategy Applies to Other ProblemsShortest path in graphDatabase join
Many Performance ConsiderationsKiefer, Volk, Lehner, TU DresdenShould do systematic comparison to other sparse matrix implementations
![Page 28: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/28.jpg)
– 28 –
Tweaking MapReduce
Map/Reduce/MergeYang, et al. Yahoo & UCLAAdd merge step to do tagged merge of data sets
E.g., getting elements from matrices A & B
Demonstrate expressive power of relational algebra
Local IterationsKambatla, et al., PurdueSupport local and global iterationsStep toward bulk synchronous model
![Page 29: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/29.jpg)
– 29 –
Global
Local
Local
Local
Global
Local
Local
Global
Local / Global Map/Reduce
Iterative Computationse.g., PageRank
Graph PartitioningPartition into clustersColocate data for each cluster
ComputationCompute solution for each cluster, holding inter-cluster values constantUpdate inter-cluster values
![Page 30: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/30.jpg)
– 30 –
Pig Project
Chris Olston, Yahoo!Part of Apache/Hadoop
Merge Database & ProgrammingSQL-like query languageSet-oriented function application
ImplementationMap onto HadoopAutomatic selection / optimization of algorithmCaptures low-level tricks programmers have devised for Map/Reduce
![Page 31: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/31.jpg)
– 31 –
Generalizing Map/ReduceE.g., Microsoft Dryad Project
Computational ModelAcyclic graph of operators
But expressed as textual program
Each takes collection of objects and produces objects
Purely functional model
Implementation ConceptsObjects stored in files or memoryAny object may be lost; any operator may failReplicate & recompute for fault toleranceDynamic scheduling
# Operators >> # Processorsx1 x2 x3 xn
• • •
Op2 Op2 Op2 Op2• • •
• • •
Opk Opk Opk Opk• • •
Op1 Op1 Op1 Op1
![Page 32: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/32.jpg)
– 32 –
Implementation Challenges
Hadoop Platform is BlessingLarge community adding features, improving performanceEasy to deployGrowing body of documentation and materialsWorks well enough for range of applications
… And CurseMap & reduce functionality hardwiredNot designed as extensible platform
Dryad Not Widely Adopted305 citations on Google Scholar vs. 1759 Map/ReduceBuilt on .NET
![Page 33: Data Intensive Scalable Computinghpdc2010.eecs.northwestern.edu/HPDC2010Bryant.pdf–3– Our Data-Driven World. Science Data bases from astronomy, genomics, natural languages, seismic](https://reader035.fdocuments.us/reader035/viewer/2022071218/60528949e66d0b4e0b02dd7f/html5/thumbnails/33.jpg)
– 33 –
Conclusions
Distributed Systems Concepts Lead to Scalable Machines
Loosely coupled execution modelLowers cost of procurement & operation
Map/Reduce Gaining Widespread UseHadoop makes it widely availableGreat for some applications, good enough for many others
Lots of Work to be DoneRicher set of programming models and implementationsExpanding range of applicability
Problems that are data and compute intensiveThe future of supercomputing?