reseña Decisiones, incertidumbrey el cerebro. Paul Glimcher-rcnp2011vol6n2-11
Supporting Load Balancing for Distributed Data-Intensive Applications Leonid Glimcher, Vignesh Ravi,...
18
upporting Load Balancing for Distribute Data-Intensive Applications Leonid Glimcher, Vignesh Ravi, and Gagan Agrawal Department of ComputerScience and Engg. The Ohio State University Columbus, Ohio - 43210
-
Upload
piers-lang -
Category
Documents
-
view
215 -
download
0
Transcript of Supporting Load Balancing for Distributed Data-Intensive Applications Leonid Glimcher, Vignesh Ravi,...
Supporting Load Balancing for DistributedData-Intensive Applications
Leonid Glimcher, Vignesh Ravi, and Gagan AgrawalDepartment of ComputerScience and Engg.
The Ohio State UniversityColumbus, Ohio - 43210
Outline
• Introduction• Motivation• FREERIDE-G Processing Structure• Run-time Load Balancing System• Experimental Results• Conclusions
April 21, 2023 2
Introduction
• Growing abundance of data– Sensors, scientific simulations and business
transactions
• Data Analysis– Translate raw data into knowledge
• Grid/Cloud Computing– Enables distributed processing
April 21, 2023 3
Motivation
• Resources are geographically distributed– Data nodes
– Compute nodes
– Middleware user
• Remote data analysis is important
• Heterogeneity of resources– Difference in network bandwidth
– Difference in compute power
April 21, 2023 4
Data Nodes
Compute Nodes
Middleware user
Grid/Cloud Environment
FREERIDE-G Processing Structure(Framework for Rapid Implementation of Datamining Engines –
Grid)
April 21, 2023 5
While( ) {
forall( data instances d) {
(I , d’) = process(d)
R(I) = R(I) op d’
}
…….
}
• A Map-reduce like system
• Remote data analysis
Middleware API
• Process
• Reduce
• Global CombineReduction Object
A Real-time Grid/Cloud Scenario
April 21, 2023 6
A B
C DCompute
Data
Run-time Load Balancing
April 21, 2023 7
Two factors of load imbalance• Computational factor, w1• Remote data transfer (wait time), w2Case 1: w1 > w2Case 2: w2 > w1We use sum of weights to account for both
thecomponents
Dynamic Load Balancing Algorithm
April 21, 2023 8
Consider every chunk, Ci
CalculateCompute cost, Cc
CalculateData transfer cost, Tc
Input Bandwidth matrix, W1 & W2
Total cost = W1*Cc + W2*Tc
If Total cost < Min
Update Min Assign Ci to Pj
Experimental Setup
Settings
• Organizational Grid
• Wide Area Network (WAN)
Goals are to evaluate
• Scalability
• Dynamic Load balancing overhead
• Adaptability to scenarios
– compute bound,
– I/O bound,
– WAN setting
Applications
• K-means
• Vortex Detection
April 21, 2023 9
April 21, 2023 10
Scalability and Overhead of Dynamic Balancing
Vortex detection
• 14.8 GB data• Organizational
setting• Bandwidth
– 50mb/sec– 100mb/sec
• 31% benefit• Overhead within
10%
April 21, 2023 10
Model Adaptability – Compute Bound Scenario
Kmeans clustering• 25.6 GB data• Bandwidth
– 50 MB – 200 MB
Best result• 75-25 combination• skewed towards work load
componentInitial (unbalanced) overhead• 57% over balancedDynamic overhead• 5% over balanced
April 21, 2023 11
Ideal CaseDynamic case
ComputeData transfer
Model Adaptability – I/O Bound Scenario
April 21, 2023 12
Kmeans clustering• 25.6 GB data• Bandwidth
– 15 mb/s– 60 mb/s
Best result• 25-75 combination• skewed towards data
transfer componentInitial (unbalanced)
overhead• 40% over balancedDynamic overhead• 4% over balanced
Model Adaptability – WAN setting
Vortex Detection • 14.6 GBBest result• 25-75 combination results
in lowest overhead (favoring data delivery component)
Unbalanced configuration• 20% overhead over
balancedOur approach• Overhead reduced to 8%
April 21, 2023 13
0500
1000
Execu
tio
n T
ime (
sec)
4 compute - 8 data 8 compute - 8 data
staric balancing 0-100 25-75
50-50 75-25 100-0
unbalanced
Conclusions
• Dynamic load balancing solution for grid environments
• Both workload and data transfer factors are important
• Scalability is good and overheads are within 10%
• Adaptable to compute-bound, I/O bound, and WAN settings
April 21, 2023 14
April 21, 2023 15
Thank You!
Questions?
Contacts:Leonid Glimcher [email protected]
Vignesh Ravi - [email protected]
Gagan Agrawal - [email protected]
DataGrid Lab
Setup 1: Organizational GridData hosted on Opteron 250’s
Processed on Opteron 254’s
2 clusters connected through two 10 GB optical fibers
Both clusters within same city (0.5 mile apart)
Evaluating:
Scalability
Adaptability
Integration overheadCompute cluster (cse-ri)
Repository cluster (bmi-ri)
DataGrid Lab
Setup 2: WANData Repository:
Opteron 250’s (OSU)
Opteron 258’s (Kent St)
Processed on Opteron 254’s
No dedicated link between processing and repository clusters
Evaluating:
Scalability
Adaptability
Compute cluster (OSU)
Repository cluster (Kent ST)
Repository cluster (OSU)
FREERIDE-G System Design
April 21, 2023 18
