SCATTER/GATHER : A CLUSTER BASED APPROACH FOR BROWSING LARGE DOCUMENT COLLECTIONS
ADaPT: An Event-Passing Protocol for Reducing Delivery Costs in Scatter-Gather Parallel Processes
description
Transcript of ADaPT: An Event-Passing Protocol for Reducing Delivery Costs in Scatter-Gather Parallel Processes
ADaPT: An Event-Passing Protocol for Reducing Delivery Costs in Scatter-Gather Parallel Processes
David M. Woollard†§
Nenad Medvidovic†
Walter M. Yamada§
Theodore W. Berger§
†Center for Software Engineering
University of Southern CaliforniaLos Angeles, California 90089-0781
§Laboratory for Neural Dynamics
University of Southern CaliforniaLos Angeles, California 90089-1451
Outline
Motivation
Established Techniques
ADaPT
Performance Comparison
Conclusions
ADaPT: An Event-Passing Protocol For Reducing Delivery Costs in
Scatter-Gather Parallel Processes
Motivation
• A computational-science section of the Center for Neural Engineering
• Part of a National Science Foundation engineering research center dedicated to biomimetic microelectronic systems
• Combines computational electrophysiology, engineering, pharmacology, and other disciplines
• Integrates empirically-measured, realistic, and biologically-inspired synaptic models for the purposeof temporal signals processing
Motivation
What is the Laboratory for Neural Dynamics?
• Biologically-inspired rather than realistic• Computationally-complex and non-linear• Signals processing application was originally a
proof of concept• Now a synergistic field for the Center
Motivation
The Dynamic Synapse
Presynapse PostsynapseNa+
Ca2+Ca2+
Action Potential input Glutamate release Action Potential output
threshold threshold
Figure 1: Electro-chemical synaptic transmission
Neuron
Synaptic Potential Summation
Feedback
• Classical NN structure
• Increased synaptic functionality
• Parameter trainingvia genetic algorithms
Motivation
Dynamic Synapse Neural Networks
K length filter b
ank
Array of K input neurons
12
3
K
APAP
AP
AP
3xK Pre-synaptic Matrix
3xK Post-synaptic Matrix
LAYER 1
LAYER 2
EPSP
EPSP
EPSP
EPSP
EPSP
EPSP
EPSP
EPSP
EPSP
EPSP
EPSP
EPSP
AP
AP
AP
Feedback Modulation
AA
Output
Neurons
Microphone Array
Captured Sound
Figure 2: 3xK 2-Layer DSNN Single Word Classifier.
Outline
Motivation
Established Techniques
ADaPT
Performance Comparison
Conclusions
ADaPT: An Event-Passing Protocol For Reducing Delivery Costs in
Scatter-Gather Parallel Processes
Motivation
• Naïve Approach• Multiple Sequential
Scatter-Gathers• With uniform computation time,
exhibits decent parallelism• With variable computation times,
significant idle time
Established Techniques
Scatter-Gather
Worke
r 1
Worke
r 2
Worke
r 3
Worke
r 4
Worke
r 5
Worke
r 6
Worke
r n -
2 Worke
r n -
1
Worke
r n
Maste
r
Worke
r 1
Worke
r 2
Worke
r 3
Worke
r 4
Worke
r 5
Worke
r 6
Worke
r n -
2 Worke
r n -
1
Worke
r n
Maste
r
Worke
r 1
Worke
r 2
Worke
r 3
Worke
r 4
Worke
r 5
Worke
r 6
Worke
r n -
2 Worke
r n -
1
Worke
r n
Maste
rM
aste
r
Co
mp
uta
tion
T
ime
Figure 3: Multi-phase evaluation of 3n genomes by n workers using naïve scatter-gathering.
• Asynchronous scattering• Reduced idle time for workers• Closer to optimal time to solution• Dynamic allocation of resources
– More difficult
Established Techniques
A More Efficient Mapping
Maste
rCo
mp
uta
tion
T
ime
Worke
r 1
Worke
r 2
Worke
r 3
Worke
r 4
Worke
r 5
Worke
r 6
Worke
r n -
2 Worke
r n -
1
Worke
r n
Maste
r
Figure 4: Multi-phase evaluation of 3n genomes by n workers using a more efficient mapping.
Outline
Motivation
Established Techniques
ADaPT
Performance Comparison
Conclusions
ADaPT: An Event-Passing Protocol For Reducing Delivery Costs in
Scatter-Gather Parallel Processes
Motivation
• Publish/Subscribe– Worker-centric, i.e. processes subscribe to the master– Data is transported (published) to workers as events– Unsubscription is possible
• Two-phase adaptive protocol– Learning phase: request-reply, monitoring of time
between requests– Aggressive phase: events are pushed to workers at regular
intervals
ADaPT
Adaptive Data-parallel Publish/Subscribe Transport Protocol
Outline
Motivation
Established Techniques
ADaPT
Performance Comparison
Conclusions
ADaPT: An Event-Passing Protocol For Reducing Delivery Costs in
Scatter-Gather Parallel Processes
Motivation
• Two protocols– Aggressive & Conservative
• Scatter/Gathers in most implementations use conservative protocols
• Analysis due to Gropp, et. al.
Performance Comparison
Message-passing costs for MPI scatters
C(MPI Scatter) = (# pop.)[3s + r(n+3e)]
Where n = event payloade = enveloper = network bandwidths = latency
Equation 1: Computation time cost in of scatters In MPI.
Performance Comparison
• Our assumption is a normally distributed population of compute times
• An ideal ordering of computations would be sortedby compute time
• How much idle time is present?
Computational Costs for Multiple scatters in MPI
C(Computation) = (# pop.)(avg. compute time) + (# workers)(avg. compute time)Equation 2: Computation time of a normally-distributed population using scatters in MPI.
Figure 5: Graph of sorted compute times of anormal distribution illustrating idle time.
• Three different costs of event-passing in ADaPT:– Subscription– Learning Phase– Aggressive Phase
Performance Comparison
Message-Passing costs for ADaPT
C(subscription) = (# workers) x [s + re]
C(learning) = (# samples) x [2s + r(n+2e)]
C(aggressive) = (# pop - # samples) x[s + r(n+e)]
Note: we assume control events to be of size e
Equation 3: Event-passing costs of ADaPT.
• An unsubscribe occurs when a worker’s event buffer is in danger of overflowing
• With ADaPT, an overflowoccurs when a worker receivesm-1 events triggering computetimes greater than the estimatedaverage (assuming a worker buffers m events)
• Conservatively, we have decidedthat workers should clear theirbuffers before resubscribing
Performance Comparison
Unsubscribe Costs for ADaPT
- We used a Monte Carlo simulation (details in paper) to determine E,the % pop with compute times > than the estimated mean given error as a function of % pop. sampled
P(unsubscribe) =
E*Pop C m-1
Pop C m-1
C(unsubscribe) = P(unsubscribe) x [2(s+re) + (m-1)(avg. compute+ Δ)]
Equation 4: Costs of worker unsubscription in ADaPT.
Performance Comparison
• Which protocol is more appropriate?
• For simplicity of comparisonwe will drop the latency termand assume the number of samples to be equal to thenumber of workers– i.e. each worker’s first
computation is monitored
Analysis (# pop - # samples)2re +
(# samples)(avg. compute time) > (# pop / m) x P(unsubscribe) x[2re + (m-1)(avg. compute time)]
Equation 5: Cost comparison of MPI vs. ADaPT.
Figure 6: Graph of inequality in Equation 5.
Outline
Motivation
Established Techniques
ADaPT
Performance Comparison
Conclusions
ADaPT: An Event-Passing Protocol For Reducing Delivery Costs in
Scatter-Gather Parallel Processes
Motivation
• ADaPT is useful when multiple scattering of data must occur due to natural aggregation– An example is the training of the DSNN using genetic
algorithms
• Worker-centric approach for reduced processor idle time
• Unsubscription is expensive but can be avoided withgreater event-buffering capabilities
• ADaPT exploits an event pattern which emerges fromthe application of a well-known architectural pattern
Conclusions
What have we shown?
Thank You