Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations
-
Upload
kieran-bowman -
Category
Documents
-
view
25 -
download
0
description
Transcript of Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations
![Page 1: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/1.jpg)
Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations
Sasa MisailovicJoint work with
Zeyuan Allen Zhu Jonathan Kelner Martin Rinard
MIT CSAIL
![Page 2: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/2.jpg)
…
… … … …
• Nodes represent computation• Edges represent flow of data
![Page 3: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/3.jpg)
…
• Functions – process individual data• Reduction nodes – aggregate data
… … … …
![Page 4: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/4.jpg)
…
min
avgavg avgavg
• Functions – process individual data• Reduction nodes – aggregate data
… … … …
![Page 5: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/5.jpg)
…
min
avgavg avgavg
Function substitution• Multiple implementations• Each has expected error/time
f2 f3f1
… … … …
![Page 6: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/6.jpg)
…
min
avgavg avgavg
Function substitution• Multiple implementations• Each has expected error/time
… … … …
![Page 7: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/7.jpg)
Sampling inputs of reduction nodes• Reductions consume fewer inputs
…
min
avgavg avgavg
… … … …
![Page 8: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/8.jpg)
…
min
avgavg
Sampling inputs of reduction nodes• Reductions consume fewer inputs
… … … …
![Page 9: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/9.jpg)
Tradeoff Space
Error
Time
![Page 10: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/10.jpg)
Tradeoff Space
Error
Time
![Page 11: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/11.jpg)
Optimal Tradeoff Curve
Error
Time Using the tradeoff curve:• Minimize time subject to error bound• Minimize error subject to time bound
![Page 12: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/12.jpg)
Our Result
Analysis Optimization
Original program
Error bound
Optimizedprogram
Transformations
• Randomized computation• Guaranteed expected error/time tradeoff• -approximation of optimal tradeoff
![Page 13: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/13.jpg)
Outline
Model of Computation
Tradeoff Curve Construction
Optimized Program Selection
Related Work
![Page 14: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/14.jpg)
Model of Computation
…
f f
t
u v
w
t
u v
w
ff ggg g… … … …
min
avg avg avg avg
![Page 15: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/15.jpg)
Model of Computation
…
f f
t
u v
w
t
u v
w
ff ggg g… … … …
min
avg avg avg avg
min
1
n
n n
f g
t
u v
w
avg avgm m
![Page 16: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/16.jpg)
Structure of Computation
• Computation nodes DAGs of functions Functions: arbitrary code Process individual inputs
• Reduction nodes Aggregation functions Average, min, max, sum…
• Computation Tree Computation nodes and
reduction nodesmin
1
n
n n
f g
t
u v
w
avg avgm m
![Page 17: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/17.jpg)
min
Accuracy-Aware Transformations
Function substitution• Multiple versions • Execute with probability • Each has error/time spec
Reduction sampling• Consume inputs
Probability of selecting eachinput:
• Derived error/time specifications
Average:
Min/max: 1
n
n n
avg avgm m
![Page 18: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/18.jpg)
min
Accuracy-Aware Transformations
Function substitution• Multiple versions • Execute with probability • Each has error/time spec
Reduction sampling• Consume inputs
Probability of selecting eachinput:
• Derived error/time specifications
Average:
Min/max: 1
n
n n
avg avgm m…
avg
![Page 19: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/19.jpg)
min
Accuracy-Aware Transformations
Function substitution• Multiple versions • Execute with probability • Each has error/time spec
Reduction sampling• Consume inputs
Probability of selecting eachinput:
• Derived error/time specifications
Average:
Min/max: 1
n
n n
avg avgm m…
avg
![Page 20: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/20.jpg)
min
Accuracy-Aware Transformations
Function substitution• Multiple versions • Execute with probability • Each has error/time spec
Reduction sampling• Consume inputs
Probability of selecting eachinput:
• Derived error/time specifications
Average:
Min/max: 1
n
n n
avg avgm m…
avg
![Page 21: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/21.jpg)
min
Program Configuration Vector
Defines transformed program • Functions: probability of
executing each version• Reductions: number of
elements to sample
1
n
n n
avg avgm m
…0.3 0.6 0.1 …
![Page 22: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/22.jpg)
min
Configuration Vector
• Specifies program version• Functions: probability of
executing each version• Reductions: number of
elements to sample
1
n
n n
avg avgm m
…0.3 0.6 0.1 …
Find optimal program=
Find configuration vector that achieves optimal accuracy vs. performance tradeoff
![Page 23: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/23.jpg)
min
1
n
n n
avg avgm m
Divide and conquer• For each subcomputation
construct tradeoff curve• Dynamic programming
Properties• Polynomial time• -approximation of
true tradeoff curve
Tradeoff Curve Construction: Algorithm
![Page 24: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/24.jpg)
min
Tradeoff Curve Construction: Algorithm
1
n
n n
avg avgm m
![Page 25: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/25.jpg)
Tradeoff Curve Construction: Algorithm
1
m m
n
n n
min
avg avg
![Page 26: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/26.jpg)
Tradeoff Curve Construction: Algorithm
1
m m
n
n n
min
avg avg
![Page 27: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/27.jpg)
Tradeoff Curve Construction: Algorithm
1
n
n n
min
avgm
![Page 28: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/28.jpg)
Tradeoff Curve Construction: Algorithm
1
n
n n
min
![Page 29: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/29.jpg)
Tradeoff Curve Construction: Algorithm
1
n
n n
min
![Page 30: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/30.jpg)
Tradeoff Curve Construction: Algorithm
1
n
n n
min
![Page 31: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/31.jpg)
Tradeoff Curve Construction: Algorithm
1
n
min
![Page 32: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/32.jpg)
Tradeoff Curve Construction: Algorithm
![Page 33: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/33.jpg)
Computation Node Optimization
Variables: • Probability to execute each version of • Range: • Sum:
(E2,T2)(E0,T0) (E1,T1)
Linear program
![Page 34: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/34.jpg)
Computation Node Optimization
Variables: • Probability to execute each version of • Range: • Sum:
Objective:
Constraint:(E2,T2)(E0,T0) (E1,T1)
Linear program
![Page 35: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/35.jpg)
Computation Node Optimization
Variables: • Probability to execute each version of • Range: • Sum:
Objective:
Constraint:(E2,T2)(E0,T0) (E1,T1)
Linear program
![Page 36: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/36.jpg)
Computation Node Optimization
Variables: • Probability to execute each version of • Range: • Sum:
Objective:
Constraint:(E2,T2)(E0,T0) (E1,T1)
Linear program
![Page 37: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/37.jpg)
The Algorithm: Reduction Nodes
Given error bound , find number of elements to sample
m
avg
![Page 38: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/38.jpg)
The Algorithm: Reduction Nodes
Given error bound , find number of elements to sample
s.t.
m
avg
(𝐸𝑠𝑢𝑏 ,𝑇 𝑠𝑢𝑏)
![Page 39: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/39.jpg)
The Algorithm: Reduction Nodes
Given error bound , find number of elements to sample
s.t.
From approximate tradeoff curve:
for
m
avg
(𝐸𝑠𝑢𝑏 ,𝑇 𝑠𝑢𝑏)
![Page 40: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/40.jpg)
The Algorithm: Reduction Nodes
Given error bound , find number of elements to sample
m
avg
(𝐸𝑠𝑢𝑏 ,𝑇 𝑠𝑢𝑏)
s.t.
• Univariate optimization problem• Analogously, minimize error subject to
![Page 41: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/41.jpg)
Approximate Tradeoff Curve
Error
Time
Bidimensional Discretization
Take elements at regular intervals
![Page 42: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/42.jpg)
Approximating Tradeoff Curve
Error
Time
Bidimensional Discretization
![Page 43: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/43.jpg)
Approximating Tradeoff Curve
Error
Time
![Page 44: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/44.jpg)
Approximating Tradeoff Curve
Randomized configuration:• Execute with probability • Execute with probability
Error
Time
𝜅 1−𝜅
𝐴
𝐵
𝐶
![Page 45: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/45.jpg)
Approximating Tradeoff Curve
Error
Time
�̂�𝐸𝜀𝐸
𝑇
![Page 46: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/46.jpg)
Approximating Tradeoff Curve
Error
Time
𝑇
�̂�
𝑇
T
![Page 47: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/47.jpg)
Properties of the Algorithm
Performance• Number of tradeoff curve points: • Most expensive operation: bidimensional discretization
Calling LP solver times Each call can have variables
Precision• Precision decreases linearly with the number of nodes • To obtain -approximation set intermediate
Space• Storing tradeoff curves:
![Page 48: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/48.jpg)
Obtaining Optimized Programs
Tradeoff curves for all subcomputations:• Each curve contains partial configuration
Probability of executing local function nodes Number of inputs to sample from reduction node Error tolerated by subcomputation
• Distribution over optimal program configurations
Incrementally construct configuration vector:• For every execution• Traverse the tree, starting from root• Time to get full vector:
![Page 49: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/49.jpg)
Related WorkAccuracy-aware transformations• Empirical justification: training/test input set
[Rinard ICS ‘06, Rinard OOPSLA ’07, Ansel et al. PLDI ‘09, Misailovic et al. ICSE ’10, Baek & Chilimbi PLDI ‘10 Hoffmann et al. ASPLOS ‘11, Sidiroglou et al. FSE ‘11]
• Probabilistic accuracy analysis for loop perforation [Misailovic et al. SAS ‘11, Chaudhuri et al. FSE ‘11]
![Page 50: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/50.jpg)
Related WorkAccuracy-aware transformations• Empirical justification: training/test input set
[Rinard ICS ‘06, Rinard OOPSLA ’07, Ansel et al. PLDI ‘09, Misailovic et al. ICSE ’10, Baek & Chilimbi PLDI ‘10 Hoffmann et al. ASPLOS ‘11, Sidiroglou et al. FSE ‘11]
• Probabilistic accuracy analysis for loop perforation [Misailovic et al. SAS ‘11, Chaudhuri et al. FSE ‘11]
Ensuring safety of transformed programs• Separating critical and approximate parts of program
[Carbin & Rinard ISSTA ’10, Sampson et al. PLDI ’11]
• Verifying relaxed semantics of programs [Carbin et al. CSAIL-TR ‘11]
Analytic properties of programs [Majumdar & Saha RTSS ‘09, Chaudhuri et al. POPL’10, Ivancic et al. MEMOCODE ‘10 , Reed & Pierce ICFP ’10, Chaudhuri & Solar-Lezama PLDI ’10 , Chaudhuri et al. FSE ’11]
![Page 51: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/51.jpg)
Summary
Model of Computation• Accuracy-aware program transformations• Effects on overall accuracy and execution time
Explore and Exploit Optimal Tradeoffs• Approximate optimal tradeoff curve construction• Polynomial, dynamic programming algorithm• Randomized program configurations to achieve tradeoffs
Envisioned Applications• Image and video processing, numerical algorithms,
queries on big data sets, machine learning, …• Optimization, fault tolerance, dynamic adaptation
![Page 52: Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations](https://reader035.fdocuments.us/reader035/viewer/2022062517/568132e3550346895d999f46/html5/thumbnails/52.jpg)