Post on 11-Jan-2022
MCDNN:AnApproximation-BasedExecutionFrameworkfor DeepStreamProcessingUnderResourceConstraints
HaichenShen(haichen@cs.washington.edu)withSeungyeop Han,Matthai Philipose,
Sharad Agarwal,AlecWolman,ArvindKrishnamurthy
UniversityofWashington MicrosoftResearch
1
When computer vision meets wearable
Consumer Manufacturing Public Safety
“That drink will get you to 2800 calories for today”
“I last saw your keys in the store room”
“Remind Tom of the party”
“You’re on page 263 of this book”
3
Deep learning makes vision work
Recognition Task face scene* object*
Accuracy 97% 88% 92%
Compute/frame (FLOPs) 1.00G 30.9G 39.3G
Do we have enough resources to run deep learning?
4
Compute@1-30fps (FLOPS) 1-30G 30-900G 40G-1.2T
* top-5 accuracy is shown in the table
But...
Resource usage for continuous vision
Imager Processor Radio
OmnivisionOV274090mW
Tegra K1 GPU290GOPS@10W= 34pJ/OP
Qualcomm SD810 LTE>800mWAtheros 802.11 a/g15Mbps@700mW= 47nJ/b
Cloud
Amazon EC2CPU c4.large 2x400GFLOPS $0.1/hGPU g2.2xlarge 2.3TFLOPS $0.65/h
5Huge gap between workload and budget
BudgetDevice power Cloud cost
30% of 10Wh for 10h = 300mW $10 person/year
Workload Deep learning 300GFLOPS @ 30GFLOPs/frame, 10fps
Compute power 9GFLOPS 3.5GFLOPS (GPU) / 8GFLOPS (CPU)
Neural network
(c)convolution(f)fullyconnect(m)maxpool(r)relu(s)softmax
6
c
r
m
c
r
m
m
r
c
r
c
r
c
f
r
f
r
f
s
imgRimgGimgB
classes+scores
Neural network ≈ matrix multiplications
7
x x x
Low rank approximation(Y. Kim, et al. 2016)
x
Matrix sparsification(S. Han, et al. 2015)
(c)convolution(f)fullyconnect(m)maxpool(r)relu(s)softmax
c
r
m
c
r
m
m
r
c
r
c
r
c
f
r
f
r
f
s
imgRimgGimgB
classes+scores
c
c
r
m m
r
f
r
c
r
f
s
imgRimgGimgB
classes+scores
Architectural changes(J. Ba, et al. 2014)
Managing the approx. / resource trade-off
‣Detailed characterization of the approximation / resource trade-off for many optimizations
‣ Two new optimizations for streaming, multi-application settings
‣New scheduling problem, Approximate Model Scheduling, with a heuristic solution
8
Outline
‣Detailed characterization of the approximation / resource trade-off for many optimizations
‣ Two new optimizations for streaming, multi-application settings
‣New scheduling problem, Approximate Model Scheduling, with a heuristic solution
9
30 40 50 60 70 80 90 100aFFuraFy (%)
101
102
103
mem
ory
(0
%)
2bM
6Fene
)aFe
Memory / accuracy trade-off
10
30 40 50 60 70 80 90 100aFFuraFy (%)
101
102
103
mem
ory
(0
%)
2bM
6Fene
)aFe
Memory / accuracy trade-off
11
Substantially reduce memory use with gradual accuracy loss
45 50 55 60 65 70 75 80 85aFFuraFy (%)
0
2
4
6
8
10
en
erJ
y (
J)
2Ej((xeF)
6Fene((xeF)
)aFe((xeF)
Energy / accuracy trade-off
Wifi xmit cost (0.5J)
LTE xmit cost (0.9J)
Compute energy budget (2.3J)
12
Always execute locally
Can execute locally under energy budget
Exceed energy budget when execute locally
energy budget = total energy / total time(10h) / requests per second(1 req/sec)
Nvidia Jetson TK1
Outline
‣Detailed characterization of the approximation / resource trade-off for many optimizations
‣ Two new optimizations for streaming, multi-application settings‣ Specialization‣ Model sharing
‣New scheduling problem, Approximate Model Scheduling, with a heuristic solution
13
Exploiting stream locality by specialization
• Standard deep neural network recognizes 4000 people• Most of videos are dominated by less than 10 faces over minutes
14
Timeline
Produce more compact models for skewed classes
Specialization runtime
15
input(withskeweddistribution)
class
compactmodel
class
fullmodel
checkfor“other”
class
input
fullmodel
Outline
‣Detailed characterization of the approximation / resource trade-off for many optimizations
‣ Two new optimizations for streaming, multi-application settings
‣New scheduling problem, Approximate Model Scheduling, with a heuristic solution
18
Energy 14
Approximate model scheduling
19
Mobile device
Cloud
Model Pool
memory
energy
cost
Accuracy
Task 1
8 2 2model 1 (90%) model 2 (80%)
5 1 1
Task 2
9 3 3model 3 (80%) model 4 (70%)
6 2 2
Memory 10
Cost 14
Goal: maximize the overall accuracy
Approximate model scheduling
20
Mobile device
Cloud
Model Pool
memory
energy
cost
Accuracy
Requests:1. task 1 2. task 2 3. task 1 4. task 1 5. task 2
Task 1
8 2 2model 1 (90%) model 2 (80%)
5 1 1
Task 2
9 3 3model 3 (80%) model 4 (70%)
6 2 2
Energy 14 Memory 10
Cost 14 à cloud, model 1à device, model 4à cloud, model 1à cloud, model 2à device, model 4
model 4
model 1 model 1 m2
model 4
1 2 3 4 5
Packing problem
Approximate model scheduling
21
Mobile device
Cloud
Model Pool
Accuracy
Requests:1. task 1 à cloud, model 12. task 2 à device, model 43. task 1 à cloud, model 14. task 1 à cloud, model 25. task 2 à device, model 46. task 1
Energy 14 Memory 10
Cost 14
model 4
model 2
memory
energy
cost
Task 1
8 2 2model 1 (90%) model 2 (80%)
5 1 1
Task 2
9 3 3model 3 (80%) model 4 (70%)
6 2 2
model 4
model 1 model 1
1 2 3
Paging problem
m2
model 4
4 5
Approximate model scheduling
• Packing problem: pick versions that satisfy energy/cost budgets" 𝑒$𝑥$&
�
&≤ 𝐸," 𝑐$𝑥$&,
�
&≤ 𝐶(𝑥$&, 𝑥$&, ∈ 0,1 , 𝑥$& 3 𝑥$&, = 0)
• Paging problem: pick versions that fit in memory∀1 ≤ 𝑡 ≤ 𝑇," 𝑠$𝑥$&
:
$;<≤ 𝑆
• Goal: maximize the accuracymaxA" " 𝑎$(𝑥$& + 𝑥$&, )
�
$
�
&
No known optimal online algorithms22
Heuristic scheduler
• Estimate future resource use and compute the budget for each request
• Account for paging cost to reduce oscillations
• Use increasingly more accurate versions of more heavily used models
23
development time
MCDNN frameworkcompiler
input typemodel schema
training/validationdata
cloudruntimeschedulerdatarouterprofiler
apps
input input
classesclasses
trainedmodelcatalog
deviceruntimeschedulerdatarouter
clouddevice
runtime
development time
MCDNN frameworkcompiler
input typemodel schema
training/validationdata
cloudruntimeschedulerdatarouterprofiler
apps
input input
classesclasses
trainedmodelcatalog
deviceruntimeschedulerdatarouter
specializer
statsspecializedmodels
clouddevice
specializationtime
runtime
Conclusion
• MCDNN makes efficient trade-offs between resource use and accuracy• Formulate the approximate model scheduling problem and
devise a heuristic algorithm• Design a generic approximation-based execution framework
for continuous mobile vision
27
Thank you! Questions?
Cloud cost / accuracy trade-off
45 50 55 60 65 70 75 80 85
aFFuUaFy (%)
100
101
102
103
104
Oate
nFy
(P
s)2bj(COouG G38)
2bj(COouG C38)
6Fene(COouG G38)
6Fene(COouG C38)
)aFe(COouG G38)
)aFe(COouG C38)
)aFe(COouG G38)
)aFe(COouG C38)
cloud GPU latency budget (9.7ms)@ $10/yr, 1 request/s@ AWS g2.2xlarge
cloud GPU latency budget (582ms)@ $10/yr, 1 request/min@ AWS g2.2xlarge
cloud CPU latency budget (7645ms)@ $10/yr, 1 request/min@ AWS c4.large
29latency budget = cost budget / cost per hour / #requests
Model sharing
30
inputintermediate
values
face ID
race age gender …
routerinput
(a) (b)
model-fragment cache