Rethinking the Camera Pipeline for Computer...
Transcript of Rethinking the Camera Pipeline for Computer...
![Page 1: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/1.jpg)
Mark Buckler, Cornell Suren Jayasuriya, CMU Adrian Sampson, Cornell
Rethinking the Camera Pipeline for Computer Visionor, Building an Approximate Camera
![Page 2: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/2.jpg)
Mobile vision is a pretty cool idea
…always on.…on your smartphone.…in real time.
object recognition object localization
image segmentation 3D structure reconstruction
localization & mapping optical character recognition
face recognition activity recognition
human pose estimation
![Page 3: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/3.jpg)
hardware efficiency
algorithmic efficiency
images vision results
![Page 4: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/4.jpg)
ISCA 2016
![Page 5: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/5.jpg)
images vision results
hardware efficiency
algorithmic efficiency
![Page 6: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/6.jpg)
imagesphotonsraw sensor data
ISP
This project: a programmable camera pipeline.
![Page 7: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/7.jpg)
Let’s approximate a camera pipeline
Design approximation into the camera sensor and the ISP
Show how to retrain vision modelsto work on the cheaper, raw data
Measure energy-accuracy trade-offslatent in real-world vision applications
![Page 8: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/8.jpg)
Camera Sensor
Image Signal Processor
Photodiode
Amplifier
ADC
Demosaicing
Denoising
White Balance
Gamut Mapping
JPEG Compression
Tone Mapping
Vision Application CPU/GPU/“VPU”
RAW image
JPEG image
![Page 9: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/9.jpg)
Photodiode
Amplifier
ADC
Demosaicing
Denoising
White Balance
Gamut Mapping
JPEG Compression
Tone Mapping
Vision Application
RAW image
JPEG image
photography mode
vision mode
Vision Application
Photodiode
Amplifier
ADC
approximate RAW image
power gated
5-bit logarithmic
![Page 10: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/10.jpg)
Reversing the pipeline
![Page 11: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/11.jpg)
norm
alize
d er
ror
0
0.25
0.5
0.75
1
1.25
1.5
1.75
2
ISP pipeline stages
original demosaic+ gamma compress
+ denoise all off
LeNet3 ResNet20 ResNet44 Farneback SGBM OpenMVG RCNN OpenFace
Sensitivity to ISP stages
cras
h
![Page 12: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/12.jpg)
Demosaicing
Denoising
White Balance
Gamut Mapping
JPEG Compression
Tone Mapping
“True” demosaicing.
![Page 13: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/13.jpg)
Demosaicing
Denoising
White Balance
Gamut Mapping
JPEG Compression
Tone Mapping
Subsampling.
![Page 14: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/14.jpg)
Demosaicing
Denoising
White Balance
Gamut Mapping
JPEG Compression
Tone Mapping
![Page 15: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/15.jpg)
Demosaicing
Denoising
White Balance
Gamut Mapping
JPEG Compression
Tone Mapping
quantize
![Page 16: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/16.jpg)
Demosaicing
Denoising
White Balance
Gamut Mapping
JPEG Compression
Tone Mapping
replace both within-sensor trickery!
![Page 17: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/17.jpg)
Sensitivity to ADC quantizationno
rmal
ized
erro
r
0.0
1.0
2.0
3.0
4.0
bits (linear quantization)8 7 6 5 4 3 2 1
LeNet3 ResNet20 ResNet44 Farneback SGBM OpenMVG RCNN OpenFace
![Page 18: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/18.jpg)
Sensitivity to ADC quantizationno
rmal
ized
erro
r
0.0
1.0
2.0
3.0
4.0
bits (logarithmic quantization)8 7 6 5 4 3 2 1
LeNet3 ResNet20 ResNet44 Farneback SGBM OpenMVG RCNN OpenFace
![Page 19: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/19.jpg)
How much energy can vision mode save?
sensor ISP vision ASIC
137.1–338.6 mW [LiKamWa]
130–185 mW [ON Semiconductor] 250 mW [Hegarty]
204 mW [TrueNorth] 590 mW [EIE]
![Page 20: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/20.jpg)
How much energy can vision mode save?
sensor ISP vision ASIC
![Page 21: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/21.jpg)
How much energy can vision mode save?
sensor ISP vision ASIC
readout (ADCs)
![Page 22: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/22.jpg)
How much energy can vision mode save?
sensor ISP vision ASIC
![Page 23: Rethinking the Camera Pipeline for Computer Visionapproximate.computer/wax2017/slides/sampson.pdf · Vision Application Photodiode Amplifier ADC approximate RAW image power gated](https://reader036.fdocuments.us/reader036/viewer/2022062507/5fc441167a31463b633ae724/html5/thumbnails/23.jpg)
Unresolved questions
Dynamic feedback loop
New signal processing toimprove learnability
Incremental costfor incremental scene changes
Data movement between sensor, ISP, and application