ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl...
-
Upload
dwight-norton -
Category
Documents
-
view
216 -
download
1
Transcript of ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl...
![Page 1: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/1.jpg)
ENERGY-PROPORTIONAL IMAGE SENSING
FOR
Robert LiKamWa
Bodhi PriyanthaMatthai
PhiliposeVictor BahlLin Zhong
CONTINUOUS MOBILE VISION
http://roblkw.comhttp://research.microsoft.com
![Page 2: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/2.jpg)
The Devil Wears Prada
CONTINUOUS MOBILE VISION
CONTINUOUS
MOBILE
VISION
2 of 33
![Page 3: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/3.jpg)
CONTINUOUS MOBILE VISION
CONTINUOUS
MOBILE
VISION
Gestures
Object Memory
3 of 33
Face Recognition
Victor
Fine-grained Localization
![Page 4: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/4.jpg)
BATTERY LIFEBATTERY DEATH
Google Glass
2-3 hours
LooxCie
2-3 hours
GoPro Hero
2-3 hours
4 of 33
![Page 5: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/5.jpg)
2000 mWh
10 h200 mW
5 of 33
![Page 6: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/6.jpg)
Image Sensor
Goal < 25 mW
Sensors
~ 5 mW
Processor
~150 mW
Network Stack
~20 mW
6 of 33
![Page 7: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/7.jpg)
Image Sensor
Goal < 25 mW
7 of 33
Image Sensor
Reality > 250 mW
![Page 8: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/8.jpg)
KEY IDEA: ENERGY α QUALITYPo
wer
Frame rate
Pow
er
Resolution
![Page 9: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/9.jpg)
ENERGY PROFILE OF AN IMAGE SENSOR
1 MP, 5 fps250 mW
0.3 MP, 15 fps245 mW
1 MP, 15 fps295 mW
< 25 mWGoal:
Reality: > 230 mW
0.3 MP, 5 fps232 mW
9 of 38
![Page 10: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/10.jpg)
ENERGY-EFFICIENTIMAGE SENSING
Image Sensor Characterization
Energy Reduction Techniques
Energy vs. Vision Performance
10 of 38
![Page 11: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/11.jpg)
IMAGE SENSOR MEASUREMENT
Camera Module
Programmable Clock (I2C)
NI DAQ Device
Power RailResistors
Power
VDD CLK
* Profiled 5 state-of-the-artimage sensors from 2 manufacturers
*
11 of 38
![Page 12: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/12.jpg)
Camera Module
Programmable Clock (I2C)
NI DAQ Device
Power RailResistors
IMAGE SENSOR MEASUREMENT
12 of 38
![Page 13: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/13.jpg)
IMAGE SENSOR WAVEFORMSActive Period
Idle Period
Analog
Digital
PLL
13 of 38
![Page 14: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/14.jpg)
IMAGE SENSOR WAVEFORMSActive Period
Idle Period
14 of 38
![Page 15: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/15.jpg)
IMAGE SENSOR WAVEFORMSActive Period
Idle Period
Pixel Count divided by
Clock Frequency
Frame Timeminus
Active Time
![Page 16: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/16.jpg)
IMAGE SENSOR PIXEL COUNT (N)
Region-of-Interest(Windowing)
Active Active
Pow
er
Time
Pow
er
Time 16 of 38
Scaled Resolution(Pixel Skipping)
![Page 17: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/17.jpg)
IMAGE SENSOR PIXEL COUNT (N)
Active Active
Pow
er
Time
Pow
er
Time
Video (30 FPS) Power vs. Resolution
17 of 38
![Page 18: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/18.jpg)
ActiveFrame
Readout
Active
Pow
er
Time
Active
IMAGE SENSOR FRAME RATE (R)
Video (0.1 MP) Power vs. FPS
ActiveFrame
ReadoutActive
Pow
er
Time
Active ActiveActive Active
1.0s
18 of 38
![Page 19: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/19.jpg)
CHARACTERIZATION CONCLUSION:
NO ENERGY PROPORTIONALITY
Video (0.1 MP) Power vs. FPS
Video (30 FPS) Power vs. Resolution
19 of 38
![Page 20: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/20.jpg)
ENERGY-EFFICIENTIMAGE SENSING
Image Sensor Characterization
Energy Reduction Techniques
Energy vs. Vision Performance
20 of 38
![Page 21: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/21.jpg)
TECHNIQUE #1: AGGRESSIVE STANDBY
Active Active
Active Active
21 of 38
![Page 22: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/22.jpg)
Active Active
Active Active
CAVEAT TO AGGRESSIVE STANDBY
Active Active
Not enough exposure time
This won’t work for long active periods,i.e., high resolution, high frame rate.
22 of 38
![Page 23: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/23.jpg)
TECHNIQUE #2: CLOCK SCALING (f)
One pixel per clock period
23 of 38
![Page 24: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/24.jpg)
TECHNIQUE #2: CLOCK SCALING (f)
Faster clock
Lower Active Time
Higher Active Power
Higher Idle Power
Slower clock
Higher Active Time
Lower Active Power
Lower Idle Power
24 of 38
![Page 25: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/25.jpg)
TECHNIQUE #2: CLOCK SCALING (f)
Active
Active
Low Pixel CountLow Frame Rate
Slowed Clock
Optimal clock frequency depends onPixel Count & Frame Rate
25 of 38
![Page 26: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/26.jpg)
AGGRESSIVE STANDBY + CLOCK OPTIMIZATION
Sped-upClock
Aggressive Standby
ActiveReadout
Active
Readout
Optimal clock frequency depends onPixel Count & Frame Rate
Exposure Time26 of 38
![Page 27: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/27.jpg)
ENERGY α QUALITYFr
am
e r
ate
(FP
S)
Resolution (MP)
30
25
20
15
10
5
021 3 4 5
Fra
me r
ate
(FP
S)
Resolution (MP)
30
25
20
15
10
5
021 3 4 5
Pow
er
(mW
)
350
300
250
200
150
100
50
0
Aggressive Standby &Clock Optimization
Unoptimized
Pow
er
(mW
)
350
300
250
200
150
100
50
0
27 of 38
![Page 28: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/28.jpg)
CPU
Driv
er
28 of 37
![Page 29: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/29.jpg)
29 of 38
![Page 30: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/30.jpg)
CURRENT IMAGE SENSOR DESIGN
Image Processor
Pixel ArrayColumn Output
Gain, ADC
Analog Signal Chain
70-85% Power Consumption
30 of 38
![Page 31: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/31.jpg)
Image Processor
Pixel ArrayColumn Output
High-Speed ADC
Mid-Speed ADC
Low-Speed ADC
Gain
HETEROGENEOUS SENSOR DESIGN
Analog Signal ChainHeterogeneous
31 of 38
![Page 32: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/32.jpg)
ENERGY α QUALITY
Default
Clock select
Standby
HW Fix
Power vs. Framerate (at 0.1 MP)
Power vs. Resolution(at 5 FPS)
Default
Clock select
Standby
HW Fix
32 of 36
![Page 33: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/33.jpg)
ENERGY-EFFICIENTIMAGE SENSING
Image Sensor Characterization
Energy Reduction Techniques
Energy vs. Vision Performance
33 of 38
![Page 34: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/34.jpg)
ENERGY vs. VISION: VISION TASK
IMAGE REGISTRATION
34 of 36
![Page 35: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/35.jpg)
ENERGY vs. VISION: PERFORMANCE Image
Registration Success
Power Reduction with software
assist
Estimated Power Reduction with hardware assist
Full VGA Resolution0.1 MP, 30 FPS 99.9% 51% 84%
Frame Rate Reduction0.1 MP, 3 FPS 95.7% 95% 98%
30% Window0.06 MP, 30 FPS 96.5% 63% 91%
Subsampled by 20.3 MP, 30 FPS 91.8% 71% 94%
185 mW
TypicalAverage Power
10 mW
With aggr. standby& optimal clock
3 mW
With heterogeneousanalog signal chain
![Page 36: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/36.jpg)
Scalable
Computer
Vision
Algorithms
IntegratedSystems
Design
Developer
SupportEnergy
Proportional
Image
Sensing
36 of 36
![Page 37: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/37.jpg)
Continuous Mobile Vision
![Page 38: ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION .](https://reader037.fdocuments.us/reader037/viewer/2022110321/56649cff5503460f949d166d/html5/thumbnails/38.jpg)
ENERGY-PROPORTIONAL IMAGE SENSINGFOR
CONTINUOUS MOBILE VISION
http://roblkw.comhttp://research.microsoft.com
Image sensors are not energy-proportional…
Fra
me r
ate
(FP
S)
Resolution (MP)
30
20
10
01 3 5
…but we can make them energy-proportional…
Aggressive Standby
Clock Optimization
Sensor Modifications
… and this is just the beginning.
CMV