Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image...
Transcript of Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image...
![Page 1: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/1.jpg)
1
Low-Power Event-driven
Image Sensor Architectures
Laurent Fesquet 1,2, Amani Darwish 1,2
Gilles Sicard 3
1 University Grenoble Alpes – TIMA – Grenoble, France
2 CNRS – TIMA – Grenoble, France
3 CEA – LETI – Grenoble, France
SIGNAL 2016
First International Conference on Advances in Signal, Image and Video Processing
June 26 - 30, 2016 - Lisbon, Portugal
![Page 2: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/2.jpg)
Internet of Things Challenges
2 Laurent Fesquet
+ more data
+ more storage
+ more communications
+ more consumption
![Page 3: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/3.jpg)
Upcoming Challenges
3
• 10% of the world
electricity consumption
• Growth is exponential
• Need fossil energy or
more nuclear power
plants
Laurent Fesquet
Where electricity is consumed in the digital universe?
![Page 4: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/4.jpg)
4
• Another way for sampling • Sampling without Shannon?
• Nonuniform Sampling
• Sampling an image • State of art: Conventional Image Sensors
• State of art: Event-based Image Sensors
• An architecture for low-power Image Sensors
• Conclusions
Outline
Laurent Fesquet
![Page 5: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/5.jpg)
Important questions
Claude Shannon and Harry Nyquist are they responsible of
the digital data deluge?
• We will not answer to this question but the big data is
today a reality!
Can we find a better sampling scheme to stem this digital
data deluge and stop the energy waste?
• We hope so !!!
5 Laurent Fesquet
Harry Nyquist and Claude Shannon
![Page 6: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/6.jpg)
Sampling is the success key
• Sampling based on the Shannon-Nyquist theorem
– Efficient and general theory… whatever the signals!
• Smart sampling techniques
– More efficient but less general
for specific signals!
– Need a more general mathematical framework
• F. Beutler, “Sampling Theorems and Bases in a Hilbert Space”, Information and
Control, vol.4, 97-117,1961
• Sampling should be specific to signals and applications
Laurent Fesquet 6
![Page 7: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/7.jpg)
7
What can we do?
ADC
A-ADC
DSP
Event-driven
DSP
Sampling Clock
ANALOG DIGITAL
x(t)
x(t) {axn, dtn}
DAC
{ayn, dtn}
A-DAC
y(t)
y(t)
ANALOG
{xn,Te} {yn, Te}
Uniform and Synchronous
Nonuniform and Event-driven (Asynchronous)
Laurent Fesquet
Direct processing of the nonuniform samples
![Page 8: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/8.jpg)
8
• Another way for sampling • Sampling without Shannon?
• Nonuniform Sampling
• Sampling an image • State of art: Conventional Image Sensors
• State of art: Event-based Image Sensors
• An architecture for low-power Image Sensors
• Conclusions
Outline
Laurent Fesquet
![Page 9: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/9.jpg)
Differently sampling
9
Respect the Shannon theorem
Instants exactly known
Information: Tsample, { ik }
In an ADC: Amplitude quantization
Many useless samples
“Level-crossing sampling” (LCSS)
Amplitudes exactly known
Information: quanta, { dtik }
In an A-ADC: Time quantization
Only useful samples
Uniform sampling Non uniform sampling
t
x(t) axn
dtxn Te
Magnitude axn-1
Dual
Magnitude
t
x(t) xn
quantum
Laurent Fesquet
![Page 10: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/10.jpg)
SNR with non-uniform sampling
• SNR for a sinusoid:
• Theoretically, noise is only due to amplitude quantization
• With non-uniform sampling, the time is quantized
• Noise only depends on the timer resolution whatever
the threshold distribution
Laurent Fesquet 10
NSNRdB .02,676,1
)log(202,11 cdB fTSNR
Number of bits
Timer period
![Page 11: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/11.jpg)
A-ADC or ADC for LCSS
• A-ADC for Non Uniform Sampling
Timer
Difference
quantifier
Up/down
Counter/LUT
Vref(t))
dtik
i(t)
ik dwn
DAC
up
Laurent Fesquet 11
If (i(t) – Vref(t)) > q/2 up
If (i(t) – Vref(t)) < -q/2 dwn
Else up = dwn = 0
+q/2
-q/2 Vref(t))
![Page 12: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/12.jpg)
A-ADC
12
A-ADC
realized with circuits
on-the-shelf. (FPGA+DAC+Comparators)
Asynchronous ADC
Synchronous ADC
E. Allier et al. 05
Aeschlimann et al., 06
Beyrouthy et al., 11
Laurent Fesquet
![Page 13: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/13.jpg)
A level-crossing flash
asynchronous analog-to-digital
converter
A-ADC testchips
13
Lowering in one step the storage, the processing,
the communications and the power consumption!
E. Allier et al., Async, 2003
Microphotography of the A-ADC
In CMOS 130 nm technology from
STMicroelectronics
Laurent Fesquet
A Clockless ADC/DSP/DAC System with
Activity-Dependent Power Dissipation
and No Aliasing
Y. Tsividis et al., ISSCC, 2008
F. Akopyan et al., Async, 2006
![Page 14: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/14.jpg)
A successful experiment
Laurent Fesquet 14
Event-driven block
(Asynchronous logic)
CMOS AMS 0.35 µm
T. Le Pelleter et al., 13
L. Fesquet et al., 14
• Context of the medical implants
• Activity patient measurements
Experiments based on real physiological signals (recorded on the patients)
![Page 15: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/15.jpg)
What we learned with this experiment
With the medical implant
• No pre-processing
• Less than 1% of data compared to the uniform sampling
• 3 orders of magnitude reduction on power
In general
• Be more specific to signals and applications
• Non-general approach, but reproducible
• Non-uniform sampling well-suited to sporadic signals
15 Laurent Fesquet
![Page 16: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/16.jpg)
16
• Another way for sampling • Sampling without Shannon?
• Nonuniform Sampling
• Sampling an image • State of art: Conventional Image Sensors
• State of art: Event-based Image Sensors
• An architecture for low-power Image Sensors
• Conclusions
Outline
Laurent Fesquet
![Page 17: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/17.jpg)
Low-power Image Sensors? Objective of this presentation
• No image sensor really dedicated to low-power
• High resolution sensors requires a high speed ADC
– High power consumption
• How to reduce the power in Image Sensors
– Use advanced technology nodes (expensive)
– Reduce the data flow / activity
– Rethink analog-to-digital conversion
Apply nonuniform sampling in 2D
Event-based image sensors
Laurent Fesquet 17
Darwish et al. 2014, 2015
Posch et al. 2008, 2011,
Delbruck et al. 2004,
Qi et al. 2004
![Page 18: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/18.jpg)
18
• Another way for sampling • Sampling without Shannon?
• Nonuniform Sampling
• Sampling an image • State of art: Conventional Image Sensors
• State of art: Event-based Image Sensors
• An architecture for low-power Image Sensors
• Conclusions
Outline
Laurent Fesquet
![Page 19: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/19.jpg)
Image sensor principles
• Based on photodiodes
• Pixel fill factor = optical quality
• All pixels are read in sequence
• Larger the sensor, higher the throughput
(fixed frame rate)
• Higher the throughput, higher the ADC
consumption
• The ADC is the first contributor of
power consumption
Laurent Fesquet 19
ADC
Pixel Sensitive
Blind
![Page 20: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/20.jpg)
Image Sensor Principles
Laurent Fesquet 20
Most common feature for an APS:
– Fill factor, SNR, Sensitivity, Dynamic, …
Pixel Photodiode
Electronics
(blind) Frame period
Time
Integration time Integration time
Reset
Vph
ADC
Reset
ADC
Luminance A
Luminance B
Pixel A Pixel B
![Page 21: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/21.jpg)
21
• Another way for sampling • Sampling without Shannon?
• Nonuniform Sampling
• Sampling an image • State of art: Conventional Image Sensors
• State of art: Event-based Image Sensors
• An architecture for low-power Image Sensors
• Conclusions
Outline
Laurent Fesquet
![Page 22: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/22.jpg)
Event-based Image Sensors
• Pixel reading on event
– Initiated by the pixel itself
– Frameless sensor
– No frame rate
• Address Event Representation (AER)
• Event-based pixel
– Asynchronous communication
– Time-to-First Spike encoding (TFS)
• Time-to-Digital Conversion (no more ADC)
• Sensor activity depends on the captured scene
• Reduced dataflow
Laurent Fesquet 22
Système de Lecture
Asynchrone
- TDC -
Système de Lecture
Asynchrone
- TDC -
![Page 23: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/23.jpg)
Event-based Image Sensors
Laurent Fesquet 23
Column Arbiter
Ref : Myat2011
Row
Arb
ite
r
• Need to manage multiple pixel requests
• Event-based Image Sensors need arbitration
• Complex Circuitry
– Exponentialy growth with the image sensor resolution
• Inaccurate time stamping
• Fixed priority arbiters are unfair
![Page 24: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/24.jpg)
Time-to-First-Spike
a bio-inspired approach
Laurent Fesquet 24
• Time represents the
information
– Spike time stamping
– Time quantization
• Advantages
– Read on events
– One integration time per pixel
– Dynmaic range controlled by
the TDC
– Robust (digital readout circuit)
• Drawbacks
– Large pixel area
– Variability of the analog part
(comparator)
Readout
system
time
Fixed voltage
reference
![Page 25: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/25.jpg)
Event-based Image Sensor: state of art
Laurent Fesquet 25
Delbrück/Lichtsteiner : Dynamic Vision Sensor (DVS)
• Neuromorphic approach
• Detect changes in pixel luminance (send a request)
• High dynamic range sensor (logarithmic pixel)
Lichtsteiner et al.
2006
![Page 26: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/26.jpg)
Event-based Image Sensor: state of art
Laurent Fesquet 26
Posch :
• Double arbiter
• Temporal redundancy reduction
• High dynamic range(~ 140 dB)
• large pixel area (89 transistors, 2 capacitances, 2 photodiodes)
• Fill factor < 14 %
Posch et al. 2011
![Page 27: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/27.jpg)
Event-based Image Sensor: state of art
Laurent Fesquet 27
Bermak / Shoushun :
• Pipelined arbiter with alternate
priority
• Local pixel reset
• Pixel is idle after reading
• Low power design
• Simple pixel design
Shoushun, Bermak 2006
![Page 28: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/28.jpg)
Event-based Image Sensor: state of art
Laurent Fesquet 28
Parameters Conventionnal Image
Sensors
Event-based Image
Sensors
Sampling Scheme Uniform Nonuniform
Integration Time One for the whole sensor One per pixel
Readout System Sequential Event-based
Protocol Sequential and Exhaustive Sequential and Event-based
(AER)
Data Flow Fixed and Redundant Nonuniform and Reduced
Pixel Complexity Simple Complex
Power Consumption Sub-optimal (redundancy) Reduced
Frame Rate Fixed Event-dependent
![Page 29: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/29.jpg)
29
• Another way for sampling • Sampling without Shannon?
• Nonuniform Sampling
• Sampling an image • State of art: Conventional Image Sensors
• State of art: Event-based Image Sensors
• An architecture for low-power Image Sensors
• Conclusions
Outline
Laurent Fesquet
![Page 30: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/30.jpg)
Changing the paradigm
in a realistic manner
• Keep the fill factor reasonable (important for industrial
products)
• Reduce the throughput without changing the frame
rate
• Remove the ADC to limit power consumption
• Replace it by a digital circuit (more easy to implement)
Laurent Fesquet 30
![Page 31: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/31.jpg)
Changing the paradigm
in a realistic manner
Laurent Fesquet 31
Spatial Redundancies
Temporal Redundancies
Suppress redundancies
Prefer Time-to-Digital
conversion
Use Event-Driven logic
(asynchronous)
• Global Clock is replaced by
local channels (handshaking)
• Power is only consumed when
data are really processed
Darwish et al., 14
Posch et al., 08
![Page 32: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/32.jpg)
Laurent Fesquet 32
R1
C1 C2 C3 C4
R2
R3
R4
P11
P22 P12
P33
P42 P44
tx tx + Δt ty ty + Δt time
Ack Ack
Invalid Pixels: Reading 1 :
• P21
Reading 2 :
• P32
• P34
• P43
Image Sensor
Behavior
![Page 33: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/33.jpg)
High-level Pixel Model
Laurent Fesquet 33
• 4-phase pixel
– Event detection
– Send requests
– Check requests
– Reset pixel
Pixel
Reset
Send
Requests
Check
Requests
Event
Detection
Global Reset
![Page 34: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/34.jpg)
Pixel behavior
Laurent Fesquet 34
Req Req Req
• Based on event-detection
• 1-level crossing sampling scheme
• Unique integration time per pixel
• Time to first spike encoding
Adjust the image
sensor sensitivity
=
High Dynamic range
A. Darwish et al., EBCCSP, 2015
A. Darwish et al., NewCAS, 2014
![Page 35: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/35.jpg)
Laurent Fesquet 35
Δ = 16
Discussion on Δt
• Δt : deadtime after time stamping.
• No other time stamping allowed during Δt even if events
occured
– Control the time stamping resolution
– Equivalent to ADC quantum ΔV
– Control the maximal number of readings or greyscale levels
• Trade-off between image quality and readout rate
Specific behavior of our sensor
![Page 36: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/36.jpg)
Performance Evaluation
Laurent Fesquet 36
• Asynchronous Image Sensor Readout Rate (AISSR) – Compute the read pixel number by our architecture (M columns x N rows)
– Reading operations bounded by the number of luminance values
– d is the image dynamic range
• PSNR (Peak Signal to Noise Ratio) – Evaluate the distorsion induced by the compression
– For grayscale image, 30dB et 50dB are prety goo values
• MSSIM (Mean Structural SIMilarity)
– Take into account the human perception (luminance, contrast, image structure)
– Best is 1, worst is 0
![Page 37: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/37.jpg)
Laurent Fesquet 37
Readout rate = 0.042 %
PSNR = 38.1dB
MSSIM = 0.919
Readout rate = 0.13 %
PSNR = 40.03dB
MSSIM = 0.999
Readout rate = 0.018 %
PSNR = 36.29dB,
MSSIM = 0.765
Readout rate = 0.004 %
PSNR = 22.51dB
MSSIM = 0.483
Δ = 2 Δ = 4
Δ = 32 Δ = 16 Δ = 8
Reference Image
Readout rate = 100%
MSSIM = 1
Readout rate = 0.008 %
PSNR = 31.46dB
MSSIM = 0.635
![Page 38: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/38.jpg)
Laurent Fesquet 38
Reference Image
Readout rate = 100%
MSSIM = 1
Readout rate = 0.029 %
PSNR = 30.34dB
MSSIM = 0.942
Readout rate = 0.084 %
PSNR = 31.1dB
MSSIM = 0.987
Readout rate = 0.012 %
PSNR = 28.36dB
MSSIM = 0.813
Readout rate = 0.006 %
PSNR = 26.24dB
MSSIM = 0.605
Readout rate = 0.003 %
PSNR = 22.13dB,
MSSIM = 0.392
Δ = 2 Δ = 4
Δ = 32 Δ = 16 Δ = 8
![Page 39: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/39.jpg)
Memory I
(Pixel Reset Time)
Verified Pixel Address
Memory II
(Integration Time)
Verified Pixel Address
Time Stamping
Block
Event-Driven
Pixel Matrix
Pixel Request
Processing Block
Verification Readout
Request
Integration time
computation
Asynchronous readout architecture
![Page 40: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/40.jpg)
How do we suppress Spatial Redundancy ? (again)
40
4 x 4 image sensor
Ack Ack Darwish et al., 14
Laurent Fesquet
![Page 41: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/41.jpg)
• 1-level crossing sampling in 2D
• Low percentage of readings per column (< 1 %)
• Image data flow reduction
• Event-driven digital circuitry
• Adjustable resolution and dynamic range
• Don’t need an ADC (power consuming)
• Intrinsic A-to-D conversion
41
What we learned with image sensors
Laurent Fesquet
![Page 42: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/42.jpg)
42
• Another way for sampling • Sampling without Shannon?
• Nonuniform Sampling
• Sampling an image • State of art: Conventional Image Sensors
• State of art: Event-based Image Sensors
• An architecture for low-power Image Sensors
• Conclusions
Outline
Laurent Fesquet
![Page 43: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/43.jpg)
General Conclusion
• Lesson 1: Determine the most efficient sampling!
• Lesson 2: Fit well Event-Driven Circuits (asynchronous)
• Lesson 3: Ultra-Low Power
• Less samples means:
– Less computation, less storage, less communications,
– less power
43 Laurent Fesquet
A more energy efficient
approach of digital processing
![Page 44: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/44.jpg)
Image Sensor Conclusion
Laurent Fesquet 44
Parameters Delbrück /
Lichtsteiner Posch
Bermak /
Shoushun Our work
Readout
Protocol
Sequential Sequential Sequential Parallel
Non-
deterministic
Non-
deterministic
Non-
deterministic Deterministic
Arbiters Yes Yes Yes No
Readout rate Pixel Rate Pixel Rate Pixel Rate Reading Rate
Data
Compression Metadata
Temporal
Redundancies
No
Compression
Spatial
Redundancies
Pixel Size to
![Page 45: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/45.jpg)
Image Sensor Pespectives
• Image sensor fabrication and test
Sampling scheme is signal-dependent
• How to directly process the image data flow?
Smart Image Sensor for Vision and Robotics
Laurent Fesquet 45
![Page 46: Low-Power Event-driven Image Sensor Architectures · 2016-12-25 · 1 Low-Power Event-driven Image Sensor Architectures Laurent Fesquet 1,2, Amani Darwish 1,2 Gilles Sicard 3 1 University](https://reader030.fdocuments.us/reader030/viewer/2022040907/5e7db8c7d2dedb599e6f818f/html5/thumbnails/46.jpg)
46
Non-uniform sampling is the future of digital universe!
Laurent Fesquet