CMOS BASED HYPERSPECTRAL IMAGING - SPIE relations/HSI imec for SPIE...CMOS BASED HYPERSPECTRAL...
Transcript of CMOS BASED HYPERSPECTRAL IMAGING - SPIE relations/HSI imec for SPIE...CMOS BASED HYPERSPECTRAL...
CMOS BASED HYPERSPECTRAL IMAGINGFOR COMPACT / LOW-COST / HIGH-VOLUME IMAGING SPECTROSCOPY
SPIE Baltimore - April 2016
public © imec 20162
WHY DO WE NEED HYPERSPECTRAL IMAGING?
1
2
n
to improve vision
and discrimination
power ...
... by using
spectral
dimension of
objects being
imaged
public © imec 20163
SPECTRAL IMAGING OPEN ONE NEW DIMENSION
POINT SPECTROSCOPY
xy
λ
Accurate spectral analysis
of one spatial pixel only
HYPERSPECTRAL IMAGING
spectral signature images revealing
objects chemical composition
1
n
Seeing RGB colors
of one image only
COLOR IMAGING
xy
R
G
B1
2
3
one wavelength dimension for each pixel ...
x
y
ʎ
public © imec 20165
CONVERGENCE OF IMAGING & SPECTROSCOPY
Hig
h d
iscr
imin
atio
n p
ow
er
Single point
with
highest
spectral
resolution
SPECTROSCOPY
High speed, high spatial resolutionIMAGING
RGB filters
on multiple pixelsIMAGING
SPECTROSCOPY
IMEC approach: spectral filters on multiple pixels
Tiled patternMosaic
pattern
OR OR Line pattern
public © imec 20166
SPECTRAL FILTERS APPLIED TO IMAGE SENSOR WAFERS BY
CMOS SEMICONDUCTOR PROCESS
IMEC hyperspectral filter structures processed at wafer-level on top of commercial CMOS
image sensor wafer (here on CMOSIS’s CMV2000 & CMV4000 sensors)
public © imec 20167
474
nm
465
nm
630
nm
624
nm
608
nm
485
nm
4x4 mosaic = 16 spectral bands
~ x16 cameras on-chip!
496
nm
510
nm
522
nm
534
nm
546
nm
586
nm
578
nm
562
nm
548
nm
600
nm
we design & manufacture spectral filters directly
onto image sensor pixels ...
... based on CMOS compatible
semiconductor process
public © imec 20168
NANO-DEPOSITED FILTERS PATTERNED @ PIXEL LEVEL!
down to 2um pixels!
1 to 100+ colors
public © imec 20169
• R&D to production environment
• 1B€ pilot-line factories on-site
• Class 1 ‘clean’ processing
• Advanced CMOS processes (22nn 7nm)
public © imec 201610
E.G. OF POSSIBLE APPROACH FABRY-PEROT SPECTRAL FILTERS
θ
θ
LWavelength selection depends on cavity length L
cos2nLk
Tx (%)
FWHM ~
5-20nm
Narrow-band & high transmission
efficiencies spectral filters
Different cavity heights = different
spectral wavelengths captured!
public © imec 201611
CUSTOM HYPERSPECTRAL IMAGING SOLUTIONS
Snapshot
Bayer Mosaic
Snapshot
TiledContinuous
Line-scan
Custom
Line-scan
Fill the sensor ONLY with
selected bands of interest
Optical filters & imager can
be customized to match final application
requirements
1 n2
Filt
er
resp
onse
FWHM
OD
Spectral filters can be integrated onto
commercial image sensor wafers
public © imec 201612
OFF-THE-SHELF SPECTRAL IMAGING SENSORS
RGB + NIR
R G B NIR
(in development)
4 bands
R G B
RGB
70+ commercial
vendors
450nm 1000nm
Multi-spectral
Snapshot mosaic & tiled
16 / 25 / 32 / 40+ bands
Linescan
100 / 150+ bands
450nm 1000nm
Hyperspectral
public © imec 201613
LINESCAN 100+BANDS HSI SENSOR DESIGN (GEN1)
Key specifications
– Spectral resolution: 100+ bands in 600-1000nm
– FWHM: ~ 10-15nm
– Spatial resolution: 2048 pixels x length of scan
– Speed: up to 340 fps (full sensor frame)
Linescan GEN1 sensor with 100+bands (from CMOSIS CMV2000, without micro-lenses)
public © imec 201614
LINESCAN 150+BANDS HSI SENSOR DESIGN (GEN 2)
Key specifications
– Spectral resolution: 150+ bands in 470-925nm
– FWHM: ~ 10-15nm
– Spatial resolution: 2048 pixels x length of scan
– Speed: up to 340 fps (full sensor frame)
Wavelength (nm)
QE (
%)
Linescan GEN2 sensor with 150+bands(from CMOSIS CMV2000, without micro-lenses)
public © imec 201615
SNAPSHOT TILED 32 BANDS HSI SENSOR DESIGN
Key specifications
– Spectral resolution: 32 bands in 600-940nm
– FWHM: ~ 10-15nm
– Spatial resolution: 256x256 RAW per band
– Speed: up to 340 fps (full sensor frame)
Snapshot tiled sensor with 32bands (from CMOSIS CMV2000 - without microlenses)
public © imec 201616
SNAPSHOT MOSAIC 4X4 HSI SENSOR DESIGN
Key specifications
– Spectral resolution: 16 bands in 460-630nm
– FWHM: ~ 10-15nm
– Spatial resolution: 512x256 RAW per band
– Speed: up to 340 fps (full sensor frame)
Snapshot mosaic sensor with 4x4 =16bands(from CMOSIS CMV2000 - without microlenses)
public © imec 201617
SNAPSHOT MOSAIC 5X5 HSI SENSOR DESIGNSnapshot mosaic sensor with 5x5 =25bands
(from CMOSIS CMV2000 - without microlenses)
0
2
4
6
8
10
12
14
600 650 700 750 800 850 900 950 1000
QE
(%
)
Wavelength (nm)
Quantum Efficiencyof hyperspectral 5x5 mosaic sensor = 25 bands sensor #XXX
band 1 band 2 band 3 band 4 band 5 band 6 band 7
band 8 band 9 band 10 band 11 band 12 band 13 band 14
band 15 band 16 band 17 band 18 band 19 band 20 band 21
band 22 band 23 band 24 band 25
Key specifications
– Spectral resolution: 25 bands in 600-960nm
– FWHM: ~ 10-15nm
– Spatial resolution: 409x216 RAW per band
– Speed: up to 340 fps (full sensor frame)
public © imec 201618
ROADMAP FOR OFF-THE-SHELF HSI SENSORS
2013 2014 2015 2016 2017+
‘RGB+NIR’
LINESCAN
‘Wedge’
GEN1 600-975nm
(100+bands)
NIR
GEN2 470-925nm
(150+bands)
VNIR
Snapshot
‘TILED’
GEN1 600-975nm (32bands)
NIR
Snapshot
‘MOSAIC’
VIS+NIR 450-875nm
(40+bands, dual-sensor)
VNIR
VIS 460-640nm
(4x4 =16bands)
VIS
NIR 600-975nm
(5x5 = 25bands)
NIR
GEN1 1-1.7um
(100+ bands)
SWIR
... to machine vision!
Bringing hyperspectral
imaging...
public © imec 201620
ECOSYSTEM PARTNERS FOR HSI SENSOR CAMERA INTEGRATION
Adimec
Tattile
PhotonFocus
Ximea
Bayspec
3D-One
VRMAGIC
Cubert
Flexible camera platform Giga/Ethernet interface
Focus on remote sensing, UAV
agriculture & machine vision
High speed camera Cameralink or CoaXPress interface
Compact optical design with ZIFF socket to
swap different HSI sensors
industrial, medical and global security
Smart camera Giga/Ethernet interface
Reprogrammable FPGA pre-processing
hypercube data inside camera
Focus on industrial machine vision
Handheld mobile platform First proof-of-concept of
IMEC’s HSI sensors integration
to mobile
Android OS open platform
USB2.0 interfaceUSB 3.0 vision camera High-speed USB3.0 camera
Ultra compact camera: 26.4 x 26.4 x 21.2 mm
27grams weight only!
Embedded vision module Giga/Ethernet interface
Motorized optics and Multi-camera head
option
Embedded HSI data processing include
storage
Smart camera GigeVision and Genicam compliant
A/B shaft encoder interface (RS-422 or HTL levels).
FPGA pre-processing
Agriculture, food sorting and industrial applications
public © imec 201621
HSI DEMO-KITS FOR EVALUATION
compact | pc-powered | ready-to-go!
public © imec 201623
HSI SOFTWARE TO SUPPORT APPLICATION DEVELOPMENT
License available to all
demo-kit customers & partners
Support including installation,
training and Q&A (remotely
and on-site)
Source-code available for
OEM integration for the
following blocks:
– Cube registration
– Reflectance calculation
– Spectral corrections
public © imec 201624
LINESCAN 100+BANDS HSI SENSOR (GEN1) CUBE EXAMPLE
HSI data-cube (after scan)
1024 pixels
2048 p
ixels
x 100+ spectral
band images!
Classified image(extracting leafs only)Sample spectra
(x3 pixels extracted from leafs)
public © imec 201625
DRIVING APPLICATIONS FOR IMEC HSI
Imaging spectroscopy analyzers
DNA sequencers / flow cytometers
Water monitoring analyzers
Blood / urine analyzers
Life-science instrumentation
Food sorting / quality grading
Pharmaceutical defect inspection
Industrial process control
Bank-note inspection
Print quality inspection
Machine vision / Optical sorting
Surgery-guided imaging
Endoscopy
Ophthalmology / retina imaging
Wounds imaging
Fluorescence microscopy
Medical imaging
Night vision systems
Fuel monitoring systems
Driver drowsiness
Automotive & Transport
Industrial gas leaks monitoring
Intrusion detection / authentication
Forensics
Anti-counterfeiting
Security / Surveillance
Remote sensing
UAVs & cube-sattelites for:
Precision agriculture
Environment monitoring
Terrestrial / maritime earth observation
... integration path into UAVs
for precision agriculture
Soil analysis
Nutrient deficiency diagnostics
Grows monitoring
Disease and weeds alerts
Yield prediction
public © imec 201629
LINESCAN SENSOR IN PRECISION AGRICULTURE
862 nm 2.5cm ground
resolution!
NDVI (vegetation index)
management
zone map
vegetation
index mapspectral imageRAW image
public © imec 201630
snapshot hyperspectral
imaging in medical
public © imec 201631
REAL-TIME OXYGENATION MAPS
View live video here https://vimeo.com/132097972
Reconstructed Hb/HbO/HBT maps extracted from HSI snapshot mosaic
4x4 = 16bands HSI sensor on epileptic patient during brain surgery
Blood Spectral SignatureSpectrometer As seem by 4x4 HSI camera
public © imec 201632
Courtesy of Perception
Park Studio
ATM / BANK NOTE INSPECTION
Hyperspectral (5x5 mosaic NIR)
RGB___________
public © imec 201633
LINESCAN HSI SENSOR FOR RICE INSPECTION
600 650 700 750 800 850 900 950 10000
50
100
150
200
250
300
350
400
450
500
wavelength (nm)
black_sample
dark_sample
background_dish
good_wholegrain
Training Set
of Hyperspectral signatures
1800
2000
2200
2400
2600
2800
3000 600
800
1000
1200
1400
1600
black spot dark
background
good
Principal Component Analysis
background_dish
black_sample
dark_sample
good_wholegrain
Object–level classification
+ standard vision software processing
dark
good
black
black
dark
good
samples
Classification Training Set
public © imec 201634
HIGH SPEED ANOMALIES DETECTION WITH
25+ BANDS SNAPSHOT MOSAIC SENSOR
View live video here https://vimeo.com/132098267
real leaf
plastic
leaf
public © imec 201635
FOOD FRESHNESS MONITORING FOR FOOD
QUALITY GRADING APPLICATIONS
100% inspection via gill’s blood spectrum
Determine food quality freshness within few hours till 4 days
public © imec 201636
SAME TECHNO INDUSTRIAL TO CONSUMER?
Life-science instrumentation
Machine vision /
Optical sorting
Medical imaging
Automotive & Transport
Security / Surveillance
Remote sensing Consumer
Consumer
ConsumerConsumer
Consumer
Consumerconnected health
public © imec 201637
HSI SOFTWARE IMAGE PROCESSING PIPELINE
Camera/Sensor
SpecificApplication Specific
Pre-Processing
Camera calibration,
registration, synchronization,
reflectance calculation...etc
Feature
Extraction &
Selection
Feature
Classification/
Recognition
Spectral un-mixing,
smoothening, PCA, SVM, k-
means clustering, ...etc
Post-Processing
Visualization, GUI... etc
HSI
‘cube’
Application Specific (to be developed
for each platform)
IMEC can support these
SOFTWARE / Application layers
public © imec 201638
IMEC COLLABORATION MODEL FOR DEVELOPING HSI
SOLUTIONS, APPLICATIONS & MARKETS
Large
data-sets
&
services
Spectral
filter
materials
& Fab
operations
Sensor
&
Filter
co-
Design
Low
level
image
proc.
Firmware
HSI
image
proc.
SW
Full
system
feasibility
study
Appli.
SW
Camera
&
System
Design