Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver...
Transcript of Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver...
![Page 1: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/1.jpg)
Fast Reactive Illumination through Rain and Snow
Sponsors: ONR, NSF, Intel, Samsung
Carnegie Mellon University
Raoul de Charette Robert Tamburo
Peter Barnum
Anthony Rowe Takeo Kanade
Srinivasa Narasimhan
![Page 2: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/2.jpg)
Driving in Snow at Night
![Page 3: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/3.jpg)
Berlin, Germany 55
Brussels, Belgium 65
Cape Town, SA 52
Chicago, USA 48
Kuala Lumpur, Malaysia 79
Paris, France 56
Pittsburgh, USA 59
Seattle, USA 62
Tokyo, Japan 50
Zurich, Switzerland 64
[ World Meteorological Organization, 30 year average ]
How many Rainy/Snowy Nights in a Year?
![Page 4: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/4.jpg)
Post-processing: De-raining and De-snowing
=
Rain frequencies
Scene frequencies
[Barnum et al, 07]
![Page 5: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/5.jpg)
Headlight that sees through Rain and Snow
Goal: High Light Throughput and Accuracy
[Similar in spirit to “Lighting up dust”, Ken Perlin]
![Page 6: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/6.jpg)
… a detour
[ Barnum, Narasimhan, Kanade, 10 ]
![Page 7: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/7.jpg)
Rain Streaks versus Rain Drops
Long Exposure Time (12 ms) Short Exposure Time (1 ms)
Rain streaks appear dense but drops are sparse
![Page 8: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/8.jpg)
Light Throughput and Camera Exposure
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 80
85
90
95
100
Exposure time (ms)
Lig
ht
thro
ug
hp
ut
(%)
0.5mm/h
2mm/h
17mm/h
![Page 9: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/9.jpg)
Operating Range: Visibility of Rain Streaks
Distance to the light source (m)
Lig
ht
inte
nsi
ty (
ftc)
0
500
1000
1500
2000
0 1 2 3 4 5 6
Headlight
DLP Projector
Halogen Light
Not visible beyond 5-6 meters from the source
![Page 10: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/10.jpg)
Reactive Illumination
operating range
camera
projector
beam splitter
capture
system latency
reactive control
![Page 11: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/11.jpg)
Capture
Frame #
Frame 1
Frame 2
Frame 3
Frame 4
TX TX Projection Process
Time (ms) 5 9 14 18 27 36
5 4 5 4 9
Projection
System Pipeline
Stage 3 Stage 2 Stage 1
Capture TX TX Process
Projection Capture TX TX Process
Capture TX TX Process
Latency
![Page 12: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/12.jpg)
![Page 13: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/13.jpg)
Prototype System in Lab
![Page 14: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/14.jpg)
Detecting, Tracking and Predicting Drops
View from the system camera
![Page 15: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/15.jpg)
Performance with Tracking Latency
0 5 10 15 20 25 0
20
40
60
80
100
Tracking latency (frames)
Acc
ura
cy (
%)
1000 Hz
500 Hz
200 Hz
100 Hz
50 Hz
![Page 16: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/16.jpg)
Performance under Detection Errors
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 90
92
94
96
98
100
Detection error (pixel)
Lig
ht
thro
ug
hp
ut
(%)
Rain (5mm/h)
Snow (1mm/h)
![Page 17: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/17.jpg)
Slow Capture and Projection
Standard headlight Smart headlight Not so
Projecting black streaks instead of white streaks
![Page 18: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/18.jpg)
High Speed Bit-Plane Projection
[Raskar et al, Narasimhan et al, Debevec et al]
What we see….or don’t see (30 Hz)
What a high-speed camera sees (2000 Hz)
[ Temporal integration ]
![Page 19: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/19.jpg)
Making Rain Disappear (90 mm/h)
System Latency
Standard headlight Smart headlight
![Page 20: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/20.jpg)
Making Rain Disappear (90 mm/h)
Standard headlight Smart headlight
![Page 21: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/21.jpg)
Performance of our system
System Latency: 13 ms Imaging exposure: 5ms Tracking latency: 1 frames
Resolution: 1024 x 1024 Car motion: 30 kph Projection rate: 120 Hz
0 5 10 15 20 25 40
50
60
70
80
90
100
Fallrate (mm/h)
Lig
ht
thro
ug
hp
ut
(%)
Snow (accuracy = 63.4%)
Rain (accuracy = 68.9%)
![Page 22: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/22.jpg)
Performance of an Ideal system
System Latency: 0 ms Imaging exposure: 1ms Tracking latency: 0 frames
Resolution: 1024 x 1024 Car motion: 30 kph Projection rate: 10000 Hz
0 5 10 15 20 25 40
50
60
70
80
90
100
Fallrate (mm/h)
Lig
ht
thro
ug
hp
ut
(%)
Rain (accuracy = 100%)
Snow (accuracy = 100%)
![Page 23: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/23.jpg)
+
-
-
-
-
High-speed computations
Projector and camera in one device?
Embedded Design for Imaging and Illumination
![Page 24: Fast Reactive Illumination through Rain and Snow€¦ · Rain Rain Go Away… • Improves driver visibility directly • Simulations suggest that the idea is feasible • Initial](https://reader033.fdocuments.us/reader033/viewer/2022042812/5fabe7ed4a9ad171f10064b8/html5/thumbnails/24.jpg)
Rain Rain Go Away…
• Improves driver visibility directly • Simulations suggest that the idea is feasible
• Initial lab prototype is encouraging
• Still a long way to go:
• Wind, turbulence, vibrations
• Making the device compact, fast moving car