Comp 790-058 Lecture 06: Overview of Autonomous...
Transcript of Comp 790-058 Lecture 06: Overview of Autonomous...
![Page 1: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/1.jpg)
Comp 790-058 Lecture 06:
Overview of Autonomous Driving
Sept 26, 2017
Sahil Narang
University of North Carolina, Chapel Hill
1
![Page 2: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/2.jpg)
University of North Carolina at Chapel Hill
Autonomous vehicle: a motor vehicle that uses artificial intelligence,
sensors and global positioning system coordinates to drive itself without the
active intervention of a human operator
Focus of enormous investment [$1b+ in 2015]
2
Tesla
Waymo
Nutonomy
Autonomous Driving
![Page 3: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/3.jpg)
Autonomous Driving: Motivation
Cars are ubiquitous
~ 1 bn vehicles for a global population of ~7 bn [est. 2010]
Car accidents can result in catastrophic costs
300 bn USD in car crashes in 2009
160 bn USD congestion related costs in 2014
Health costs
33k fatalities, 2 million+ injuries in 5.4 million crashes in 2010
Premature deaths due to pollution inhalation
3
![Page 4: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/4.jpg)
Autonomous Driving: Levels of Autonomy
0: Standard Car
1: Assist in some part of driving
Cruise control
2: Perform some part of driving
Adaptive CC + lane keeping
3: Self-driving under ideal conditions
Human must remain fully aware
4: Self-driving under near-ideal conditions
Human need not remain constantly aware
5: Outperforms human in all circumstances
4
![Page 5: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/5.jpg)
Structure
History of Autonomous Driving
Main Components
Other Approaches
Other Issues
5
![Page 6: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/6.jpg)
Structure
History of Autonomous Driving
Through the years (1958-2007)
Current State of the Art
Main Components
Other Approaches
Other Issues
6
![Page 7: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/7.jpg)
Autonomous Driving: Levels of Autonomy
https://www.youtube.com/watch?v=E8xg5I7hAx4
7
![Page 8: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/8.jpg)
Autonomous Driving: Through the years
Magic Highway (1958)
https://www.youtube.com/watch?v=L3funFSRAbU
8
![Page 9: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/9.jpg)
Autonomous Driving: Through the years
CMU NavLab (1986)
https://www.youtube.com/watch?v=ntIczNQKfjQ
9
![Page 10: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/10.jpg)
Autonomous Driving: Through the years
DARPA Grand Challenge 2004
https://www.youtube.com/watch?v=wTDG5gjwPGo
10
![Page 11: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/11.jpg)
Autonomous Driving: Through the years
DARPA Grand Challenge 2005
https://www.youtube.com/watch?v=7a6GrKqOxeU
11
![Page 12: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/12.jpg)
Autonomous Driving: Through the years
DARPA Grand Challenge 2007
Focus on urban driving
https://www.youtube.com/watch?v=8NIx7Y4EgQg
12
![Page 13: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/13.jpg)
Autonomous Driving
Urban driving is particularly challenging
13
![Page 14: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/14.jpg)
Structure
History of Autonomous Driving
Through the years (1958-2007)
Current State of the Art
Main Components
Other Approaches
Other Issues
14
![Page 15: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/15.jpg)
Autonomous Driving: State of the Art Today
Automated road shuttles
Vehicles operate in segregated spaces
Simple car-following strategies
https://www.youtube.com/watch?v=Byk8LcPovOQ
15
![Page 16: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/16.jpg)
Autonomous Driving: State of the Art Today
Google’s Waymo
https://www.youtube.com/watch?v=TsaES--OTzM
16
![Page 17: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/17.jpg)
Structure
History of Autonomous Driving
Main Components
Perception
Planning
Control
Other Approaches
Other Issues
17
![Page 18: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/18.jpg)
Autonomous Driving: Main Components
18
![Page 19: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/19.jpg)
Autonomous Driving: Main Components
19
Perception
collect information and extract relevant knowledge from the
environment.
![Page 20: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/20.jpg)
Autonomous Driving: Main Components
20
Planning
Making purposeful decisions in order to achieve the robot’s higher order
goals
![Page 21: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/21.jpg)
Autonomous Driving: Main Components
21
Control
Executing planned actions
![Page 22: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/22.jpg)
Structure
History of Autonomous Driving
Main Components
Perception
Planning
Control
Other Approaches
Other Issues
22
![Page 23: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/23.jpg)
Autonomous Driving: Perception
Sensing Challenges
Sensor Uncertainty
Sensor Configuration
Weather / Environment
23
![Page 24: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/24.jpg)
Autonomous Driving: Challenges in Perception
Sensor Misclassification
“When is a cyclist not a cyclist?”
When is a sign a stop sign?
Whether a semi or a cloud?
24
![Page 25: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/25.jpg)
Autonomous Driving: Perception
Environmental Perception
LIDAR
Cameras
Fusion
Other approaches
RADAR, Ultrasonic sensors
25
![Page 26: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/26.jpg)
Autonomous Driving: Perception
Environmental Perception
LIDAR
Cameras
Fusion
Other approaches
RADAR, Ultrasonic sensors
26
![Page 27: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/27.jpg)
Autonomous Driving: Perception using LIDAR
Light Detection and Ranging
Illuminate target using pulsed laser lights, and measure reflected pulses
using a sensor
27
![Page 28: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/28.jpg)
Autonomous Driving: Perception using LIDAR
LIDAR Challenges
Scanning sparsity
Missing points
Unorganized patterns
Knowledge gathering can be
difficult
28
![Page 29: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/29.jpg)
Autonomous Driving: Perception using LIDAR
Data Representation
Point clouds
Features: lines, surfaces etc
Grid based approaches
29
![Page 30: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/30.jpg)
Autonomous Driving: Perception using LIDAR
Knowledge Extraction
3D point cloud segmentation
Classification
30
![Page 31: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/31.jpg)
Autonomous Driving: Perception using LIDAR
Knowledge Extraction
3D point cloud segmentation
Edge based
Region based
Model based
Attribute based
Graph based
Classification
31
![Page 32: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/32.jpg)
Autonomous Driving: Perception using LIDAR
Knowledge Extraction
3D point cloud segmentation
Edge based
Region based
Model based
Attribute based
Graph based
Classification
32
![Page 33: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/33.jpg)
Autonomous Driving: Perception using LIDAR
Knowledge Extraction
3D point cloud segmentation
Edge based
Region based
Model based
Attribute based
Graph based
Classification
33
![Page 34: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/34.jpg)
Autonomous Driving: Perception using LIDAR
Knowledge Extraction
3D point cloud segmentation
Edge based
Region based
Model based
Attribute based
Graph based
Classification
34
![Page 35: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/35.jpg)
Autonomous Driving: Perception using LIDAR
Knowledge Extraction
3D point cloud segmentation
Edge based
Region based
Model based
Attribute based
Graph based
Classification
35
![Page 36: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/36.jpg)
Autonomous Driving: Perception using LIDAR
Knowledge Extraction
3D point cloud segmentation
Edge based
Region based
Model based
Attribute based
Graph based
Classification
36
![Page 37: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/37.jpg)
Autonomous Driving: Perception using LIDAR
Knowledge Extraction
3D point cloud segmentation
Classification
Few methods use point clouds directly
High memory and computational costs
Less robust
37
![Page 38: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/38.jpg)
Autonomous Driving: Perception using LIDAR
Knowledge Extraction
3D point cloud segmentation
Classification
Multi-class labelling using SVM
VoxNet: 3D CNN
38
![Page 39: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/39.jpg)
Autonomous Driving: Perception using LIDAR
LIDAR in practice
Velodyne 64HD lidar
https://www.youtube.com/watch?v=nXlqv_k4P8Q
39
![Page 40: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/40.jpg)
Autonomous Driving: Perception
Environmental Perception
LIDAR
Cameras
Fusion
Other approaches
RADAR, Ultrasonic sensors
40
![Page 41: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/41.jpg)
Autonomous Driving: Perception using Cameras
Camera based vision
Road detection
Lane marking detection
Road surface detection
On-road object detection
41
![Page 42: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/42.jpg)
Autonomous Driving: Perception using Cameras
Camera based vision
Road detection
Lane marking detection
Road surface detection
On-road object detection
42
![Page 43: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/43.jpg)
Autonomous Driving: Perception using Cameras
Challenges in Lane Detection
Road conditions
Singularities
Worn-out markings
Directional arrows
Warning text
Zebra crossing
Environment conditions
Shadows from cars and trees
Weather effects
43
![Page 44: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/44.jpg)
Autonomous Driving: Perception using Cameras
44
Challenges in Lane Detection
![Page 45: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/45.jpg)
Autonomous Driving: Perception using Cameras
General approach to lane detection
Lane line feature extraction
Model fitting
Vehicle pose estimation
45
![Page 46: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/46.jpg)
Autonomous Driving: Perception using Cameras
General approach to lane detection
Lane line feature extraction
Gradient based methods
Pattern finding
Model fitting
Vehicle pose estimation
46
![Page 47: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/47.jpg)
Autonomous Driving: Perception using Cameras
General approach to lane detection
Lane line feature extraction
Gradient based methods
Pattern finding
Model fitting
Vehicle pose estimation
47
![Page 48: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/48.jpg)
Autonomous Driving: Perception using Cameras
General approach to lane detection
Lane line feature extraction
Gradient based methods
Pattern finding
Model fitting
Vehicle pose estimation
48
![Page 49: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/49.jpg)
Autonomous Driving: Perception using Cameras
General approach to lane detection
Lane line feature extraction
Model fitting
Vehicle pose estimation
49
![Page 50: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/50.jpg)
Autonomous Driving: Perception using Cameras
General approach to lane detection
Lane line feature extraction
Model fitting
Parametric
Semi-parametric
Particle Filters
Vehicle pose estimation
50
![Page 51: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/51.jpg)
Autonomous Driving: Perception using Cameras
General approach to lane detection
Lane line feature extraction
Model fitting
Parametric
Semi-parametric
Particle Filters
Vehicle pose estimation
51
![Page 52: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/52.jpg)
Autonomous Driving: Perception using Cameras
General approach to lane detection
Lane line feature extraction
Model fitting
Vehicle pose estimation
52
![Page 53: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/53.jpg)
Autonomous Driving: Perception using Cameras
Camera based vision
Road detection
Lane marking detection
Road surface detection
On-road object detection
53
![Page 54: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/54.jpg)
Autonomous Driving: Perception using Cameras
Approaches to Road surface detection
Feature-based
Deep learning
54
![Page 55: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/55.jpg)
Autonomous Driving: Perception using Cameras
Approaches to road surface detection
Feature-based
Feature extraction
Segmentation
Classification
May not be robust
Deep learning
55
![Page 56: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/56.jpg)
Autonomous Driving: Perception using Cameras
Approaches to road surface detection
Feature-based
Deep learning
Direct pixel/block labelling
High memory and computation
requirements
Annotated data
Black box
56
![Page 57: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/57.jpg)
Autonomous Driving: Perception using Cameras
Camera based vision
Road detection
Lane marking detection
Road surface detection
On-road object detection
57
![Page 58: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/58.jpg)
Autonomous Driving: Perception using Cameras
On-road object detection
Pedestrian, cyclists, other cars
Challenging due to the various types,
appearances, shapes, and sizes of the
objects
58
![Page 59: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/59.jpg)
Autonomous Driving: Perception using Cameras
On-road object detection
Pedestrian, cyclists, other cars
Challenging due to the various types,
appearances, shapes, and sizes of the
objects
Deep learning methods are far superior
59
![Page 60: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/60.jpg)
Autonomous Driving: Perception using Cameras
Mobileye
Country road: https://www.youtube.com/watch?v=ywvJqKVcnDA
Highway: https://www.youtube.com/watch?v=_ZH5Taq6mvw
Rain: https://www.youtube.com/watch?v=39QMYkx89j0
Pedestrians: https://www.youtube.com/watch?v=H_wMyUEeIzQ
60
![Page 61: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/61.jpg)
Autonomous Driving: Perception using Sensor Fusion
LIDAR
3D measurements
Impervious to illumination changes
Prone to noise
Hard to extract knowledge
Cameras
Provide rich appearance details in 2D
Affected by illumination/ weather
61
![Page 62: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/62.jpg)
Autonomous Driving: Vehicle Localization
Determining the pose of the ego vehicle and measuring its own motion
Fusing data
Satellite-based navigation system
Inertial navigation system
Map aided localization
SLAM
62
![Page 63: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/63.jpg)
Structure
History of Autonomous Driving
Main Components
Perception
Planning
Control
Other Approaches
Other Issues
63
![Page 64: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/64.jpg)
Autonomous Driving: Main Components
64
Planning
Making purposeful decisions in order to achieve the robot’s higher order
goals
![Page 65: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/65.jpg)
Autonomous Driving: Planning
Compare to Pedestrian Techniques:
Route Planning: road selection (global)
Path Planning: preferred lanes (global)
Maneuver-search: high level maneovers
(local)
Trajectory planning: Lowest level of
planning (local)
65
![Page 66: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/66.jpg)
Autonomous Driving: Route Planning
Determine the appropriate macro-level route to take
Typically road level i.e. which roads to take
Katrakazas: “Route planning is concerned with finding the best global route from a
given origin to a destination, supplemented occasionally with real-time traffic
information”
66
![Page 67: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/67.jpg)
Autonomous Driving: Path Planning
Determine the appropriate geometric waypoints to follow when driving
Katrakazas: “a path is a geometric trace that the vehicle should follow in order to
reach its destination without colliding with obstacles. ”
67
![Page 68: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/68.jpg)
Autonomous Driving: Maneuver Planning
Determine the appropriate ‘attitude’ or posture of the vehicle. Decides which behavior
the vehicle is performing at any time
Katrakazas: “high-level characterization of the motion of the vehicle, regarding the
position and speed of the vehicle on the road. Examples of maneuvers include ‘going
straight’, ‘turning’, ‘overtaking’ etc”
68
![Page 69: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/69.jpg)
Autonomous Driving: Trajectory Planning
Katrakazas: “Trajectory planning(also known as trajectory generation) is concerned
with the real-time planning of the actual vehicle’s transition from one feasible state to
the next, satisfying the vehicle’s kinematic limits based on vehicle dynamics and
constrained by the navigation comfort, lane boundaries and traffic rules, while
avoiding, at the same time, obstacles including other road users as well as ground
roughness and ditches”
69
![Page 70: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/70.jpg)
Autonomous Driving: Planning
70
![Page 71: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/71.jpg)
Autonomous Driving: Route Planning
Extensive literature in traffic simulation and civil engineering domain
In general, graph search (i.e. A-Star, Dijkstra)
Includes self-aware routing
Travel-network optimization
Dynamic Traffic Assignment
See: “Self-Aware Traffic Route Planning”, David Wilkie, Jur Van den
Berg, Ming Lin, Dinesh Manocha
71
![Page 72: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/72.jpg)
Autonomous Driving: Path Planning
RRT and Lattice methods are most-well represented
72
![Page 73: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/73.jpg)
Autonomous Driving: Path Planning
73
![Page 74: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/74.jpg)
Autonomous Driving: Maneuver Planning
Decision making far more complex due to interactions with other traffic
participants
Required to anticipate behavior of other participants
Existing techniques
POMDPs
MCDM – Multi-Criteria Decision Making
DFA – Deterministic Finite Automata
Game theory approach (perfect information game, ends in crash)
Driving corridors
74
![Page 75: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/75.jpg)
Autonomous Driving: Trajectory Planning
Compute a trajectory, according to the chosen path and maneurver, which
satisfies:
Kinematic constraints
Dynamic constraints
Collision-free constraints
Passenger comfort constraints
Common approaches
Driving corridor optimization
Sampling-based planning
75
![Page 76: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/76.jpg)
Structure
History of Autonomous Driving
Main Components
Perception
Planning
Control
Other Approaches
Other Issues
76
![Page 77: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/77.jpg)
Autonomous Driving: Control Planning
Convert plans into actions
Provide inputs to the hardware level to generate the desired motion
Common Approaches
Proportional-Integral-Derivative (PID) controller
Model Predictive Control (MPC)
77
![Page 78: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/78.jpg)
Structure
History of Autonomous Driving
Main Components
Other Approaches
Other Issues
78
![Page 79: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/79.jpg)
Autonomous Driving: End-End Approaches
Nvidia PilotNet
Deep learning to directly map video frames to control
79
![Page 80: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/80.jpg)
Structure
History of Autonomous Driving
Main Components
Other Approaches
Other Issues
80
![Page 81: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/81.jpg)
Autonomous Driving: Other Issues
Other challenges:
Communication
Coordination
Ethical Issues
Trolley Problem
81
![Page 82: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/82.jpg)
Autonomous Driving: Other Issues
Other challenges:
MIT “Moral Machine” [https://goo.gl/RL4pr5]
82
![Page 83: Comp 790-058 Lecture 06: Overview of Autonomous Drivinggamma.cs.unc.edu/courses/planning-f17/SLIDES/lec06_autonomous… · Sept 26, 2017 Sahil Narang University of North Carolina,](https://reader035.fdocuments.us/reader035/viewer/2022071111/5fe7433233c2f2578d27590e/html5/thumbnails/83.jpg)
Autonomous Driving: Other Issues
Civil Engineering / Ethics
Traffic impacts?
Pro: Vehicles should respond appropriately to traffic reducing jams
Con: Many more vehicles per person possible
People may not own cars?
Pro: Less emission? Less Traffic?
Con: Less access?
83