PERCEPTION FOR AUTOMATED AND ASSISTED...
Transcript of PERCEPTION FOR AUTOMATED AND ASSISTED...
PERCEPTION FOR AUTOMATED AND ASSISTED DRIVING
DR.-ING. MICHAEL DARMS GROUP RESEARCH / HEAD OF SENSORS AND FUSION IROS – 28.09.2015
2 Volkswagen AG | Konzernforschung
ENVIRONMENT PERCEPTION
Motivation
• Highly automated driving functions impose increased requirements on the performance of the perception module
• Perception module has to cope with contradictory requirements (comport vs. safety systems)
• Development of a modular and extensible perception architecture
• Environment model has to be independent of specific ADAS function
• Implementation of unified interface to function
Goals
3 Volkswagen AG | Konzernforschung
Representation of the world around the vehicle
Static Environment
Road / Lanes
Dynamic Objects
Road Graph
WORLD MODEL
4 Volkswagen AG | Konzernforschung
OBJECT ESTIMATION
Mono Camera Laser Radar
• Estimating of moving/ movable objects and dynamic model
• Benefit from individual sensor detection capabilities
5 Volkswagen AG | Konzernforschung
GRID ESTIMATION
Ultrasonic Front Camera
2D Layer Fusion
• Estimate Free Space
• Estimate Occupied Space
• Exclude Movable/ Moving Objects
6 Volkswagen AG | Konzernforschung
ROAD ESTIMATION
Highway Lane Markings Other Vehicles
Rural Roads Lane Markings Other Vehicles Curbs, sward
Inner City Lane Markings Other Vehicles Curbs Traffic Lights, Trafic Signs Digital Maps
• Interpret where in the world vehicles should drive using feature cues from the environment
7 Volkswagen AG | Konzernforschung
SCENE ESTIMATION
RoadGraph • Output of the perception modules is
integrated into one model • Roadgraph is main interface to
driving functions Challenges • Fusion of road estimation and
context knowledge into unified, consistent and comprehensive model
• Scene Estimation for urban areas, especially in complex intersection scenarios
• Handling of traffic participants that do not behave as expected
8 Volkswagen AG | Konzernforschung
V-CHARGE: AUTONOMOUS VALET PARKING AND CHARGING FOR E-MOBILITY
Automated valet parking and charging • no time-consuming search for parking spots any more • driverless valet service • no human intervention Fully automated driving • in mixed-traffic scenarios • in indoor and outdoor parking lots and parking garages
without GPS
9 Volkswagen AG | Konzernforschung
V-CHARGE: AUTONOMOUS VALET PARKING AND CHARGING FOR E-MOBILITY Close-to-series sensors
Stereo- Camera
Area- view
Ultra- sonic
Car2x
10 Volkswagen AG | Konzernforschung
OBJECT ESTIMATION USING FISH-EYE CAMERAS
University of Parma, VisLab
11 Volkswagen AG | Konzernforschung
GRID ESTIMATION USING FISH-EYE CAMERAS
C. Häne, T. Sattler, M. Pollefeys, Obstacle Detection for Self-Driving Cars Using Only Monocular Cameras and Wheel Odometry
12 Volkswagen AG | Konzernforschung
GRID ESTIMATION WITH ALL SENSORS
06/25/2015 22
Ultrasonic Grid Stereo Grid
Motion Stereo Grid Fused Grid
13 Volkswagen AG | Konzernforschung
GRID ESTIMATION WITH ALL SENSORS
06/25/2015 22
Ultrasonic Grid Stereo Grid
Motion Stereo Grid Fused Grid
14 Volkswagen AG | Konzernforschung
ROAD ESTIMATION: KNOWING WHERE THE ROAD IS VIA LOCALIZATION
14
Setup
15 Volkswagen AG | Konzernforschung
APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION
15
Survey Run
Furgale et al., ICRA2014
16 Volkswagen AG | Konzernforschung
APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION
16
Survey Run
Online Run
Survey Run
Furgale et al., ICRA2014
17 Volkswagen AG | Konzernforschung 17
Furgale et al., ICRA2014
18 Volkswagen AG | Konzernforschung
APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION
18
Furgale et al., ICRA2014
19 Volkswagen AG | Konzernforschung
APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION
19
Furgale et al., ICRA2014
20 Volkswagen AG | Konzernforschung
APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION
20
Furgale et al., ICRA2014
21 Volkswagen AG | Konzernforschung
October 13, 2013 15:15
22 Volkswagen AG | Konzernforschung
November 8, 2013 10:26
23 Volkswagen AG | Konzernforschung
December 6, 2013 13:50
24 Volkswagen AG | Konzernforschung
July 31, 2014 15:45
25 Volkswagen AG | Konzernforschung
APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION
25
Summary
Mapping / Fusion Localization
Database
Summary map for localization
Localization output & raw data
Full-scale map
Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis
26 Volkswagen AG | Konzernforschung
APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION
26
Find the most useful Landmarks:
Most observed landmarks from a single session:
Landmarks observed over the most different sessions:
Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis
27 Volkswagen AG | Konzernforschung
APPROACH: INTERPRETING WHERE THE ROAD IS – ROAD ESTIMATION
Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis Mühlfellner et al.: Summary Maps for Lifelong Visual Localization, JFR, 2015
19.0
9.
04.1
0.
11.1
0.
17.1
0.
23.1
0.
31.1
0.
07.1
1.
08.1
1.
12.1
1.
19.1
1.
03.1
2.
06.1
2.
10.1
2.
12.1
2.
14.0
1.
04.0
4.
14.0
4.
25.0
5.
28.0
4.
30.0
4.
05.0
5.
14.0
5.
21.0
5.
26.0
5.
28.0
5.
11.0
6.
30.0
6.
02.0
7.
16.0
7.
23.0
7.
30.0
7.
31.0
7.
100%
75%
50%
25%
0%
1 2 3 4 5 8 9 6 7 10
11
12
Full Multi-Session Map Summary Map
Date, Arrows Indicate Mapping Sessions
% S
ucce
ssfu
l Loc
aliz
atio
ns
Success Rate
28 Volkswagen AG | Konzernforschung
APPROACH: INTERPRETING WHERE THE ROAD IS – ROAD ESTIMATION
Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis Mühlfellner et al.: Summary Maps for Lifelong Visual Localization, JFR, 2015
19.0
9.
04.1
0.
11.1
0.
17.1
0.
23.1
0.
31.1
0.
07.1
1.
08.1
1.
12.1
1.
19.1
1.
03.1
2.
06.1
2.
10.1
2.
12.1
2.
14.0
1.
04.0
4.
14.0
4.
25.0
5.
28.0
4.
30.0
4.
05.0
5.
14.0
5.
21.0
5.
26.0
5.
28.0
5.
11.0
6.
30.0
6.
02.0
7.
16.0
7.
23.0
7.
30.0
7.
31.0
7.
1e7
1e6
1e5
1 2 3 4 5 8 9 6 7 10
11
12
Full Multi-Session Map Summary Map
Date, Arrows Indicate Mapping Sessions
Land
mar
ks T
otal
(Log
. Sca
le)
Number of Landmarks per Session
29 Volkswagen AG | Konzernforschung
CES DEMO DRIVE 2015
− About 900 km on Highways − 5 Journalists as drivers using the automated vehicle
Bakersfield / 5.1.2015 / 7:00
San Francisco / 4.1.2015 / 11:00
Las Vegas / 5.1.2015 / 16:00
CANV / 5.1.2015 / 12:00 Interstate I5 / 4.1.2015 / 15:00
30 Volkswagen AG | Konzernforschung
1 4
5
2x Laser-scanner
4x Ultrasonic 1x 3D-video-camera
4x Top View 4x Mid-range-radar
2x Long-range-radar
1x Stock GPS 2x Short Range Radar
2
2
3
3
4
2
1 2 3 4 5 6 7 8
6
7
8
1
TECHNOLOGY – TODAY
31 Volkswagen AG | Konzernforschung
KEY CHALLENGE: WHERE IS THE ROAD?
Simple Marking features, simple lane geometries
Medium Marking features, more complex geometries (smaller radius of curvature, not parallel geometries)
Complex Arbitrary features, arbitrary lane geometries
Robust and highly available marking features low demands
Marking features with more complex geometries medium-level demands
No clear road markings High demands on the scene interpretation
31
Töpfer, Spehr et al: Efficient Scene Understanding for Intelligent Vehicles Using a Part-Based Road Representation
32 Volkswagen AG | Konzernforschung
INPUT AND OUTPUT OF THE ROAD ESTIMATION • Spatial and temporal reasoning
Features
Patches
Dig. map
Result
Lanes
Road Estimation
Spat
ial a
nd te
mpo
ral
reas
onin
g
Standards
A-priori
Sensors
Road model
Credits Spehr et al
33 Volkswagen AG | Konzernforschung
INTEGRATION OF A-PRIORI KNOWLEDGE
• Standards: • Guidelines for building roads, freeways and street • Use of country-dependent policies (e.g. in Germany „Straßenbaunormen DIN
EN 1423 und DIN EN 1424“)
• Deviations are modeled with appropriate probability distributions
• Digital maps:
• Information beyond the detection range of the sensors • Used for integrating the expected road geometry and topology during
spatial reasoning.
34 Volkswagen AG | Konzernforschung
SENSOR SETUPS
• High-level sensor information (Lanes) • Covers a wide range in the vehicle’s environment • Trajectories of other vehicles • Lane output of external preprocessing units
• Low-level sensor information (Patches/ Features) • Spatially restricted features in the vehicle’s environment such as
boundary features • But also features like grid cells (occupied yes/no), color values of a
camera image
35 Volkswagen AG | Konzernforschung
HIERARCHICAL MODEL OF A SINGLE SCENE
35
The graphical model comprises hidden random variables 𝒙𝒙𝑖𝑖 and observations 𝒚𝒚𝑖𝑖 • Hidden variables represent parts and sub-parts of a scene encoded by the root node • Variables are continuous, multi-dimensional, and multi-modal
Edges encode probabilistic dependencies between pairs of variables
2-Lane Road Scene
Features
Right Lane Left Lane
Lane-Segment Lane-Segment
Patches
36 Volkswagen AG | Konzernforschung
INFERENCE DEPTH-FIRST MESSAGE PASSING bottom-up = generating a
hypothesis top-down = verifying the
hypothesis Example: 11. Verifying the road hypothesis (top-down)
36
37 Volkswagen AG | Konzernforschung
IS THE ROAD DRIVEABLE – DISTANCE TO STOP
Velocity
50 km/h
70 km/h
100 km/h
130 km/h
Deceleration
2 m/s2 54 m 103 m 205 m 342 m
10 m/s2 23 m 38 m 66 m 101 m
0
100
200
300
2 4 6 8 10
Ove
rall
Stop
ping
Dis
tanc
e in
m
Deceleration in m/s2
Geschwindigkeit in km/h
50
70
130
100
t1=0,3s r0=10ms-2 / 0,7s
38 Volkswagen AG | Konzernforschung
Sensors G
atew
ay
Vehicle
zFAS - modular & scalable architecture
CAN
/ F
LEXR
AY /
ETH
ERN
ET /
…
APPLICATION FOR SERIES PRODUCTS
39 Volkswagen AG | Konzernforschung
Today: Release testing of todays ADAS with up to 2 million test km and 1.000 test drivers
data basis: 2.000.000 km, > 1.000 drivers; source: Dr. Markus Fach et al., Daimler AG, VDI/VW Gemeinschafts-Tagung, 2010)
Tomorrow: Increasing system complexity of future ADAS will increase diversity of relevant test scenarios Forecast for high automation: 100 Million km = 0,67 x average distance sun earth = 5,6 light minutes Costs for such release testing: several 100 Million EUR source: Prof. Winner et al., Darmstädter Kolloquium „Mensch und Fahrzeug“, 2011
Objective: Sustainable and affordable concept for test and release of future ADAS
System Activation frequency per 10.000 km km till activation Distance Warning 40 - 60 170 – 250
Breaking Assistance (BAS) Plus 0,5 - 1 10.000 – 20.000
PRE-SAFE breaking, level 1 0,1 – 0,2 50.000 – 100.000
PRE-SAFE breaking, level 2 0 -
Ensuring reliability – challenges
40 Volkswagen AG | Konzernforschung
Extension Function range /
Security requirements
SW Module- Tests
SIL- Tests
HIL- Tests
Veh. Tests
Test levels/ Function range
Com
plex
ity
SW Module-Tests
SIL-Tests
HIL-Tests
Veh. Tests
The effort of vehicle tests rises disproportional with increasing functions range/ safety requirements
Reduction of the vehicle tests effort, through shifting the test from the street to the simulation. => Virtual Test Drive virtual
test drive
Test levels and complexity
41 Volkswagen AG | Konzernforschung
vehicle model
sensor models
virtual camera environment model video Inter- face
image processing
sensor data fusions
application SW - longit. control
- lateral control - path planner
vehicle SW
Simulation via Virtual Test Drive
42 Volkswagen AG | Konzernforschung
CONCLUSION
Knowledge about the location of the road is a key factor for automated driving and future driver assistance systems
Interpretation based approaches using environment sensors work well in easy to medium challenging scenarios
Using additional map information leads to more robust results
Localization techniques are currently used to solve the most complex scenarios
Drivability estimation at long ranges for high speed driving is still challenging
New sensor principles and machine learning approaches are one way for solving this topic
Testing environment perception is one of the key challenges
Shifting tests from street to simulation reduces vehicle test efforts significantly
Centralized ECUs like the zFAS help facilitating testing procedures
THANK YOU FOR YOUR ATTENTION