Perceiving and understanding the world for ADAS and Autonomous _Peter Labaziewicz_v4

TI Information – Selective Disclosure TI Information – Selective Disclosure

Perceiving and understanding the world for ADAS and Autonomous TU-Automotive ADAS & Autonomous October 3- 4, 2016

Peter Labaziewicz

Texas Instruments

http://events.tu-auto.com/autonomous/2016/Public/Sessions.aspx




TI Information – Selective Disclosure

Perceiving and understanding the world

One of the most critical functions a vehicle will have in the age

of ADAS and autonomous driving is to accurately perceive and

understand the world around it.

2


Autonomous driving steps

3

Perceive Understand Act

Sensors

Sensor Fusion

Machine Learning

CNN is not be all end all

Processing

Challenges & Approaches


Accurately perceiving the world

4

Local Perception Perceive local dynamic

environment at present time

Eg. pedestrians, vehicles etc.

• Onboard Sensors

Localization Determine the vehicle

position relative to a global

coordinate system

• High-resolution mapping

with precise GPS & IMU

inputs establishes location

of stationary environment

Communication • V2V, V2E, V2I

World

Perception


Attributes of sensor sets - passive sensors

5

Limitations:

• Passive sensors are affected by the environment

• The 3rd Dimension - Depth

Mono cameras have difficulty in

sensing distance

BUT - Stereo cameras can easily

and robustly detect distance

Cameras Passive sensors – Sense reflected or emitted radiation

• Visible image cameras: Operate in the visual spectrum

• Infra-Red image cameras: Operate outside the visual

spectrum such as Near IR or Thermal (Far IR)


Examples

• Ultrasonic: Cheap, commonly used for short range distance measurement

• Radar: Cost & size coming down due to 77GHz CMOS radar advances

• Lidar: Cost & size coming down due to advances in solid state LiDAR design

• Time of Flight: High resolution & accuracy, short range (up to10’s meters)

• Structured Light: High resolution & accuracy, short range

Attributes of sensor sets - Active sensors

6

Active sensors emit radiation and measure responses of reflected signals • Able to obtain measurements anytime, under any weather condition

• Provide depth information

• Range is limited by emitted energy


Challenges to accurately perceive the world

7

Lighting Rain

Snow Fog/Smog

Night

Dirt


Challenges to accurately understand the world

8

Google car examples

Daniel Rosenband, Google - Hot Chips 2016

conference presentation (http://www.hotchips.org/ )

Construction

Scale

Material properties

Repeated structures

http://www.hotchips.org/

http://www.hotchips.org/


Comparison of sensing technologies

9

Camera Radar LiDAR

Object Detection + +

Pedestrian Detection +

Weather Conditions - +

Lighting Conditions - + +

Dirt - + -

Velocity + -

Distance - Accuracy Stereo + +

Distance - Range +

Data Density +

Classification + -

Fusion

+

+

+

+

+

+

+

+

+


Sensor fusion

10

For increased accuracy and robustness under a wide variety of conditions,

data from multiple sensor types of complementary modalities needs to be

fused to view the same scene


Algorithm view of fusion system

6x-10x

Cameras

6x-10x

Radar

1x-4x

LIDARs

Thermal/IR

8x-12x

Ultrasonic

GPS IMU

Sensor Processing

Sensor Processing

Sensor Processing

Sensor Processing

Sensor Processing

Perception

Perception

Perception

Perception

Perception

FUSION • Sensor Fusion

• Localization

• Mapping

Maps

PLANNING AND CONTROL

• Path planning

• Motion planning

• Vehicle controls

• Acceleration

• Brake

• Steering

FEEDBACK TO DRIVER

• Visualization/Display

• Warnings

Driver monitoring

Object information

• Location

• Velocity

• Type

Raw data

• Pixels

• Point clouds


Balancing all the key requirements

12

COMPUTE PERFORMANCE

BANDWIDTH

I THINK WE HAVE

STRUCK THE RIGHT

BALANCE

Power/Thermal


Heterogeneous processing architecture

13

ARM/DSP are needed

for High-level vision stages

of the algorithm. • ARM Cortex Axx:

• Scalable RISC

• Data Fusion

• Memory Coherency

DSP: • VLIW SIMD+MIMD

• Data Fusion

EVE Vector Coprocessor: • High Bandwidth

• Pixel Operations

• SIMD Parallelism

• Energy Efficiency

Hardware Acceleration: • High Bandwidth

• Pixel Operations

• HW Acceleration

• Configurable


6x-10x

Cameras

6x-10x

Radar

1x-4x

LIDARs

Thermal/IR

8x-12x

Ultrasonic

GPS IMU

Sensor Processing

Sensor Processing

Sensor Processing

Sensor Processing

Sensor Processing

Perception

Perception

Perception

Perception

Perception

FUSION

• Sensor Fusion

• Localization

• Mapping

Maps

PLANNING AND CONTROL

• Path planning

• Motion planning

• Vehicle controls

• Acceleration

• Brake

• Steering

FEEDBACK TO DRIVER

• Visualization/Display

• Warnings

Driver monitoring

Efficient mapping of compute to heterogeneous cores

HWA (ISP, …)

Vector Processor (EVE)

DSP

ARM

GPU


Distributed Systems • State of the art today

• Processing on the edge – processor

close to sensor

• Easy to manage - clear ownership

• Power and size constraints

Hybrid “Centralized” System • High speed RAW data transmission to node

processor

• Aggregate common sensor nodes into Central

Fusion ECU (each processor node transmits

object data to fusion ECU)

• Easy to manage - clear ownership

• Easily scalable

• Sensing units can be very small

• Power and size can be relaxed

Fully Centralized Processing • Massive compute can be used

• Complex and difficult to scale to lower

cost vehicles

• Difficult to manage multi-vendor

collaboration on one chip

• Multiple vendor systems on one chip

raises safety & responsibility issues

ADAS/Autonomous vehicle system architectures

SV

Distributed

Interior Camera

Parking Camera

LR Radar

Front Camera

MultiMode

Radar

Hybrid Centralized

Fusion

SV SV SoC

Fusion

Fully Centralized

?


Scene Understanding

16

Feature

Extraction Classification

Scene

Labeling

Object

Detection

• Multiple sensors

• Sensor Fusion • Distance

• Velocity

• Edges

• HoG

• Etc.

• Traditional • SVM, KNN Adaboost,

LDA, HMM etc.

• Deep Learning

• CNN

Understanding http://mscoco.org

http://mscoco.org


Deep Learning - CNN is the ultimate classifier

• Imitates the brain with neural networks that gather information and react to it independently

• Given vast amounts of annotated data the computer learns how to solve the problem

• After a training process on a powerful compute platform, can be implemented in an embedded processor

• Require relatively simple MAC processors with very high compute & bandwidth requirements

– This makes CNN a good candidate for specialized HWA

17

Trained

Features “Trained”

Classifier

Car

“hello”

34818

Cascaded layers

Machine Learning Machine Learning

Hierarchical representation Mathematical model Domain independent

Convolutional neural networking Huge Data

Training

offline

Embedded Output


But - CNN is not the be all end all

18

Intriguing properties of neural networks, by Christian Szegedy, Wojciech Zaremba,

Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus (2013)

• Neural Networks do fail, and for mysterious reasons

Correctly classified Incorrectly classified Image perturbation

Nguyen A, Yosinski J, Clune J. Deep Neural Networks are Easily Fooled: High

Confidence Predictions for Unrecognizable Images. In Computer Vision and

Pattern Recognition (CVPR ’15), IEEE, 2015.

Images that are unrecognizable to humans, but that state-of-

the-art DNNs trained on ImageNet believe with 99.6%

certainty to be a familiar object

For Autonomous driving failure is NOT an option!

Rich data sets from multiple sensors can dramatically

increase classifier accuracy and can dramatically

simplify classification

http://arxiv.org/abs/1312.6199


Putting it all together

19

You need to perceive the world around you

• Multi-modal sensor fusion provides robust local perception in all conditions

• Cameras, Radar and LiDAR will be the key complementary sensing technologies for object fusion

You need to understand the world around you • Robust image understanding is essential for autonomous driving

• Machine learning is the tool that enables understanding

• CNN is not the only answer to all problems

You need to process the data • Partitioning all functions within reasonable processing, bandwidth & power constraints is a challenge

• Heterogenous architectures provide an efficient approach

• TI TDAx SoC’s provide an optimized solution for ADAS and Autonomous Driving

Visit ti.com/ADAS for more information.

Perceiving and understanding the world for ADAS and Autonomous _Peter Labaziewicz_v4

Documents

Transcript of Perceiving and understanding the world for ADAS and Autonomous _Peter Labaziewicz_v4