CANVAS - michigan.gov · michigan state university college of engineering canvas c onnected and a...

MICHIGAN STATE UNIVERSITY COLLEGE OF ENGINEERING

CANVAS CONNECTED and

AUTONOMOUS

NETWORKED

VEHICLES for

ACTIVE

SAFETY


CANVAS Autonomous Vehicle Platform


CANVAS Research & Student Engagement

Research Faculty & Grad/Undergrad Students

Educa,on & Student Engagement CAV Related Courses CAV Prac:cal Training CAV Students’ Club

CAV Interna:onal Compe::ons


CONNECTED & AUTONOMOUS NETWORKD VEHICLES for ACTIVE SAFETY


CANVAS Autonomous Vehicle: Sensing

Lidars

Cameras

Radars


Autonomous Vehicle: Architecture

Sensing Decision

Simultaneous Localiza6on & Mapping

Percep6on

Communica6on V2V / V2I

Autonomous


canvas RESEARCH: Mapping, Localization & Multimodal sensing

SEAMLESSLY INTEGRATING MOBILITY, SAFETY & SECURITY IN AUTONOMOUS & CONNECTED VEHICLES

CONNECTED & AUTONOMOUS NETWORKED VEHICLES FOR ACTIVE SAFETY

n multi-modality sensingSignal sensing from multiple lidars, radars & cameras

n sensor & data fusionIntra- & inter-modality fusion of sensed signals

n (joint) deep learningObject detection, recognition & motion forecast using deep learning

n internal sensingHuman-vehicle interaction & human sensing

research focus:

regular image

thermal image

face alignment

head pose estimation

eye gaze estimation

expression recognition

body tracking

wearable sensor

voice recognition

angry

afraid

surprised

happy

sad

disgusted

inference

Thermal Image

Regular Image


Simultaneous Localization and Mapping (SLAM)


Autonomous Driving on Snow


CANVAS Object Detection & Classification in Snowy Conditions


Autonomous Vehicle: Perception Piece of the CANVAS “mind”

Sigmoid

FC

Euclidean

Loss2

AveragePool

DepthConcat

Conv

Conv

Conv

Conv

Conv

Maxpool

Conv

DepthConcat

Conv

Conv

Conv

Conv

Conv

MaxPool

Conv

MaxPool

DepthConcat

Conv

Conv

Conv

Conv

Conv

Maxpool

Conv

DepthConcat

Conv

Conv

Conv

Conv

Conv

MaxPool

Conv

DepthConcat

Conv

Conv

Conv

Conv

Conv

Maxpool

Conv

DepthConcat

Conv

Conv

Conv

Conv

Conv

Maxpool

Conv

DepthConcat

Conv

Conv

Conv

Conv

Conv

MaxPool

Conv

MaxPool

DepthConcat

Conv

Conv

Conv

Conv

Conv

Maxpool

Conv

DepthConcat

Conv

Conv

Conv

Conv

Conv

MaxPool

Conv

MaxPool

DepthConcat

Conv

Conv

LocalRespNorm

MaxPool

Conv

Input

Sigmoid

FC

Euclidean

Loss1

AveragePool

FC

Sigmoid

FC

Euclidean

Loss0

AveragePool

FC

Example of a Deep Neural Network


Sensor Fusion (Lidar + Radar)


Sensor Fusion (Lidar + Video)

§  Automatic fusion of Lidar + images §  Generating dense depth maps that can be used to

training deep neural network §  Defining deep learning architecture, and training

and testing it

Smart interpolated depth Lidar + Image


canvas RESEARCH: RADAR

Antenna RF IC

Radar Sensor packaging

Radar Fusion, Learning, Classifica6on intra-‐ and inter-‐sensor data

Radar System radar arrays

EMI/ EMC

Simula6on &

Modeling Other sensing

modali6es


CANVAS Advanced Radar Research

§  CANVAS Radar Research Objectives §  Detection and classification of

pedestrians, vehicles, animals, etc… §  Classification of people and activities

§  Approach §  Leverage micro-motion signatures

present in Doppler radar §  Further develop interferometric radar

mode developed by MSU faculty §  Research distributed radar systems

for advanced detection and classification

Simulated radar micro-Doppler signature of a walking person, showing distinct time-frequency signatures

Joint 30 GHz interferometer/Doppler measurements of a walking person


canvas RESEARCH: deep learning Pedestrian Detection

17

MSU Video: SDS on 30 classes Caltech Test Video: SDS R-CNN


canvas RESEARCH FOCUS: deep learning


Situational Awareness: Overview Observe

Act Understand

An6cipate

•  Multi-sensor system (LIDAR+camera) •  Car Detection and Localization •  Human Detection and Localization •  Identifying Types of Pedestrians/Objects •  Forecasting Human-Car Dynamics •  Forecasting Human-Human Dynamics •  Take Preemptive Actions •  Collision avoidance


Situational Awareness: Simulating the World In Closing

Simulating The World...

Figure 1.1: Top row: A number of example renders from the large database of 3D CADmodels available on the online database turbosquid.com. Bottom Row: Renders frommodels of similar objects found in the free Trimble 3D Warehouse. The models found inturbosquid enable much higher quality rendering. By using a high end rendering softwareand accurate models one can create a large amount of photo-realistic renders that can beused to trained object detectors.

1.2.3 Ontological Supervision for Fine Grained Classification of StreetView Storefronts

Modern search engines receive large numbers of business related, local aware queries,such as “Mexican Restaurants in Pittsburgh”, “Laundromats around me open now”, etc.These queries are best answered using accurate, up-to-date, business listings, that containrepresentations of business categories. Creating such listings is a challenging, and neverending, task as businesses often change hands or close down. For businesses with streetside locations one can leverage the abundance of street level imagery, such as GoogleStreet View, to automate the process. However, while data is abundant, labeled data is not;the limiting factor is creation of large scale labeled training data.

In this work, we utilize an ontology of geographical concepts to automatically prop-agate business category information and create a large, multi label, training data for finegrained storefront classification. We match street level imagery to known business in-formation using both location and textual data extracted from images. We fuse infor-mation from the ontology to propagate category information such that each image ispaired with labels with different levels of granularity. Our learner, which is based onthe GoogLeNet/inception deep convolutional network architecture and classifies 208 cat-egories, achieves human level accuracy.

7

Figure 1.2: A fully rendered scene. To our knowledge, we are the first to utilize this levelof realism in synthetic data generation.

1.3 Broad Impact

There are a number of ways in which this work will impact the broad computer visioncommunity: (1) The output of our classification method can be used as the first step in afully automated solution for object-model matching, which is a fundamental operation inmany computer vision, graphics, and robotics applications. In graphics, it can be usedfor photo-manipulation – inserting new objects into images, and manipulating currentones [Kholgade et al., 2014]. In computer vision, it will aide 3D reasoning of scenesand augmented reality, while in robotics it can be used for robot manipulation. (2) Byemploying rendered images of 3D models, and automatically assigning labels, a limitingtask in computer vision, that of collecting large labeled datasets, is effectively eliminated.The use of 3D CAD model databases for computer vision will enable tasks such as imageclassification, object detection, pedestrian tracking, and human pose estimation.

Moreover, we feel that insights gained from the work done in this thesis go beyond theuse of 3D CAD models, and are applicable to the broad computer vision community. Themain takeaway is that for constructing training sets, a computer vision researcher needsto be a true Renaissance Woman, employing skills not only of a machine learning hacker,but also those of natural language analysis, human computer interaction, library studies,behavioral economics, and even philosophy. There is a lot of high quality informationthat has already been curated, or created, by people using significant amounts of labor. Byvisually-grounding our tasks to the physical world we are able to leverage these knowledgesources.

8

Figure 3.6: Example results from all of the datasets used. Each row shows input images(top) and overlaid pose estimation results (bottom). (a) Results using a Known 3D model,(b) results using an Unknown 3D model, and (c) failure cases.

31

Y. Movshovitz-Attias, V.N. Boddeti, Z. Wei and Y. Sheikh, 3D Pose-by-Detection of Vehicles via Discriminatively Reduced

Ensembles of Correlation Filters, BMVC 2014

Vishnu Boddeti (CMU) Situational Awareness February 26, 2016 50 / 1


V2V & I2V: Sensor Fusion


canvas RESEARCH: V2X sensor fusion







research focus:

regular image

thermal image

face alignment


eye gaze estimation


body tracking

wearable sensor

voice recognition

angry

afraid

surprised

happy

sad

disgusted

inference


canvas RESEARCH: internal sensing







research focus:

regular image

thermal image

face alignment


eye gaze estimation


body tracking

wearable sensor

voice recognition

angry

afraid

surprised

happy

sad

disgusted

inference


CANVAS Education & Student Engagement

Research Faculty & Grad/Undergrad Students

Educa,on & Student Engagement CAV Related Courses CAV Prac:cal Training CAV Students’ Club

CAV Interna:onal Compe::ons


CANVAS Education & Student Engagement Examples of CANVAS related courses/areas

§  Deep Learning, Machine Learning & AI •  Theoretical and practical algorithmic-oriented courses

§  Robotics for CAVs •  Statistical signal processing & autonomous systems

§  Computer Vision and Image Processing •  Both in CSE and ECE

§  Radars and Antennas •  Advanced EM theory & radar-dedicated courses

§  Control, Communications and Networking •  Distributed and non-linear control and wireless

communications and networking


CANVAS Education & Student Engagement CANVAS Students’ Club & Practical Training


CANVAS Education & Student Engagement CANVAS Students’ Club

§  Attracted ~ 50 undergraduate students §  Breakdown by class

§  Senior 52% §  Junior 27% §  Soph. 12%

§  Breakdown by major §  Elect. & Comp. Eng. 45% §  Comp. Sci. & Eng. 36% §  Mech. Eng. 9%


CANVAS Education & Student Engagement NHTSA Enhanced Safety Vehicle (ESV) Competition


CANVAS Education & Student Engagement Examples of CANVAS related areas


CANVAS Education & Student Engagement SAE/GM AutoDrive Challenge MSU is one of only six US universi6es selected


Thank You!

canvas.msu.edu

CANVAS - michigan.gov · michigan state university college of engineering canvas c onnected and a...

Documents

Transcript of CANVAS - michigan.gov · michigan state university college of engineering canvas c onnected and a...