Christian LAUGIER – e-Motion project-team 1

EE--Motion Motion TeamTeam--ProjectProject““Geometry and Probability for motion and actionGeometry and Probability for motion and action””

INRIA Grenoble Rhône-Alpes

&

Laboratory of Informatics of Grenoble (LIG)

Scientific leader : Christian LAUGIER (DR1 INRIA)

http://emotion.inrialpes.fr

christian.laugier@inrialpes.fr

Christian LAUGIER – e-Motion project-team 2

Context & Scientific challengeContext & Scientific challenge

• Overall challengeOverall challenge Human-Centered Robotics

ITS for improving safety & comfort & efficiency Personal Assistant & House Keeping & Rehabilitation

• Main MotivationsMain Motivations

� Important socio-economic perspectives => Transport, Aging society, Medical care &

Rehabilitation, Human assistance, Intelligent home …

� Increasing interest of industry => Automotive industry, Robots, Health sector, Services …

� Challenging research topics => Dynamic world, Robust perception, Safety, Human Aware

Motion, Complex Human-Robot interactions …

� Robotics state-of-the-art & Progress in ICT Technologies (computers, sensors, micro-

nano technologies, energy …) make this challenge potentially reachable

Robotics Technologies Robotics Technologies –– Current LimitationsCurrent Limitations

=> DARPA Grand Challenge 2004

� Significant step towards Motion Autonomy

� But still some “Uncontrolled Behaviors” !!!!

Current Autonomous robots are able to exhibit quite impressive skills …. BUT

they are NOT adapted to human environments and they are often UNSAFE !

Some technologies are almost ready for use in some restricted anSome technologies are almost ready for use in some restricted and/or d/or

protected public areas protected public areas …….. BUT.. BUT

�� Open environments are still beyond the state of the artOpen environments are still beyond the state of the art

�� Safety is still not guaranteedSafety is still not guaranteed

�� Too many costly sensors are still requiredToo many costly sensors are still required

=> URBAN Challenge 2007

� A large step towards urban road environments

� But still some accidents, even at low speed !!!

New challenges to be addressed New challenges to be addressed

� Dynamicity & Uncertainty

� Interpreting ambiguities

� Prediction of future states

=> Avoiding future collisions !!!

e.g. Traffic scene understanding

�� Human environment understandingHuman environment understanding

�� Share control & Safe Interaction with humanShare control & Safe Interaction with human

Human beings are unbeatable in taking decisions in complex situations

Technology is better for “simple” but “fast” control decisions (ABS, ESP …)

Human – Robot interaction issue has to be addressed

Driver monitoring

Patient – Wheelchair interaction

Christian LAUGIER – e-Motion project-team 5

Robotics Experimental PlatformRobotics Experimental Platform

Parkview Cycab & Simulator

KoalaAutonomous Wheelchairs Industrial Experimental Vehicles

Commercialized by

Robosoft

Commercialized by

Bluebotics

Equipped Toyota Lexus

Christian LAUGIER – e-Motion project-team 6

Main Topics & Achievements Main Topics & Achievements (2005(2005--09)09)11 PhD, 2 books, 20 journal papers, 3 patents & technological transfers

Observation

Prediction

Risk-based navigation

Partial Motion Planning&

Inevitable Collision States

Autonomous navigation & Safety (3 theses)

Probabilistic Risk Assessment(coop. Probayes & Toyota)

Learning & World change prediction(coop. ETH Zurich)

Prediction & Risk assessment (2 theses)

Efficient 3D Multi-resolution Mapping & Localization using “Tensor maps”

(coop. Perception, IBEO)

Dense Mapping & Localisation (1 thesis)

Vision based Detection & TTC(Coop. Prima)

Perception & Situation awareness (5 theses)

Robust Perception (BOF) & DATMO(Coop. Probayes & Toyota)

Parameters estimation from noisy sensor data

(2 on going theses)

• Continuous symmetry (appli. to calibration)• Fusion of IMU & Vision

(Coop. ETH Zurich & Bluebotics)

Christian LAUGIER – e-Motion project-team 7

Main Topics & Achievements Main Topics & Achievements (Theme 3, 2005(Theme 3, 2005--09)09)

Action selection & Attention focusing[Koike 06]

Bayesian learning[LeHy 07] [Dangauthier 08]

Bayesian models of Superior Colliculus[Colas et al. 09] (coop. LPPA)

Human Perception of Shape from Motion [Colas 06] (coop. LPPA)

Sensory-motor systems & Handwriting[E. Gillet PhD Thesis] (coop. LPN)

Robots

Living systems

Prey & Predator scenario

6 PhD, 1 Book, 9 journal papers (Robotics & Neurosciences), 1 start-up (Probayes)

Brain controlled wheelchair [Rebsamen 09](coop. NUS Singapore)

Christian LAUGIER – e-Motion project-team 8

Key perception concepts for safe navigation: Key perception concepts for safe navigation: Situation Awareness & Risk AssessmentSituation Awareness & Risk Assessment

Bayesian Perception

Probabilistic Risk Assessment

Equipped Toyota Lexus

Ibeo Lux

Stereo camera

IMU + GPS+ Odometry

ADAS & Autonomous Driving=> Cooperation Probayes, Toyota, Renault …

Human-Centered Navigation=> INRIA PAL & ICT-Asia PAMM

Bayesian Perception Bayesian Perception –– Processing Uncertainty & DynamicityProcessing Uncertainty & Dynamicity

Bayesian Occupation Filter paradigm (BOF)Patented by INRIA & Probayes, Commercialized by ProbayesPatented by INRIA & Probayes, Commercialized by Probayes

Prediction

EstimationOccupied space

Freespace

Unobservable space

Concealed space

(“shadow” of the obstacle)

Sensed moving obstacle P( [Oc=occ] | z c)

c = [x, y, 0, 0] and z=(5, 2, 0, 0)

Occupancy grid

� Continuous Dynamic environment modelling using one or

several sensors

� Grid approach based on Bayesian Filtering

� Estimates at each time step the Occupation & Velocity

probabilities of each cell in a “space-velocity” grid

� Uses Probabilistic Sensor & Dynamic models

=> More robust to Sensing errors & Temporary occultation

=> Designed for Sensor Fusion & Parallel processing

BOFBOF

Christian LAUGIER – e-Motion project-team

[Coué & al IJRR 05]

Application to Detection & Tracking

(coop Toyota & Denso)

Driving Experimentations Driving Experimentations –– INRIA Lexus PlatformINRIA Lexus Platform

Toyota Lexus LS600h

2 Lidars IBEO Lux

Stereo camera TYZX

Inertial sensor / GPS Xsens MTi-G

Dell computer + GPU + SSD memory

GPS track example(Using Open Street Map)

Sensor Fusion experiment: Sensor Fusion experiment: Stereo + 2 LidarsStereo + 2 Lidars

Front view from left camera Fusion result using BOF

OG from left Lidar OG from right Lidar OG from Stereo

[Perrollaz et al 10] [Paromtchik et al 10]Movie

Conservative Collision Anticipation using the BOFConservative Collision Anticipation using the BOFTracking + Conservative hypotheses

Autonomous Vehicle Parked Vehicle (occultation)

Thanks to the prediction capability of the BOF technology, the Autonomous Vehicle “anticipates” the

behavior of the pedestrian and brakes (even if the pedestrian is temporarily hidden by the parked vehicle)

Collision Risk Assessment Collision Risk Assessment –– Problem statementProblem statementBehavior Prediction + Probabilistic Risk Assessment

Consistent Prediction & Risk Assessment requires to reason about :

� History of obstacles Positions & Velocities (perception or communications)

� Obstacles expected Behaviors e.g. turning, overtaking, crossing ...

� Road geometry e.g. lanes, curves, intersections … using GIS

TTC-based crash warning is not sufficient !

False alarm !

Conservative hypotheses

Previous observations

Collision Risk Assessment Collision Risk Assessment –– Functional ArchitectureFunctional Architecture

Estimate the probability of

the feasible driving behaviors

Probabilistic representation of

a possible evolution of a car

motion for a given behavior

Probabilistic Collision Risk:

Calculated for a few seconds

ahead from the probability

distributions over Behaviors

Recognition & Realization

Patent INRIA & Toyota 2009 [Tay 09] [Laugier et al 11]

Motion Prediction: Learn & Predict paradigmMotion Prediction: Learn & Predict paradigm

• Observe & Learn “typical motions”

• Continuously “Learn & Predict”� Learn => GHMM & Topological maps (SON)

� Predict => Exact inference, linear complexity

Experiments using Leeds parking data

Euron PhD Thesis Award 07 (Dizan Vasquez)

Christian LAUGIER – Keynote FSR’09, Boston

Probabilistic Collision Risk AssessmentProbabilistic Collision Risk Assessment

•• Behaviors :Behaviors : Hierarchical HMM (learned)

BehaviorPrediction

e.g. Overtaking => Lane change, Accelerate …

GP: Gaussian distribution over functions

Prediction: Probability distribution (GP) using mapped

past n position observation

• Motion Execution & Prediction :Motion Execution & Prediction : Gaussian Process

[[Tay & Laugier & Mekhnacha 11]PhD Thesis Tay Meng Keat + Patent Toyota & Inria & Probayes (2010)

High-level Behavior predictionfor other vehicles

(Observations + HMM)

Ego vehicleRisk estimation

(Gaussian Process)

Behavior Prediction

(HMM)Observations Behavior models

+Prediction 0

0,1

0,2

0,3

0,4

0,5

0,6

Overtaking TurningLeft TurningRight ContinuingStraightAhead

Behaviour Probability

Behavior belief table

RiskAssessment

(GP)Road geometry (GIS) + Ego vehicle trajectory to evaluate

Collision probability for ego vehicle

Behavior belief table for each vehicle in the scene

0

0,1

0,2

0,3

0,4

0,5

0,6

Ov ertaking TurningLeft TurningRight ContinuingStraightAhead

Behaviour Probability +Evaluation

Experimental validation:

Toyota Simulator + Driving device

Ego vehicle

An other vehicle

Collision Risk Assessment Collision Risk Assessment –– Simulation resultsSimulation results

Equipped Toyota Lexus

Ibeo Lux

Stereo camera

IMU + GPS+ Odometry

Collision Risk Assessment Collision Risk Assessment –– Experimental results (Real data)Experimental results (Real data)

Behaviors prediction on a highway (Real time)

Cooperation Toyota & Probayes

Performance summary (statistics)

Maneuvers Maneuvers prediction at roads intersectionsprediction at roads intersectionsCooperation Stanford & Renault

� Scenario� A vehicle is approaching, then crossing an intersection

� Available information

=> perception, previous mapping, communication ...

� Digital map of the road network

� State of the vehicle: position, orientation, turn signal

� Associated uncertainty

� Objective� At any t, estimate the manoeuvre intention of the driver of the approaching vehicle

(e.g. turn left), using states information and extracted information from the digital

map

[Lefevre & Laugier & Guzman IV’11]

Maneuvers Prediction at Road IntersectionsManeuvers Prediction at Road IntersectionsCooperation Stanford & RenaultCooperation Stanford & Renault

– Digital map obtained using Google Map, an annotated using the RNDF format

– Typical paths are obtained with a 3D laser (velodyne), by observing real traffic

– 40 recorded trajectories have been manually annotated

– 2 datasets have been constructed with these trajectories, by automatically annotating

the turn signal

• 40 trajectories with consistent turn signal

• 40 trajectories with inconsistent turn signal

Stanford’s Junior Vehicle (parked)

Intersection 1 Intersection 2

Experimental evaluation : Qualitative ResultsExperimental evaluation : Qualitative Results

� Consistent turn signal � Inconsistent turn signal

– Definitions

• mA, mB = most probable manoeuvre and second most probable manoeuvre

• Undecidable prediction: P(mA) - P(mB) ≤ 0.2

• Incorrect prediction: P(mA) - P(mB) > 0.2 and mA is incorrect

• Correct prediction: P(mA) - P(mB) > 0.2 and mA is correct

– Results on 2 datasets (40 trajectories each)

� Consistent turn signal � Inconsistent turn signal

Entrance Exit

Experimental evaluation : Quantitative ResultsExperimental evaluation : Quantitative Results

e-Motion contributions

on Mobility Assistance

Anne Spalanzani

Arturo Escobedo

Jorge Rios Martinez

Christian Laugier

INRIA Rhône-Alpes

Navigation of a wheelchair taking into account the

context of use : main challenges

• Study of the needs : who might benefit from an Autonomous Wheelchair?

• The wheelchair is a robot : autonomous navigation

– Uncertain and incomplete knowledge of the environment

– Ability to predict the behavior of the obstacles (which can be humans)

• The wheelchair transports a person

– Person/wheelchair communication

– Integration of social conventions in the navigation decision

– Autonomous/Semi-autonomous navigation

• Validation of the proposed system

Scientific challenges• The static environment is unknown

⇒ Construction of maps of the environment

• Mobile obstacles are not known, but they follow typical patterns

⇒ Detection & Tracking + Prediction on-line

⇒ Learning of typical patterns

• Deal with dynamic and uncertain environments

⇒ Navigation decisions based on a risk criteria (Risk-RRT Fulgenzi 08)

⇒ Social conventions with proxemics constraints (Personal Space, Interaction) (Rios 2011)

Fulgenzi C., Tay C., Spalanzani A., Laugier C. “Probabilistic navigation in dynamic environment using Rapidly-exploring Random Trees and Gaussian

Processes”, IEEE/RSJ 2008 International Conference on Intelligent RObots and Systems, 2008

Rios-Martinez, J., Spalanzani, A., Laugier, C.: Probabilistic autonomous navigation using risk-rrt approach and models of human interaction. In: Proceedings of

the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (2011)

Risk-RRT (C. Fulgenzi)

26

Human aware Navigation (J. Rios Martinez)

� Personal Space

� Space of interaction

� Navigation among humans based on risk and comfort

Pconf (q) = Pcoll (q) ° Ppers ° Pinter

Not tak

ing

into

account

interactio

ns

Tak

ing in

to

account

interactio

ns

Mobility Assistance (A. Escobedo)

• Navigation system adapted to the person (elder people,

disabled, poly disabled…)

– Autonomy-semi autonomy

– Interacting with a wheelchair

– From following to accompanying …

28