• DECEMBER 2019

Workflow based exploration of parameter space and

reliability analysis for automated driving

Roland Niemeier, Gilles Gallee, Bernard Dion

2

Agenda

• Introduction, Motivation

• Reliability Analysis:

Limit State Function and the Probability of Failure

• Application to Driving Scenarios

• Ansys Autonomy

Introduction,

Motivation

How many miles to be driven?

There is a crucial need for smart reliability analyses for vehicle ADAS/AD Function validation.

Source: Nidhi Kalra, Susan M. Paddock: Driving to Safety, www.rand.org

4 ©2020 Ansys, Inc. / Confidential

5 ©2020 Ansys, Inc. / Confidential

Industry leaders have defined the principles for developing safe autonomous vehicles

Using systematic

development practices to

ensure safe designs

Safety by Design

Establishing rigorous

statistical validation cases

Safety by Validation White paper

co-authored by

Aptiv, Audi, Baidu,

BMW, Continental,

Daimler, FCA, Here,

Infineon, Intel & VW

Safety First for Automated Driving (SaFAD)

Source: “Safety First for Automated Driving” (Aptiv, Audi, Baidu, BMW, Continental, Daimler, FCA, Here, Infineon, Intel & VW)

I. Safety by Design:

Implementation of robust

system design.

II. Verification of requirements:

All requirements from I are met.

Known scenarios are covered

& system behaves as

expected.

III. Validation, Statistical

demonstration: Build the

statistical argument to confirm

safety across known and

unknown scenarios. 100 %

reliability of the system & 100

% confidence in a given level of

reliability are not possible.

IV. Post-deployment

observation: Includes field

monitoring of safety

performance and security of the

automated driving system

Focus on MBSE

Focus on Simulation

Need for a statistical

approach

Reliability Analysis:

Limit State Function and the Probability of Failure

• Optimization is introduced into virtual prototyping for more than 20 years

• Robustness evaluation and reliability analysis are key methodologies for safe, reliable and robust products

• The combination leads to robust design optimization (RDO) strategies

• The complementary of reliability is the probability of failure. This can be computed taking into account the scattering, variations of the input. Failure can be defined by exceeding a certain threshold, a limit …

• Applications for example in ADAS/AD, Microelectronics, ...:

– Driving Scenarios

– Solder Joint Fatigue

– …

Reliability and the Probability of Failure

Reliability Analysis with Limit State Functions

X1

X2

g=0

• Robust for arbitrary limit state functions

• Confidence of the estimate is very lowfor small failure probabilities

• Monte Carlo Sampling works well only for Sigma level ≤ 2

• Advanced methods in optiSLang for Reliability Analysis (Sigma level > 2):

Sigmalevel

PF

Number of designsfor cov(PF) = 10%

2 2.3E-2 4 400

3 1.3E-3 74 000

4.5 3.4E-6 29 500 000

First Order Reliability Method Adaptive RSMAdaptive Importance Sampling

Monte Carlo Simulation

Directional Sampling

A simple example: Mishra’s Bird Function

• Comparison of refined Monte Carlo (left) based Failure Probability calculation with Adaptive Sampling method (middle, right) helped to reduce simulations runs from about 500.000 to 3.000

• To define a limit state is most important for the reduction (Subspace)

• The limit state may have separations (Several limit lines)

Test function used for events in advanced driver assistance systems

© Dynardo GmbH

• Based on most probable failure points

obtained by FORM

• Sampling density is centered at this “design

point”

• For almost linear limit state function and

accurate design point

ISPUD is efficient even in higher dimensions

➢ Multiple design points (local minima) are

supported

➢ May be able to mitigate error due to

linearization in FORM

Importance Sampling Using Design Point (ISPUD)

Application to

Driving Scenarios

Reliability Analysis for Driving Scenarios

Source left picture and formula: http://www.pegasusprojekt.deSource right picture: Rasch, M. et al: Safety Assessment and Uncertainty Quantification of

Automated Driver Assistance Systems; NAFEMS World Congress, Quebec, 17-20 June 2019

A Logical Scenario: Approaching jam end with lane cross of leading vehicle

Source: Rasch, M. et al: Safety Assessment and Uncertainty Quantification of Automated Driver Assistance Systems; NAFEMS World Congress, Quebec, 17-20 June 2019

• Logical scenarios described by stochastic input parameters(e.g. Jam end speed, ego speed, lead vehicle speed, laneoffsets, number of lanes, lead vehicle class, ,..)

• Specific traffic situation e.g. jam end• Real ECU code as solver (Software-in-the-loop

simulations), which includes sensors, vehicle data as wellas data from other ECU‘s installed in the vehicle

• Failure probability shall be estimated for each individual scenario

Analysis based on MOP (Metamodel of Optimal Prognosis)

• Partially low local CoPs (CoP – Coefficient of Prognosis; prediction quality) • Assumption special physical and control mechanisms in these regions

Source: Niemeier, R. et al: New Reliability Methodologies for Driving Scenarios; 6th European Expert Workshop on Reliability of Electronics and Smart Systems EuWoRel 2018, Berlin, 1 Oct – 2 Oct 2018

Analysis with Anthill and Parallel Coordinates Plots

• Some output parameters are used for the steering and therefore have impact on other output parameters

• Analysis provided excellent indication which parameters are used for steering

Source: Niemeier, R. et al: New Reliability Methodologies for Driving Scenarios; 6th European Expert Workshop on Reliability of Electronics and Smart Systems EuWoRel 2018, Berlin, 1 Oct – 2 Oct 2018

Automated Workflow for Reliability Analysis• Loop over threshold values (fragility curves) by custom algorithm

• Robustness sampling (before the loop)

• Estimate failure probability from robustness sample

• Start reliability analysis only for small probability

• Loop until minimal (target) probability is reached

• Note: Parameter reduction with meta models based on global sensitivities should be

considered very carefully

Source: Niemeier, R. et al: New Reliability Methodologies for Driving Scenarios; 6th European Expert Workshop on Reliability of Electronics and Smart Systems EuWoRel 2018, Berlin, 1 Oct – 2 Oct 2018

Reliability Analysis for Logical Scenario Approaching Jam End

TTC = 1.0 Samples Pf CoV Beta

MCS 30.000 1.61*10-2 4.5% 2.14

AS 8.000 1.30*10-2 5.8% 2.22

ISPUD 2.000 (+6.400 FORM) 1.70*10-2 6.8% 2.12

TTC = 0.5 Samples Pf CoV Beta

MCS 14.010.000 2.86*10-5 5.0% 4.02

AS 16.000 2.85*10-5 8.4% 4.05

ISPUD 4.000 (+4.500 FORM) 3.03*10-5 8.8% 4.01

TTC = 0.4 Samples Pf CoV Beta

MCS 39.420.000 2.54*10-6 10.0% 4.56

AS 16.000 2.81*10-6 9.1% 4.54

ISPUD 7.000 (+5.500 FORM) 2.31*10-6 9.5% 4.58

Comparison of Efficiency with Monte Carlo Sampling (MCS), Adaptive Sampling (AS) and Importance Sampling Using Design Point (ISPUD) including First Order Reliability Method (FORM)

Conclusion:

Advanced Reliability methods

only needs a thousandth of

designs for small probabilities

of failures in comparison to

Monte Carlo Sampling

here: 28,500 runs for

Adaptive Sampling + ISPUD

versus 39.420.000 runs for

Monte Carlo

Therefore new Advanced

Reliability Analyses are

feasible and appropriate for

standard usage for driving

scenarios

Reference: “Safety Assessment of Automated Driver Assistance Systems, M. Rasch (Daimler AG), T. Most (Ansys), RDO Journal, Issue 2, 2019,

https://www.dynardo.de/fileadmin/Material_Dynardo/dokumente/broschuere/JournalArtikel/RDOJournal_2_2019_ADAS.pdf

ADAS L3 scenario based using reliability analysis

19

Ref: Rasch, M.: Simulative validation of automated driver assistance systems using reliability analysis; 16th WOSD, 2019, Weimar, Germany

Daimler have been implemented Ansys optiSLang for automation of driving scenario-based evaluation

Result is a solid workflow considering robustness evaluation and reliability analysis for parameterized driving scenarios in a way that is much more efficient than Monto-Carlo Sampling.

Source: M. Rasch (Daimler AG), Simulative validation of automated driver assistance systems using reliability analysis, WOSD, Weimar, 2019

Ansys Autonomy:a comprehensive solutionfor ADAS/AD Design and Validation

Ansys - BMW GroupTechnology Partnership

“Ansys And BMW Group Partner To Jointly Create The Industry's First Simulation Tool Chain For Autonomous Driving”

New agreement drives development of autonomous driving technology for the BMW iNEXT, the next-generation autonomous vehiclehttps://www.ansys.com/about-ansys/news-center/06-10-19-ansys-bmw-group-partner-jointly-create-simulation-tool-chain-autonomous-driving

• Long term agreement• Level 3 / 4 • iNext Launch 2021

Ansys will assume exclusive rights to the simulation toolchain technology for commercialization to a wider marketas part of Ansys Autonomy.

Image source: BMW Press Photos Website

Ansys Autonomy for Reliability Analysis

Closed-Loop Simulation

ODD Definition

Too

lch

ain

Val

idat

ion

Drive Analytics

Result Analytics

Drive Data

SUT

Scenario

Creation &

Variation

Test

Plan

Data Lineage

Data Ingestion Data Ingestion

Auto & Manual Labeling

Standard GT Conversion

Anonymization

Standard GT Conversion

Test Fleet Customer Fleet

Cloud Infrastructure and Services

22

Fusion

HW SW

Actuators

HW SW

Sensors

HW SW

HMI / HUD

HW SW

ADAS/AD Function

HW SW

GPS

Radar

Camera

Ultrasonic

Lidar

Addressing all aspects of an ADAS/AD system to ensure both Performance and Safety

Summary and Outlook

24

- Smart reliability analysis is key in order to strongly reduce the number of necessary concrete simulated scenarios

- Customers have successfully applied these algorithms for driving scenarios within Ansys optiSLang workflows

- Ansys will bring into a new tool as part of VRXPERIENCE product family, Test Space Analytics, that integrates these reliability analyses.

For more information - Please, contact us, come to the Ansys Booth