Hazard Analysis for Autonomous Systems and …...CRA Risk Forum 4 October 2016 Hazard Analysis for...

© WMG, The University of Warwick, 2016

CRA Risk Forum 4 October 2016

Hazard Analysis for Autonomous

Systems and Development of Test

Scenarios Gunny Dhadyalla

WMG, University of Warwick, UK

CRA’s Risk Forum 2016

Stratford upon Avon, UK

4 October 2016


CRA Risk Forum 4 October 2016 CRA Risk Forum 4 October 2016

Agenda

Introduction

Test Methodology

Hazard Analysis

Test Scenarios

Conclusions



An academic department within the science faculty

Established in 1980 by Professor Lord Bhattacharyya as Warwick Manufacturing Group to facilitate technology transfer and knowledge creation for Industry

500+ people (800+ university and industry) working in 6 buildings

Training over 1,500 individuals in the UK and abroad (from school to post experience)

Co-located with JLR & TMETC

Professor Lord Bhattacharyya

Founder and Chairman of WMG

WMG



Going Driverless….. Good idea?



Towards Autonomy – the stages

No new vehicles are now being sold in the US at level 0

Image: http://automotive.tomtom.com/en/highly-automated-driving



Intelligent vehicle technology will bring benefits

Safety argument

Improved energy efficiency, air quality, reduced congestion

Greater productivity: average UK driver spends 235 hours behind the wheel

Independent mobility for all

Huge new business opportunities for many sectors (£51bn global market)



Introduction: Safety

7

In the UK, by 2030, 2,500 lives could be saved, and more than 25,000

serious accidents prevented



Introduction: Acceptance

Over 90% of all on-road accidents occur due to human error

However, customer uptake of existing autonomous systems has been slow

Any benefit from various levels of autonomous systems can be realized only if drivers use such systems

Important to understand the factors that influence (users’) acceptance of automated systems

8



Introduction: What is the future?

Accidents in ‘self driving cars’ well below the average for human drivers…but: – reputational damage

– resulted in death

– incurred expensive recalls

– raised issues with trust

Full autonomy will only be possible after: – The legal and ethical framework to support is in place

– People accept and trust the technology, and we understand how they will use it

– Technical solutions are affordable and dependable



Introduction: Aspects of System Acceptance

Adapted from: Khastgir, S., Birrell, S., Dhadyalla, G. and Jennings, P., 2017. Calibrating Trust to Increase the Use of Automated Systems in a Vehicle. In Advances in Human Aspects of Transportation (pp. 535-546). Springer International Publishing.



Introduction: Testing an autonomous system in real-time

Research Question: How?

– Test methods

Research Question: What?

– Hazard Analysis: Identification and classification of hazards to be tested

– Test scenarios: Scenarios creating the identified hazards



How?: Existing Test Methodology

Absence of standard test methods, different test setups have been developed:

– Vehicle Hardware-in-the-Loop (VEHiL)

• Vehicle is mounted on a chassis dynamometer

– Vehicle-in-the-Loop (ViL)

• Use of augmented reality

– Driving Simulators

• Extensively used to understand the driver perception of the systems

– Test Track/Real World driving

• Cost and time intensive to test large number of scenarios



Vision: To test or evaluate any new technology (infrastructure, communications and on-vehicle) in representative real world conditions with a “driver” in the loop

WMG 3xD Simulator



Hazard Analysis

ISO 26262-2: 2011



Hazard Analysis

“Sufficient level of skills, competencies” : subjective interpretation

Groups of experts discuss/debate and reach a conclusion

Challenges with current Hazard Analysis methods: – Inter-rateability variation

– Intra-rateability variation



Hazard Analysis

How do you overcome intra-rateability and inter-rateability variation in hazard analysis?

Objectify the hazard analysis approach

– By framing rules for categorizing hazards and giving them ratings

– Rules for giving Severity (S), Exposure (E) and Controllability (C) ratings



Hazard Analysis: Objectification

Parametrization for rating

– Severity (S)

– Exposure (E)

– Controllability (C)

Sample Parametrization

for Controllability rating



An aside on Controllability of autonomous cars

Speed

Co

ntr

olla

bili

ty



Test Scenarios: The challenge

Software content comprises:

– 80% to 90% of vehicle innovations

– 40% of production costs

– 50% to 70% of embedded systems R&D costs

Premium car 100 million lines of code

Boeing 787 6.5 million lines of code

Boeing 777 4 million lines of code

F-35 Joint Strike 5.7 million lines of code

F-22 Raptor 1.7 million lines of code



Test Scenarios: The challenge

Complexity

– Autonomous driving systems sensors and control systems without driver intervention (SAE Level 5)

– Diversity of driving, communications and environmental conditions

Real world testing is not feasible

– Mileage

– “Corner cases” sporadic, infrequent and difficult to recreate

– Human resource and cost constraints

Who wants to be a test driver?

Google Self-Driving Car Project Monthly Report February 2016 “…Our car had detected the approaching bus, but predicted that it would yield to us because we were ahead of it.”

Chris Urmson – Ex-CTO Self-Driving Cars, Google March 2016

…his team “implemented 3,500 new tests to make sure this won't happen again.”

Facts and figures At the time of this crash Google had driven 1,452,177 autonomous miles since 2009.

At the end of July 2016, they had driven 1,842,496 autonomous miles.



Karla N & Paddock S 2016 Driving to Safety: How Many Miles of Driving Would it Take to

Demonstrate Autonomous Vehicle Reliability RAND Corporation

How many miles does it take to test an autonomous vehicle?



Intelligent Test Case Generation

In order to tackle the challenge of sample space explosion, a new approach to test scenario creation is needed

We need to be smart about the way we create and run test cases

23



Vitaq Test Case Scenarios

24

Test Action

Test Action Test Action

Test Action

Check

Check

Each action/check is

modelled using a Vitaq built-

in class

Test Action

Test Action

Check

Action parameters are randomly generated

according to Test specified rules

All Possible next

actions/checks are

connected

Build up a simple model that

covers a vast number of

possible test sequences

Seed=1 Seed=2

Test Action

GB2508447A

For a given start seed the

randomly selected sequence is

stable

Run with many seeds to get many new test cases and scenarios and parameter values



Continuous

automated tests Test cases

Test case n stimulus Test case 9 stimulus

Test case 8 stimulus

Test case 7 stimulus Test case 6 stimulus Test case 5 stimulus Test case 4 stimulus Test case 3 stimulus Test case 2 stimulus Test case 1 stimulus stimulus

Has this been tested?

Covered Covered Monitor Monitor

HIL Simulator

Functional coverage

check check cov cov

test scenarios

Connecting Vitaq to the simulator

Controlled

Random

Simulator

Scenarios

Vitaq Input Connected

Vehicle

in the

Loop

Directed

Random

Runtime

Control



Connecting Vitaq to the simulator

Starting scenario can be created following Vitaq rules

– number, speed and path of vehicles

– braking/acceleration of vehicles

– environmental conditions

Runtime interaction with the simulator

– apply driving input

– receive data as if from sensors

– stimulus created from rules

Control properties of 802.11p communication

26



Summary

New technology and vehicles are coming sooner than we might think, bringing benefits for all of us.

But we need to ensure they are secure, safe and robust in complex real world environments To do this, we will need new infrastructure, real world trials, and new methods too…. WMG is developing a new and unique capability for virtual prototyping, to reduce R&D costs and accelerate commercialisation


CRA Risk Forum 4 October 2016 CRA Risk Forum 4 October 2016

Thank you for your attention!

Gunwant Dhadyalla [email protected]

gdhady

Acknowledgements

mailto:[email protected]

Hazard Analysis for Autonomous Systems and …...CRA Risk Forum 4 October 2016 Hazard Analysis for...

Documents

Transcript of Hazard Analysis for Autonomous Systems and …...CRA Risk Forum 4 October 2016 Hazard Analysis for...