Safety and Automated Driving SystemsKyle Vogt, Cruise, October 28, 2015

The issue at hand: How do you know a vehicle is safe enough?

2

What exists? The Law

FMVSS

3

NHTSA / DMV

Turns out, it doesn’t tell us

What exists? Standards

4

Industry Standards

Best Practices

Digging deeper…Combine everything that exists Meet it or Beat it

5

AVs

Problem is you could follow what exists, and still end up with an unsafe product because ADS’s add complexity that the existing base does not address.

We have to do more.

c

Where are some of the holes?

6

c

Hazard Analysis MethodsLack of specificity

Behavioral Requirements

Why are traditional hazard analysis techniques not enough?

Bottom up approach misses system-level problems – component interactions

Rely on chain of event causation model

Not good for software

1

2

3

8

How to address those problems

Hazards

Design and implementation

Safety

STPA

Testing and Validation

9

What’s challenging about ADSs? Complexity

How do you build safe complex systems? Solid process and simulation

The path to driverless

10

Naturalistic testing is extremely useful, and key for validation, but cannot accomplish necessary coverage

To get on the road, the only legitimate way to know a system is safe will be through extensive simulation

Thank you.

11

Appendix A: Example of weakness of traditional hazard analysis

Bhopal accident• December 1984, pesticide

plant gas leak in Bhopal, India, exposing over 500,000 people to toxic gas

• None of the safety devices worked

• Seems incredibly unlikely that all safety systems would fail at once, but in reality, they were not independent failures

• Traditional methods only consider limited set of causes and miss the more systemic causes (e.g. financial pressure, poor hiring, failure to heed prior warnings)

12

Appendix B: Reactor example

Chemical reactor: • Computer controls (1) flow of catalyst

into reactor and (2) flow of water into reflux condenser to cool off reaction

• Sensors warn of problems• If fault detected, computer

programmed to leave variables as is, and simply sound alarm

13

Extra - actions taken from STPA results

Comprehensive analysis of specific social human driving behaviors

Crucial point is requirements, have to have top notch requirements that are derived from safety constraints

Create behavioral situations that you can test against, and Test them, extensive simulation

Real world experience

14