UNCLASSIFIED

Target Level of Safety and

Reusable Safety Arguments

for UAS Integration into

Civil Airspace

5th Annual Operational

Analysis Workshop

15 June 2011

Russell Wolfe

Acquisition & Technology Development Group Lead

Modern Technology Solutions, Inc.

Wolfe_UAS Safety Criteria_NATO OAW_15June2011

2 UNCLASSIFIED

UAS Airspace Integration

Challenges

• Unmanned Aircraft Systems (UAS) are more widely used than ever before in aviation history but possess no inherent ability to “see and avoid” to prevent collisions as required by regulations.

• There are three key items inhibiting UAS manufacturers and acquisition organizations from defining performance and safety requirements for UAS operations within the NAS: 1. Inability to derive quantitative sense

and avoid (SAA) requirements and standards from existing regulations

2. Lack of a defined Target Level of Safety (TLS) for UAS

3. Limited guidance on what comprises a Safety Case and how to present the substantiating evidence

3 UNCLASSIFIED

Regulatory Guidance 14 CFR Regulations relating to See & Avoid

Sec. 91.111 - Operating near other aircraft.

(a) No person may operate an aircraft so close to another aircraft as

to create a collision hazard.

Sec. 91.113 - Right-of-way rules: Except water operations.

(b) General. When weather conditions permit, regardless of whether an operation is conducted under instrument flight rules or visual flight rules, vigilance shall be maintained by each person operating an aircraft so as to see and avoid other aircraft. When a rule of this section gives another aircraft the right-of-way, the pilot shall give way to that aircraft and may not pass over, under, or ahead of it unless well clear.

The qualitative nature of the current regulations are not suitable for establishing

requirements for unmanned aircraft “sense and avoid” technical solutions

4 UNCLASSIFIED

Justification for establishing

a Target Level of Safety for UAS

FAA SAA Workshop Final Report “Recommendation 5.5: FAA and DOD should charter a group to define the process,

methods and tools for analyzing UAS SAA as set out in this report. This should include

establishing the numerical probability for the Target Level of Safety (TLS) applicable to UAS

SAA and how the TLS should be allocated between operational procedures and systems and

equipment for substantiating compliance.”

National Aeronautics R&D Plan “Goal 6: Develop capabilities for UAS NAS integration (Near Term <5 yrs): • Develop a flight safety case modeling capability including data collection methods

• Define the appropriate target level of safety and the process for evaluation”

OSD UAS Task Force Airspace Integration IPT Charter “Section 2.4.2 - Systems Integration SIPT: The Systems Integration SIPT will be responsible for identifying acquisition solutions for

airspace integration. Initial focus of this SIPT will be, but not limited to:

….

5. Developing a repeatable and quantifiable approach for demonstrating that a UAS can

meet a target level of safety that will satisfy the Department’s requirements for the type of

UAS and class of airspace operations are intended.”

OSD = Office of the Secretary of Defense

5 UNCLASSIFIED

OSD-led Safety Criteria

and Assessment Project

Project Objectives Participants**

Outcome / Results

• Assess various Safety Guidelines and

approaches to determine the best one for

providing a comprehensive, robust collision

risk assessment of UAS NAS operations

• Conduct a series of workshops to review

current risk levels and to determine a

midair collision Target Level of Safety

(TLS) for a range of UAS types in various

airspace environments

• Establishment of a repeatable/quantifiable

safety methodology with reusable safety

arguments

• Guidelines for how to conduct UAS Safety

Analysis

• Recommended midair collision TLS (10-x)

for DoD UAS to routinely access the NAS

• Profile specific safety analysis reports for

LOS, Terminal, Lateral, Vertical, Operating

Area and Dynamic Operations

** DoD participation includes members from OSD, Service

Safety Centers, AW Directorates, Programs of Record

Schedule

6 UNCLASSIFIED

UAS Target Level of Safety Defining how safe the UAS must be

• Purpose: – Conduct a series of workshops to review current risk levels and to determine

a DoD midair collision Target Level of Safety for a range of UAS groups in various airspace environments

• Workshop Objectives: 1. Provide assistance to the OSD AI IPT Safety Methodology activity to provide

consistency between the recommended methodology and evaluation criteria

2. Conduct a thorough review of historical midair and near-midair collision

statistics to provide an unmitigated NAS risk baseline

3. Determine what factors (e.g. airspace class, environmental, UAS Group)

should be used to establish a TLS value(s) and select the appropriate one(s)

4. Establish a methodology to quantitatively establish a TLS value(s) for

anticipated DoD UAS operations

5. Identify a set of tools that can be used to conduct analysis for determining

whether the TLS value(s) has been met

6. Provide a consolidated DoD recommendation to the future FAA-led TLS

Workshops scheduled to kick-off in 2011

7 UNCLASSIFIED

UAS Target Level of Safety Approach

• Approach

– Conduct a series of workshops attended by military, FFRDC and Industry

subject matter experts in the areas of safety, airworthiness and operations

– Identify past efforts that have defined/used a target level of safety approach

and how it was implemented

– Establish any assumptions, terms and definitions that will be used as the

foundation for developing a TLS for military UAS

– Consider all feasible TLS concepts, parameters and units of measure and

determine the best combination to use

– Derive a DoD midair collision TLS for a range of UAS groups in various

airspace environments

– Provide substantiating evidence supporting any TLS value(s)

Recommended TLS Value(s) for

UAS Operations within the NAS

Group 1 Group 2 Group 3 Group 4 Group 5

Class A na na na 10-8 10-8

Class B 10-5 10-7 10-8 10-9 10-9

Class C 10-5 10-5 10-6 10-8 10-8

Class D 10-4 10-4 10-5 10-7 10-7

Class E 10-4 10-4 10-5 10-7 10-7

Class G 10-4 10-4 10-5 10-7 10-7

Define TLS Criteria Derive TLS Value(s)

Yes

No

OwnshipAvoidance

IntruderAvoidance

ATCDirections

Encounter Threshold

Tau > XNMAC|

Encounter

Encounter

OwnshipInducing

IntruderInducing

OwnshipInducing

IntruderInducing

IntruderInducing

OwnshipInducing

Encounter

MAC|NMAC MAC

NMAC

Encounter

Encounter

8 UNCLASSIFIED

TLS Workshop Discussion Topics Potential TLS Concepts

• Should TLS be defined as a “Target” or as a “Threshold”?

• Should TLS be a single value or a range of values?

• Should TLS be established for the SAA system or the UAS?

• Should TLS be defined for the SAA system alone or for all protection layers?

• Do we use an existing collision risk matrix or define a new one?

• Should TLS be based on historical accident statistics for manned aircraft?

VS.

1 x 10-n For all UAS

VS.

VS.

VS.

VS.

Likelihood Qualitative Quantitative

Probable Anticipated to occur one or more times during the entire system/operational life of an item.

Probability of occurrence per operational hour is greater than

1 x 10-5

Remote Unlikely to occur to each item during its total life. May occur several time in the life of an entire system or fleet.

Probability of occurrence per operational hour is less than 1 x

10-5, but greater than 1 x 10-7

Extremely Remote Not anticipated to occur to each item during its total life. May occur a few times in the life of an entire system or fleet.

Probability of occurrence per operational hour is less than 1 x 10-7 but greater than 1 x 10-9

Extremely Improbable So unlikely that it is not anticipated to occur during the entire operational life of an entire system or fleet

Probability of occurrence per operational hour is less than 1 x 10-9

MIL-STD-882

FAA System

Safety Handbook

9 UNCLASSIFIED

TLS Workshop Discussion Topics Potential TLS Parameters

• Class of Airspace

• Surface Population Density

• Altitude

• UAS Group

• UAS Momentum (mass x velocity)

• Positive Control (IFR / VFR)

• Airspace Density

• Maneuverability

10 UNCLASSIFIED

TLS Workshop Discussion Topics Potential Units of Measure

• Examples – midair collisions / flight hour

– loss of separation / flight hour

– NMACs / flight hour

– fatal accidents / mission

– all accidents / mission

– Others

exposure ofunit

eventTLS

11 UNCLASSIFIED

TLS Workshop Findings / Recommendations

• TLS Concept Decisions:

– Target v. Threshold: Decision was that the TLS value(s) should establish the DoD threshold(s) that should be attained to gain access to the NAS

– Single v. Multiple Values: Multiple Values should be derived, however, the exact number or values will be determined by our future analysis results

– SAA Function or UAS : a TLS will be derived for the SAA Function

– SAA System alone v. All protection layers: ALL Protection Layers should be considered in deriving a TLS

– Use existing risk matrix or define a new one: Agreed by all that this is out of scope and a Service decision

– Should TLS be based on historical accident data: Yes

• TLS Parameter Decisions

– Class of Airspace, UAS Group, and IFR/VFR are the preferred TLS parameters

– Airspace Density and Maneuverability should also be considered

• Unit of Measure Decision

– The TLS unit of measure should be Midair Collisions per Flight Hour

12 UNCLASSIFIED

TLS Workshop Findings / Recommendations

• TLS has previously been applied to existing aviation systems – JAA Requirements on Aircraft

Accident Rates (~1980) – Precision Runway Monitor (1989)

• Two existing modeling environments capable of conducting this type of analysis

– MIT/LL: Collision Avoidance System Safety Assessment Tool (CASSATT)

– MTSI: Sense and Avoid Flight Encounter Simulation Toolset (SAFESTTM)

Pilot

response

model

Manned Aircraft

Unmanned Aircraft

Pilot

response

model

Comm

Sense and avoid

system

TCAS

UAS

EnvironmentModels

Dynamic Simulation

Visual

acquisition

Tim

e H

isto

rie

s &

M

etr

ics

TCASComm

Random

Situations

Aircraft

dynamic

model

UAS

dynamic

model

Airspace Models

13 UNCLASSIFIED

TLS Workshop Findings / Recommendations

• Airspace analysis shows large variations in collision risk based upon:

– Region

– Altitude

– Proximity to major airports

– Cooperative/Non-cooperative

• Determined that 6 study areas should be analyzed based on data variations & operational considerations

– Airspace Class

– Altitude

Airspace Group Lateral Distance

(nm)

Vertical Distance

(ft)

A 1 500

B & C ½ 300

D ½ 300

E above 10k ft. 1 300

E below 10k ft. 1 300

G 1 300

14 UNCLASSIFIED

Next Steps TLS Analysis (currently in work)

• Goal of TLS analysis is to provide substantiation to all TLS values

• Analysis conducted using the MIT/LL CASSATT

and MTSI SAFEST modeling & simulation tools – Data verification/comparison run completed

• Quantifying NAS ambient probability of encounter – Analyzing altitude, airspace class, geographic location

– Results will be used in a fault tree analysis and to

identify if multiple TLSs are reasonable

• Upcoming tasks will focus on nodes of fault trees – Fault tree being constructed to aid in deriving TLS

– Leveraging existing fault trees built for similar

purposes

• Probabilistic Risk Assessment – Encounter probability used as Initiating Event (IE)

– Mitigation analysis using Event Sequence Diagram

Yes

No

OwnshipAvoidance

IntruderAvoidance

ATCDirections

Encounter Threshold

Tau > XNMAC|

Encounter

Encounter

OwnshipInducing

IntruderInducing

OwnshipInducing

IntruderInducing

IntruderInducing

OwnshipInducing

Encounter

MAC|NMAC MAC

NMAC

Encounter

Encounter

15 UNCLASSIFIED

Next Steps Develop Re-usable Safety Arguments

• Once the safety approach and TLS values are adopted by DoD, re-usable safety arguments will be developed for six airspace access profiles as defined within the DoD UAS Airspace Integration Plan.

Terminal Area

Operating Area Dynamic

Visual Line of Sight

Vertical Lateral

16 UNCLASSIFIED

Next Steps Operators, Regulators, Developers

• Planning to conduct a series of follow-on activities to ensure TLS values and safety assessment processes can be leveraged and used by operators, regulators and developers. – Working with owner of MIL-STD-882 to integrate Safety Case

recommendations

– Safety Case support to Services

• Navy BAMS

• Cherry Point MCAS

• Cannon AFB

– RTCA SC-203 support

• Plenary and Workgroup meetings

• Co-lead for WG3 FAA SAA Workshop

MASPS

Standards

DevelopmentOrganizations

(e.g. RTCA, ASTM, SAE)

MOPS TSOsDesign Standards

for Manufacturers and Developers

Developers

Service

Safety Centers

Operators

Program of

Record (POR)

Requirements

Common

Safety Case

Methodology

Service Unique

Safety Guidance(Army, Navy, AF)

Capability

Based Assessment

ICDCDD /

CPD

AI

KPP

FAA

Unmanned AircraftProgram Office

RegulatorsNotice for Proposed

Rulemaking (NPRM)

Rule Modification

(14 CFR xxx)

Modify

Advisory Circularsand/or FAA Orders

Document Change

Proposals (DCP)

Formal Rule-

Making Process

Revised Rules

Interpretation

AoA Safety Case

TLS Workshop

Recommendations

AI – Airspace Integration

AoA – Analysis of Alternatives

CDD – Capability Development Document

CPD – Capability Production Document

ICD – Initial Capabilities Document

KPP – Key Performance Parameters

MASPS – Minimum Aircraft System Performance Specification

MOPS – Minimum Operational Performance Standard

TSO – Technical Standard Order

17 UNCLASSIFIED

Summary

• Provided an overview of an on-going OSD-led effort to establish two items essential for UAS airspace integration: – Common Safety Methodology for conducting UAS Safety Analysis

– Target Level(s) of Safety for DoD UAS

• Discussed the TLS Workshop initial findings and recommendations

• Discussed the TLS Analysis Plan currently in work

• Defined what some of the next steps will entail

Establishing the safety criteria and methodology for conducting UAS safety

analysis are critical elements in solving the “see and avoid” challenge.

18 UNCLASSIFIED

Contact Information

• Russell Wolfe Acquisition & Technology Development Group Lead Modern Technology Solutions, Inc. 5285 Shawnee Road, Suite 400 Alexandria, VA 22312

– 703-564-3828

– russ.wolfe@mtsi-va.com