Dynamic Aviation Risk Management System (DARMS): A...

3710 McClintock Avenue, RTH 314 ~ Los Angeles, CA 90089-2902 ~ (213) 740-5514 ~ www.usc.edu/create

National Center for Risk and Economic Analysis of Terrorism Events

University of Southern California

Dynamic Aviation Risk Management System (DARMS): A Proof of Concept

Study Examining the Role of Multi-Attribute Utility

Dr. William J. Burns (Co-PI), Dr. Robin Dillon-Merrill (Co-PI) and

Dr. Richard John (Co-PI)

DRAFT Report to Transportation Security Administration (TSA)

"This research was supported by the United States Department of Homeland Security through the National Center for Risk and Economic Analysis of Terrorism Events (CREATE) under Cooperative Agreement No. 2010-ST-061-RE0001. However, any opinions, findings, and conclusions or recommendations in this document are those of the authors and do not necessarily reflect views of the United States Department of Homeland Security or the University of Southern California."

Cooperative Agreement No. 2010-ST-061-RE0001

Department of Homeland Security

March 26, 2015

http://www.usc.edu/create

ii

ABOUT CREATE

Now in its eleventh year of operation, the National Center for Risk and Economic Analysis of Terrorism

Events (CREATE) was the first university-based Center of Excellence (COE) funded by University

Programs of the Science and Technology (S&T) Directorate of the Department of Homeland Security

(DHS). CREATE started operations in March of 2004 and has since been joined by additional DHS

centers. Like other COEs, CREATE contributes university-based research to make the Nation safer by

taking a longer-term view of scientific innovations and breakthroughs and by developing the future

intellectual leaders in homeland security.

CREATE's mission is to improve our Nation's security through research and development of advanced

models and tools to evaluate risks, costs and consequences of terrorism and natural and man-made

hazards and to guide economically viable investments in homeland security. We are accomplishing our

mission through an integrated program of research, education and outreach that is designed to inform

and support decisions faced by elected officials and governmental employees at the national, state, and

local levels. We are also working with private industry, both to leverage the investments being made by

the Department of Homeland Security in these organizations, and to facilitate the transition of research

toward meeting the security needs of our nation.

CREATE employs an interdisciplinary approach merging engineers, economists, decision scientists, and

system modelers in a program that integrates research, education and outreach. This approach encourages

creative discovery by employing the intellectual power of the American university system to solve some

of the country’s most pressing problems. The Center is the lead institution where researchers from

around the country come to assist in the national effort to improve homeland security through analysis

and modeling of threats. The Center treats the subject of homeland security with the urgency that it

deserves, with one of its key goals being producing rapid results, leveraging existing resources so that

benefits accrue to our nation as quickly as possible.

CREATE develops models, analytical tools, methodologies and software, and tests these tools in case

analyses, representing critical homeland security investment and policy decisions. Due to the cross-

cutting nature of this research in risk, economics, risk management and operations research, CREATE

serves the need of many agencies at the DHS, including the Transportation Security Administration,

Customs and Border Protection, Immigration and Customs Enforcement, FEMA and the US Coast

Guard.. In addition, CREATE has developed relationships with clients in the Offices of National

Protection and Programs, Intelligence and Analysis, the Domestic Nuclear Detection Office and many

State and Local government agencies. CREATE faculty and students take both the long-term view of

how to reduce terrorism risk through fundamental research, and the near-term view of improving the cost-

effectiveness of counter-terrorism policies and investments through applied research.

Please visit www.usc.edu/create for more information.

http://www.usc.edu/create

DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)

iii

Table of Contents

About CREATE ...................................................................................................................................................... ii

Executive Summary ............................................................................................................................................... 1

Introduction .............................................................................................................................................................. 4

Multi-Attribute Utility (MAU) Inputs .............................................................................................................. 6

MAU Analysis Output ........................................................................................................................................ 24

Modeling Uncertainty ......................................................................................................................................... 37

Adversary MAU ................................................................................................................................................... 46

Implementing DARMS: Challenges .............................................................................................................. 52

Conclusions ........................................................................................................................................................... 55

Future Research .................................................................................................................................................... 57

References .............................................................................................................................................................. 59


1

Executive Summary

Background. The threat of international terrorism and its focus on attacking U.S.

commercial airlines continues to loom large nearly thirteen years following the events of

September 11th

. The Transportation Security Administration (TSA) was created in 2001 to

address this threat and today has as its core mission “Protect the Nation’s transportation systems

to ensure freedom of movement for people and commerce” Inherent in this mission has been the

assessment and management of the risks surrounding threats to commercial airlines.

In the last three years the TSA has begun to consider carefully the operational costs of a

“one size fits all” screening policy. Responding to the complexity of assessing the threat posed

by passengers and cargo, the TSA could benefit from expanding the Risk-Based Security

approach to include the development of a system-wide architecture that would allow the

assessment of risk on a flight-by-flight basis and make risk-based decisions in real time. In this

regard, the TSA is now investigating the Dynamic Aviation Risk Management System

(DARMS) initiative, and hence the motivation for this study.

Goals. This proof of concept study had four goals: 1) Explore an approach to articulating

and measuring fundamental objectives that lead to an understanding of consequences the TSA

(and other stakeholders) care about; 2) Investigate the uncertainty surrounding credible threats

and flight vulnerability so as to suggest approach to calculating system wide risk to domestic

commercial aviation; 3) Illustrate how the Current and DARMS approaches to passenger

approach can be compared on objectives identified in this study and 4) Identify areas needing

further investigation.

Approach. Objectives and attributes were identified to develop a multi-attribute utility

model (MAU). The search began broadly starting with the TSA’s overarching strategic objective

“Protect the Nation’s transportation systems to ensure freedom of movement for people and

commerce.” and then focused more narrowly on objectives pertaining to aviation security and

specifically the DARMS initiative. Attribute measures, scales and consequences were selected

and assessed based on informal conversations with colleagues from TSA and Deloitte and

publically available information. Uncertainty about credible threats and flight vulnerability (the

probability a security system can be defeated assuming a credible threat) was explored by

decomposing the assessment of the probability of a successful attack into relevant component

parts (e.g., risk classification, threat detection during screening) using probability trees. A

probability function was derived to calculate system wide risk. A multi-attribute utility function

was used as an example of how to compare the Current and DARMS approaches across attribute

measures. Attribute weights (e.g., Figure 1) and other components of the model were based on

the perspective of one of the research team for illustration purposes. However, a more definitive

MAU model can be developed in the future involving experts at the TSA using the methods

outlined in this report.

Objectives and Attribute Measures. The following fundamental objectives were

proposed for the analysis presented in this study: 1) Security Effectiveness, 2) Passenger

Satisfaction, 3) Economic Costs of Security Breaches, 4) Operational Efficiency, 5) Operational

Costs, 6) TSO Job Satisfaction and 7) Aviation Industry Vitality Costs. For this study it appeared

better to think broadly and inclusively about the number of objectives and corresponding


2

attributes with the idea that the list could be paired down as needed at a later time. Twenty four

attribute measures of these high level objectives, along with their assessed weights, were used to

construct the multi-attribute utility functions. Examples of these attribute measures include

fatalities, passengers’ perception of security, economic costs of a significant security breach,

passenger throughput, operational costs (average cost of passenger classification and screening),

TSO morale and airline operating revenue costs.

Uncertainty: Threats and Vulnerability. Probability trees were constructed as an

illustration of how flight vulnerability could be decomposed and its component parts assessed.

Assessment of branch probabilities was guided not by sensitive information but rather by a

reasonable ordering of relative probability magnitudes (e.g., passengers posing a threat will be

much less likely to received expedited screening, standard screening lanes will have a higher rate

of threat detection than expedited screening lanes, DARMS with its proposed sophisticated

countermeasures will reduce flight vulnerability overall). As it turned out, vulnerabilities using

the probability trees calculations were similar to TSSRA assessments, and were estimated to be

about 10% (Current being slightly higher than DARMS).

Using these vulnerabilities and speculations about the average number of domestic

credible threats per year the TSA might encounter, a system wide probability of at least one

successful attack was calculated. To accomplish this, two recursive probability functions were

derived based on the binomial distribution, one assuming that credible threats operate

independently (standard assumption) and the other that their efforts are correlated in some way.

Assuming for illustration, that threats occur independently and there are two credible threats in a

given year, the system wide chance of at least one successful attack is about 19%. The recursive

expressions in equations (1) and (2) also allow for assessments regarding uncertainty of number

of credible threats i, and degree of correlated threats . These parameters permit a broad range

of system wide estimates. Notice that equation (1) is a special case of the more general

expression in equation (2) when =0. When =1 equation (2) reduces to p the flight

vulnerability.

P(X1|n,p) = i (1-(1-p)i ) for nx; 0 otherwise; i=0,1,2 …n. (1)

P(X1|n,p, ) = i {1-((1-p)(p+(1-p))i-1

)} for nx; 0 otherwise; i=1,2 …n. (2)

where X is the number of successful attacks; n is the number of credible threats; I is the

probability of n credible threats, i =1 for i=0,1,2 …n and P(X1|n,p, )=0 for n=0; p is the

flight vulnerability; and (–1 to 1) is the correlation between threats.

Key Findings. MAU calculations tend to favor a DARMS approach regardless of

whether there is a successful attack on a commercial airline or not. However, this finding is

based on the assessed attribute weights of one of the research team and was meant to illustrate

an approach to making this comparison not to yield a definitive conclusion. What this study does

demonstrate are two things worth noting. First, there appear to be conflicting objectives. That is,

the DARMS approach may do better on security effectiveness, operational efficiency and

operational costs and the Current approach may do slightly better on passenger satisfaction,

somewhat better on economic costs following a successful attack and clearly better TSO job

satisfaction regardless of attack outcome. These observations in part, emerged out of

conversations with TSA and Deloitte colleagues.


3

Second, there is a great deal of uncertainty surrounding consequence estimates of these

attribute measures as well as probability estimates of a successful attack. One of the striking

observations was how critically dependent the probability of a successful attack is on the number

of credible threats in a given year. This speaks to the pivotal role of deterrence.

Future Research. The next step is to take the approach illustrated in this study, and

involve key stakeholders in careful elicitations. These next steps are outlined in detail in the

report.


4

Introduction

Perspectives on Risk and Screening Procedures. The threat of international terrorism

and its focus on attacking U.S. commercial airlines continues to loom large nearly thirteen years

following the events of September 11th

. The Transportation Security Administration (TSA) was

created in 2001 to address this threat and today has as its core mission “Protect the Nation’s

transportation systems to ensure freedom of movement for people and commerce” Inherent in

this mission has been the assessment and management of the risks surrounding threats to

commercial airlines. Many agencies within the U.S. Department of Homeland Security (DHS),

including the TSA, describe the risk of terrorist attacks as a function of three components: threat,

vulnerability and consequences. Because of the difficulty of assessing this threat, and the dire

consequences that would likely follow a successful attack on an airline, the TSA initially sought

to reduce the Nation’s vulnerability by adopting a policy of screening all passengers regardless

of their individual risk level.

Risk-Based Security. In the last several years the TSA has focused on moving beyond

the “one size fits all” policy. Screening passengers according to their threat level (an assessment

of their criminal intent and capability) should allow the TSA to allocate resources in a way that

increases security and is more cost-effective in the long term. A tactical outgrowth of such a

Risk-Based Security strategy is the implementation of expedited screening for passengers who

are deemed low risk. Expedited screening can be less intrusive, more convenient, quicker and

less costly than standard screening, thus allowing more security resources to be allocated to those

passengers not receiving expedited screening. Passengers going through expedited screening can

still be subjected to random augmented screening to reduce the chance that this procedure will be

gamed. The TSA has set a strategic goal of moving significantly more of the traveling public

through expedited screening so resources can be dedicated to passengers that the TSA knows less

about in terms of their potential threat.

Dynamic Aviation Risk Management System (DARMS). Responding to the

complexity of assessing the threat posed by passengers and cargo, the TSA believes it could

benefit from expanding the Risk-Based Security approach to include the development of a

system-wide architecture that would allow the assessment of risk on a flight-by-flight basis and

make risk-based decisions in real time. Such an endeavor would require the calculation and

integration of risk over a number of dimensions and the adjustment of risk using a collection of

mitigation options on a flight-by-flight basis according to government-determined risk tolerance

levels.

Project Goal

The goal of this proof of concept study was to create an objectives and attributes

hierarchy that can serve to guide meaningful comparisons among approaches to commercial

aviation security including the Current and DARMS approaches. Hence, this report sought to

demonstrate how assessments could be made across attribute consequences to articulate the pros

and cons of each alternative.

This study sought to address two challenges:

1) Illustrate how DARMS might be examined from a strategic level perspective. As such,

the report describes the procedures used to identify strategic objectives important to the TSA and

its stakeholders and to develop performance measures of these objectives so as to compare


5

DARMS to what is currently being done in terms of passenger risk classification and screening.

For this purpose an example Multi-attribute utility (MAU) model is presented.

2) Illustrate how uncertainty with regard to attack outcomes and consequences can be

represented. Hence, an example probability tree that decomposes vulnerability into relevant

components is depicted.

Contribution of Multi-attribute Utility Approach

A key aspect of the game theoretic component of the DARMS approach is consequence

assessment for both the attacker and the defender. Consequences are contingent on the attacker’s

1) target selection and mode of attack (including the “no attack” option), 2) the defender’s

selection of countermeasures, and 3) the outcome of any attack initiated. How both the attacker

and the defender view these consequences is evaluated relative to each’s fundamental objectives.

Inevitably, even when one focuses on only one adversary, the fundamental objectives can still be

in conflict, and trade-offs are necessary (e.g., maximize security and maximize checkpoint

efficiency). One critical aspect of estimating consequences is determining the trade-offs among

conflicting objectives for both the attacker and defender. For most if not all consequences,

uncertainty is endemic and impacts all objectives. Additionally, accounting for risk attitudes of

both the attacker and the defender will be important. This can be done with utilities functions

that capture the preferences of decision makers.

The tasks required to develop utility functions are described below.

1. Identify attackers. It is important to identify adversaries with the fundamental objective

of attacking/crashing a passenger airplane flying within the United States. In this study

adversaries are limited to organizations with clear intent and significant capabilities.

2. Identify defender stakeholders. While the TSA is the primary stakeholder, it is important

to identify other stakeholders closely involved in aviation transportation defense that

should be included in the DARMS model. These stakeholders include passengers,

Transportation Security Officers (TSOs), and the Aviation Industry.

3. Identify fundamental objectives of an adversary and the defender. We selected objectives

both of the adversary and the defender with the guidance of the TSA and the Deloitte

team. Adversary objectives may vary by adversary, but we focused on those that include

maximizing direct and indirect economic costs to the defender, maximize deaths,

maximize one-time government costs, e.g., purchasing new security equipment, and

maximize psychological impacts to the public. Defender objectives may vary by

stakeholder group, but will likely include (but are not limited to) minimize

countermeasure costs, minimize deaths, minimize direct (short term) economic cost

associated with a successful attack, minimize indirect (long term) economic cost

associated with a successful attack, and minimize intrusiveness of countermeasures on

the flying public. Fundamental objectives for both the selected adversary and the

defender stakeholder group(s) were constructed in direct consultation with the TSA and

other SMEs and are discussed in the next section.

4. Identify attack path alternatives for specific adversaries, including specific targets and

modes of attack. Working in conjunction with Dr. Milind Tambe’s CREATE game theory

team, specific countermeasures were selected on a flight-by-flight basis by the TSA. As

noted earlier, a subset of both adversary and defender alternatives were selected for the


6

proof of concept demonstration. Specifically, the attack modality was construed to be an

adversary carrying a non-metallic explosive on their person or carry-on trying to board a

domestic flight.

5. Construct attribute scales. For the fundamental objectives for both the selected adversary

and selected defender, attribute scales (or simply attributes) provide a metric for

describing the consequences quantitatively. These attribute scales were constructed in

direct consultation with the TSA and other SMEs (e.g., Deloitte team).

6. Assess probability distributions. As an illustration of approach, probability distributions

describing the consequences on each attribute scale for both the selected adversary

(number of credible threats per year only) and defender stakeholder(s) were assessed.

These consequence distributions were conditional on defender “current” and “DARMS”

airport screening alternatives, and on other exogenous uncertainties related to attack

success.

7. Elicit utility functions. In consultation with the TSA and other SMEs, utility functions for

each of the attribute scales are constructed that represent characteristics of adversary and

defender preferences. All single attribute utility functions for this study are assumed to be

linear.

8. Assess trade-off weights. In consultation with the TSA and other SMEs, trade-offs among

the attribute scales associated with both adversary and defender objectives were elicited.

As an illustration, trade-off weights were developed based on the priorities of one

member of the research team. Additionally, this proof of concept assumed that attribute

scales could be aggregated using a weighted additive model.

Multi-Attribute Utility (MAU) Inputs

Defining Objectives

First, fundamental objectives were identified. We followed the procedures suggested by

Keeney and von Winterfeldt (2011) in which they developed a multi-attribute value model to

guide decisions relevant to the U.S. Department of Homeland Security. Essentially, the search

began broadly beginning with the TSA’s overarching strategic objective “Protect the Nation’s

transportation systems to ensure freedom of movement for people and commerce” (TSA, 2015)

and then focused more narrowly on objectives pertaining to aviation security and specifically the

DARMS initiative. The Deloitte consultant team to the Office of the Chief Risk Officer,

developed a DARMS business case in which they proposed six fundamental objectives critical to

TSA aviation security and the DARMS initiative:

1) Security Effectiveness,

2) Operational Efficiency,

3) Passenger Satisfaction,

4) Industry Vitality,

5) Fiscal/Policy Issues and

6) Enterprise Efficiency.


7

Some of these objectives had multiple attribute scales. For example, Operational Efficiency was

measured by cycle time and both quality and frequency of alarms; Passenger Satisfaction was

measured by touch rate, divesture, CBRA (checked baggage resolution area) rate (i.e., how many

checked bags need to be opened and hand checked), wait time and perceptions of security;

Industry Vitality was measured by cost to the airline/airport industry to implement and

Fiscal/Policy Issues was measured by security costs per unit and regulatory/compliance costs.

Enterprise Efficiency was measured by the number of systems tracking passenger risk.

Beginning with the objectives suggested by the Deloitte team, preliminary conversations

with TSA SMEs and findings from Keeney and von Winterfeldt (2011) the following

fundamental objectives were proposed for the analysis presented in this study:

1) Security Effectiveness,

2) Passenger Satisfaction,

3) Economic Costs of Security Breaches,

4) Operational Efficiency,

5) Operational Costs,

6) TSO Job Satisfaction and

7) Aviation Industry Vitality Costs.

This study attempted to be both broad and inclusive about the number of objectives and

corresponding attribute scales with the idea that the list could be pared down as needed at a later

time. In the following paragraphs each objective and its one or more attribute scales will be

discussed. For a complete of list of objective definitions and attribute scale values please see

Tables 1 and 2.

Table 1. Definitions of TSA Objectives and Attributes

Objectives/Attribute Scales Definition

Security Effectiveness Minimize casualties and breaches of security in sterile areas in the airport and the aircraft cabin and prevent the catastrophic loss of an aircraft through the use of risk-based procedures involving deterrence, passenger pre-screening, detection and mitigating response.

Fatalities Lives lost as direct result of a security breach

Injuries Injuries incurred as direct result of a security breach

Security Breaches Inside Airport Sterile Area

Any entry into a sterile area by someone with the intent and capability of seriously harming other people within this area.

Security Breaches inside Cabin of Aircraft

Any entry into an aircraft cabin by someone with the intent and capability of seriously harming other people within this area.

Deterrence1 Risk-based procedures that prevent an attempt to breach security by

increasing the perceived costs of defeating or engaging the security system or by decreasing the perceived likelihood of defeating the security system.


8


Detection sensitivity1 Area under a plot of P(Detection/Threat) or (screening sensitivity)

versus P(Detection/Non-Threat) or (1-screening specificity). Perfect classification has an area of 1 and random classification as an area of .5. The plot is generated for all screening thresholds. Similar plots were first used during World War II for the analysis of radar signals and are an important part of signal detection theory. The technical name of this type of plot is a receiver operating characteristic curve and is well-known as a ROC curve.

Passenger Satisfaction Maximize passenger satisfaction during security screening through the use risk-based procedures that facilitate passenger confidence, perception that they have been treated fairly and at a reasonable cost of their time, hassle and intrusion.

Average Wait time The average number of minutes a passenger can expect from the point they enter the passenger security queue until they are they are through the screening procedures (i.e. typical wait time).

Variance Wait time The variation in minutes a passenger can expect from the point they enter the passenger security queue until they are they are through the screening procedures (i.e. predictability of wait time).

Touch Rate: Passengers Touched by TSO

The percentage of passengers being touched by a TSO during the screening process (i.e. intrusiveness level).

Divesture The extent to which a passenger has to take off or take out carry-on items during the screening process (i.e. hassle factor).

Passenger Perceptions of Fairness

The extent to which a passenger feels that they have received the appropriate level of screening relative to other passengers and relative to their investment in various “trusted traveler” programs (i.e. equity level).

Passenger Perceptions of Security Effectiveness

The extent to which a passenger feels airport and airline security is effective as defined above (i.e. “system is working”).

Economic Costs of Security Breaches in Airport Sterile Areas or Aircraft Cabin

Minimize total economic costs to the U.S. due to loss of life, injuries, property damage, emergency response, business disruption, additional security measures and public response as a direct result of security breaches.

Total Consequences The sum of all economic consequences in dollars over a one-year time horizon due to security breaches.

Operational Efficiency Minimize the use of input resources such as TSOs, equipment and airport floor area to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance.

Passenger Throughput The number of passengers moving from the point they enter the passenger security queue until they are they are through the screening procedures in an hour.


9


Passenger Cycle Time The number of minutes from the point a passenger enters the passenger security queue until they are they are through the screening procedures (same as wait time).

Passenger FTE The average number of TSOs required to screen a passenger.

Passenger False Alarm Rate

The percentage of passengers triggering either a false positive or a minor true positive (i.e. non-security breach).

TSO Utilization Rate Percentage of time TSOs spend screening passengers.

Responsiveness The number of minutes needed to reduce the average wait time to a desired level (e.g., 10 minutes).

Resilience The number hours to restore commercial aviation system to normal operating capacity following a breach of security.

Operational Costs Minimize the costs to the TSA of all procedures involved with achieving a desired level of security effectiveness, passenger satisfaction and regulatory compliance minus the costs of significant security breaches.

Total Passenger Security Costs

The sum of all costs to the TSA in dollars over a one-year time horizon due to operations relating to security effectiveness, passenger satisfaction and regulatory compliance (e.g., FTE, Equipment, FAMS, Pre-screening). Not included are the costs of responding to significant security breaches that are covered under Economic Costs.

TSO Job Satisfaction Maximize TSO job satisfaction in terms of morale, perceptions of that airport and airline security is effective and low turnover rate.

TSO Morale The extent to which a TSO feels a commitment to the TSA and its mission.

TSO Perceptions of Security Effectiveness

The extent to which a passenger feels airport and airline security is effective as defined above (i.e. “system is working”).

TSO Turnover The percentage of TSOs leaving the TSA per year.

Aviation Industry Vitality Costs

Minimize costs to U.S. airports and airlines from additional TSA security compliance requirements as the TSA attempts to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance.

Costs of Reconfiguration at Airport Checkpoints

The sum of all costs to U.S. airports and airlines over a one-year time horizon due to additional security compliance requirements from the TSA as the TSA attempts to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance (e.g., redo the checkpoints). These do not include the impact from major security breaches that are covered under Economic Costs.

Reduction of Operating Revenue from

The reduction of U.S. airline revenue from all sources (e.g., passenger, checked baggage, cargo) over a one-year time horizon due to


10


Additional Security Compliance

additional security compliance requirements from the TSA as the TSA attempts to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance. These do not include the impact from major security breaches that are covered under Economic Costs.

Reduction of Revenue Passenger Miles from Additional Security Compliance

2

The reduction of the number of miles flown by a paying passenger on a U.S. airline over a one-year time horizon due to additional security compliance requirements from the TSA as the TSA attempts to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance. These do not include the impact from major security breaches that are covered under Economic Costs.

Reduction of Revenue per Available Seat Mile from Additional Security Compliance

2

The reduction of U.S. airline revenue per available seat over a one-year time horizon due to additional security compliance requirements from the TSA as the TSA attempts to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance. These do not include the impact from major security breaches that are covered under Economic Costs.

1 Are included in this list because they represent an important aspect of their corresponding objective but

were excluded from the MAU analysis because their effect was represented in the probability trees. 2 Are included in this list because they represent an important aspect of their corresponding objective but

were excluded from the MAU analysis because their effect was largely represented in the other two attributes.

Security Effectiveness. While the primary focus of this objective is to prevent the

catastrophic loss of an aircraft, it is construed more broadly to include significant security

breaches inside the sterile areas of a U.S. airport or cabin of a commercial aircraft inside the

United States. The reasoning is straightforward—significant security breaches can have large

disruptive economic effects. The attributes for security effectiveness include fatalities (0–400 per

year), injuries (0–400 per year), security breaches in sterile areas (0–90 per year), and aircraft

cabins (0–5 per year). Deterrence and area under the detection sensitivity curve (measure of

detection effectiveness depicting a plot of true positives to false positives) are included in Table

1 because they are critical components of security effectiveness. They are not included in the

MAU analysis directly because their contribution is captured in the probability trees discussed in

a later section.

Impact of DARMS: Discussions about the potential impact of the DARMS approach

suggested that within the security effectiveness objective, there should be a gain in security

effectiveness due to an optimal mix of game theory inspired countermeasures and better

intelligence shared about flight-by-flight risks. However, detection probabilities may not

necessarily improve with the DARMS approach as it seeks to adjust risk across flights. As an

example, screening settings for low risk passengers might be relaxed and hence detection

probabilities decrease accordingly.

Passenger Satisfaction. With regards to passengers, TSA must promote confidence in

the TSA’s ability to provide security while maintaining fair and not overly burdensome

procedures to board an aircraft. Attribute scales for passenger satisfaction include average wait


11

time (5–20 minutes per passenger), variation in wait time (0–10 minutes per passenger), touch

rate (0–10%), divesture (1–6 bins per passenger), perceptions of fairness (1–10 constructed

scale) and perceptions of security effectiveness (1–10 constructed scale).

Impact of DARMS: Discussions about the potential impact of DARMS on passenger

satisfaction were mixed across SMEs. Considering each attribute scale, average wait time is

predicted to decrease under DARMS unless the increased real time risk assessment caused

throughput delays. In contrast, variation in wait time is expected to increase under DARMS

because of the inability of passengers to fully predict whether or not they will receive expedited

screening. Touch rate and divesture are expected to decrease under DARMS largely due to the

increased percentage of passengers receiving expedited screening. Perception of fairness and

perceptions of security effectiveness are both expected to decrease under DARMS. With

increased risk classification comes the possible perception among passengers that not everyone is

being treated equally (and hence fairly) and possible confusion over the use of randomized

procedures. While a game theory guided approach to countermeasures may optimally reduce

flight risk, this approach will also be hard to explain to passengers. Surveys conducted over the

last two years by the risk perception and risk communication group at CREATE (Burns &

Dillon-Merill 2013a, 2013b) have consistently shown that expedited screening is perceived by

many as less effective than standardized screening.

Economic Costs of Security Breaches in Airport Sterile Areas or Aircraft Cabin.

Research by the economic and risk perception groups at CREATE (Burns et al., 2013; Giesecke,

Burns, Rose, Barrett, & Griffith, 2014; Giesecke et al., 2012; Rose et al., 2015) have shown the

major contributor to economic costs following a terrorist attack is public reaction. Thus, one

significant objective of the TSA in attempting to prevent terrorist attacks is to minimize the costs

associated with the catastrophic loss of an aircraft. In addition to the catastrophic event, though,

the more common and disruptive significant security breaches inside the sterile areas of a U.S.

airport or cabin of a commercial aircraft inside the United States. needs to also be considered.

This objective has only one attribute scale: the consequence in dollars ($0–$100B per year) and

is predicated on security breaches and most particularly the catastrophic loss of an aircraft.

Impact of DARMS: Barring any significant security breach, DARMS is not expected to

have a material impact on economic costs. However, should a terrorist attack occur it is predicted

the economic costs under DARMS would be considerably greater due to a significant loss of

public confidence and possible Congressional outcry (e.g., “TSA had a screening procedure in

place that completely prevented a successful attack and then switched to DARMS to reduce

operational costs”).

Operational Efficiency. TSA needs to minimize the use of resources as operationally

efficient as possible at all times. Attributes scales to capture the various operational efficiency

dimensions include passenger throughput (200–500 passengers per hour assuming two lanes

each for standard and expedited screening), passenger cycle time (5–10 minutes per passenger

screening—it is the same as passenger wait time but SMEs felt it provided added insight for

evaluating operational efficiency), passenger FTE (,02–.20 TSOs per passenger screening),

passenger false alarm rate (0–10%), TSO utilization rate (50–100%), responsiveness to reducing

unexpected increases in passenger wait times (10–60 minutes) and resilience, that is the ability to

restore system capacity following a major security breach (1–48 hours).


12

Impact of DARMS: DARMS is expected to improve operational efficiency across almost

all attribute scales. Throughput, cycle time, and passenger false alarms would improve because

more passengers would be receiving expedited screening. TSO utilization rate is predicted to

increase because DARMS with its more complex risk assessments would place greater demands

on TSO skills. Situational factors occasionally produce higher than desired passenger wait times.

It is thought in these situations that DARMS will have an increased capacity to respond quickly

to reduce passenger wait time. It is also thought that DARMS would improve system resilience

because of increased ability to share risk information and improvements in operational

efficiency. Passenger FTE is expected to increase due to the increased complexity of DARMS

(Note, this is the only operational efficiency attribute scale that DARMS does not improve).

Operational Costs. Given a constrained (and ever contracting budget), TSA must

minimize the costs of all procedures involved with achieving a desirable level of security

effectiveness, passenger satisfaction and regulatory compliance. The attribute scales for

operational cost are the costs of maintaining security effectiveness, passenger satisfaction and

regulatory compliance ($6–$11 per passenger screening). It does not include the economic costs

of significant security breaches (which are captured separately, see above).

Impact of DARMS: DARMS is expected to reduce operational costs due to improvements

in operational efficiency. However, there was concern raised that the complexity of DARMS

might require more personal and sophisticated procedures and thus increase operational costs

off-setting some of the improvements.

TSO Job Satisfaction. In addition to concern for the passengers and the aviation

industry, TSA needs to maximize the job satisfaction of the transportation security officers

(TSOs). The TSOs are on the frontline in airports, conduct the passenger and baggage screening

procedures and represent the TSA to the traveling public. Attribute scales for quantifying TSO

job satisfaction include TSO morale (1–10 constructed scale), perceptions of security

effectiveness (constructed scale 1–10) and TSA employee turnover (10–50%). DARMS is

expected to decrease TSO morale and perceptions of security effectiveness and hence also

increase job turnover.

Impact of DARMS: DARMS will place more complex demands on TSOs in real time and

so there could be lower job satisfaction. Likewise, the DARMS approach might appear less safe

to TSOs for the same reasons it might appear less safe to passengers.

Aviation Industry Vitality Costs. The intent is to minimize costs to airports and airlines

as they adjust to additional security requirements posed by the TSA. We include two attribute

scales to capture the potential impact to this objective: the costs of reconfiguration at airport

checkpoints ($0–$1B per year) and the reduction of operating revenue from additional security

compliance ($0–$1B per year). The former represents costs that the airports are likely to pass

along to airlines and the later represents possible reduction in passenger demand because of

increased ticket prices. Reduction of revenue passenger miles from additional security

compliance and reduction of revenue per available seat mile from additional security compliance

are measures the airline industry routinely use to measure performance, however, they are not

used in this analysis because they were thought to be highly related to the first two measures.

Impact of DARMS: DARMS is expected to have no material impact on these costs in the

absence of significant security breaches. However, like economic costs described above it is


13

predicted DARMS would result in higher costs to the aviation industry following a significant

security breach (especially the catastrophic loss of an aircraft) as the TSA seeks to regroup.

Table 2 summarizes the objectives, the attributes with scales, and the impact of DARMS relative

to the current system.


14

Table 2. TSA Objectives, Attributes, Scales and DARMS Impacts

Objectives1 Metric

Scale Units

Predictions for DARMS Impact

Security Effectiveness

Fatalities Number of people/year2 0–400 Game theory model might help

reduce number for DARMS-not sure.

Injuries Number of people/year2 0–400 Game theory model might help

reduce number for DARMS-not sure.

Security Breaches Inside

Airport Sterile Area

Number of security breaches/year3 0–90 Game theory model could help

reduce number with DARMS.

Security Breaches Inside

Cabin of Aircraft

Number of security breaches/year 0–5 Game theory model could help reduce number with DARMS

Deterrence RAND Scale (“no meaningful deterrence” to “highest level of deterrence”)

4

0–5 Less deterrence if perception of security decreases, More if perception of security increases (harder to beat the randomness)

Detection Sensitivity Plot of P(Detection/Threat) versus P(Detection/Non-Threat). Larger is better. Area maximum of 1.

4a

.5–1 Game theory model predicts larger area under curve due to optimal mix of counter measures.

Passenger Satisfaction

Average Wait Time Average Minutes/passenger screening

5–20 Decrease under DARMS – the more people sent through expedited screening, the lower the average wait time

Variation Wait time Standard Deviation in Minutes/passenger screening

0–10 May Increase under DARMS if passengers sometimes have expedited screening and sometimes don’t

Touch Rate: Passengers Touched by TSO

Percentage of passengers 0–10% Decrease under DARMS – less AIT used, less alarms, lower touch rate

Divesture Number of Bins Used/passenger screening

5

1–6 Decrease under DARMS – the more people sent through expedited screening, the less bins


15

Objectives1 Metric

Scale Units


Perceptions of Fairness Constructed scale (1–10)6 1–10 Perception hard to predict –

may increase (if see random selections as fair), may stay the same. Probably will decrease (if see others getting “better” treatment

Perceptions of Security Effectiveness

Constructed Scale (1–10)6 1–10 Perception hard to predict –

TSA certainly wants people to perceive as at least as safe if not more safe, DARMS probably will be perceived as less safe

Economic Costs of Security Breaches in Airport Sterile Areas or Aircraft Cabin

Total Consequences Dollars/year7 $0B–

$100B With a significant security breach DARMS might cause more costly public reaction.

Operational Efficiency

Passenger Throughput Passengers/hour8

200–500

Increase under DARMS

Passenger Cycle time Minutes/passenger screening9 5–20 Decrease under DARMS

Passenger FTE TSOs/passenger screening10

.02–.20

Pre-check lanes require less but DARMS may increase due to complexity.

Passenger False Alarm Rate

Percentage of passengers 0–10% Less with less AITs

TSO Utilization Rate Percentage time TSOs engaged in screening

50–100%

Could increase because of complexity.

Responsiveness Minutes needed to reduce passenger wait time to a desired level (e.g., 10 minutes).

10–60 Better under DARMS because of throughput increases.

Resilience Hours to restore commercial aviation system to normal operating capacity following a breach of security in a U.S. airport or aircraft cabin.

1–48 Better under DARMS (assumes increased throughput will aid in returning to normal faster)

Operational Costs

Total Passenger Security Costs (e.g., FTE, Equipment, FAMS, Pre-screening)

Dollars/passenger screening11

$6–$11

Once past fixed costs, DARMS could be operationally cheaper – less fixed equipment


16

Objectives1 Metric

Scale Units


TSO Job Satisfaction

TSO Morale Constructed Scale (1–10) 1–10 Decrease under DARMS

TSO Perceptions of Security Effectiveness

Constructed Scale (1–10) 1–10 Decrease under DARMS

TSO Turnover Employee Percentage per year 10–50%

Decrease under DARMS

Aviation Industry Vitality Costs

Costs of Reconfiguration at Airport Checkpoints

Dollars/year12

$0–$1B

Larger fixed cost to get to DARMS capability

Reduction of Operating Revenue from Additional Security Compliance

Dollars/year13

$0–$1B

Some additional cost associated with sharing flight information, other costs.

1 Based largely on Keeney and von Winterfeldt (2011) and von Winterfeldt and O’Sullivan (2006) papers;

DARMS Business Case Quantification Framework-October 29 2014 (Deloitte Team, 2015). 2 Range based on available 2014 Estimates.

3 Range based on a conversation with Jeff Price, author of Practical Aviation Security (Price & Forest,

2013) in which he provided a ballpark estimate of the number of serious security breaches in airports nationwide to be from 30 to 60 per year. 4 Based on RAND scale reported in RAND Report entitled “Understanding the Role of Deterrence in

Counterterrorism Security” by Andrew R. Morral and Brian A. Jackson (2009). The scale is takes into account the perceived costs by adversaries to overcome a security system, the perceived costs to engage a security system and the perceived likelihood of defeating a security system. It ranges from “no meaningful deterrence” to “highest level of deterrence.” This attribute was not used in the MAU analysis. 4a

ROC standard for Receiver Operating Curve. Each point on the curve represents a ratio of Probability (Detection/Threat) to Probability (Alarm/No Threat). The curve is generated by computing these ratios for a wide range of signal thresholds. The larger the area under the curve the more use the detection system is. That is, across a wide array of thresholds the chance of detecting true threats is larger than false alarms. This attribute was not used in the MAU analysis. 5 Based on conversation with Kristen Best and Todd Trafford (from TSA): 1–2 bins for expedited

passenger screening and 3–4 bins for standard passenger screening.

6 This metric is based on a scale (1–10) developed to measure perceived safety of commercial airline

travel under TSA standard versus expedited screening procedures. The mean rating under in national samples is approximately 7. 7 Range based on available 2014 Estimates.

8 Based on conversation with Kristen Best and Todd Trafford (from TSA): 250 passengers per hour for

expedited passenger screening and 110–120 passengers per hour for standard passenger screening. 9 Cycle time in this case is exactly the same metric as passenger wait time and a close variation of

passenger throughput. According to Kristen Best, it provides an additional perspective to the throughput metric.


17

10 Based on conversation with Kristen Best and Todd Trafford (from TSA): 12 TSOs for two expedited

passenger screening lanes and 13 TSOs for two standard passenger screening lanes (12/250/hr = .048 and 13/110/hr = .118 respectively). 11

Based on conversation with Kristen Best and Todd Trafford (from TSA) to look at total TSA aviation budget and passengers screened: Total TSA budget for 2013 for aviation was approximately $5.3B and about 640M domestic and international passengers were screened in 2013. Hence the average security cost per passenger was about $5.2B/640M = $8.26 assuming 100% of the aviation budget went toward security. The report can be found on

http://www.dhs.gov/sites/default/files/publications/MGMT/FY%202014%20BIB%20-%20FINAL%20-508%20Formatted%20(4).pdf 12

In 2013 for all U.S. Carriers domestically and internationally Operating Income was approximately $200B. Suppose TSA compliance requirements caused airports to pass along their checkpoint remodeling costs to the airlines by one half percent of their Operating Revenues or $1B. 13

In 2013 for all U.S. Carriers domestically and internationally Operating Income was approximately $200B. Let’s suppose that additional security compliance requirements would cause airline ticket prices to increase resulting in a one half percent decrease in Operating Revenues or $1B.

Consequences and Uncertainties of Consequences

Consequences were approximated to illustrate how a MAU analysis could be applied to

this evaluation of DARMS. The assessment did take into account much publically available

information and conversations with SMEs. Ultimately there were four alternative-scenarios of

interest: consequences under the current approach with no successful attack, consequences under

the current approach with a successful attack, consequences under DARMS with unsuccessful

attack and consequences under DARMS with a successful attack. Note that this study does not

distinguish between unsuccessful attack and no attack, an important point discussed later in the

report.

Those objectives for which it is believed there might be significant uncertainty associated

with the consequences, even in the case where a successful attack occurred, are assessed on a

three-point percentile scale (5%, 50% and 95%). For example, given that a successful attack has

occurred, the fatalities under DARMS was assessed to be 50 at the 5th

percentile, 175 at the 50th

percentile and 350 at the 95th

percentile. Three point assessments were completed for fatalities,

injuries, security breaches inside the sterile area of airports and aircraft cabin, passenger and

TSO perceptions of security effectiveness, TOS morale and turnover. These percentile

assessments were used to eventually generate a single point estimate for each attribute across all

four contingencies. Table 3 describes the three point consequence assessments and Table 4

details the single point consequence assessments. The consequences from Table 4 were used to

compute the individual attribute utilities to be discussed next.

http://www.dhs.gov/sites/default/files/publications/MGMT/FY%202014%20BIB%20-%20FINAL%20-508%20Formatted%20(4).pdf

http://www.dhs.gov/sites/default/files/publications/MGMT/FY%202014%20BIB%20-%20FINAL%20-508%20Formatted%20(4).pdf


18

Table 3. TSA Consequence Uncertainties Assessments

Objectives Attributes

Current Unsuccessful

Attack

Current Successful

Attack

DARMS Unsuccessful

Attack

DARMS Successful

Attack

Percentile 5%/50%/95% 5%/50%/95%

Security

Effectiveness

Fatalities 0 50/175/350 0 50/175/350

Injuries 0 50/175/350 0 50/175/350

Breaches-Airport

30/45 /60 30/45/60 27/40.5/54 27/40.5/54

Breaches-Cabin

1/2.5/4 1/2.5/4 .5/1.25/2 .5/1.25/2

Passenger

Satisfaction

Perceptions

Security

5/7/9 1.5/2/5 4/6.3/8 1/1/4

Economic Costs $B

0 10/75/90 0 10/85/95

TSO Job

Satisfaction

TSO Morale 4/6/8 1/2/4 3/4/7 1/1/2

Perceptions-Security

4/6/8 1/2/4 3/4/7 1/1/2

TSO Turnover Rate

15/20/25 20/30/40 20/22/30 25/35/45


19

Table 4. TSA Consequences for Alternative-Outcome Contingencies

Objectives Attributes Current Unsuccessful

Attack

Current Successful

Attack

DARMS Unsuccessful

Attack

DARMS Successful

Attack


Fatalities 0 184.25 0 184.25

Injuries 0 184.25 0 184.25

Breaches-Airport 45 45 40.5 40.5

Breaches-Cabin 2.5 2.5 1.25 1.25


Average Wait Time 7.5 7.5 7.12 7.12

Variation Wait Time 5 5 5.5 5.5

Touch Rate 5 5 4 4

Divesture 2.5 2.5 2 2

Perceptions-Fairness 7.5 7.5 6 6

Perceptions-Security 7 2.46 6.19 1.56

Economic Cost $B 0 65.75 0 72.98


Throughput 350 350 385 385

Cycle Time 7.5 7.5 7.12 7.12

Passenger FTE .08 .08 .09 .09

Passenger False Alarm 5 5 4 4

TSO Utilization Rate 75 75 90 90

Responsiveness 30 30 24 24

Resilience 2 36 1.5 24

Operational Cost $ 8.26 8.26 7.43 7.43


TSO Morale 6 2.19 4.37 1.19

Perceptions-Security 6 2.19 4.37 1.19

TSO Turnover Rate 20 30 23.11 35

Industry Vitality Cost $B

Reconfiguration Costs .05 .25 .06 .30

Reduction Revenue Costs

.05 .25 .06 .30


20

Single Attribute Utilities and Weights

Utility Assessment. For purposes of this study it is assumed that all utilities are linear in

consequences, which assumes that values are neither marginally increasing or decreasing in the

consequence scales units, and risk neutrality. The consequence scales for each attribute were

constructed in a way that a linear utility function assumption is reasonable. Given the size of the

U.S. transportation system, one might expect stakeholders would be risk neutral. Further

assessment would be necessary to estimate both the nature and degree of non-linearity in each

attribute scale utility function. The common assumption of risk aversion is complicated by the

natural loss frame placed on this problem, which often leads to risk seeking utility functions.

Hence, utility functions for each attribute scale were constructed as a linear mapping of the

attribute range onto a 0–1 scale respecting least preferred and most preferred end points:

U(X) = (X – worst-level) / (Best-level – Worst-level)

For example, the utility function for fatalities mapped a 0–400 range onto a 0–1 scale with the

linear function U(fatalities) = 1 – .0025*(fatalities). This utility function would assess zero

fatalities as a 1 and 184 fatalities as depicted in Table 4 would be a utility of .54. These single

attribute utilities for each of the four alternative-scenarios considered were then weighted and

combined to form a multi-attribute utility function.

Weight Assessment. An additive MAU model was assumed for purposes of this

demonstration. This model assumes that attribute scales are utility independent. Again, attributes

were constructed in a manner to make this a reasonable assumption. Additional assessments

would be required to identify attribute dependencies and estimate non-additive MAU model

parameters. There is ample research (von Winterfeldt & Edwards, 1986) that additive MAU

models are often a good approximation of non-additive MAU models, and avoids potential

assessment errors problematic for non-additive MAU models.

The weights were assessed using a swing weight approach (Clemen & Reilly, 2001;

Eisenfuhr, Weber, & Langer, 2010). The assessment began with determining high-level weights

for each of the seven objectives. This was done according to the following standard steps for

assessing swing weights:

1) Consequence ranges for each objective were considered (including the more detailed

attribute scales);

2) Each objective was imagined at its least preferred level across all its attribute scales;

3) Each objective in turn, was then imagined at its most preferred level across all its

attribute scales;

4) The eight hypothetical consequence sets (i.e. least preferred on all consequence levels)

and seven sets in which one objective was at its most preferred consequence levels were

reviewed and ranked;

5) A rating of 0 was given to the set in which all objectives were at their least preferred

levels, a 100 was given to the objective with the highest ranking and all other objectives

were given a rating between 0 and 100 consistent with their ranking and

6) The ratings were summed and then normalized to produce weights for the set of

objectives that summed to 1.


21

This procedure was then repeated for each set of attribute scales within each objective (if there

was more than one) producing lower-level weights that summed to 1. Finally, each lower-level

weight was multiplied by its corresponding objective weight to determine its effective weight in

the multi-attribute utility function. Utilities and weights can be seen in Table 5. It should be

noted that this assessment was based on the perceptions and priorities of one researcher and not

stakeholders at the TSA and aviation industry.

Table 5. Weights and Consequence Utilities at the Attribute Scale Level


Attribute Weight % Utilities


Attack

Current Successful

Attack

DARMS Unsuccessful

Attack

DARMS Successful

Attack


Fatalities 8.9 1.00 .539 1.00 .539 Injuries 6.2 1.00 .539 1.00 .539 Brch. Airport 3.4 .500 .500 .550 .550 Brch. Cabin 4.4 .500 .500 .750 .750


Avg. Wait Time

2.4 .833 .833 .858 .858

Var. Wait Time

1.6 .500 .500 .450 .450

Touch Rate 2.8 .500 .500 .600 .600 Divesture 2.0 .700 .700 .800 .800 Fairness 3.1 .722 .722 .556 .556 Security 4.1 .667 .162 .577 .062

Economic

Cost $B

13.8 1.00 .343 1.00 .270


Throughput 1.5 .500 .500 .617 .617 Cycle Time 1.5 .833 .833 .858 .858 FTE 2.2 .694 .694 .611 .611 False Alarm 2.1 .500 .500 .600 .600 TSO Utilization

.90 .500 .500 .800 .800

Responsive 1.5 .600 .600 .720 .720 Resilience 1.8 .978 .255 .989 .510

Operational

Cost $

18.4 .548 .548 .714 .714


TSO Morale 6.0 .556 .132 .374 .021 Security 4.8 .556 .132 .374 .021 Turnover Rate 4.2 .750 .500 .672 .375


Reconfiguration .80 .950 .750 .945 .700 Revenue Costs 1.5 .950 .750 .945 .700


22

Objectives Hierarchy

Figure 1 displays the seven fundamental objectives with their corresponding attribute

measures. Differing from the format in previous tables above, the objectives are displayed

according to their assessed weights (e.g., security effectiveness is shown first followed by

operational costs and so forth) and attribute scales within each objective are similarly ordered.

For example, within security effectiveness, fatalities are given the largest weight followed by

injuries, security breaches inside an aircraft cabin and security breaches inside the sterile area of

an airport. Similarly, for passenger satisfaction, perceptions of security are given the largest

weight followed by perceptions of fairness, touch rate, average wait time, divesture and lastly

variation in wait time. It should be noted that an attribute measure may have a large relative

weight within its objective but its overall influence will be also determined by the weight

assigned to the objective. For example, the attribute measure reduction of operating revenue has

a weight of .667 however the overall influence will still be relatively small because its objective

has a weight of only .023.


23

Figure 1. Objectives Hierarchy with Attribute Scales

Fatalities

0.388

Injuries

0.271

Security Breaches inside aircraft cabin

0.194

Security Breaches inside sterile Area

0.146


0.230

Operational Costs

0.184

Passenger Perceptions of Security

0.253


0.190

Passenger Touch Rate

0.177

Avg Wait Time

0.152

Divesture

0.127

Variation in Wait Time

0.101


0.161

TSO Morale

0.400

TSO Perceptions of Security

0.320

TSO Turnover

0.280


0.149

Economic Costs

0.138

Passenger FTE

0.195

Passenger False Alarms

0.182

Resilience

0.156

Passsenger Throughput

0.130

Responsiveness

0.130

Cycle Time

0.130

TSO Utilization Rate

0.078


0.115

Reduction of Operating Revenue

0.667

Reconfiguration Costs

0.333

Industry Vitality

0.023

OVERALL

1.000


24

Multi-Attribute Utilities

To combine the utilities modeled at the attribute scale level into a single multi-attribute

utility function for each of the four alternative-scenarios the weighted additive model was used.

Adding the utilities assumes that each utility can be assessed without considering consequence

levels of the other attribute scales, which can be a strong assumption, but additivity has been

shown to be a good approximation of more general multiplicative or multi-linear utility

functions; examination of stakeholders’ preferences can be addressed at a later time. As an

example, using values from Table 5, an abbreviated version of the multi-attribute utility function

for the Current approach assuming a successful attack is given in equation (1):

Utility (X) = .089 * U (fatalities) + … + .015 * U (airline operation revenue costs) = .470. (1)

The corresponding multi-attribute utility for DARMS was .492. A detailed analysis of these

inputs follows in the next section.

MAU Analysis Output

Ranking of the Four Alternative-Scenarios

Figure 2 displays the overall utilities under each of the four alternative-scenarios. From

this figure, one can see in this assessment that DARMS has a higher overall utility than the

Current approach whether there is a successful attack or not, and the difference is about the same

regardless of what attack outcome occurs. The color-coded bar chart allows a comparison of the

DARMS and Current approaches across the two attack outcomes and also shows the relative

contribution of each objective to overall utility (i.e. larger bars indicate greater contribution to

overall utility). First, one can see that most objectives have a markedly lower utility if a

successful attack occurs with the exception of operational efficiency and industry vitality which

are only somewhat lower. Also, DARMS performs better on security effectiveness, operational

costs and operational efficiency over both attack outcomes. In contrast, the Current approach

performs better on TSO satisfaction over both attack outcomes. Passenger satisfaction is about

the same for DARMS and the Current approach over both outcomes (passenger perceptions of

security within this objective fares worse under DARMS should there be a successful attack).

Economic cost is the same for DARMS and the Current approach if no attack occurs, but

DARMS performs worse if an attack occurs. In this assessment, this figure suggests that

DARMS fares better overall regardless of whether an attack occurs however it does not fare best

across all objectives.


25

Figure 2. Relative Contribution to Overall Alternative-Scenario Utilities.

Impact of Weight on Overall Utility

Figure 3 depicts the weights at the attribute scale level in order of their relative

contribution to overall utility. These weights reflect not only the relative assessed weight within

each objective but also the relative assessed weight of each objective. In this assessment,

operational costs (18.4%) and economic costs (13.8%) clearly stand out and their corresponding

consequences comprise almost a third of the overall utility. Consequences from these two

attribute scales together with fatalities (8.9%), injuries (6.2%), TSO morale (6.0%), TSO

perceptions of security effectiveness (4.8%), security breaches inside the aircraft cabin (4.5%)

and TSO turnover (4.2%) comprise two thirds of the overall utility. The consequences from these

attributes together with passenger perceptions of security effectiveness (4.1%), security breaches

inside airport sterile areas (3.4%), and passengers perceptions of fairness (3.1%) account for over

eighty percent of the overall utility-that is, half of the attributes impact 80% of the utility.

Ranking for OVERALL Objectives

AlternativeScenario

DARMS, No Successful Attack

Current, No Successful Attack

DARMS, Successful Attack

Current, Successful Attack

Utility

0.749

0.726

0.493

0.471



Industry Vitality

Operational Costs

Economic Costs




26

Figure 3. Ranking of Attribute Weights

Figures 4 and 5 display the contribution of each attribute scale to overall utility. With an

unsuccessful attack, the attribute scales that most favor DARMS are operational costs and

security breaches inside the cabin of an aircraft while TSO morale and perceptions of security

most favor the Current approach. With a successful attack, operational costs and security

breaches inside the cabin of an aircraft again most favor DARMS while economic costs and TSO

morale most favor the Current approach.

Measure

Operational Costs

Economic Costs

Fatalities

Injuries

TSO Morale



TSO Turnover





Avg Wait Time

Passenger FTE


Divesture

Resilience

Variation in Wait Time

Reduction of Operating Revenue


Responsiveness

Cycle Time


Reconfiguration Costs

Weight

18.4

13.8

8.9

6.2

6.0

4.8

4.5

4.2

4.1

3.4

3.1

2.8

2.4

2.2

2.1

2.0

1.8

1.6

1.5

1.5

1.5

1.5

0.9

0.8


27

Figure 4. Relative Contribution of Attribute Differences to overall Alternative-Scenario

Utilities with Unsuccessful Attack

OVERALL Objectives Utility for DARMS, No Successful Attack


Total Difference

0.749

0.726

0.024

Total Difference

Operational Costs


TSO Morale




TSO Turnover




Divesture

Passenger FTE

Responsiveness



Other

Difference

0.024

0.031

0.011

-0.011

-0.009

-0.005

-0.004

-0.003

0.003

0.003

0.002

0.002

-0.002

0.002

0.002

0.002

Current, No Successful Attack DARMS, No Successful Attack

Other 0.000


28

Figure 5. Relative Contribution of Attribute Differences to overall Alternative-Scenario

Utilities with Successful Attack

Illustrations of Tradeoffs

Economic Costs Equivalents. As an illustration of how to equate a set of attributes with

the value of a particular attribute, Table 6 displays tradeoffs between economic costs and the

other attributes as assessed by one of the research team. For each attribute an economic cost is

provided that is equivalent to moving from the attribute’s least preferred consequence to its most

preferred consequence. For example consider fatalities. In this assessment, the following two sets

of consequences are equally preferred by the team member providing the assessments: (400

fatalities/yr., $0 economic costs/yr.) and (0 fatalities/yr., $64.8B/yr.). In other words, there is a

willingness to incur an economic cost from a significant security breach of $64.8B per year to

reduce fatalities from 400 to 0 per year. The largest economic cost equivalent is $100B to move

from an operational cost of $9.75 per passenger screening to $6.00 per passenger screening. The

least economic cost equivalent is for airport screening reconfiguration costs. Here the economic

costs equivalent is $5.6B to move from $1B to $0B in reconfiguration costs. In terms of

economic costs tradeoffs, the top five attributes are operational costs, fatalities, injuries, TSO

morale and TSO perceptions of security effectiveness.

OVERALL Objectives Utility for DARMS, Successful Attack


Total Difference

0.493

0.471

0.022

Total Difference

Operational Costs


Economic Costs

TSO Morale


TSO Turnover


Resilience





Divesture

Passenger FTE

Responsiveness



Other

Difference

0.022

0.031

0.011

-0.010

-0.007

-0.005

-0.005

-0.005

0.005

-0.004

0.003

0.003

0.002

0.002

-0.002

0.002

0.002

0.002

Current, Successful Attack DARMS, Successful Attack

Other -0.001


29

Table 6. TSA Attribute Equivalents Expressed in Economic Costs

Objectives Attributes Units

Least Preferred

Most Preferred

Equivalent Economic Costs $B


Fatalities Number of people/year 400 0 $64.8

Injuries Number of people/year 400 0 $45.3

Breach. Airport Number of security breaches/year 90 0 $24.4

Breach. Cabin Number of security breaches/year 5 0 $32.2


Avg. Wait Time Avg. Minutes/passenger screening 20 5 $17.8

Var. Wait Time Standard Deviation in Minutes/passenger screening

10 0 $11.7

Touch Rate Percentage of passengers 10% 0% $20.7


6 1 $14.8

Fairness Constructed scale (1–10) 1 10 $22.2

Security Constructed Scale (1–10) 1 10 $29.5

Economic

Cost $B

Dollars/year


Throughput Passengers/hour

200 500 $10.8

Cycle Time Minutes/passenger screening 20 5 $10.8

FTE TSOs/passenger screening .2 .02 $16.2

False Alarm Percentage of passengers 10% 0% $15.1

TSO Utilization Percentage time TSOs engaged in screening

50% 100% $6.5

Responsive Minutes needed to reduce passenger wait time to a desired level (e.g., 10 minutes).

60 10 $10.8


48 1 $13.0

Operational

Cost $

Dollars/passenger screening $9.75 $6.00 $100.0


TSO Morale Constructed Scale (1–10) 1 10 $43.2


Turnover Rate Employee Percentage per year 50% 10% $30.3


Reconfiguration Dollars/year $1B $0B $5.6

Revenue Costs Dollars/year $1B $0B $11.1


30

Operational Costs Equivalents. Table 7 displays tradeoffs between operational costs

per passenger screening and the other attributes. In this assessment the baseline operational cost

per passenger screening is $6.00. As before, two sets of equally preferred consequences are

presented. For example, for fatalities the following two consequence sets are equally preferred

(400 fatalities/yr., $6.00) and (0 fatalities/yr., $8.43). In other words, there is a willingness to

incur an additional $2.43 per passenger screening to move from 400 fatalities per year to 0

fatalities per year. In terms of operational costs the top five attributes are economic costs,

fatalities, injuries, TSO morale and TSO perceptions of security effectiveness.

Table 7. TSA Attribute Equivalents Expressed in Operational Costs


Least Preferred

Most Preferred

Equivalent Operational

Costs

Security

Effectiveness

Baseline $6.00/screening

Fatalities Number of people/year 400 0 $8.43

Injuries Number of people/year 400 0 $7.70

Breach. Airport Number of security breaches/year 90 0 $6.91

Breach. Cabin Number of security breaches/year 5 0 $7.21

Passenger

Satisfaction

Avg. Wait Time Avg. Minutes/passenger screening 20 5 $6.66

Var. Wait Time Standard Deviation in Minutes/passenger screening

10 0 $6.44

Touch Rate Percentage of passengers 10% 0% $6.78


6 1 $6.56

Fairness Constructed scale (1–10) 1 10 $6.83


Economic

Cost $B

Dollars/year $9.75


Throughput Passengers/hour

200 500 $6.41

Cycle Time Minutes/passenger screening 20 5 $6.41

FTE TSOs/passenger screening .2 .02 $6.61

False Alarm Percentage of passengers 10% 0% $6.57

TSO Utilization Percentage time TSOs engaged in screening

50% 100% $6.24

Responsive Minutes needed to reduce passenger wait time to a desired level (e.g., 10

60 10 $6.41


31


Least Preferred

Most Preferred

Equivalent Operational

Costs

minutes).


48 1 $6.49

Operational

Cost $

Dollars/passenger screening $9.75 $6.00

TSO Job

Satisfaction

TSO Morale Constructed Scale (1–10) 1 10 $7.62


Turnover Rate Employee Percentage per year 50% 10% $7.13

Industry

Vitality

Cost $B

Reconfiguration Dollars/year $1B $0B $6.21

Revenue Costs Dollars/year $1B $0B $6.42

Overall Utility and Single Attribute Equivalents. In addition to comparing the four

alternative-scenarios in terms of utility, it can be more intuitive (perhaps) to compare them in

terms of the consequences of a single attribute. This is done by setting all other attribute scales to

their most preferred levels and then moving the attribute scale of interest to a level that produces

the same overall utility as before. That is, the level of attribute scale of interest is adjusted so that

the following equality holds true:

Utility (proposed consequence settings) = Utility (all other attribute settings at their most

preferred levels, adjusted attribute of interest level).

For example, the proposed set of consequences for an unsuccessful attack under DARMS

has an overall utility of .749. Looking specifically at economic costs, if all attribute scales are set

to their preferred level and economic costs are increased to $181.7B from a proposed level of

$0B (due to unsuccessful attack)—this adjustment also produces an overall utility of .749. The

Current approach under an unsuccessful attack has a utility of .726 and an equivalent economic

cost of $198.8B. Hence the difference in utilities from DARMS to Current approach is .023 or

equivalently $17.1B in economic costs per year. The difference in utilities from DARMS and the

Current approach under a successful attack is .022 or equivalently $15.9B in economic costs per

year. In terms of operational costs, under an unsuccessful attack the difference in equivalent

operational costs per passenger screening favors DARMS by $0.63 and under a successful attack

by $0.57 per passenger screening. In terms of fatalities, under an unsuccessful attack the


32

difference in equivalent fatalities favors DARMS by 105 fatalities per year and under a

successful attack by 98 fatalities per year.

Table 8. TSA Overall Utility and Single Attribute Equivalents: Economic Costs, Operational Costs and Fatalities


DARMS Unsuccessful

Attack


Attack

DARMS Successful

Attack

Current Successful

Attack

Overall Utility .749 .726 .493 .471

Economic Costs $B

$181.7B $198.8B $367.4B $383.3B

Operational Cost $

$12.82 $13.45 $19.77 $20.34

Fatalities 1123 1228 2270 2368

Sensitivity Analysis for Objective Weights

Security Effectiveness. It is often helpful to examine how sensitive models are to

changes in assessment inputs such as the weights. Figure 6 displays how the utility for each of

the four alternative-scenarios changes as the objective weight for security effectiveness changes.

The vertical scale is the overall utility for each of the four alternative-scenarios, the horizontal

scale is the percent weight placed on security effectiveness and the vertical bar is the proposed

weight for security effectiveness (23%). Comparing DARMS and the Current approach under no

successful attack, over the full range of weights for economic costs DARMS has a larger utility

score than the Current approach. The same is true for a successful attack.

Figure 6. Sensitivity Analysis of Security Effectiveness

Utility

Percent of Weight on Security Effectiveness Objectives

0.886

0.455

0 100






33

Passenger Satisfaction. Figure 7 displays how the utility for each of the four alternative-

scenarios as the weight for passenger satisfaction changes. The vertical bar shows the proposed

weight for passenger satisfaction (16.1%). Comparing DARMS and the Current approach under

no successful attack DARMS has a larger utility than the Current approach until the weight

approaches about 60%. For a successful attack, DARMS has a larger utility until about 50%.

Figure 7. Sensitivity Analysis of Passenger Satisfaction

Utility

Percent of Weight on Passenger Satisfaction Objectives

0.771

0.459

0 100






34

Economic Costs. Figure 8 displays how the utility for each of the four alternative-

scenarios as the weight for economic costs changes. The vertical bar shows the proposed weight

for economic costs (13.8%). Comparing DARMS and the Current approach under no successful

attack, over the full range of weights for economic costs DARMS has at least as large a utility

score as the Current approach. However, for a successful attack, as the weight for economic costs

increases past about 30% the utility for DARMS is less than that for the Current approach.

Figure 8. Sensitivity Analysis of Economic Costs

Utility

Percent of Weight on Economic Costs Measure

1.000

0.270

0 100






35

Operational Efficiency. Figure 9 displays how the utility for each of the four alternative-

scenarios as the weight for operational efficiency changes. The vertical bar shows the proposed

weight for operational efficiency (11.5%). Notice that across the full range of weights DARMS

has a larger utility than the Current approach regardless of the attack outcome.

Figure 9. Sensitivity Analysis of Operational Efficiency

Utility

Percent of Weight on Operational Efficiency Objectives

0.752

0.460

0 100






36

Operational Costs. Figure 10 displays how the utility for each of the four alternative-

scenarios as the weight for operational costs changes. The vertical bar shows the proposed

weight for operational costs (18.4%). Comparing DARMS and the Current approach, DARMS

has a higher utility than the Current approach when weight exceeds 5% regardless of the

outcome of the attack.

Figure 10. Sensitivity Analysis of Operational Costs

TSO Job Satisfaction. Figure 11 displays how the utility for each of the four alternative-

scenarios as the weight for TSO job satisfaction changes. The vertical bar shows the proposed

weight for TSO job satisfaction (14.9%). Comparing DARMS and the Current approach under

no successful attack DARMS has a larger utility than the Current approach until the weight

approaches about 25%. For a successful attack, DARMS has a larger utility until about 30%.

Figure 11. Sensitivity Analysis of TSO Job Satisfaction

Utility

Percent of Weight on Operational Costs Measure

0.766

0.443

0 100





Utility

Percent of Weight on TSO Job Satisfaction Objectives

0.800

0.120

0 100






37

Aviation Industry Vitality. Figure 12 displays how the utility for each of the four

alternative-scenarios as the weight for aviation industry vitality changes. The vertical bar shows

the proposed weight for aviation industry vitality (14.9%). Comparing DARMS and the Current

approach under no successful attack DARMS has a larger utility than the Current approach until

the weight approaches about 80%. For a successful attack, DARMS has a larger utility until

about 30%.

Figure 12. Sensitivity Analysis of Aviation Industry Vitality

Modeling Uncertainty

Probability Trees

Currently the TSA uses Transportation Sector Security Risk Assessment TSSRA

vulnerability estimates generated by experts within the TSA and airline industry. Essentially

these experts were asked what the chance would be of an adversary successfully executing an

attack for each of a large number of scenarios. They assumed the attackers had clear intent and

capability. That is, the threat level was assumed to be virtually certain for a particular flight.

These global assessments provide a useful perspective, especially when done by separate groups

of experts as was done in the TSSRA assessments.

In addition to global assessments, it is often helpful to decompose probabilities into

component parts that are easier to assess, explain to stakeholders and perhaps control through

targeted countermeasures. This study develops a probability tree that decomposes the probability

of an attack on a commercial flight into the following components: 1) how the passenger is

classified into low risk (receives expedited screening) vs. unknown risk (receives standardized

screening) in the current approach or more risk classifications in the DARMS approach; 2) how

well the countermeasures (e.g., machines, canines, BDOs) detect the threat (i.e., signal or no

signal) and 3) can the adversary successfully carry out the attack from the sterile area .

Utility

Percent of Weight on Industry Vitality Objectives

0.950

0.464

0 100






38

Figure 13. Probability Tree for Current Approach


39

Figure 14. Probability Tree for DARMS Approach


40

As shown in Figures 13 and 14, preliminary judgments were used to parameterize the

potential improvements that DARMS might have over the Current approach. For purposes of

illustration a number of assumptions were made as follows: 1) Passenger Classification.

Passengers that might pose a flight threat are much less likely to be classified as low risk and

hence less likely to receive expedited screening than passengers posing no risk; 2)

Countermeasures Detection. Passengers posing a flight threat will have higher detection rates

than passengers posing no threat, regardless of risk classification. Also, passengers classified as

an unknown risk (receive standard screening) will have higher detection rates than those

passengers classified as low risk and 3) Attack Risk. Even if a passenger posing a threat enters a

sterile area the chance of completing a successful attack is still about 50% (e.g., shoe bomber

attempt, Christmas day bomber attempt).

Probability Tree: Current Approach Assuming Passengers Present a Threat

Figure 13 portrays the probability tree for the Current approach. The following

assumptions are made:

Assuming a passenger poses a flight threat, the chance of being classified as a low risk

passenger is 1% and for those passengers posing no threat it might be about 50% (not

shown).

For those passengers posing a flight threat and classified as a low risk the probability of

being not detected is 40% while those posing a flight threat but classified as an unknown

risk has a probability of being not detected of 20%.

Assuming a passenger poses a flight threat, if they enter a sterile area they have about a

50% chance of succeeding in the attack.

Instructive for this assessment, given a true threat to a flight, the probability of a successful

attack or the flight vulnerability is about 10.1%. This probability will be important when

considering system wide risk.

Probability Tree: DARMS Approach Assuming Passengers Present a Threat

Figure 14 portrays the probability for the DARMS approach. The same assumptions and

calculations apply here as with the Current approach except for two differences. First, it is

assumed that DARMS does 2% better in terms of detecting threats and thwarting attack attempts

by passengers who pose a flight threat and who enter a sterile area. Second, consistent with the

intent of DARMS, passengers are classified into lower risk and lowest risk groups. DARMS has

plans to divide passengers into four low risk groups but only two groups are used here for

illustration. Given a credible threat to a flight the chance of a successful attack here is 9.9%

(about 2% improvement over the Current approach).

Both the current and DARMS probability trees assume there is a threat. The same

approach could be used to consider the misclassification of no-threat individuals as dangerous

(i.e., false alarm). While no probabilities were assigned Figure 15 demonstrates the structure of a

probability tree that could be used to examine the false alarm problem.


41

Figure 15. Probability Tree for the No Threat Branch

System Wide Risk

Calculating System Wide Risk. Using the probability trees for the Current and DARMS

approaches, simplifying assumptions are made to calculate the system wide probability of at least

one successful attack. Here are the assumptions:

For the same attack scenario (e.g., non-metallic body bomb), the vulnerability per flight

is similar across domestic flights in airports with sufficient resources and technology

(e.g., departure from JFK, National, O’Hare);

the probability of a successful attack in any given year is a function of flight vulnerability

and the number of credible threats per year.

Assuming Credible Threats Are Independent. To compute the probability of at least one

successful attack per year system wide, a binomial distribution was used in the following way:

1) Suppose n is the number of credible threats per year;

2) p is a common flight vulnerability, that is, the probability of an adversary defeating a

flight’s countermeasures;

3) i is the probability of nth

credible threat per year where i = 0,1, 2 …n; and

4) X is the number of successful domestic attacks per year.

Assume that credible threats are independent of each other. The probability that there are no

successful attacks in a given year is expressed by the binomial formula in equation (2):

P(X=0| n,p) = (n!/x!(n-x)!) px (1-p)

n-x . (2)

For X=0 this reduces to (1-p)n. Hence, the probability of at least one successful attack for a given

number of credible threats n is expressed by equation (3):


42

P(X1|n,p) = 1- (1-p)n (3)

Now relaxing the assumption that the number of credible threat per year n is known with

certainty, assume n can be represented by a probability distribution characterized by I where

i=1. Then the probability of at least one successful attack in a given year is expressed by

equation (4):

P(X1|n,p) ={i (1-(1-p)i ) for nx; 0 otherwise; i=0,1,2 …n. (4)

This is a recursive formula that computes the probability for a least one successful attack for up

to n credible threats. For example, the flight vulnerability for the Current is approach is assessed

to be .101 from above (10.1%). Investigating the case for up to n=2 credible threats per year,

suppose it is believed that the chances of zero threats, one threat or two threats are 10%, 40% and

50% respectively or 0= .1, 1=.4 and 2=.5. Then the probability of at least one successful attack

in a given year is expressed by equation (5):

P(X1|n,p) = 0(1-(1-p)0

+ 1(1-(1-p)1 + 2(1-(1-p)

2 = 0+.4(1-(1-.101)+ .5(1-(1-.101)

2 = .136 or a

13.6% . (5)

Using the assessed vulnerability under DARMS probability this would be about .134 or 13.4%.

Allowing Credible Threats to be Correlated. This case relaxes the assumption that the

credible threats are independent, and reconsiders the same case but where credible threats are

allowed to correlate. Drawing upon the recursive function from a generalized binomial

distribution (Drezner & Farnum, 1993) the chance of no successful attacks for a given number of

credible threats n is expressed in equation (6) where is the correlation between credible threats

and ranges from –1 to 1 and i is the number of credible threats up to n:

P(X=0)|n,p,) = (1-p)(p+(1-p))i-1

(6)

The chance of at least one successful attack in a given year is expressed by equation (7):

P(X1|n,p, ) = {i {1-(1-p)(p+(1-p))i-1

} for nx; 0 otherwise; i=1,2 …n. (7)

Notice for the case where =0 this probability function reduces to the probability expression

under the assumption of independence. For =1, the probability function reduces to p the flight

vulnerability which assumes one credible threat. Hence, with perfect positive correlation the

probability of a successful attack does not change as a function of actual number of threats. It is

as though they all operate as one threat. For all >0, P(X1|n,p, ) < P(X1|n,p). That is, if

credible threats are positively correlated the chance of at least one successful attack is bounded

above by the probability under the assumption of independence and bounded below by p under

the assumption of perfect positive correlation.. The reverse is true for <0 then P(X1|n,p, ) >

P(X1|n,p). Consider the same case as above but credible threats are positively correlated with

=.5. Now P(X1|n,p, ) = .113 or 11.3% down from the 13.6% estimate given above under the

assumption of independence. The corresponding DARMS calculations would be .112 or 11.2%.

Summary of Probability Calculations. As a way to explore how flight vulnerability

could be used to provide insight on system wide risk to domestic commercial aviation this study

began with a few simplifying assumptions: 1) Under the Current approach flight vulnerability is

similar across flights for major domestic airports. This can be determined by expert assessment

as was done in the TSSRA report or in conjunction with a probability decomposition approach as


43

was illustrated in this report; 2) The probability of at least one successful attack in a given year

depends critically on the number of credible threats per year. This information rests largely in the

hands of the intelligence community. However, the probability distribution of threats can be

incorporated into broader probability calculations and this study simply described the uncertainty

about the number of credible threats as an assessed discrete probability i. Notice, that if i =1

for any value of number of threats n then the probability expression reduces to a standard

binomial calculation for fixed number of n. Also note, there are several ways to represent this

uncertainty including assuming credible threats follow a Poisson distribution as illustrated below

in the report’s simulation results; 3) The study began with the assumption that credible threats

are independent, as is required by a standard binomial distribution, and calculations of system

wide probabilities of a successful attack were computed. Then the assumption of independence

was relaxed and a recursive expression was derived to calculate the probability of at least one

successful attack assuming credible threats are correlated. Note, it appears reasonable to think

that credible threats are most likely positively correlated. With this assumption, the probability

expressions derived in this study suggest that the probability of a successful attack will be

bounded by the above calculations assuming independence and 5) Under the DARMS approach

it is not yet clear whether flight vulnerability will be similar across flights in the way the Current

approach is. This will depend on how flight risk is adjusted system wide as a matter of policy.

It’s difficult to imagine however, that countermeasures flight-by-flight will differ so markedly

that flight vulnerability varies significantly. If vulnerability across flights does differ then the

mathematical tractability of the above expressions becomes challenging. This heterogeneity

across flights can be addressed by relaxing the assumption of common vulnerability estimate p

and correlation coefficient but this would require the aid of computing algorithms to sum

across a very large number of flights.

Assuming a Fixed Number of Credible Threats and Independence Across Threats.

This study derived a probability expression for the probability of a successful attack that

incorporates uncertainty about the distribution of threats and correlation across threats. However,

this study does not estimate values for or . Hence, for purposes of illustration it is assumed

that on average there will be 2 credible threats per year. This assumption, along with the

vulnerability estimates, suggest the chance of at least one successful attack per year is about

19.2% and 18.8% for the Current and DARMS approaches respectively. This assumes with

certainty that the number of credible threats per year will be two that is 2=1 for n=2.

Simulating the Probability of a Successful Attack. To demonstrate variation about

probability estimates Monte Carlo simulations were done. Using the vulnerability calculations

from the probability tree described above and assuming credible threats follow a Poisson

distribution the probability of at least one successful attack per year was simulated. Means of 2,

10 and 25 credible threats per year were examined. Figures 16–18 show the simulated

distributions respectively. The average probabilities of a successful attack in these distributions

were 18%, 64% and 92% but of course there is considerable variation about these means and that

is the point.


44

Figure 16. Simulation of Probability of at Least One Successful Attack with an Average of

2 Credible Threats per Year

Figure 17. Simulation of Probability of At Least One Successful Attack with an Average of


Figure 18. Simulation of Probability of At Least One Successful Attack with an Average of



45

Figure 19 displays the cumulative probability of at least one successful attack per year as

a function of number of credible threats and the correlation between threats under the Current

approach. These calculations are deterministic, that is, they assume the number of threats in a

given year is known. This graph demonstrates how quickly the probability of a successful attack

can increase as a function of number of credible threats per year. Notice that assuming

independence past 22 credible threats the probability of at least one successful attack in a year is

over 90%. At 40 credible threats the probability is about 99%. With a correlation of .5 these

probabilities would be about 70% and 88% respectively. This graph suggests that if adversaries

were to launch a large number separate attacks within minutes of each other, there is a sizable

and sobering chance they would succeed. This result speaks to the critical role of actionable

threat intelligence and deterrence. The best policy is to reduce the motivation to commit such

acts and to prevent those who are determined from reaching the domestic airports.

Figure 19. Probability of At Least One Successful Attack As a Function of Credible

Threats and Correlation Between Threats

Expected MAU Comparison of DARMS and Current Approaches

Figure 20 shows a probability tree of the comparison of Current and DARMS approaches

from a system wide perspective. The tree has been reduced to the four alternative-scenarios or

end nodes. The chance of at least one successful attack per year was computed for 2 credible

threats in a given year; these probabilities are 19.2% and 18.8% respectively. From Table 8

above, the multi-attribute utilities for the Current and DARMS approaches for an unsuccessful

attack are .726 and .749 respectively. Likewise, multi-attribute utilities for a successful attack are

.471 and .493. Thus, DARMS is preferred under either scenario. Aggregating over the two

scenarios, the expected utility for the Current approach is .677 and for DARMS is .701. In this

assessment, considering preferences and uncertainties, DARMS is the more attractive alternative.

0.000

0.200

0.400

0.600

0.800

1.000

0 20 40 60

Pro

ba

bil

ity

Credible Threats Per Year

Probability of Successful Attack

Independence

Correlation=.5

Correlation=.7

Correlation= 1


46

Figure 20. Decision Tree of Current and DARMS Approaches

Adversary MAU

Defining Objectives

Adversary objectives and attribute scales are identified in the same manner as for the TSA

described above. These are informed by Keeney and von Winterfeldt (2010) and informal

conversations with colleagues at the TSA. The objectives hierarchy is assumed to be

considerably simpler than for the TSA. Definitions of the objectives and attributes are shown in

Table 9, and scales and ranges are shown in Table 10. In this assessment there are two

objectives: Growth of the Terrorist Organization and Military Outcomes. In terms of former, the

intent is to maximize status, financial resources and support among the Muslim community. This

objective has four attribute scales:

1) status as formidable force in the middle east (1–3 constructed scale);

2) financial contributions ($0–$1M per year);

3) new recruits to attack U.S. aviation (0–120 per year) and

4) winning the hearts and minds of Muslims (1–3 constructed scale).

In terms of the latter, the intent is to maximize harm to the United States. by inflicting casualties

and economic losses as well as driving up aviation security costs and fear among the American

public. This objective has six attribute scales:

1) attacks on U.S. airports and airlines (0–5 per year);

2) U.S. fatalities (0–400 per year);

3) U.S. injuries (0–400 per year);

4) economic costs ($0–$100B per year);

5) operational costs ($0–$365M per year) and


47

6) and fear among U.S. public (0–60M per year).

Table 9. Definitions of Adversary Objectives and Attributes

Objectives/Attributes Definitions

Growth of Terrorist Organization

Maximize the status, financial resources, recruiting environment and number of Muslims in support of the organization goals and activities.

Status as formidable force in Middle East

The extent to which the organization is perceived in the Muslim world as capable of carrying out significant attacks on U.S. targets.

Financial contributions The number of dollars given to support organization in anticipation of or directly following an attack on a U.S. target.

New recruits to attack U S airports and

airlines

The number of new recruits willing to support or carry out organizational objectives in anticipation of or directly following an attack on a U.S. target.

Hearts and minds of Muslims

The number of Muslims who have favorable impression of organization’s goals and activities in anticipation of or directly following an attack on a U.S. target.

Military Outcomes Maximize the harm to the U.S. by inflicting casualties, economic consequences, increased aviation security operating costs and fear among American population.

Attacks on U.S. airports and airlines

The number of successful attacks per on a U.S. airport or airline.

American Fatalities Lives lost as direct result of an attack

American Injuries Injuries incurred as direct result of an attack

Economic Costs The sum of all economic consequences in dollars over a one-year time horizon due to attacks on U.S. airports and airline.

Operational Costs for aviation security

The sum of all costs to the TSA in dollars over a one-year time horizon due to operations relating to security effectiveness, passenger satisfaction and regulatory compliance (e.g., FTE, Equipment, FAMS, Pre-screening). Not included are the costs of responding to significant security breaches that are covered under Economic Costs.

Fear among Americans

The number of fearful Americans as direct result of an attack.


48

Table 10. Adversary Objectives, Attributes, Scales and DARMS Impacts

Objectives1 Metrics

Scale Units Predictions for DARMS Impact


Status as formidable force in middle east

Change in status/year

2

1–3 If successful attack will increase under DARMS-more significant defeat of U.S. security.

Financial contributions

Dollars/year $0–$1M If successful attack will increase under DARMS-more significant defeat of U.S. security.

New recruits to attack U.S. airports and airlines

New Recruits/year3 0–120 If successful attack will increase under DARMS-more

significant defeat of U.S. security.

Hearts and minds of Muslims

Change in number of Muslims/year

2

1–3 If successful attack will increase under DARMS-more significant defeat of U.S. security.

Military Outcomes

Successful Attacks on U.S. airports and airlines

Number of targets/year

4

0–5 Decrease under DARMS due to Game Theory countermeasures.

American Fatalities

Number of Americans/year

0–400 Possibly less number under DARMS-not sure.

American Injuries

Number of Americans/year

0–400 Possibly less number under DARMS-not sure.

Economic Costs Dollars/year $0–$100B

If successful attack will increase under DARMS due to increased public reaction.

Impact on Operational Costs for Aviation Security

Dollars/year5 $0–

$365M Minimal or no perceived impact of DARMS for adversary.

Fear among Americans

Number of fearful people/year

6

0–60M If successful attack will increase under DARMS-more significant defeat of U.S. security.

1 Based largely on the Keeney and von Winterfeldt (2010) paper.

2 Scale based on a change in status from current level. The scale is 1 for decrease in status, 2 for no

change in status and 3 for increase in status. 3 There were about 1.6B Muslims worldwide in 2010 according to a Pew study

http://www.pewresearch.org/fact-tank/2013/06/07/worlds-muslim-population-more-widespread-than-you-might-think/. Conservatively, at least seventy five percent are 18 years or older or 1.2B. Let’s suppose that following a successful attack at most one person in a million of this adult population is motivated to

http://www.pewresearch.org/fact-tank/2013/06/07/worlds-muslim-population-more-widespread-than-you-might-think/

http://www.pewresearch.org/fact-tank/2013/06/07/worlds-muslim-population-more-widespread-than-you-might-think/


49

carry out the goals of the organization or 1200. Suppose further ten percent of these have willingness and capability of attacking U.S. airports and airlines or 120. 4 Suppose out of the possible 90 security breaches inside the sterile areas of U.S. airports per year two

percent are significant or about 1. Likewise, suppose out of the possible 5 security breaches inside the aircraft cabin per year fifty percent are significant or about 3. Combined this would total 5. 5 This is the combined checkpoint reconfiguration and operational costs. In 2013 the aviation budget was

approximately $5.3B. Also, in Table 2 aviation reconfiguration and loss of operating revenue costs combine to range from $0–$2B. Together these costs would add to $7.3B. Suppose terrorist organizations with common goals are able to influence these costs by up to 5%/year then the costs would range from $0 to $365M per year. 6 According to the U.S. Census, there were approximately 235M U.S. residents 18 years and older. Let

suppose a successful attack caused 25% of this population to have a high level of fear for at least one month following the attack or almost 60M people. Data taken from U.S. Census: http://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?src=bkmk.

Table 11 displays the assessed consequences for adversary attributes for each of the four

alternative-scenarios. Consequences for status in the middle-east, financial contributions,

recruiting success and winning the hearts and minds of Muslims were based on reasonable

speculation. For example, unsuccessful attacks are predicted to result in minimal positive

consequences from the adversary’s perspective and were treated the same for the Current and

DARMS approaches. Also, consequences for a successful attack are predicted to be larger under

a DARMS approach because it would represent a more substantial defeat of U.S. security efforts.

Consequences for fatalities, injuries, and economic costs are predicted to be about the same as

the assessment from the TSA perspective given above. Successful attacks on airports and airlines

are treated as a combination of significant security breaches inside airport sterile areas and

aircraft cabins. As a simplification, the assumption is made that an adversary would most care

about a breach inside the cabin of commercial aircraft and so were treated the same as a cabin

breach under the TSA perspective. Fewer breaches per year are predicted under the DARMS

approach. Impact on security operational costs is predicted to be minimal with no successful

attacks. With a successful attack it is assumed that the adversary might hope for a five percent

increase in security costs to airports and airlines and this was predicted to be the same under the

Current and DARMS approaches. Lastly, under a successful attack the number of fearful

Americans is taken to be about twenty five percent of the adult population. Because a successful

attacks under DARMS might represent a more substantial defeat, fear is predicted to be greater

than under the Current approach.


50

Table 11. Adversary Consequences for Alternative-Outcomes Contingencies



Attack

Current Successful

Attack

DARMS Unsuccessful

Attack

DARMS Successful

Attack


Status 1 2 1 3

Financial Contributions $M

$0 $.75M $0 $1M

New Recruits 0 90 0 120

Hearts and Minds 1 2 1 3

Military Outcomes

Successful Attacks-

U.S. airports and airlines

2.5 2.5 1.25 1.25

Fatalities 0 184.25 0 184.25

Injuries 0 184.25 0 184.25

Economic Cost $B $0B $65.75B $0B $72.98B

Operational Cost $M

$0M $365M $0M $365M

Fear among Americans M

0M 50M 0M 60M

Adversary Utilities

Attribute weights and utilities are displayed in Table 12. First, the weights for adversary

attributes are assessed as they were for the TSA perspective. The most weight is given to success

in recruiting people to attack U.S targets and inflicting economic costs on the United States. Far

less weight is given to impacting operational costs than under the TSA’s perspective. Computing

the multi-attribute utility for each of the four alternative-scenarios is computed as before. The

MAU calculations for the Current approach with no successful attack is .059 and for the

DARMS approach is .029. From the adversary’s perspective the Current approach is more

attractive because of the prediction that a larger number of security breaches would occur with

this security approach. However, the reverse is true under a successful attack; the Current

approach has a utility of .630 and the DARMS approach has a utility of .794. This occurs largely

because a successful attack would represent a more substantial defeat of U.S. security efforts and

the resultant consequences would be considered to be decidedly more favorable from an

adversary’s perspective. Additionally, operational costs are given less weight here than under the

TSA perspective above. Hence, DARMS is less favored on this attribute. These results suggest


51

the need to consider treating the defender and adversary utility functions, not as a zero-sum

game, but rather as different preference functions.

Table 12. Adversary Weights and Consequence Utilities

Objectives

Attributes

Attribute

Weight

%

Current

Unsuccessful

Attack

Current

Successful

Attack

DARMS

Unsuccessful

Attack

DARMS

Successful

Attack

Growth of Terrorist

Organization

Status 11.5 .000 .500 .000 1.00

Financial

Contributions $M

8.6 .000 .750 .000 1.00

New Recruits 14.3 .000 .750 .000 1.00

Hearts and Minds 10.0 .000 .500 .000 1.00

Military Outcomes

Successful Attacks-

U.S. airports and

airlines

11.7 .500 .500

.250 .250

Fatalities 8.8 .000 .461 .000 .461

Injuries 5.9 .000 .461 .000 .461

Economic Cost $B 14.6 .000 .658 .000 .73

Operational Cost $M 4.4 .000 1.00 .000 1.00

Fear among Americans M 10.2 .000 .833 .000 1.00


52

Implementing DARMS: Challenges

From this research, two significant challenges were identified that need to be considered

(and perhaps further studied) as a DARMS-approach moves from conceptual pilot modeling to

future planned implementation:

1) the homogeneity of flights and

2) the current airport operational limitations.

Heterogeneity versus Homogeneity

A DARMS-approach is better than the current approach when flights are more

heterogeneous. If flights are homogeneous (no different values/passengers for different flights),

then the current approach and a future DARMS approach from a game-theory perspective are

equivalent in terms of impact to security. The challenge is that on most days within a single

airport, flights may be largely homogeneous. Table 13 details characteristics of flights

originating from U.S. airports, specifically describing how the flight characteristic can influence

either the probability of a successful attack from the terrorist’s perspective or the consequence of

a successful attack. Additionally, some characteristics of the flight will not be knowable to the

adversary in advance (such as FFDO or FAM on-board) and some may be only knowable to the

TSA with very short notice. These characteristics are labeled in Table 13.

The differences among flights from one airport are: the destination city, the aircraft type

(correlated with destination city), the airline, the departure gate, the time of day, and the travelers

and crew on-board the flight. Some airports have multiple TSA screening areas, so in some

airports that would also be a difference. Finally, cargo could also create differences but was not

considered in this phase of the study.

Based on current perceptions of terrorist values, it seems unlikely that one airline would

be targeted versus another (i.e., Delta versus United). Additionally, departure gate is only

relevant for those airports that have multiple TSA screening areas, where passengers may receive

different screening services. Neither of these factors will create heterogeneity among flights.

The destination city and aircraft type will vary by flight but in a common pattern every

day (i.e., American Airlines uses a 737 aircraft for the daily 6:30pm IAD-LAX flight). So

screening algorithms that focus resources on larger airplanes flying to higher “valued”

destinations would still be fairly constant day-to-day.

The remaining significant difference among flights is the travelers and crew on-board.

Some will increase the risk (i.e., selectees) and some will decrease the risk (i.e., FAM, FFDO,

and armed federal and local law enforcement officers). Selectees already receive more screening,

but algorithms could increase screening for others on flights with selectees.

Because the travelers are the major significant difference for a flight, more research is needed to

determine how passengers can be further categorized into a finer gradient of risk, since this is

where a DARMS-approach can have value above the current screening methods.


53

Table 13. Differences Across Flights

Implication to probability of success from terrorist perspective

Implication to consequences

Knowable to adversary in advance

Knowable to TSA in advance

Other notes

Origin City Yes if smaller airports have lesser security, or overseas originated

Not separate from aircraft type (i.e., if airport only has commuter flights) unless symbolic value

Yes Yes Constant for each airport being considered

Destination City No Yes if some cities are higher value target

Yes Yes

Aircraft Type (# of passengers)

No Yes – bigger number of passengers

Yes but imperfectly – last minute aircraft substitutions

Yes Correlated with origin – destination cities, bigger planes on hub-to-hub and coast-to-coast flights

Airline No No – assuming no one airline is higher value target

Yes Yes

Passengers in different risk categories (connecting vs original, traveling military)

Yes – some passengers increase risk (connecting, high risk), some passengers lower risk – military personnel

No – unless some elaborate assassination attempt

Only those within group

Yes

Armed passengers (fed and local law enforcement)

Yes – reduce risk

No No Yes but short notice

FFDO (armed flight crew)

Yes – reduce risk

No No Yes but may be short notice


54

Implication to probability of success from terrorist perspective

Implication to consequences

Knowable to adversary in advance

Knowable to TSA in advance

Other notes

FAM Yes – reduce risk

No No Yes

Time of Day Yes – if more or less crowded security affects screening

No Yes Yes

Departure Gate/Concourse

No (not separate from different screening)

No Yes – may change

Yes – may change

Correlated with airline

Different screening area (if airport has different areas)

Depends on screening

No Yes – may change

Yes – may change things

Would be airport specific, DCA has separate areas by concourse

Current Operational Limitations

A member of the research team visited BWI airport and examined the separate TSA

screening checkpoints (A, B, C, and D) guided by Todd Trafford (Deputy Federal Security

Director, BWI Airport) and Kristen Best (Operations Improvement Branch, Office of Security

Operations). A significant challenge for implementing a future DARMS-approach will be

maintaining positive passenger control between the TDC (Travel Document Checker) and the

screening machines. If machines are intended to screen passengers based on an algorithmically

determined risk level, then when the passenger places their baggage on the X-ray and then either

pass through the AIT or the walk-through metal detectors, the screening devices need to know

the appropriately risk level for each individual and adjust based on risk.

As currently configured at BWI, pier D would need to be completely reconstructed. Pier

D currently has no queue area. The line wraps down the hallway. On the day of our visit, they

had capacity for 7 lanes but are losing 2 shortly for a few years because of airport remodeling to

create a concourse D/E connector. Once passengers pass the TDC, they enter a huge crowd of

travelers waiting to be sorted into lanes to approach the belt. This area appeared much like a

“mosh pit” at a pop concert. Positive passenger control would be close to impossible with the

current design unless passengers re-identified themselves at the screening machines.

On the other hand though, pier C is the newest and most efficient check point of the four

at BWI. This checkpoint has 9 lanes (they were operating 7 at our visit). Pier C has a great deal

of queueing space and correspondingly very little waiting between the TDC and X-ray belts and


55

passenger screening machines. Pier C could more easily be able to do some degree of positive

passenger control.

Finally, at pier A, the TSA is testing a CAT (credential authentication terminal). The

CAT is capable of providing the TDC with much more information about the passenger after

scanning their identification. Completing the piloting of such technology and deploying it in the

field will be a critical step in implementing a DARMS-approach because TDC will potentially

need more information that can be discretely encoded on a boarding pass to determine approach

screening based on risk.

Conclusions

Summary

TSA Objectives and Attributes. Seven fundamental objectives are identified based on

discussions with colleagues at the TSA and a brief review of the security literature. These were:

1) security effectiveness; 3) passenger satisfaction, 4) economic costs of a significant security

breach; 4) operational efficiency; 5) TSO job satisfaction; 6) operational costs and 7) aviation

industry vitality. Twenty eight performance measures were proposed for consideration and

twenty four were used in the MAU analysis. Likewise, attribute scales are developed and

consequence ranges are assessed for each. Objective and attribute weights are assessed from one

member of the research team as an illustration. The most weight is given to attributes such as

operational costs, economic costs of security breaches and measures of security effectiveness

such as fatalities and injuries.

Consequences and Utilities. For contingency successful attack and an unsuccessful

attack scenario, consequences were assessed for the Current approach and DARMS using

suggestions from colleagues at the TSA and discussions among the research team members.

DARMS fared better on all measures of security effectiveness and almost all measures of

operational efficiency and operational costs regardless of the outcome of significant security

breaches. The Current approach fared better on variation of passenger wait time and perceptions

of fairness and security, TSO resources (FTE), all measures of TSO job satisfaction and

measures of aviation industry vitality regardless of the outcome of significant security breaches.

Focusing on economic costs favors the Current approach should there be significant security

breach. Overall, regardless of whether a significant security breach occurs, DARMS had a higher

utility.

Calculating System Wide Probability of a Successful Attack. It is assumed that the

cumulative probability of a successful attack would follow a binomial distribution and be a

function of flight vulnerability to attack and the number of credible threats per year system wide.

As an illustration, assuming on average two credible threats, the chance of at least one successful

attack per year under the Current approach is 19.2%. Note, if both the uncertainty surrounding

number of credible threats and the possible correlation among these threats are considered than

this estimate would be smaller. On the other hand, assuming independent threats and certainty

Figure 19 above graphically portrays how the chance of a successful attack increases as a

function the number of credible threats. Past twenty two threats per year the upper bound chance

of at least one successful attack climbs to over 90%.

Adversary Objectives, Attributes and Utilities. Two objectives are identified for an

adversary: 1) growth of the terrorist organization and 2) military outcomes. Ten performance


56

attributes are used in the MAU analysis. Likewise, attribute scales were developed and

consequence ranges were assessed for each. Objective and attribute weights are assessed from

the perspective of one of the research team as an illustration. The most weight is given to

attributes such as success in recruiting members to attack U.S. targets and inflicting economic

costs on the United States. When comparing the Current and DARMS approaches, it is noted

with an unsuccessful attack the Current approach was more attractive, while with a successful

attack the DARMS approach is more attractive. This difference is based on a prediction that

defeating DARMS would represent a more significant victory for an adversary than the defeating

the Current approach. It also suggests the need to represent the utility function of the TSA and

adversary in a game theory context as distinct preference functions.

Caveats

This study was undertaken as a proof of concept, that DARMS could be compared to the

Current approach on a number of dimensions important to the TSA. The comparison was done at

a strategic level not tactical level. The analysis provided is not meant to provide definitive

conclusions about DARMS or the probability of a successful attack on the domestic commercial

aviation. Instead it is intended to describe a general approach to studying these issues and layout

the kinds of information, assessments and challenges that would be involved in a more complete

analysis. There were a number of simplifying assumptions that were made that need to be

addressed in future analyses.

Representing Outcomes and Consequences. The probability trees decompose a flight’s

vulnerability to a successful attack into meaningful components. This decomposition is intended

to articulate a number of uncertainties that might be useful in exploring threat levels, and

classification and detection (or false alarm) rates. The assessed probabilities are primarily

intended as an illustration, but the calculated vulnerability is quite similar to that provided by

TSA and industry experts in other contexts.

The probability trees represent passengers posing a flight threat and attack outcomes are

defined as unsuccessful attack and successful attack. This representation presumes there is some

level of threat to a flight, that is, an adversary with motivation and capability had intended to

defeat the security system connected with a particular flight. Because this study does not specify

the criminal actions, it is left undefined as to what an unsuccessful attack means apart from the

system was not defeated. In future analyses it would be important to carefully define such actions

because that would make for a more meaningful assessment of outcome consequences. For

example, even a passenger discovered outside the passenger security area with a bomb would

generate significant public response and produce economic costs. Additionally, a successful

attack is treated in the aggregate but in fact has a number of consequence levels. For example,

there are different consequences for a significant security breach inside the airport, a breach

inside the aircraft cabin and the complete loss of an aircraft.

In this study, the assessment of outcome consequences makes no distinction between no

attack and unsuccessful attack. In actuality there might be a large difference between these two,

and this should be addressed carefully in future work.

Representing Attribute Weights and Utilities. Weights were assessed by one of the

research team members with careful attention to consequence ranges and conversations with

colleagues at the TSA. However, this is not a good substitute or proxy for a careful elicitation of


57

the many stakeholders connected with comparing the Current and DARMS approaches.

Likewise, this study assumed a risk neutral attitude for all measures. In future work this

assumption should be examined. For example, it might well be that for many attributes the TSA

is risk neutral but there may be other attributes for which the TSA is risk averse or even risk

seeking. Finally, the study also assumes the multi-attribute utility functions are additive. That is,

preferences on individual attribute utilities are independent of levels on other attributes. In

practice, this assumption is often made as a simplifying approximation but in future work this

should be examined carefully.

Representing Probabilities. To illustrate how to calculate the system wide chances of at

least one successful attack on a commercial airline per year three assumptions are made: 1) the

flight vulnerability to attack is uniform across a wide range of flights in airports large enough to

have sufficient resources and technology (e.g., assumption might not apply to flights between

small cities like Boise and Cedar Rapids with less access to technology); 2) the number of

credible threats per year can be assessed (e.g., i) or perhaps follow a Poisson distribution and 3)

the system wide probability was calculated using a binomial distribution using estimates of

vulnerability and number of credible threats. The binomial distribution assumes credible threats

are independent. That is, the presence of one credible threat does not influence the likelihood of

witnessing other threats in a particular year. It is not clear this assumption would hold. For

example, several criminal cells could be coordinating attacks. This study addressed this

assumption by deriving a probability expression that allows for correlated threats by drawing

from the work a generalized binomial distribution (Drezner & Farnum, 1993). In this study, the

probability calculations primarily serve to illustrate an approach to calculating the system wide

probability of a successful attack and to demonstrate this probability is highly sensitive to the

number of credible threats in a given year. This speaks to the importance of deterrence.

Future Research

This study has outlined the basic types of information and analyses that could be helpful

in examining how DARMS compares with the Current approach and other alternative aviation

security approaches as articulated for example by the Deloitte team working inside the Office of

the Chief Risk Officer. What is needed in future work is to follow up on this foundation as

follows:

Fundamental Objectives and Performance Attributes. The most important task

moving forward is to achieve agreement and buy in throughout the TSA about the objectives and

performance metrics to be used to guide strategic decisions about DARMS and a number of

other related projects. Taking the list of objectives and attribute measures described here as a

starting point, stakeholders throughout the TSA and within the aviation industry should be

involved in a discussion about the relevance and completeness of the proposed list. On reflection,

additional performance attributes may be suggested and others discarded as not useful. Likewise,

it would be helpful to get greater clarity and agreement on attribute definitions. Getting

consensus on definitions is critical and also very difficult. During discussions with TSA

colleagues it was apparent that opinions differed on issues as fundamental as how to measure

security effectiveness.

With an agreed upon list of objectives and attributes the next step is to further develop the

scales used to measures these attributes. In this study some scales were natural such as number of

fatalities during a significant security breach and customer wait time in minutes. As such the


58

consequence ranges were straightforward to think about. Other scales such as operational costs

and economic costs were in dollars. The consequences here were also straightforward but being a

composite of several factors more analysis is needed to appropriately represent the ranges. For

example, how will indirect economic consequences from public reaction be computed? Still

other scales such as passenger perceptions of security and TSO morale were constructed and may

need to be calibrated with other behavioral measures to adequately understand consequence

ranges. The CREATE risk perception and risk communication team has done some survey work

on passenger perceptions that might be helpful in this regard.

Once consequence scales and ranges are determined, key stakeholders can be queried

regarding the relative weights given each attribute. In this study a member of the research team

used swing weighting to assess attribute weights and assumed independence of attributes.

However, a more through elicitation procedure should be used involving stakeholders at the

TSA. Preference dependencies should be examined to determine the actual structure of the multi-

attribute utility function.

Uncertainty and Outcomes. In this study uncertainty has been represented largely in

terms of probability trees that account for classification rates, detection (false alarm) rates and

ability to execute an attack once in a sterile area of an airport or aircraft. This offers a reasonable

approach to decomposing flight vulnerability into understandable and perhaps manageable

components. This procedure should be pursued with more depth with a more careful assessment

of branch probabilities generated by subject matter experts inside the TSA. Also, detection rates

were thought about very broadly here but the set of countermeasures that would materially

influence the detection rates for a particular type of threat (e.g., non-metallic explosive carried on

body) should be specifically described.

The definition of a successful attack should be developed with more fidelity than was

done in this study. While it is true that any significant criminal activity inside a airport sterile

area or aircraft could be construed as a successful attack these outcomes need to be defined more

carefully. For example, attacks carried out in the airport may have different consequences than

an equivalent attack inside the aircraft while in flight. What about near misses in which security

officials skillfully thwart an attack in progress? Also, do attempted attacks that were

unsuccessful produce the same consequences as no attempted attacks? Again, this may be

context dependent. An attempted attack that is decisively and skillfully thwarted may have

temporary impacts in terms of economic and operational costs but have long term benefits in

terms of deterrence. To properly understand the implications of different security protocols these

issues need to be investigated.

In this study consequences for the four alternative scenarios were assessed by the

research team using three factors as a guide: 1) the attribute consequence range, often a

midpoint; 2) unsuccessful versus successful attack and 3) Current versus DARMS approach.

Experts within the TSA should be queried about these consequences and a more nuanced

conversation should take place about the predicted efficacy of the DARMS approach over the

Current approach on certain critical attributes.

System wide probability of a successful attack against commercial aviation was

examined with a number of simplifying assumptions. These were preliminary calculations to

identify and better understand the possible factors driving this probability. Binomial distributions

were used assuming independent and correlated credible threats. As a start this analysis was


59

useful because it demonstrates how sensitive the chance of a successful attack is to number of

credible threats per year. This observation points to the importance of properly assessing the

distribution of threats.

Adversary Objectives and Attributes. The adversaries represented here were thought to

be religious extremists, organized and possess sufficient resources to be credible threats. Other

types of adversaries (e.g., lone wolf) would have different priorities and this should be

investigated. The objectives and attributes were identified and scaled in much the same manner

as for the TSA. Some attributes such as fatalities and number of new recruits had natural scales

while others such as status in the Middle East had constructed scales. As mentioned, constructed

scales need to be calibrated to better understand what impact the consequences might have. For

these, it might be useful to go to the literature on motivations behind terrorism for insight.

References

Burns, W. J., & Dillon-Merrill, R. (2013a). Public perception of TSA security checkpoint

procedures and safety: Results from a nationwide survey. Presentation at Transportation

Security Administration Headquarters on August 1, Pentagon City, Arlington County,

Virginia.

Burns, W. J., & Dillon-Merrill, R. (2013b). Public perception of TSA security checkpoint

procedures and safety: Results from two recent and independent nationwide surveys.

Presentation at Transportation Security Administration Headquarters on December 12,

Pentagon City, Arlington County, Virginia.

Burns, W. J., Slovic, P., John, R., Rosoff, H., Rose, A., Avetisyan, M., Chan,O. & Sellnow, T.

(2013). Public risk perception surveys: Risk perception, public confidence and economic

impacts of risk amplification following a terrorist event. Report delivered to

Transportation Security Administration on June 30, Pentagon City, Arlington County,

Virginia.

Clemen, R. T., & Reilly, T. (2001). Making hard decisions with decision tools. Pacific Grove,

CA: Duxbury.

Deloitte Team. (2015). DARMS Data Call Overview. Powerpoint slides describing DARMS

business case, January 21, 2015.

Drezner, Z., & Farnum, N. (1993). A generalized binomial distribution. Communications in

Statistics: Theory and Methods, 22, 3051–3063.

http://dx.doi.org/10.1080/03610929308831202

Eisenfuhr, F., Weber, M., & Langer, T. (2010). Rational decision making. Berlin, DE: Springer–

Verlag.

Giesecke , J. A., Burns, W. J., Barrett, A., Bayrak, E., Rose, A., Slovic, P., & Suher, M. (2012).

Assessment of the regional economic impacts of catastrophic events: CGE analysis of

resource loss and behavioral effects of a RDD attack scenario. Risk Analysis, 32, 583–

600. http://dx.doi.org/10.1111/j.1539-6924.2010.01567.x

Giesecke, J., Burns, W. J., Rose, A., Barrett, A., & Griffith, M. (2014). Regional dynamics under

adverse physical and behavioral shocks: The economic consequences of a chlorine

http://en.wikipedia.org/wiki/Pentagon_City






60

terrorist attack in the Los Angeles financial district. In P. Nijkamp, A. Rose, & K. Kourtit

(Eds.), Regional science matters: Studies dedicated to Walter Isard. Berlin, DE:

Springer–Verlag. http://dx.doi.org/10.1007/978-3-319-07305-7_16

Keeney, G. L., & von Winterfeldt, D. (2010). Identifying and structuring the objectives of

terrorists. Risk Analysis, 30, 1803–1816. http://dx.doi.org/10.1111/j.1539-

6924.2010.01472.x

Keeney, R. L., & von Winterfeldt, D. (2011). A value model for evaluating homeland security

decisions. Risk Analysis, 31, 1470–1487. http://dx.doi.org/10.1111/j.1539-

6924.2011.01597.x

Morral, A. R., & Jackson, B. A. (2009). Understanding the role of deterrence in

counterterrorism security. RAND Corporation occasional paper series. Santa Monica,

CA: RAND Corporation. Retrieved from

http://www.rand.org/content/dam/rand/pubs/occasional_papers/2009/RAND_OP281.pdf

Price, C., & Forrest, J. (2013). Practical aviation security: Predicting and preventing future

threats. Waltham, MA: Butterwork-Heinemann.

Rose, A., Avetisyan, M., Chan, O., Rosoff, H., Burns, W. J., & Slovic, P. (2015). The role

of behavioral responses in the total economic consequences of terrorist attacks on U.S.

air travel targets. Manuscript submitted for publication.

Transportation Security Administration (TSA). (2015). Mission statement. Official website of the

U.S. Department of Homeland Security. Retrieved from http://www.tsa.gov/about-

tsa/mission on February 12, 2015.

von Winterfeldt, D., & Edwards, W. (1986). Decision analysis and behavioral research. New

York, NY: Cambridge University Press.

von Winterfeldt, D., & O’Sullivan, T. (2006). Should we protect commercial airplanes against

surface-to-air missile attacks by terrorists? Decision Analysis, 3, 63–75.

Dynamic Aviation Risk Management System (DARMS): A...

Documents

Transcript of Dynamic Aviation Risk Management System (DARMS): A...