THE EFFECTIVENESS OF MULTI-CRITERIA INTELLIGENCE MATRICES IN INTELLIGENCE ANALYSIS

LINDSEY N. JAKUBCHAK

A Thesis

Submitted to the Faculty of Mercyhurst College

In Partial Fulfillment of the Requirements for

The Degree of

MASTER OF SCIENCEIN

APPLIED INTELLIGENCE

DEPARTMENT OF INTELLIGENCE STUDIESMERCYHURST COLLEGE

ERIE, PENNSYLVANIAMAY 2009

i

ii

DEPARTMENT OF INTELLIGENCE STUDIESMERCYHURST COLLEGE

ERIE, PENNSYLVANIA

THE EFFECTIVENESS OF MULTI-CRITERIA INTELLIGENCE MATRICES IN INTELLIGENCE ANALYSIS

A ThesisSubmitted to the Faculty of Mercyhurst CollegeIn Partial Fulfillment of the Requirements for

The Degree of

MASTER OF SCIENCEIN

APPLIED INTELLIGENCE

Submitted By:

LINDSEY N. JAKUBCHAK

Certificate of Approval:

_________________________________Kristan J. WheatonAssistant ProfessorDepartment of Intelligence Studies

_________________________________James BreckenridgeChairman/Assistant ProfessorDepartment of Intelligence Studies

_________________________________Phillip J. BelfioreVice PresidentOffice of Academic Affairs

MAY 2009

iii

iv

Copyright © 2009 by Lindsey N. JakubchakAll rights reserved.

v

“The work of managers, of scientists, or engineers, of lawyers-the work that steers the course of society and its economic and government organizations-is largely work of making decisions and solving problems. It is work of choosing issues that require

attention, setting goals, finding or designing suitable courses of action, and evaluating and choosing among alternative actions.”

-Nobel Laureate Herbert Simon

“Life is the sum of all your choices. History equals the accumulated choices of all mankind.”

-Albert Camus

vi

DEDICATION

This work is dedicated to:

-My parents for their continued support and confidence in me and everything I have done throughout my life.

-My friends who have greatly impacted the person that I am today.

-My classmates who are constantly pushing me to achieve new levels of academic and professional excellence.

vii

ACKNOWLEDGEMENTS

I would like to acknowledge my thesis advisor and primary reader, Kristan Wheaton, for

his guidance throughout the thesis process and throughout my graduate studies at

Mercyhurst College.

I would like to thank Professor Hemangini Deshmukh for her assistance with the

statistical analysis on this work.

I would also like to thank Justin Hiskey for his assistance with editing my thesis, and for

his overall support throughout my graduate studies

Finally, I would like to acknowledge Joshua Peterson for making his work available to

me, and for his advice throughout the process.

viii

ABSTRACT OF THE THESIS

The Effectiveness of Multi-Criteria Intelligence Matrices in Intelligence Analysis

By

Lindsey N. Jakubchak

Master of Science in Applied Intelligence

Mercyhurst College, 2009

Professor Kristan J. Wheaton, Chair

While there is a substantial body of literature related to the use and effectiveness

of Multi-Criteria Decision Making (MCDM) in the general sense, there is none regarding

the use of this method in intelligence analysis. This study discusses relevant literature on

the conventional form of MCDM and addresses the differences necessary to convert

MCDM to an intelligence methodology (tentatively titled Multi-Criteria Intelligence

Matrices-MCIM). Additionally, a controlled experiment was conducted to test the

hypothesis that MCIM is a valuable method to use in intelligence analysis. Findings

suggest that MCIM is likely a valuable method to use when conducting analysis, though

more research is recommended to confirm these results.

ix

TABLE OF CONTENTS

Page

COPYRIGHT PAGE …………………………………………………………... iii

QUOTES………………………………………………………………………... iv

DEDICATION…………………………………………………………………. v

ACKNOWLEDGMENTS……………………………………………………… vi

ABSTRACT……………………………………………………………………. vii

TABLE OF CONTENTS………………………………………………………. viii

LIST OF FIGURES…………………………………………………………….. xi

CHAPTER

1 INTRODUCTION……………………………………………… 1

2 LITERATURE REVIEW……………………………………….

Intuition vs. Structured Methodology……..……………. Benefits of Structured Methodology in Intelligence…….

Testing Structured Methodology in Intelligence…….…..MCDM………………………...…………………………The MCDM Process ……………………….……….…...

Validity of MCDM……………………………..…….…..Converting MCDM to MCIM……………………………Hypotheses ………………………………………………

4

468912182425

3 METHODOLOGY………………………………………………

Research Design………………………………………..Selection of Subjects…………………………………...

Recruitment of Subjects………………………………..Process…………………………………………………

Control Group…………………………………………. Experiment Group…………………………….............. Data Analysis Procedures……………………………..

27

27282830303133

x

4 RESULTS………………………………………………………..

Pre-Test Findings……..……………………………….....Post-Test Findings……………………………….……....Product…………………………………………...………Process……………………………………………………

Timeliness………………………………………………... Analytic Confidence…………………………………….. Objectivity and the Incorporation of Alternative Analysis………….. Logical Argumentation…………………………………. Bullet Point Analysis……………………………………. Potential of Future Use…………………………………..

36

363840424345

46464748

5 CONCLUSIONS………………………………………………...

Pre-Test and Post-Test Conclusions……………………... Process and Product Conclusions……………………….. A Step Forward…………………………………………..

50

505253

BIBLIOGRAPHY………………………………………………………………. 56

APPENDICES………………………………………………………………….. 59

APPENDIX 1: Analytic Methods Experiment Sign-Up Form….. 60

APPENDIX 2: Institutional Review Board Proposal Form…….. 61

APPENDIX 3: Participation Consent Form. …………………… 66

APPENDIX 4: Experiment Section 1 Handout (Control Group).. 67

APPENDIX 5: Analytic Methods Experiment Links…………… 69

APPENDIX 6: Pre-Test Questionnaire (Control and Experiment Groups) ………………………………………… 70

APPENDIX 7: Post-Test Questionnaire (Control Group)…………………….……………………. 72

APPENDIX 8: MCDM Participation Debriefing……………….

74

APPENDIX 9: MCDM Lecture Notes………………….……… 75

APPENDIX 10: Experiment Section 2 Handout (Experimental

xi

Group)………………………………………… 77

APPENDIX 11: Post-Test Questionnaire (Experimental Group)…………………………………………

APPENDIX 12: Statistical Data………………………………..

LIST OF FIGURES

80

82

xii

Page

Figure 1.1 General Example Matrix Used in MCIM 13

Figure 1.2 Step Two in the MCIM Process 15

Figure 1.3 Steps Three and Four in the MCIM Process 16

Figure 1.4 Steps Five and Six in the MCIM Process 17

Figure 1.5 Completed Example Matrix 18

Figure 3.1 Subjects by Class Year 29

Figure 3.2 Demographic Breakdown by Experiment Group 33

Figure 4.1 Control Group Results 41

Figure 4.2 Experiment Group Results 41

Figure 4.3 Length of Analysis Completion Time 44

Figure 4.4 Levels of Analytic Confidence 45

Figure 4.5 Bullet Point Analysis 48

xiii

1

CHAPTER 1:

INTRODUCTION

We all make decisions on a daily basis. In fact, we often make so many decisions

that we may fail to recognize we are even doing so; it is an act that has become second

nature to us. Some decisions, such as what we choose to eat for breakfast in the morning,

or which road we use to travel to work, are typically made with our “gut feeling” or

intuition. While these decisions may require the balancing of multiple factors (how

hungry we are or how much time we have), they require little forethought or analysis.

These decisions are not life altering, and they do not generally have a long-term affect on

us. On the other hand, decisions such as what kind of car we buy, or what career path we

embark on, are decisions that are affected by multiple criteria and may alter our life in

some way. In the field of intelligence, decisions made by others can affect matters of

national security, law enforcement and business operations. These decisions are directly

related to the safety and survival of both nations and organizations; therefore, they

mandate appropriate analysis.

Since the ability to make the “right” decision or predict an adversary’s most likely

course of action (COA) can have heavily influence the society in which we live, it is

imperative that an analyst makes sure that he or she fully grasps the relevant information

to provide effective analysis to a decision maker. One method of ensuring appropriate

evaluation of significant criteria, as well as the assessment of suitable alternatives, is

through the use of a structured methodology, such as MCDM.

2

MCDM is a generic term used to encompass a broad range of analytical

methodologies that use matrices as the basis for their conclusions. More specifically,

MCDM is an internally focused decision support method that assesses possible COAs

based on the evaluation of multiple criteria, goals, and objectives of conflicting nature. It

provides substance to a decision maker, as it allows for selecting an option based on the

appropriateness of those alternatives weighted against one another.

Given recent intelligence failures, substantial emphasis has been placed on

improving intelligence analysis and capabilities. In the Vision 2015 document, the

Director of National Intelligence specifically addresses the need to improve the

Intelligence Community (IC) and “create decision advantage.”1

One way to create decision advantage, as well as improve competitive analysis,

and counterintelligence capabilities, is by gaining insights into the thoughts of one’s

competitor or adversary through the use of Multi-Criteria Intelligence Matrices (MCIM).2

MCIM is a twist to the conventional form of MCDM, as it focuses not on an

organization’s own COA but rather on the likely COA of others outside the organization.

This method, once validated in an intelligence environment, might promote a more

efficient way of structuring intelligence data and may increase an analyst’s ability to

learn what is happening with his or her adversary or the environment in which they are

working in. Additionally, it may ensure that an analyst examines all relevant COAs

allowing for more thorough analysis of a situation. Finally, it may help an analyst escape

analytic pitfalls, and provide more credibility to an analyst’s work by providing a logical,

transparent process to a decision maker in the form of an easy to understand matrix.1 United States Government, Director of National Intelligence, Vision 2015, Available at

http://www.dni.gov/Vision_2015.pdf; Internet (Accessed 20 March 2009).2 MCIM is used to make the distinction between externally focused matrix analysis and MCDM,

or internally focused matrix.

3

While there is a substantial body of literature related to the use and effectiveness

of MCDM in the general sense, there is none regarding the use of this method in

intelligence analysis. The purpose of this experimental research is to determine the

effectiveness of using MCIM as an analytical methodology in the field of intelligence.

Currently no research has been conducted on the topic, and this study hopes to take the

first steps of many in increasing an analyst’s ability to thoroughly examine all possible

COAs in a timely manner as well as accurately forecast what COA one’s adversary is

likely to choose.

4

CHAPTER 2:

LITERATURE REVIEW

Some individuals will always argue that nothing can compete with the use of an

analyst’s “gut feeling” or intuition to make decisions, or that on occasion, there just isn’t

time to apply a structured methodology to an intelligence problem. After all, most

analysts are working under a deadline and cannot afford to incorporate time-consuming

methodologies that are difficult to use. However, in failing to implement a structured

method of problem solving, an analyst may be sacrificing quality analysis.

Intuition vs. Structured Methodology

In Michael R. LeGault’s book, Think: Why Crucial Decision Can’t Be Made in

the Blink of an Eye, he suggests that individuals should use their knowledge and abilities

to make informed decisions, rather than base decisions merely on intuition or impulse. 

He further stresses the importance of critical or complex thinking by stating, “It can be

used simply to dig into things to enhance one’s awareness and discernment.”3 This quote

can be directly related to the analytic process in intelligence. When an analyst draws

conclusions based on their own experiences or intuition, he or she may fail to account for

conditions or factors in their adversary’s environment that may lead to a different

conclusion. By systematically reviewing all pertinent information, through the use of a

structured analytic method, an analyst may increase his or her chance of creating accurate

analytic judgments.

3 LeGault, Michael R, Think: Why Crucial Decision Can’t Be Made in the Blink of an Eye, New York, Threshold Editors, 2006, Page 305.

5

Ernest Forman, Professor of Management Science at George Washington

University, also highlights the benefit of structured thinking over intuitive thinking. He

states, “Although the vast majority of everyday decisions made intuitively are adequate,

intuition alone is not sufficient for making complex, crucial decisions.”4 He argues that

“organizations that use modern decision support methods can gain and maintain a

competitive edge in leading and managing global business relationships that are

influenced by fast changing technologies and complicated by complex interrelationship

between businesses and governments.”5 Forman’s position highlights the need for

teaching and implementing competitive decision making techniques, or structured

analytic methods.

Like Forman, an additional argument by LeGault demonstrates the importance of

teaching structured methodology early on in an analyst’s career. He states, “It [critical

thinking] is a skill that every individual is born with; however, it requires development

upon laying a foundation.”6 Therefore, if an analyst learns how to effectively use

analytical methodologies in the early stages of his or her career, they may expand and

develop the critical thinking skills necessary to make better decisions and more accurate

forecasts throughout their career. It is also important to recognize here that intuitive

decision making is not always available or beneficial to entry-level intelligence

professionals. These individuals often lack experience in their field, thus suggesting the

need for a more organized approach to deal with complex problems, in order to produce

better analysis.

Benefits of Structured Methodology in Intelligence4 Forman, Ernest, Decision by Objectives (How to Convince Others That You Are Right, Available

at http://mdm.gwu.edu/forman/DBO.pdf; Internet.5 Ibid.6 LeGault, Page 32.

6

Numerous experts in the field of intelligence have discussed the potential benefits

of using structured methodology in analysis. According to Stephen Marrin, “the primary

value of analytic techniques or structured methods is that they provide a way to account

for the analytic judgment; an analytic ‘audit trail’ as it were.”7 Thus, thru the

implementation of a structured methodology, the analyst’s thought process becomes more

transparent to others. Marrin also states, “With an analytic audit trail, analysts and their

colleagues can discover the sources of analytic mistakes when they occur and evaluate

new methods or new applications of old methods. In this way, structured methods make

it possible to advance the analytic tradecraft.”8 In other words, if the desired outcome

was not achieved, an analyst or decision maker can review what went wrong, or learn

how to approach and evaluate problems better in the future.

According to Richards Heuer, the use of structured methodology in intelligence

analysis may also help to prevent analytic pitfalls, such as the satisficing strategy or

groupthink.9 One structured methodology in particular, the Analysis of Competing

Hypothesis (ACH), requires analysts to weigh alternatives, in order to reach some type of

conclusion. Additionally, ACH provides a structured outline as to how an analyst

reached a specific conclusion, thus providing substance to a decision maker regarding his

or her estimates.

In regards to ACH, Heuer that stated the following benefits the methodology:

7 Marrin, Stephen, “Intelligence Analysis: Structured Methods or Intuition,” American Intelligence Journal, Page 7, Summer 2007.

8 Ibid, Page 7.9 Heuer, Richards, Psychology of Intelligence Analysis [book on-line] (Langley, Virginia: Central

Intelligence Agency Center for the Study of Intelligence, 1999), Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf; Internet, Page 109.

7

This procedure [ACH] leads you through a rational, systematic process that avoids some common analytical pitfalls. It increases the odds of getting the right answer, and it leaves an audit showing the evidence used in your analysis and how this evidence was interpreted. If others disagree with your judgment, the matrix can be used to highlight the precise areas of disagreement. Subsequent discussion can then focus productively on the ultimate source of the differences.10

Thus, structured methods such as ACH provide analysts with a more efficient way to

approach and organize complex problems

Structured methodology may also help an analyst record all relevant pieces of

information. Sometimes, ideas and thoughts that analysts do not write down may get lost

or forgotten about. In fact, George A. Miller stated that “the number of things most

people can keep in working memory at one time is seven, plus or minus two.” 11 Thus, by

implementing a structured method, an analyst would have an organized way to keep track

of information so that it is not lost or forgotten.

Structured methodology in intelligence may also facilitate an analyst’s

understanding of how pieces of a problem are related. Heuer stated the following

regarding coping with complexity in analysis: “put all the parts down on paper or a

computer screen in some organized manner such as a list, matrix, map, or trees so that we

and others cans see how they interrelate as we work with them.”12 Thus, Heuer is

10 Heuer, Richards J., Psychology of Intelligence Analysis [book on-line] (Langley, Virginia: Central Intelligence Agency for the Study of Intelligence, 1999), Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf; Internet, Page 109.

11 Miller, George. A., “The Magic Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information,” The Psychological Review, Vol. 63, No. 2 (March 1996). As cited by Heuer, Richards Jr. in “Taxonomy of Structured Analytic Techniques,” Paper prepared for the International Studies Association 2008 Annual Convention, March 26-29, 2008, San Francisco, CA.

12 Heuer, Richards, Jr., “Taxonomy of Structured Analytic Techniques,” Paper prepared for the International Studies Association 2008 Annual Convention, March 26-29, 2008, San Francisco, CA, Available at http://www.allacademic.com//meta/p_mla_apa_research_citation/2/5/4/1/2/pages254125/p254125-1.php; Internet; (Accessed 20 November 2008).

8

affirming the idea that structured methodology may provide some organized way for

analyst to visualize and evaluate data and see how pieces of criteria relate to each other.

Testing Structured Methodology in Intelligence

An experiment conducted in 2000 by MSgt Robert D. Folker, Jr. of the United

States Air Force (USAF) specifically addressed the use of structured methodology in

intelligence. The base of Folker’s research revolved around an experiment designed to

test the effectiveness of structured methodology in providing “correct” intelligence

analysis, by comparing it to intuitive analysis.13

Folker's subjects were various students at the Joint Military Intelligence College

(JMIC), which he divided into two groups: a control group and an experimental group.

His experimental group was trained on a structured methodology and asked to

incorporate that methodology in their analysis, while the control group was not asked to

use a specific methodology. Folker gave both groups the same two hypothetical

intelligence scenarios based off of facts from a real world problem. By forming the basis

of his experiment around an event that actually took place, Folker was able to assess an

analyst’s ability to pick the “correct” COA. Upon review of the results, Folker’s found

that the group of analysts, who used structured methodology in their analysis, provided

significantly better analysis than the group who used no formal structured methodology.14

He also concluded that these results were not linked to “rank, experience, education or

13 Folker, MSgt Robert D, Jr., “Intelligence Analysis in Theater Joint Intelligence Centers: An Experiment in Applying Structured Methods,” Joint Military Intelligence College, Washington DC, NDIC Press 2000. Available at http://www.fas.org/irp/eprint/folker.pdf; Internet, Page 15

14 Ibid, Page 32

9

branch of service.”15 Folker further stated, in order to be effective, intelligence analysts

must be properly trained on the specific type of methodology.16

Folker’s research was important as it reinforced the idea that the implementation

of structured methodology can assist analysts in breaking down problems and producing

better analysis.

MCDM

In the Psychology of Intelligence Analysis, Richards Heuer, also distinguishes

between merely “sitting down and thinking about a problem and really analyzing a

problem.”17 Like many others, he suggests that in order to fully analyze a complex

problem, an individual must break it down into its’ smaller components, evaluate each

portion, and then piece the subcomponents back together to make a decision.18 One

method of doing just that is through the use of decision support systems such as MCDM.

As noted earlier, MCDM is a generic term used to encompass a broad range of

analytical methodologies that use matrices as the basis for their conclusions. More

specifically, MCDM is an internally focused decision support method that assesses COAs

based on the evaluation of multiple criteria, goals, and objectives of conflicting nature.

Prior to demonstrating the validity of MCDM, as well as judging the effectiveness of

MCIM, it is necessary to begin by providing a brief overview of the traditional process.

As noted earlier, the conventional process of MCDM has numerous variations.

15 Folker, Page 3216 Folker, Page 3217 Heuer, Richards, Psychology of Intelligence Analysis, [book on-line] (Langley, Virginia: Central

Intelligence Agency Center for the Study of Intelligence, 1999), Available from https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf: Internet, Page 94.

18 Ibid, Page 94.

10

Dr. Thomas Saaty developed the Analytic Hierarchy Process (AHP), one variation

of MCDM. AHP allows decision makers to structure decisions hierarchically to

determine which one most suits their needs. Typically, the overall goal or decision to be

made, is stated at the top of the hierarchy, criteria used evaluate the alternatives are in the

middle, and the various COA are listed at the bottom. In regards to the benefits of his

methodology Saaty states:

The AHP is about breaking a problem down and then aggregating the solutions of all the sub problems into a conclusion. It facilitates decision making by organizing perceptions, feelings, judgments, and memories into a framework that exhibits the forces that influence a decision. In the simple and most common case, the forces are arranged from the more general and less controllable to the more specific and controllable.19

Another variation of MCDM is the WSM. According to research conducted by

Evangelos Triantaphyllou, the WSM is “probably the most commonly used approach,

especially in single dimensional problems.”20 Additionally, Triantaphyllou states that this

method works best “in single dimension problems where all the units are the same, (e.g.

dollars, feet, seconds)” and “difficulty arises when it is applied to multi-dimensional

decision making problems.”21

The WPM is another popular MCDM variation that is similar to the WSM model.

In fact, the main factor that distinguishes the WPM from the WSM is that the WPM

19 Saaty, Thomas L, “How To Make A Decision: The Analytic Hierarchy Process,” Available at http://sigma.poligran.edu.co/politecnico/apoyo/Decisiones/curso/Interfaces.pdf; Internet (Accessed 24 March 2009).

20 Ibid, Page 6.21 Triantaphyllou, Evangelos and Stuart H. Mann, “An Examination of the Effectiveness of Multi-

Criteria Decision Making Methods: A Decision Making Paradox,” Available At http://www.csc.lsu.edu/trianta/index.html?http://www.csc.lsu.edu/trianta/Books/DecisionMaking1/Book1.htm; Internet (Accessed 20 March 2009).

11

applies multiplication to criterion, in order to compare alternatives, where as the WSM,

uses addition.22

The staff study is also a MCDM variant that is commonly studied and utilized in

the military community. As outlined in the United States Army Field Manual (FM) 101-

5, Staff Organizations and Operations, there are several types of military briefs including

information, decision, mission and staff.23 FM 101-5 states that the decision brief,

designed to arrive at the solution to a problem/decision, “requires that an analyst be

prepared to present his assumptions, facts, alternative solutions, reason for choosing the

recommended solution, and the coordination involved.” The staff study format, outlined

in Appendix D of FM 101-5,24 provides the means to do this by outlining multiple COAs,

reviewing pertinent criteria, and evaluating the most appropriate solution.

22 Triantaphyllou, Evangelos and Panos M. Parlos (ed.), Multi-Criteria Decision Making Methods: A Comparative Study, The Netherlands: Kluwer Academic Publishers, 2000, Page 8.

23 The United States Army. Field Manual 101-5. Staff Organization and Operations. 31 May 1997. Page 115. Available at http://www.fs.fed.us/fire/doctrine/genesis_and_evolution/source_materials/FM-101-5_staff_organization_and_operations.pdf; Internet.

24 Ibid, Page 105.

12

The MCDM Process

Although MCDM is an umbrella term used for a collection of methodology types,

they all involve the idea of decision alternatives, or COAs, determined by numerous

criteria weighted against each other. In MCDM, emphasis is on placing value, or

judgment of an item’s worth and desirability, on pieces of criterion, a standard on which

a judgment or decision may be based.25 By providing a breakdown of the conventional

MCDM process, I hope to demonstrate that this methodology is relatively easy to

understand, and when used properly, it is an effective way to approach complex

problems.

For this research, I have chosen to use a MCDM model derived from the Army’s

staff study. This model seemed like an appropriate choice given its simplicity and ability

to accommodate a large amount of unstructured data, which is common to intelligence

analysis.

In general, the basic parts of the MCDM process can be broken down into the

following eight steps:

1. Requirement/Question and Collection2. Establish Possible COAs3. Establish Screening Criteria4. Screen COAs5. Establish Evaluation Criteria6. Weight Evaluation Criteria7. Evaluate COAs8. Make Recommendations

25 Webster’s Dictionary Online, Available at http://www.merriam-webster.com/dictionary/criteria; Internet.

13

Requirement/Question and Collection

The first step in MCDM, establishing a clearly defined question, is essential. An

analyst simply cannot have a good analysis without a clearly defined question. In this

step, it is equally important that the analyst or decision maker form the question in a way,

which will elicit possible COAs. For the purpose of working through the conventional

process of MCDM, I have chosen to explain each step by addressing the question of

which vehicle to buy.

MCDM normally uses a matrix to display the results of the analysis. The general

matrix used in MCDM normally looks like the one below, where COA represents the

various options realistically open to the decision maker and criteria represents the various

relevant ways in which a decision maker will judge each COA.

Figure 1.1 Example of a General Matrix Used in MCDM

 Question: Which kindof vehicle should I buy?

Criteria 1:

Criteria 2:

Criteria 3:

Criteria 4:

Total/ (RankOrder):

COA 1:          

COA 2:          

COA 3:          

COA 4:          

COA 5:          

COA 6:          

14

Establish Possible COA

The second step in the MCDM process is establishing relevant possible COAs.

According to numerous experts, there are two categories for further classifying MCDM

problems based on the possible COAs: Multi-Objective Decision Making (MODM) and

Multi-Attribute Decision Making (MADM).26 The primary difference between these two

categories is that with MODM problems, the decision alternatives are endless; whereas,

MADM problems focus on distinct decision spaces, or predetermined COAs.27 In the

case of deciding what car to buy, the example question falls in line with the idea of

MODM, as there are endless possibilities of cars for an individual to choose from, and an

individual is not required to choose from pre-determined COAs. In this step, it is

important for an analyst to identify as many COAs as possible, since there is no “correct”

number of alternatives. The range of possible COAs merely derives from the extent of

the problem and the creativity of the analysts in solving the problem.

The following example matrix outlines six types of vehicles that an individual

may purchase. It is important to note that this is not an all inclusive list, and is merely an

abridged list for the purpose of demonstrating the use of matrices.

26 Triantaphyllou, Evangelos and Panos M. Parlos (ed.), Multi-Criteria Decision Making Methods: A Comparative Study, The Netherlands: Kluwer Academic Publishers, 2000, Page 1.

27 Ibid, Page 1.

15

Figure 1.2 Step Two in the MCDM Process

Question: Which kindof vehicle should I buy?

Criteria 1:

Criteria 2:

Criteria 3:

Criteria 4:

Total/ (RankOrder):

COA 1:Family compact carCOA 2:Mini VanCOA 3:Sports carCOA 4:Station wagonCOA 5: SUVCOA 6:Motorcycle

Establish Screening Criteria and Screen Criteria

After completion of the second step in the MCDM process, an individual might be

overwhelmed at the plethora of COAs available; therefore, he or she must begin to

eliminate certain COAs by establishing screening criteria. It is necessary to distinguish

this step separate from the previous step, which involved establishing possible COAs.

On occasion, individuals subconsciously rule out possible COAs, without even

recognizing they are going through a screening process. For example, when choosing

which car to buy, a student may not even consider options that may be out of range for

either financial reasons (too costly) or due to appearance, (it is not a dream car),

thus,ruling out possible alternatives before having the opportunity to compare it against

other COAs.

In the example case, screening criteria for our aforementioned problem may be

“must be vehicle two wheeled vehicle.” By establishing “must have/be” guidelines for

COAs an individual can quickly eliminate unlikely possible COAs. Based on the

16

previous evaluation criteria, the possible COA, “motorcycle” may be eliminated as it is

not a four wheel vehicle. The following matrix shows the elimination of one possible

COA (a motorcycle), after implementing the screening criteria.

Figure 1.3 Steps Three and Four in the MCDM Process

 Question: Which kindof vehicle should I buy?

Criteria 1:

Criteria 2:

Criteria 3:

Criteria 4:

Total/ (RankOrder):

COA 1:Family Compact CarCOA 2:Mini VanCOA 3:Sports CarCOA 4:Station WagonCOA 5: SUVCOA 6:Motorcycle

X X X X X

Establish Evaluation Criteria and Weight Evaluation Criteria

The next steps in the process, establishing evaluation criteria and weighting the

evaluation criteria, allows the analyst to provide rank to the remaining COAs. At this

point, the analyst uses evaluation criteria to establish that “all things being equal, this is

the best COA for me." The analyst should develop a system of rank that stresses the

significance of one criterion over another. The analyst can provide rank as either a

tangible means, things that he or she can measure, or by intangible means, things that he

or she cannot easily measure.

Evaluation criteria for the stated problem may be specific, “must cost less than

$25,000,” or it may be generalized, “with all other things being equal, the most

17

inexpensive car is the best.” This step involves evaluating what an analyst or decision

maker really cares about, or what is most important in achieving the desired outcome.

Additionally, this step allows an analyst to provide a system to weight the most important

criteria to them. For example, if the individual was a student of limited financial means,

price may be the most important criteria when looking to buy a vehicle. Therefore, the

price category will be multiplied by two, in order to demonstrate its significance. The

following matrix outlines the criteria, while applying weight and rank order:

Figure 1.4 Steps Five and Six in the MCDM Process

 Question: Which kindof vehicle should I buy?

Criteria 1:PriceCheaper isBetter(x2)

Criteria 2:SizeRoom forfamily

Criteria 3:SafetyForFamily

Criteria 4:AppealAttractiveLook

Total/ (RankOrder):

COA 1:Family Compact Car

1 (2) 4 1 2 9

COA 2:Mini Van

2 (4) 1 2 4 11

COA 3:Sports Car

5 (10) 5 5 1 21

COA 4:Station Wagon

3 (6) 3 3 5 17

COA 5: SUV

4 (8) 2 4 3 17

Evaluate COA and Make Recommendations

After establishing evaluation criteria, it is necessary to provide a system of rank

order to the possible COAs by establishing a scale. When addressing our example

question, this task could be as simple as ranking the cars most likely to purchase, to least

18

likely to purchase, by utilizing a scale from 1 to 3, 1 to 5, or 1 to 10. The following

completed matrix highlights the rank order of the possible COAs:

Figure 1.5 Completed Example Matrix

 Question: Which kindof vehicle should I buy?

Criteria 1:PriceCheaper isBetter(x2)

Criteria 2:SizeRoom forfamily

Criteria 3:SafetyForFamily

Criteria 4:AppealAttractiveLook

Total/ (RankOrder):

COA 1:Family Compact Car

1 (2) 4 1 2 9 (1)

COA 2:Mini Van

2 (4) 1 2 4 11 (2)

COA 3:Sports Car

5 (10) 5 5 1 21 (5)

COA 4:Station Wagon

3 (6) 3 3 5 17 (3)

COA 5: SUV

4 (8) 2 4 3 17 (3)

After ranking the possible alternatives, the individual now has a structured format,

which provides substance to their decision. The family compact car appears to be the

most idea vehicle, followed by the minivan, station wagon and SUV, and sports car.

Upon completion of the first MCDM analysis, one COA that may be chosen is a

sports car. It is important to note, that upon determining the first ideal COA, a second

matrix analysis may be completed in order to determine the more specific type of sports

car.

Validity of MCDM

19

In his book, Multi-Criteria Decision Making Models: A Comparative Study

(2000), Evangelos Triantaphyllou explores some of the variations on MCDM to

determine what the most “ideal” variation is to use. Since it was not possible to include

every possible MCDM variation, Triantaphyllou focused on the more popular methods,

such as AHP, WSM, and the WPM, noted above. Although unrealistic to determine the

most “ideal” method to use in a given situation, Triantaphyllou’s extensive research of

the variations of MCDM provides strong evidence that there is a plethora of methods

available which allow an efficient and organized approach to effective decision making.

Triantaphyllou also points out the irony in his research commenting, “Deciding on which

one is the best method can be viewed as an MCDM problem itself whose solution

requires the use of the best MCDM.”28

Valerie Belton and Theodor Stewart also provide an excellent overview of the

various types of MCDM models and theories that have developed over the past 25 years,

in their book Multiple Criteria Decision Analysis: An Integrated Approach (2002). The

authors reinforce the idea that although MCDM cannot guarantee a “correct” answer, the

methodology should “facilitate decision makers’ learning about the understanding of the

problem faced, about their own, other parties’ and organizational priorities, values and

objectives and through exploring these in the context of the problem to guide them in

identifying a preferred COA.”29 The book concludes with the authors’ opinion regarding

the future of MCDM, specifically stating that integration of various perspectives and

28 Triantaphyllou, Evangelos, “Can We Always Determine the Right Alternatives in Business Problems,” 18 August 2002.

29 Belton, Valerie and Theodor J. Stewart, Multiple Criteria Decision Analysis: An Integrated Approach, The Netherlands: Kluwer Academic Publishers, 2002, Pages 2-3.

20

identifying common strengths and weaknesses, is essential to the methodology’s “growth

and success.”30

Aside from the research surrounding MCDM as outlined in the aforementioned

academic books, MCDM has multiple applications in the real world, contributing to such

fields as finance and economics, environmental management and marketing.31 In his

article “Decision making with the analytic hierarchy process,” Saaty justifies the

application of AHP by citing examples where the methodology has been used in the real

world.

The following examples are just a few of the ones outlined by Saaty, and are

taken directly from his publication:

British Airways used the AHP in 1998 to choose the entertainment system vendor for its entire fleet of airplanes.

In 2001, the methodology was used to determine the best relocation site for the earthquake devastated Turkish city Adapzari.

Xerox Corporation has used AHP to allocate close to a billion dollars to its research projects.

In sports, it was used in 1995 to predict which football team would to go to the Superbowl and win. The AHP was also applied in baseball to analyze which players should be retained on a team.

In 1999, the Ford Motor Company used the AHP to establish priorities for criteria that improve customer satisfaction. Ford gave Expert Choice Inc, an Award for Excellence for helping them achieve greater success with its clients.32

The application of MCDM problems has also surfaced in financial planning

journals. In the March 2008 Journal of Financial Planning, William Z. Suplee and

Steven R. Dzubow discuss the importance of using structured methodology in helping

30 Belton and Stewart, Pages 333-343.31 Zopounidis, Constantin and Doumpos, Michael, “Multiple-Criteria Decision Making,” Thomson

Corporation, 2006. 32 Satty, Thomas , “Decision making with the analytic hierarchy process,” Int. J. Services

Sciences, Vol. 1, No. 1, 2008, Available at http://inderscience.metapress.com/media/pgwf2qtuyg3xnvnhxnby/contributions/0/2/t/6/02t637305v6g65n8.pdf; Internet (Accessed 26 March 2009).

21

their clients make critical financial decisions. In their article, “Using Multiple-Criteria

Decision Analysis to Simply the Financial Planning Process,” they demonstrate the value

of using matrix analysis by a parent and child deciding on the best college for the child to

attend through the process of evaluating a plethora of criteria including tuition costs,

graduation rate, campus life, student/teacher ratio and post-graduation placement.33

In addition to financial journals, the relevance of MCDM has also appeared in

environmental management journals. In their article, the “Application of Multicriteria

Decision Analysis in Environmental Decision Making,” in The Integrated Environmental

Assessment and Management Journal, Gregory A. Kiker, Todd S. Bridges, Arun

Varghese, Thomas P. Seager and Igor Linkov discuss the application of MCDM in

various environmental projects. The authors highlight numerous environmental case

studies, by outlining the specific MCDM method used, the decision context (a range of

topics in the areas of environmental management, stakeholder involvement and

contaminated sites) and the funding agency (a range of universities, as well as US and

international government agencies).34 This article is mentioned as it demonstrates the

flexibility and use of MCDM in a multiplicity of environmental topics conducted by an

array of organizations.

A surplus of research and studies has also been conducted as to the application

and relevance of MCDM methods in engineering. In the fields of civil and

environmental engineering, the Elimination and Choice Translating Reality (ELECTRE)

33 Dzubow, PhD and William Z. Suplee IV, CFA, CFP, ChFC, CAS, “Using Multiple-Criteria Decision Analysis to Simplify the Financial Planning Process,” Journal of Financial Planning. March 2008, Available at http://www.library.idsc.gov.eg/GUI/Globals/Upload/BULLETIN_ATTACHMENT/92/e-files/manegement%20and%20economics/using%20multiple.pdf; Internet ( Accessed 5 December 2008).

34 Kiker, Gregory A., Todd S. Bridges, Arun Varghese, Thomas P. Seager and Igor Linkov. “Application of Multicriteria Decision Analysis in Environmental Decision Making.” Integrated Environmental Assessment and Management. Volume 1, Number 2, pages. 95-108. 2005, Available athttp://www.allenpress.com/pdf/ieam-01-02_95_108.pdf; Internet. (Accessed 20 November 2008).

22

II and III, are cited as two MCDM variations that are widely used.35 While conducting

his research in regards to MCDM in engineering, Xiaoting Wang mentions several past

case studies in which MCDM methods were applied. Once again, the variety of topics

covered highlights the flexibility in the application of MCDM problems. Some of the

real life case studies utilized in Wang’s research included “choosing a solid waste

management system [Hokkanen, J., and P. Salminen (1997)], choosing an alternative fuel

system for land transportation [Poh, K.L., and B.W. Ang, (1999), and selection of an

alternative electricity power plant [Leyva-Lopes, J.C., and E. Fernandez-Gonzalez

(2003)].”36

In their article “Multi-Criteria Decision Making Process for Buildings,” J.D.

Balcomb and A. Curtner demonstrate MCDM’s value in the building design planning

process. The authors suggest that by evaluating criteria such as life cycle cost,

architectural quality, functionality, air quality, and maintainability, various building

designs can be evaluated to ensure maximum sustainability. The authors indicate that as

fundamental decisions are being made, the methodology has the ability “to assist design

teams in prioritizing their goals, setting performance targets and evaluating design

options to ensure that the most important issues affecting building sustainability are

considered.”37 Additionally, the paper highlights the various strengths of the

methodology including ease of use, its inexpensive nature, its ability to document how

35 Wang, Xiaoting, “Study of Ranking Irregularities When Evaluation Alternatives By Using Some ELECTRE Methods and a Proposed New MCDM Method Based on Regret and Rejoicing,” A Thesis Submitted to the Graduate Faculty of the Louisiana State University and Mechanical College. August 2007. Available online at http://etd.lsu.edu/docs/available/etd-07112007-012708/unrestricted/Wang_thesis.pdf; Internet (Accessed 24 March 2009), Page iv.

36 Ibid, Page 25.37 Balcomb, J.D. and A. Curtner, “Multi-Criteria Decision-Making Process for Building,” paper to

be presented at the American Institute of Aeronautics and Astronautics Conference, Las Vegas, Nevada, July 24-28. 2000. Available online at http://www.nrel.gov/docs/fy00osti/28533.pdf; Internet (Accessed 23 March 2009), Page 1.

23

the team arrived at conclusions, as well as its ability to provide visual representation of

the team’s thought process which can be interpreted by almost anyone.38

Research conducted by Stacy Gilchrist, Master of Science in Applied Intelligence

from Mercyhurst College, examined the use of the military staff study in intelligence by

transforming the conventional problem solving process into an intelligence-focused

methodology.39 For the purpose of his research, Gilchrist applied the methodology to the

following question regarding the Mara Salvatrucha gang (MS-13): “How will MS-13

respond or react to the current law enforcement efforts led by the Immigration and

Customs Enforcement (ICE) branch of the Department of Homeland Security (DHS),

specifically Operation Community Shield?”40

Gilchrist noted that first and foremost, the intelligence-focused staff study was

decision maker focused and provided a means to track the analyst’s thought process.41 A

second strength noted of the intelligence-focused staff study by Gilchrist is that the

process is structured and organized.42 A final strength that Gilchrist noted was that the

intelligence-focused staff study produced objective analysis, stating that cognitive biases

are likely to occur less frequently, due to the individual evaluation of each criterion in

regards the overall objective.43 Gilchrist also noted possible weaknesses including the

potential for the methodology to be both time-consuming and contain flawed results.44

38 Balcomb, J.D. and A. Curtner, Pages 2 and 8.39 Gilchrist, Stacy. “Staff Study,” (Research Paper for Advanced Analytic Techniques,

Mercyhurst College).40 Ibid, Page 3.41 Ibid, Page 1.42 Ibid, Page 343 Ibid, Page 1.44 Ibid, Page 1.

24

Gilchrist’s findings are important as they address the issues that many analysts

face when working with structured methodology. Additionally, Gilchrist’s research

provides valuable insight to the use of the intelligence-focused MCIM matrix.

Converting MCDM to MCIM

Since the conventional form of MCDM has proven to be valuable, why not

replicate the process, while making slight modifications, in order to create a valuable

structured methodology that may lead to the production of better intelligence analysis?

The only difference with using the methodology in intelligence is that the analyst would

apply the method to an external entity-to best predict the likely COA an enemy or

competitor will choose.

MCIM would likely have several benefits for intelligence analysts. Just as Heuer

states in regards to ACH, although the use of MCIM can never guarantee a “correct”

answer, it may increase the odds of choosing the COA one’s adversary or competitor is

most likely to pursue, based on a thorough breakdown of all relative pieces of

information.45 Additionally, the analyst can provide the decision maker with a supporting

matrix that outlines their thought process, which may give more credibility to their work.

The matrices can also highlight the areas of disagreement or conflict between two

individuals, by identifying the weight of importance that an analyst places on each

45 Heuer, Richards J., Psychology of Intelligence Analysis [book on-line] (Langley, Virginia: Central Intelligence Agency for the Study of Intelligence, 1999), Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf; Internet (Accessed 11 November 2008), Page 109.

25

individual criterion.46 Finally, a decision maker or analyst can adjust the matrix to

accommodate new data and “what-if” scenarios.

Assuming the role of an external entity would also benefit intelligence analysis.

In fact, this is something the army has already been doing for years through the Red

Team methodology, a tactic frequently used to increase the odds of predicting how one’s

opponent is likely to react in a specific situation.47 Additionally, in his book, Strategy-

Specific Decision Making: A Guide for Executing Competitive Strategy, William Forgang

states that “external analysis has the ability to deter an organization from pursuing an

option they might have otherwise chosen to accept.”48 Thus, by using structured methods

with an external focus such as MCIM, an analyst might gain insight to factors outside of

a decision maker’s immediate surroundings, and possible deter them from costly or poor

decisions they would otherwise have chosen to pursue.

Hypotheses

The following experiment seeks to demonstrate that MCIM can prove itself as a

useful intelligence analysis methodology. Specifically, I hypothesized the following:

Hypothesis 1: The experimental group using MCIM to conduct analysis will

have a more accurate product than the students in the control group.

46 Heuer, Richards J., Psychology of Intelligence Analysis [book on-line] (Langley, Virginia: Central Intelligence Agency for the Study of Intelligence, 1999), Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf; Internet, Page 109.

47 2008 U.S. Army Posture Statement. Information Papers: Red Team Education and Training, Available at http://www.army.mil/aps/08/information_papers/prepare/Red_Team_Education_and_Training.html; Internet (Accessed 26 September 2008).

48 Forgang, William G., Strategy-Specific Decision Making: A Guide for Executing Competitive Strategy. M.E. Sharpe, Inc., Armonk, New York, 2004.

26

Hypothesis 2: The analytical process for the experimental group using MCIM

will be more transparent than the control group.

Hypothesis 3: The experimental group using MCIM will complete their analysis

faster than the control group.

Hypothesis 4: The experimental group using MCIM will have a higher level of

analytic confidence in their product than the control group.

Hypothesis 5: The experimental group using MCIM will be more objective

during the analytic process and examine more alternative possibilities.

Hypothesis 6: The experimental group using MCIM will provide better logical

argumentation than the control group.

27

CHAPTER 3:

METHODOLOGY

In order to test the given hypotheses, I designed an experiment to assess whether

or not MCIM methodology improves intelligence analysis. When the experiment was

completed, I evaluated the analyst’s process and product, including the length of time it

took the student-analysts to complete analysis and their level of analytic confidence. This

methodology section will detail the research design of this experiment.

Research Design

The experiment I conducted broke the subjects into two groups. One of the

groups was an experimental group (comprised of 24 student-analysts), and one was a

control group (comprised of 21 students-analysts). I gave the control group a 10 minute

instruction/information period regarding the experiment tasking, and then asked them to

complete an analysis of a real life intelligence scenario. I taught the experimental group

the method of MCDM/MCIM over the course of approximately 35 minutes, gave them a

10 minute instruction/information period regarding the experiment tasking, and then

asked them to provide an analysis regarding the same real life intelligence scenario. The

only difference between the two groups was that the experimental group received

instruction regarding MCDM/MCIM and then was asked to complete the analysis using

that methodology. I gave the two groups access to the same resources and provided them

the same amount of time to complete their analysis.

28

Selection of Subjects

I used students at the Mercyhurst College Institute for Intelligence Studies

(MCIIS) as my research subjects. Mercyhurst College, located in Erie, Pennsylvania, is

the originator of a four-year program specifically designed to prepare students for entry-

level careers in business, law enforcement and national security intelligence analysis

Additionally, Mercyhurst College offers a master’s program, with a thesis requirement.

Since the students of Mercyhurst College are training to be efficient analysts, they could

arguable serve as a proxy for entry-level analysts in the intelligence community, who

would have been difficult, (if not impossible), to recruit due to security clearances and

access problems.

Recruitment of Subjects

In order to recruit participants for my experiment, I approached several professors

in the intelligence department asking for permission to recruit student-analysts during

their designated class time. Upon receiving their approval, and approximately one and a

half weeks prior to the start of my experiment, I gave a brief oral presentation at the

beginning of several classes that included information regarding the time, place and

subject matter of the experiment. The students were informed that the experiment would

be conducted during the second week of winter term. The presentation also indicated that

several professors in the department would be offering extra credit for their class if

students chose to participate in the experiment. Offering extra credit may have persuaded

several students to participate when they otherwise might not have. Upon agreeing to

participate, each student-analyst was handed a sign-up sheet (see Appendix 1) and asked

29

to choose the appropriate date and time that they were available to participate. The

students could choose from either a Wednesday or Thursday night time slot based on

their availability or preference. The time frame on both days was kept the same so that

the student-analysts were unable to determine which group was the control group and

which group was the experimental group. If students did not have a preference for

availability, they selected a third choice, in which they were randomly assigned a day to

complete their experiment.

The experiment was open to all students in the intelligence department, including

both undergraduate and graduate students (See Figure 3.1 for breakdown by of the

subjects by class year). A total of 45 students participated in experiment. Although the

experiment was open to all class years, no freshmen participated in the experiment.

Figure 3.1 Subjects by Class Year

30

Process

Prior to conducting my experiment, I needed approval from the Mercyhurst

College Institutional Review Board (IRB). The IRB requires any student anticipating

conducting an experiment involving human subjects to submit a proposal for review (See

Appendix 2). The purpose of the experiment proposal is to outline the experiment and

identify any possible risks, if any, which participants may come into contact with as a

result of participating in the experiment.

After receiving approval from the IRB, I also had to secure consent from

participants in the experiment. At the beginning of each experiment session, I outlined

what the experiment entailed. I then provided each student-analyst with a participation

consent form (See Appendix 3). The consent form asked for the student’s name, class

year, and name of their professor/professors, in order for me to pass along the appropriate

information to award them extra credit. Before starting each group’s section of the

experiment, I answered any questions that the subjects may have had regarding the

experiment. The process of each group was carried out the same way, only adjusting

those items necessary to test the effectiveness of the methodology.

Control Group

The first group of participants that I used to test my hypotheses on was the control

group. When the sessions started, I provided the group with a brief set of instructions and

provided them with a handout regarding a real life intelligence scenario. The handout

provided the subjects with room to write their analysis, as well as a section to identify

their level of analytic confidence (See Appendix 4). I provided the subjects with a

31

handout containing a list of websites to use as a starting point for their research (See

Appendix 5). I informed the students that these links were merely a starting point for

their analysis, and it was their choice if they wanted to use the links for information. I

asked the students to carefully read through the problem and gain an understanding of the

requirements. I then handled any questions they had regarding the process of the

experiment. After gaining an understanding of the expectations, and prior to completing

their tasking, I asked the subjects to complete a pre-test questionnaire and provide brief

demographic information, both of which I used to further compare the experimental and

control groups (See Appendix 6). I then gave the group two hours to complete their

analysis using whichever sources and method they preferred. Upon completing their

analysis, I asked the control group to complete a post-test questionnaire (See Appendix

7), and then provided them with a debriefing form (See Appendix 8).

Experimental Group

The second group of participants that I used to test my hypothesis was the

experimental group. At the beginning of the session, the experimental group received a

35 minute long presentation on MCDM/MCIM methodology, where the subjects worked

through examples of MCDM/MCIM problems. In addition, the group received a handout

on MCDM/MCIM that they were able to refer back to later on in the experiment (See

Appendix 9). I gave the experimental group the same 10 minute instruction period as the

control group, regarding the tasking. In addition, I provided them with a handout (See

Appendix 10) which asked for each subject to provide analysis for the same intelligence

related problem within a period of two hours. The only difference between the two

32

groups was that I asked the experimental group to complete their analysis using MCIM.

The handout provided room for the student-analysts to write their analysis and to indicate

their level of analytic confidence; however, it also instructed each subject to create an

MCIM matrix in Excel format. I provided the experimental group with an example of

what the matrix should resemble when their product was completed. Additionally, they

had the option of completing an outline, which asked for possible COAs, a listing of

screening criteria, the eliminated possible COAs and the evaluation criteria. After

receiving instruction, but prior to completing the requirement, I asked the experimental

group to complete the same pre-test questionnaire as the control group. Upon completion

of their analysis, I asked each member of the experimental group to fill out a post-test

questionnaire (See Appendix 11), which differed slightly from the control group, and

received the same debriefing form.

In order to obtain minimal outside influence with the results, I intended to keep

the control and experimental group as similar as possible as far as number of participants

and class year (See Figure 3.2 for a breakdown demographic breakdown of subjects by

group). Although the number of participants in each group was similar, (a total of 21

student-analysts participated in the control group and a total of 24 student-analysts

participated in the experimental group section), there was an unpreventable discrepancy

regarding the number of students in each class year. Specifically, there were a greater

number of first year graduate students within the experimental group. This was due to

uncontrollable scheduling conflicts regarding evening classes and meetings, as well as the

availability of students who wanted to participate in the experiment. In order to identify

if this discrepancy had any effect on the results, specific questions related to familiarity

33

and experience with analytic methods were addressed in both the pre-test and post-test

questionnaires.

Figure 3.2 Demographic Breakdown by Experiment Group

Data Analysis Procedures

Since the intent of the experiment was to test whether MCIM is a valuable

methodology to use in the field of intelligence, I evaluated several factors upon

completion of the experiment. Prior to the review of each analyst’s process and product,

it was necessary for me to draw some conclusions from both the pre-test and post-test

questionnaires in order to identify possible outside influence. A majority of the questions

required the students to rate their answer on a 5 level Likert scale. I recorded the number

of responses for each numeric category in a spreadsheet, found the percentage of subjects

34

who answered each number within each group, and figured out the average of both

groups. By comparing the averages, I was able to pinpoint any differences between the

two groups and test whether or not these differences were statistically significant at the

5% level.

In addition to these questionnaires, I asked each student-analyst to indicate their

level of analytic confidence. When determining his or her level of analytic confidence,

each student-analyst was asked to consider the following seven factors:49

1. Use of Structured Method (s) in Analysis2. Overall Source Reliability3. Source Corroboration/Agreement4. Level of Expertise on Subject/Topic & Experience5. Amount of Collaboration 6. Task Complexity7. Time Pressure

In order to measure analytic confidence, I asked the students to rate their level of

analytic confidence by marking their level of confidence on the modified scale shown

below.50

Low |------------------------------------------------------| High

The scale was 10 centimeters in length, and they placed a slash mark where they

felt their level of analytic confidence fell. In order to compare levels of analytic

confidence between groups, I measured each analyst’s slash mark to the nearest half

centimeter. I then determined the minimum and maximum level of analytic confidence in

the experimental group and control group, as well as the average level within each group.

49 Peterson, Joshua. “Appropriate Factors to Consider When Assessing Analytic Confidence in Intelligence Analysis.” (Master's Thesis, Mercyhurst College, March 2008)

50 Ibid

35

Upon completion of their analysis, I also asked each student-analyst to record the

amount of time it took them to complete their analysis (in minutes). In order to make

comparisons regarding length of time to complete their analysis, I found the average

amount of time, as well as the minimum and maximum amount of time within each

group. Finally, I reviewed each individual analysis, in order to make comparisons

regarding the thoroughness of their process, and the accuracy of their product.

36

CHAPTER 4:

RESULTS

Statistical analysis was conducted on the results in order to determine if the

findings were significant. If the results indicated a P-value < 0.05, then the results were

considered significant, meaning it is unlikely to have occurred by chance. If the results

indicated a P-value > 0.05, then the results were considered not significant, meaning it

was likely to have occurred by chance. This section will discuss the results of both the

pre-test and post-test questionnaires, as well as the results of the control group and

experimental group viewed both individually and compared as a whole. A more

complete review of the statistical data can be found in Appendix 12. Additionally, the

consequences of some of these results are discussed further in Chapter 5.

Pre-Test Findings

I distributed identical pre-test questionnaires to the control group and the

experimental group prior to the analysts beginning their research. The subjects were

asked to answer questions on a 5 level Likert scale. The purpose of the questions was to

determine if there were any significant factors that would explain differences between the

two group’s results that were not due to the experiment itself. Specifically, these

questions were designed to provide insight as to the subject’s level of interest in

participating in the experiment, as well as their level of familiarity and interest associated

with the topic (Russia’s relationship with OPEC).

37

The first question was designed to determine the level of interest regarding

participation in the experiment between the two groups. Both groups expressed roughly

the same level of interest regarding participation in the experiment, with the control

group expressing an average interest level of 3.48 (out of 5) and the experimental group

expressing an average interest level of 3.75 (out of 5). Statistical analysis revealed a P-

value of 0.586, meaning that this result was not significant. Extra credit appeared to be a

driving force behind the majority of subject participation with the control group average

at 3.9 (out of 5) and the experimental group average at 4.25 (out of 5). Specifically, extra

credit appeared to be important to the undergraduate and first year graduate students.

Only four students indicated that extra credit was “not at all important,” all of whom were

second year graduate students.

The next two questions were designed to see if one group had more knowledge in

regards to the subject matter (Russia’s relationship with OPEC), or interest in the topic,

prior to conducting research. Results indicated that the control group was more

knowledgeable on the topic, with an average score of 2.5 (out of 5), compared to the

average score of 1.7 (out of 5) expressed by the experimental group, a difference found to

be statistically significant with a P-value of 0.012. The control group also indicated that

they were slightly more interested in the topic, with an average control group score of

3.48 (out of 5) and an experimental group average of 2.83 (out of 5), a difference also

found to be statistically significant with a P-value of 0.032.

38

Post-Test Findings

As discussed in Chapter 3, the control group and experimental group were

required to fill out a post-test questionnaire upon completion of their analysis. Each

group received several identical questions. These questions were designed to provide

insight as to the frequency that each subject uses analytic methods when providing

analysis and their opinion regarding the ability of analytic methods to improve

intelligence analysis. Additionally, the questions were designed to determine the

subject’s opinion in regards to having adequate information and time to complete his or

her analysis, and to assess their overall enjoyment of the process. When I asked the

student-analysts how frequently they use structured analytic methods when solving

intelligence related problems, the control group responded with an average of 3.14 (out of

5) and the experimental group responded with an average of 3.5 (out of 5). This result

has a P-value of 0.150, thus this difference was not found to be statistically significant.

In addition, I asked the two groups to rate their level of enjoyment during the process.

The control group average was 3.7 (out of 5) and the experimental group’s average was

3.46 (out of 5). This result yielded a P-value of 0.101, thus the difference was not found

to be statistically significant. These results suggest that both the control group and the

experimental group felt that they had access to a reasonable amount of information that

was necessary to evaluate the problem to its full extent, and that the use of the

methodology by the experimental question did not seem to overly burden the student-

analysts.

I asked the student-analysts to determine if they were given enough time to

complete their analysis. The control group’s average response was 4.43 (out of 5) and

39

the experimental group responded with an average of 4.83 (out of 5), a difference that

was statistically significant with a P-value of 0.025. This result suggests that the

experimental group felt that they had more than enough time to complete the estimate, a

result that is particularly interesting when compared to the actual time each group took to

complete the estimate. (See the section on timeliness discussed later in this chapter).

Aside from the questions previously discussed, I asked both groups several

questions that were designed to gain feedback regarding the experiment set-up, as well as

gauge the student’s prior exposure to analytic methods and desire to use structured

methodology in future analysis. These questions were not tested for statistical

significance as they did not likely have any influence regarding the actual process or

product of the student’s analysis. These questions merely had to do with the experiment

itself and therefore, there was no basis for comparison.

In the post-test questionnaire, I asked the student-analysts in both groups how

familiar they were with the process of MCIM. It was predictable that the experimental

group would indicate that they were more familiar with the process of MCIM (after all

they did just receive instruction on the methodology). The control group rated a level of

familiarity as an average of 2.29 (out of 5) with six students claiming they had “no

familiarity at all with the methodology.” The experimental group rated a level of

familiarity of 3.29 (out of 5); with three students claiming they had “no familiarity at all

with the methodology.” Based on the experimental group’s responses it is likely that the

three students who indicated that they had “no familiarity at all with the methodology”

interpreted the question to mean familiarity prior to the experiment session. After all, the

40

students had just received instruction on the methodology and completed their analysis

using the methodology.

Despite the difference in level of familiarity regarding MCIM, both groups did

indicate very similar levels of familiarity with another structured analytic method, the

ACH. The control group indicated an average of 4.29 (out of 5), with no subjects

providing a response of 1 or 2 (responses that would have indicated no familiarity with

the methodology, or minimal at best). The experimental group indicated an average of

4.42 (out of 5), with no subjects provided a response of 1, 2, or 3 (responses that would

have indicated no familiarity with the methodology, or minimal at best). In spite of these

statistics indicating that the students had some level of familiarity with analytic methods,

no student-analyst in the control group actually used a structured method during their

analysis.

Product

I asked the student-analysts the following question: “How is Russia likely to seek

to interact with OPEC after the December 17th, 2008 meeting in Algeria.” Figure 4.1

expresses the COAs derived from the 21 members of the control group. Figure 4.2

expresses the COAs derived from the 24 members of the experiment group. However,

one student-analyst in the experiment group provided two answers; therefore the data in

that chart reflects 25 responses instead of 24.

41

Figure 4.1 Control Group Results

Figure 4.2 Experimental Group Results

The pie charts show that the experimental group arrived at a broader range of

possible COAs. The control group came up with five possible COAs; whereas, the

experimental group came up with seven possible COAs. Four of the COAs were roughly

the same between the two groups (Russia will closely coordinate with OPEC regarding

production, Russia will build a reserve to store oil, Russia will develop stronger ties with

OPEC, and Russia will engage in future talks with OPEC). The majority of the student-

analysts in both groups predicted that Russia will closely coordinate with OPEC

regarding oil production. This difference in number of suggested possible COAs may

imply that the use of MCIM aided the student-analysts in generating more creative and

42

unique solutions to the problems, rather than choosing the first COA that seemed logical

without reviewing alternatives.

When I originally picked the real world scenario (Russia’s relationship with

OPEC after the December 17th 2008 meeting) for the student-analysts to provide their

analysis, I chose a situation that I felt would have a clear cut answer within a few weeks

of the meeting’s termination. This has proved to not be the case up to this point. With no

clear cut answer at this time, it is impossible to evaluate the method with regard to the

accuracy of its product.

Process

Despite the lack of a clear cut answer to the question, it is beneficial to examine

the process by which both the control and experimental group derived their conclusions.

The Intelligence Community Directive (ICD) Number 203 is a document that manages

“the production and evaluation of national intelligence analysis.”51 While the entire

document does not apply to my research, several of the IC Analytic Standards outlined in

section D4 are worthy of noting. These standards include timeliness, expression of

analytic confidence, objectivity and the incorporation of alternative analysis, as well as

the use of logical argumentation.

Timeliness

The ICD recognizes the importance of timeliness in intelligence analysis. The

document emphasizes that analytic products have the obligation to meet the time 51 Intelligence Community Directive, Number 203. Analytic Standards (Effective June 21, 2007)

Available at: http://docs.google.com/gview?a=v&attid=0.1&thid=11f1d65194a3fe1b&mt=application%2Fpdf&pli=1; Internet.

43

standards set by the consumer, to ensure that they can be actionable products. This is

why I choose to evaluate the length of time it took each analyst to complete their work, as

there would be no purpose in utilizing a time consuming method; especially, if it is not

likely to yield better results.

Both groups were given two hours to complete their analysis. At the end of each

experiment section, each subject also recorded the actual time it took them to complete

their analysis, not including the instruction session. The results regarding time were

interesting. In the control group, the amount of time to complete analysis ranged from a

minimum of 45 minutes to a maximum of 105 minutes, with the average time at 70

minutes. The experimental group completed their analysis slightly faster, with a

minimum completion time of 30 minutes, a maximum completion time of 90 minutes,

and an average completion time of 58.96 minutes. The average length of completion

time was a result found to be statistically significant, with a P-value of 0.036. This result

suggests that the experimental group was able to structure and complete their analysis in

a notably more time efficient way, almost 20% faster, than the control group. Figure 4.3

represents the completion time for 20 student-analysts in the control group (one member

failed to provide their length of completion time) and for the 24 student-analysts in the

experimental group.

Figure 4.3 Length of Analysis Completion Time

44

Upon completion, the student-analysts were also asked if this was enough time to

complete their analysis. The control group indicated an average response of 4.43 (out of

5) and the experiment group indicated a response of 4.83 (out of 5), a difference that was

found to be statistically significant with a P-value of 0.025. The results regarding time

are important for two reasons. Not only were the student-analysts able to complete their

analysis more quickly, but their perception for the process was that they had more than

enough time.

Analytic Confidence

The ICD also mandates that analytic products express the analyst’s level of

confidence in their analytic judgment. After analysis was completed, each student-

analyst was asked to rate his or her level of analytic confidence.

The control group had a minimum level of analytic confidence of 1, a maximum

level of analytic confidence level of 9, and an average response of 5.93. The

45

experimental group expressed a slightly higher level of analytic confidence, with a

minimum level of 1.5, a maximum level of 9, and an average of 6.31. This difference

was not found to be statistically significant, with a P-value of 0.2335. Figure 4.4 displays

the levels of analytic confidence for both groups.

Figure 4.4 Level of Analytic Confidence

Objectivity and the Incorporation of Alternative Analysis

The ICD also mandates that analysts remain objective in their work and that their

analytic products incorporate alternative analysis when appropriate. This was one of the

primary differences I saw between the control group and the experimental group. While

a few students in the control group provided one or two alternative COAs, the majority of

the student-analysts merely provided one COA with few comparisons to any alternatives,

thus, not providing any insight to whether or not alternative solutions were considered.

In the experimental group, the student-analysts, who used MCIM, provided a list of all

possible COAs, and identified the importance of specific criterion or various factors to

46

those COAs. Their matrices highlighted strengths and weaknesses of each COA allowing

a hypothetical decision maker (or in this case, researcher) to compare each COA against

the alternatives. The matrices received from the experimental group provided a much

clearer outline regarding the thought process of each student-analyst and allowed me to

determine what alternative factors and COAs that the student-analysts considered.

Additionally, by providing a list of alternatives, I can assume that the student-analysts

took a more comprehensive look at different options, and did not merely find information

to support the first COA they found fitting.

Logical Argumentation

The ICD requires that key findings and analysis be supported by appropriate facts

and information, as well as address any divergent ideas. As part of the research process, I

asked both groups of student-analysts to use bullet points to explain their analysis.

Although the bullets provided some indication as to the thought process of the student-

analyst, the matrices completed by the experimental group were much easy to understand

and visibly conveyed the written analysis provided.

Bullet Point Analysis

I also asked the students to provide bullet point analysis so that I could capture the

data later on. One method I used to determine the thoroughness of analysis was by

evaluating the number of bullet points that each student-analyst used. I found the average

number of bullet points for each group to determine if there was a difference between

groups. The control group had a total of 146 bullet points, used a minimum of three

47

bullet points, a maximum of 11 bullet points, and an average of 6.95 bullet points. The

experimental group had a total of 139 bullet points, used a minimum of three bullet

points, a maximum of 15 bullet points and an average of 5.79 bullet points. This

difference was found to be statistically significant, with a P-value of 0.0415. I did not

evaluate the quality of each bullet point in this step; however, I do recognize that each

bullet point may not have been entirely relevant to the analysis. I am assuming that the

amount of insignificant bullet points between the two groups was roughly the same;

therefore, they would cancel each other out. Figure 4.5 provides a statistical breakdown

of the number of bullet points used by each member of the control and experimental

groups.

Figure 4.5 Bullet Point Analysis

48

Potential of Future Use

Since MCIM is relatively new to the field of intelligence, I wanted some feedback

as to the student’s opinion of future use of the methodology. Upon completion of their

analysis, I asked the experimental group several post-test questions designed to determine

if the student-analysts had enough instruction to complete the process, determine the ease

of using MCIM, and assess the likelihood that each analyst would use MCIM in the

future. When I asked the student-analysts if they felt like they had received enough

instruction to complete the process using MCIM (the group received a 35 minute lecture

on the process of MCIM in which I provided background information, and the group

worked through two example problems), they responded with an average answer of 4.5

(out of 5). Most of the student-analysts indicated this method was relatively simple to

use, rating it with an average score of 3.83 (out of 5). The majority of the student-

analysts also indicated that this was a method they would likely use again in the future,

with an average score of 3.67 (out of 5). Thus, even with minimal instruction time, the

student-analysts felt that the methodology was relatively easy to understand and use, and

it is something they would consider using in future intelligence analysis.

49

CHAPTER 5:

CONCLUSIONS

The purpose of this experiment was to test the value of using MCIM when solving

intelligence related problems. By controlling the method in which student-analysts

worked through an intelligence scenario, I was able to evaluate the significance of MCIM

in intelligence analysis. These factors included the analyst’s thought process and

product, their level of analytic confidence, as well as the length of time it took to

complete their analysis.

Pre-Test and Post-Test Conclusions

The pre-test and post-test questionnaire proved to be valuable, as they were an

indicator as to some of the factors outside of the experiment that may have contributed to

the results in some way. Additionally, they gave me a basis for which to compare the

base level of knowledge and level of interest in participation between the two groups,

prior to beginning research.

One interesting conclusion drawn from the pre-test was that although the

experimental group indicated a lower level of knowledge in regards to the topic (Russia’s

relationship to OPEC) and expressed a lower level of interest with the topic, both of

which were found to be statistically significant, they were able to arrive at a broader

range of possible COAs. This result suggests that despite the control group having a

jump start on the research, the experimental group was able to provide roughly the same

possible COAs, as well as a few alternatives. Another interesting conclusion drawn after

50

reviewing the products of the two groups occurred in regards to time. The experimental

group, who used MCIM, was able to complete their analysis in a more time efficient way

(faster), while still perceiving that they had more than enough time (according to the pre-

test). Specifically, the average completion time for the control group was 70 minutes and

the average time for experimental group was 58.6 minutes. Therefore, when looking at

the big picture, although the experimental group seemed less knowledgeable and less

interested, they were able to arrive at a more complete list of relevant possible COAs, and

they completed their analysis in less time.

Finally, the bullet point analysis indicated that the control group averaged

approximately one more bullet point in the analysis part of their product. This difference

was found to be statistically significance with a P-value of 0.0415. Although this result

suggests that the control group was able to provide more analysis than the experimental

group, it is important to remember that the worth of each bullet point was not assessed.

It is also entirely possible that the student-analysts in the experimental group felt that the

matrix was able to more accurately support their analysis, thus, they provided less bullet

point reasoning.

The results from the post-test suggest that all student-analysts participating in the

experiment had some level of familiarity with structured analytic methodologies. In

addition to familiarity with structured analytic methods, the results of the post-test

indicated that all of the subjects that participated in the experiment have actually used

these methods when conducting prior analysis. They also expressed that these methods

improve intelligence analysis on some level. Despite these factors, no one in the control

group utilized any sort of structured method when completing their analysis. There is no

51

apparent reason as to why this happened. The student-analysts may have felt that

structured methodology wasn’t necessary for this type of problem, or perhaps they felt

that using an informal process was both quicker and easier. Any future research or

experiments conducted regarding the use of structured methodology in intelligence

analysis, should include a post-test question asking why the student either chose to use, or

chose not to use methodology. By asking this question, more information could be

gathered as to what makes an analyst chose the method that they did.

Process and Product Conclusions

As previously mentioned, when I originally picked the real world scenario

(Russia’s relationship with OPEC after the December 17th, 2008 meeting) for which the

student-analysts to provide their analysis, I chose a situation that I felt would have a clear

cut answer within a few weeks. Although this has proved to not be the case, the use of a

MCIM model has proven to be beneficial in a variety of ways. The matrix clearly

explains how each analyst in the experimental group arrived at their conclusion. The

student-analysts outlined the criteria that he or she felt to be important and explained how

significant each of these factors was at arriving at their decision by the use of a weighting

system. Additionally, if any of the criteria should change as time progresses, the matrix

is easily modified by adding or eliminating criteria, or by merely adjusting the weights to

arrive at an alternative COA. In other words, all the hard work has already been done on

the problem, and only minor changes would need to be made to suggest an alternative

COA needed due to changes in the adversary’s environment or priorities. Since no one in

the control group used any form of structured methodology to arrive at their conclusion,

52

their decision maker (or in this case, researcher) would have a minimal idea of how each

arrived at their conclusion. Although a majority of the control group’s bullet point

analysis does support the COA that they chose to be most likely, there is minimal

mention of any other possible COA, or of the significance of each factor they considered

to support their predicted COA. If there was a change in the adversary’s environment

that would suggest the need for an alternative COA, the student-analysts in the control

group would need to go back and figure out what is the next most likely solution based

on the importance of each factor that they considered.

In general MCIM is beneficial to the field of intelligence due to its flexibility and

versatile nature. The use of MCIM is not specific to a given intelligence field, therefore

it is applicable to business, law enforcement and national security. Additionally, it is an

appropriate method to use when working with both qualitative and quantitative data.

A Step Forward

While there are a number of potential benefits with using this methodology, there

are also several prospective weaknesses. Although the matrix seeks to identify the most

important criteria involved in the decision making, it has the potential to be flawed as the

analyst is doing the scoring of the criteria. Therefore, the process of assigning weight to

pieces of criteria has the potential to be based on the analyst’s intuition and may contain

bias. Additionally, an analyst must know how and when to make a distinction when

weighing different criteria. For example, I would like to refer back to the example of

buying a car that I discussed in Chapter 2. How would an analyst rank a difference in

price for values that are close, such as $25,000 and $25,001? In other words, at what

53

point do they numbers quit being the same? Finally, this process seeks to forecast

something in which the analyst has no actual control over. Individuals only have total

control over what they do, but not competitors, enemies, or criminals. Thus, even with a

detailed, structured approach that hopes to identify the COA that an opponent is most

likely to chose, it is possible that the adversary will go with a COA that is completely

random based on their intuition or “gut feeling.” Heuer draws attention to this idea when

he said, “Caution is in order, however, whenever one thinks of predicting or even

explaining another person’s decision, regardless of whether the person is American or

foreign. People do not always act rationally in their own best interests. Their decisions

are influenced by emotions, habits, what others might think, or values that others many

not be aware of.”52

In all, the benefits of using a structured methodology in intelligence analysis, as

well as the overall general positive responses received from the student-analysts

regarding the use of MCIM, suggest that this topic is worthy of continued research.

Larger sample sizes would be beneficial in any future experiments in order to confirm

hypotheses surrounding use of the method. Additionally, in future experiments, analysis

should be provided for a real world situation that would have a clear cut answer within a

short time frame of the analysis in order to further evaluate MCIM’s ability to aid

decision makers in efficient forecasting.

Despite the possible flaws with the methodology, the results of this experimental

research highlight the potential benefits of using structured analytic methods, such as

MCIM, during the analytic process. Although the use of MCIM can never guarantee a

“correct” answer, it may increase the odds of choosing the most likely answer, by

52 Heuer, Richards, J., Email. February 12, 2009.

54

providing a more efficient way of handling unstructured data in both qualitative and

quantitative form. Additionally, matrix analysis provides a clear explanation to a

decision maker regarding the thought process of each analyst. By identifying the weight

that an analyst places on each individual criterion, MCIM can highlight the areas of

disagreement between two individuals, and it is easily adjusted to accommodate new data

or “what-if” scenarios. Additionally, as demonstrated in the experiment, using the

MCIM methodology may allow an analyst to structure and complete their analysis in a

more time efficient.

While the jury is still out regarding the accuracy of the analysis, MCIM can help

teach an analyst the importance of considering the significance of certain factors. This

study hopes to take the first steps of many in increasing an analyst’s ability to thoroughly

examine all possible COAs in a timely manner, as well as accurately forecast what COA

one’s adversary is likely to choose.

55

BIBLIOGRAPHY

Balcomb, J.D. and A. Curtner. “Multi-Criteria Decision-Making Process for Building.” Paper to be presented at the American Institute of Aeronautics and Astronautics Conference, Las Vegas, Nevada, July 24-28. 2000. Available online at http://www.nrel.gov/docs/fy00osti/28533.pdf; Internet; accessed 23 March 2009.

Belton, Valerie and Theodor J. Stewart. Multiple Criteria Decision Analysis: An Integrated Approach. The Netherlands: Kluwer Academic Publishers, 2002.

Dzubow, PhD and William Z. Suplee IV, CFA, CFP, ChFC, CASL. “Using Multiple-Criteria Decision Analysis to Simplify the Financial Planning Process.” Journal of Financial Planning. March 2008. Available at http://www.library.idsc.gov.eg/GUI/Globals/Upload/BULLETIN_ATTACHMENT/92/e-files/manegement%20and%20economics/using%20multiple.pdf; Internet; accessed 5 December 2008.

Forgang, William G., Strategy-Specific Decsion Making: A Guide for ExecutingCompetitive Strategy. M.E. Sharpe, Inc., Armonk, New York, 2004.

Forman, Ernest, Decision by Objectives (How to Convince Others That You Are Right). Available at http://mdm.gwu.edu/forman/DBO.pdf; Internet.

Gilchrist, Stacy. “Staff Study.” Research Paper for Advanced Analytic Techniques, Mercyhurst College.

Heuer, Richards, Jr. Email. 12 February 2009.

Heuer, Richards, Jr. Psychology of Intelligence Analysis [book on-line]. Langley, Virginia: Central Intelligence Agency Center for the Study of Intelligence, 1999. Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf; Internet.

Heuer, Richards, Jr. “Taxonomy of Structured Analytic Techniques.” Paper preparedfor the International Studies Association 2008 Annual Convention, March 26-29, 2008, San Francisco, CA. Available at http://www.allacademic.com//meta/p_mla_apa_research_citation/2/5/4/1/2/pages254125/p254125-1.php; Internet; accessed 20 November 2008.

Intelligence Community Directive, Number 203. Analytic Standards (Effective 21 June 2007). Available at http://docs.google.com/gview?a=v&attid=0.1&thid=11f1d65194a3fe1b&mt=application%2Fpdf&pli=1; Internet.

56

Kiker, Gregory A., Todd S. Bridges, Arun Varghese, Thomas P. Seager and Igor Linkov. “Application of Multicriteria Decision Analysis in Environmental Decision Making.” Integrated Environmental Assessment and Management. Volume 1, Number 2, Pages 95-108. 2005. Available at http://www.allenpress.com/pdf/ieam-01-02_95_108.pdf; Internet; accessed 20 November 2008.

LeGault, Michael R. Think: Why Crucial Decision Can’t Be Made in the Blink of an Eye. New York, Threshold Editors, 2006. Page 305.

Marrin, Stephen. “Intelligence Analysis: Structured Methods or Intuition.” American Intelligence Journal. Page 7. Summer 2007.

Peterson, Joshua. “Appropriate Factors to Consider When Assessing Analytic Confidence in Intelligence Analysis.” (Master's Thesis, Mercyhurst College, March 2008).

Satty, Thomas. “Decision making with the analytic hierarchy process,” Int. J. Services Sciences, Vol. 1, No. 1, 2008. Available at http://inderscience.metapress.com/media/pgwf2qtuyg3xnvnhxnby/contributions/0/2/t/6/02t637305v6g65n8.pdf; Internet; accessed 26 March 2009.

Saaty, Thomas L. “How To Make A Decision: The Analytic Hierarchy Process.” Available at http://sigma.poligran.edu.co/politecnico/apoyo/Decisiones/curso/Interfaces.pdf; Internet; accessed 24 March 2009.

Simon, Herbert and Associates. “Decision Making and Problem Solving.” Reprinted with permission from Research Briefings 1986: Report of the Research Briefing Panel on Decision Making and Problem Solving. 1986 by the National Academy of Sciences. Published by National Academy Press, Washington, DC. Available at http://dieoff.org/page163.htm; Internet; accessed 20 November 2008.

The United States Army. Field Manual 101-5. Staff Organization and Operations. 31 May 1997. Available at http://www.fs.fed.us/fire/doctrine/genesis_and_evolution/source_materials/FM-101-5_staff_organization_and_operations.pdf; Internet. Page 115.

The United States Army 2008 U.S. Army Posture Statement. Information Papers: Red Team Education and Training. Available at http://www.army.mil/aps/08/information_papers/prepare/Red_Team_Education_and_Training.html; Internet; accessed 26 September 2008.

The United States Government. Director of National Intelligence, Vision 2015. Available at http://www.dni.gov/Vision 2015.pdf; Internet.

57

Triantaphyllou, Evangelos and Stuart H. Mann. “An Examination of the Effectiveness of Multi-Criteria Decision Making Methods: A Decision Making Paradox.” Available at http://www.csc.lsu.edu/trianta/index.html?http://www.csc.lsu.edu/trianta/Books/DecisionMaking1/Book1.htm; Internet.

Triantaphyllou, Evangelos. “Can We Always Determine the Right Alternatives in Business Problems.” 18 August 2002.

Triantaphyllou, Evangelos and Panos M. Parlos (ed.). Multi-Criteria Decision Making Methods: A Comparative Study. The Netherlands: Kluwer Academic Publishers, 2000.

Wang, Xiaoting. “Study of Ranking Irregularities When Evaluating Alternatives By Using Some ELECTRE Methods and a Proposed New MCDM Method Based on Regret and Rejoicing,” A Thesis Submitted to the Graduate Faculty of the Louisiana State University and Mechanical College. August 2007. Available online at http://etd.lsu.edu/docs/available/etd-07112007-012708/unrestricted/Wang_thesis.pdf; Internet; accessed 24 March 2009.

Webster’s Dictionary Online; available from http://www.merriamwebster.com/dictionary/criteria; Internet.

Zopounidis, Constantin and Doumpos, Michael. “Multiple-Criteria Decision Making.” Thomson Corporation. 2006.

58

APPENDICES

59

Appendix 1:

Analytic Methods Experiment Sign-Up Form

Name:

Class Year:

Phone Number:

E-mail Address:

Instruction Session Dates/Times:(Please select the day you are able to participate)

Wednesday, 10 December 2008 5:00-8:00pm ____

Thursday, 11 December 2008 5:00-8:00pm ____

If you are able to participate on either day please ____check here, and you will be randomly assigned a day

*** I will send you an e-mail confirming your experiment day and time

Upon completion, please return this form to Lindsey Jakubchak or Travis Senor in CIRAT.

Contact Info:ljakub00@mercyhurst.edu

60

Appendix 2:

Institutional Review Board Proposal Form

Date Submitted:10/10/2008

Investigator(s):Lindsey Jakubchak

Investigator Address:

Investigator(s) E-mail:ljakub00@mercyhurst.edu

Investigator Telephone Number:

Advisor's Name (if applicable):Kristan Wheaton

Advisor’s E-mail:kwheaton@mercyhurst.edu

Advisor's Signature of Approval:[X] Place X here if advisor hasapproved research

Title of Research Project:The Effectiveness of Multi-Criteria Decision Making (MCDM) in the Field of Intelligence

Date of Initial Data Collection:TBD, anticipate October-December 2008

________________________________________________________________________

Please describe the proposed research and its purpose, in narrative form:

Multi-Criteria Decision Making (MCDM) is a generic term used to encompass a broad range of analytic methodologies that use matrices as the basis for their conclusions. More specifically, MCDM is an internally focused support system that confronts real world decisions based on the evaluation of multiple criteria, goals, and objectives of conflicting nature. Additionally, it provides substance to a decision maker as it allows for selecting an option based on the appropriateness of those alternatives weighted against the others.

While there is a substantial body of literature related to the use and effectiveness of MCDM in the general sense, there has been little research done in the field of intelligence. The purpose of this proposed research and experiment is to take an alternative look at this methodology. In other words, if MCDM was switched to an external focus, and used in the field of intelligence, is it likely that this methodology

61

would be a valuable way to predict what course of action one’s adversary is likely to choose?

I have developed an experiment which I feel will test this hypothesis, in both process and product, when solving intelligence related problems.

I plan to test if MCDM truly is a viable methodology by using intelligence analysts (both undergraduate and graduate students in the intelligence program at Mercyhurst College) to solve a real world problem.

Indicate the materials, techniques, and procedures to be used (submit copies of materials):Materials:

Exercise ScenariosWriting UtensilsPre-Test and Post-Test Questionnaire

Procedure:

One month prior to the experiment, I will solicit both undergraduate and graduate students at Mercyhurst College to participate through department wide e-mails, fliers and through short information sessions at the beginning of classes (with permission of the designated professor). Students will be selected on a first come, first serve basis. One week prior to the study, I will send out reminders to those who have volunteered to participate. I will send out another reminder the day before the study.

During the actual study, I will begin by going over the directions. I will then explain what is expected of the participants, and how they will be getting credit for their participation. After the introduction, I will pass out a pre-test questionnaire (attached at the end of this form), the study’s materials, and instruct the students to begin their analyses. I will then provide the students with one week in which they can complete their analysis and provide a time in which they can return their finished product to me. This process will be the same for both groups (the control group and the experimental group), with the only exception being that the experimental group will receive a one hour long instructional session on the how to use a variant of MCDM in an intelligence context (in order to complete their analysis), after the introduction.

Following the completion of the exercise, I will ask the participants to fill out a questionnaire (attached at end of this form) and provide feedback regarding both the topic and the experiment.

I plan to conduct my experiment two times, on two different nights. They will vary only in which handout I give them. There will be one experimental group and a control group.

62

All groups will be given the exact same scenario; I will just vary the methodology utilized to solve the problem. (Please see the forms below)

63

1. Do you have external funding for this research (money coming from outside the College)? Yes[     ] No[X]

Funding Source (if applicable):      

2. Will the participants in your study come from a population requiring special protection; in other words, are your subjects someone other than Mercyhurst College students (i.e., children 17-years-old or younger, elderly, criminals, welfare recipients, persons with disabilities, NCAA athletes)? Yes[     ] No[X]

If your participants include a population requiring special protection, describe how you will obtain consent from their legal guardians and/or from them directly to insure their full and free consent to participate.

N/A

Indicate the approximate number of participants, the source of the participant pool, and recruitment procedures for your research:

I plan to have approximately 75 participants. I plan to recruit undergraduate and graduate students in the intelligence studies department through a department-wide email, fliers hung in the building, and by information sessions held at the beginning of classes (with permission of the designated professor). I will select the students on a first come, first serve basis.

Will participants receive any payment or compensation for their participation in your research (this includes money, gifts, extra credit, etc.)? Yes[X] No[     ]

If yes, please explain:

In the past, most of the intelligence professors have been willing to grant extra credit for participating in an experiment.

3. Will the participants in your study be at any physical or psychological risk (risk is defined as any procedure that is invasive to the body, such as injections or drawing blood; any procedure that may cause undue fatigue; any procedure that may be of a sensitive nature, such as asking questions about sexual behaviors or practices) such that participants could be emotionally or mentally upset? Yes[     ] No[X]

Describe any harmful effects and/or risks to the participants' health, safety, and emotional or social well being, incurred as a result of participating in this research, and how you will insure that these risks will be mitigated:

None.

64

4. Will the participants in your study be deceived in any way while participating in this research? Yes[     ] No[X]

If your research makes use of any deception of the respondents, state what otheralternative (e.g., non-deceptive) procedures were considered and why they weren't chosen:

N/A

5. Will you have a written informed consent form for participants to sign, and will you have appropriate debriefing arrangements in place? Yes[X] No[     ]

Describe how participants will be clearly and completely informed of the true nature and purpose of the research, whether deception is involved or not (submit informed consent form and debriefing statement):

Prior to the training sessions, participants will be provided with a general overview of what will occur during the session as well as the consent form, which will also describe what is expected of them. Following the administrative questionnaire, participants will be provided with a debriefing statement that will explain how the results from the session will be used (please see forms at the end of this proposal).

Please include the following statement at the bottom of your informed consent form: “Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Mr. Tim Harvey, Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 824-3372.”

6. Describe the nature of the data you will collect and your procedures for insuring that confidentiality is maintained, both in the record keeping and presentation of this data:

Names are not required for my research and thus no names will be used in the recording of the results or the presentation of my data. Names will only be used to notify professors of participation in order for them to correctly assign extra credit.

7. Identify the potential benefits of this research on research participants and humankind in general.

Potential benefits include:

For participants:For some students, this experiment will provide an opportunity to learn a new analytical tool that they may use in intelligence related decision making. For other students, it will provide an opportunity to practice decision making skills that they have learned in the classroom or developed in the real world. Students are often asked to utilize various

65

analytical tools in order to gain a more thorough understanding of the problem at hand, and this may be an effective methodology for doing that.

For the Intelligence Community (IC):This experiment hopes to prove that MCDM is an effective methodology to use when making intelligence related decisions. In doing so, the IC would benefit from a methodology that allows our nation’s decision makers to make more informed decisions.

Please submit this file and accompanying materials to the IRB Chair, Tim Harvey, via electronic mail (tharvey@mercyhurst.edu) for review.

Appendix 3:

Multi-Criteria Decision Making Participation Consent Form

The purpose of this research is to test the effectiveness of a particular method, in both process and product, when making intelligence related decisions. Your participation involves an instruction period, completion of a short intelligence analysis, and filling out a pre and post test questionnaire. This process should take no longer than 3 hours. Your name WILL NOT appear in any information disseminated by the researcher. Your name will only be used to notify professors of your participation in order for them to assign extra credit.

There are no foreseeable risks or discomforts associated with your participation in this study. Participation is voluntary and you have the right to opt out of the study at any time for any reason without penalty.

I, ____________________________, acknowledge that my involvement in this research is voluntary and agree to submit my data for the purpose of this research.

_________________________________ __________________Signature Date_________________________________ __________________Printed Name Class

Name(s) of professors offering extra credit and class(es) you are enrolled in: (Professor Breckenridge, Professor Marrin, Professor Welch, and Professor Wheaton) ________________________________________________________________________

Researcher’s Signature: __________________________________________________________

If you have any further question about methodology used, or this research, you can contact me at ljakub00@mercyhurst.edu

Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Tim Harvey; Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 824-3372. tharvey@mercyhurst.edu

Lindsey Jakubchak, Applied Intelligence Master’s Student, Mercyhurst College

Kristan Wheaton, Thesis Advisor, Mercyhurst College

Appendix 4:

Experiment Section #1 (Control Group)

You are a national security analyst specializing in Russian relations. Your director is interested in the outcome of the upcoming meeting of the Organization of Petroleum Exporting Countries (OPEC), taking place in Algeria on December 17th, 2008. Please provide an answer to the following question:

How is Russia likely to seek to interact with OPEC after the December 17th, 2008 meeting in Algeria?

STEP 1: Please complete the following analysis in the space provided below. *You may use any sources you would like

It is likely that Russia will seek to-________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

STEP 2: Please list your reason(s) for your analysis: *You do not need to provide your sources

STEP 3: Please rate your level of analytic confidence.

Important Information:Analytic Confidence:

Analytic Confidence reflects the level of confidence an analyst has in his or her estimates and analyses. It is not the same as using words of estimative probability, which indicate likelihood. It is possible for an analyst to suggest an event is virtually certain based on the available evidence, yet have a low amount of confidence in that forecast due to a variety of factors or vice versa.

As you are considering your level of analytic confidence, please consider these seven factors:

1. Use of Structured Method(s) in Analysis2. Overall Source Reliability3. Source Corroboration/Agreement4. Level of Expertise on Subject/Topic & Experience5. Amount of Collaboration6. Task Complexity7. Time Pressure

Mark your analytic confidence on the scale below.

Low |-----------------------------------------------------------------------| High

Appendix 5:

Analytic Methods Experiment Links

Current Situation: Although Russia is not currently an OPEC member, it does maintain a

relationship with the organization.

Links to get you started… You are not limited to these sources; they are merely starting points to familiarize

yourself with the topic! Feel free to use whatever sources you would like!

Monthly Oil Market Report-November 2008:Source: Organization of the Petroleum Exporting Countries (OPEC)http://www.opec.org/home/Monthly%20Oil%20Market%20Reports/2008/pdf/MR112008.pdf

Non-OPEC Oil Production:Source: Council on Foreign Relationshttp://www.cfr.org/publication/14554/nonopec_oil_production.html

Organization of the Petroleum Exporting Countries (OPEC):Source: OPEC Homepagehttp://www.opec.org/home/

Outlook for Non‐OPEC Oil Supply Growth in 2008‐2009:Source: Energy Information Administrationhttp://www.eia.doe.gov/emeu/steo/pub/special/2008-non-opec-oil-supply.pdf

Russia Energy Data, Statistics and Analysis - Oil, Gas, Electricity, Coal:Source: Energy Information Administrationhttp://www.eia.doe.gov/emeu/cabs/Russia/pdf.pdf

Appendix 6:

Pre-Test Questionnaire (Control Group and Experimental Group)

Answer the following questions by circling the best response. Please answer all the questions honestly.

On a scale from 1 to 5, where 1 represents not interested at all and 5 represents extremely interested, how interested are you in participating in this experiment?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not knowledgeable at all and 5 represents extremely knowledgeable, how knowledgeable are you with this topic?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not interested at all and 5 represents extremely interested, how interested are you in this topic?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not interested at all and 5 represents extremely interested, how interested are you in learning about analytic methodologies?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not enough time and 5 represents more than an adequate amount of time, do you think you will have enough time to complete this analysis?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not important at all and 5 represents extremely important, how important was offering extra credit in participating in this experiment?

1 2 3 4 5

What is your class rank?

FR SO JR SR G1 G2

Please provide any additional comments or feedback in the space provided:________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Appendix 7:

Post-Test Questionnaire (Control Group)

Answer the following questions by circling the best response. Please answer all the questions honestly.

On a scale from 1 to 5, where 1 represents not at all familiar and 5 represents extremely familiar, how familiar are you with the process of Multi-Criteria Decision Making (MCDM)?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not at all familiar and 5 represents extremely familiar, how familiar are you with the process of Analysis of Competing Hypothesis (ACH)?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not at all and 5 represents extremely frequently, how frequently do you use structured analytic methods in solving intelligence related problems?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents no improvement at all and 5 represent a significant improvement, how much do you feel analytic methods improve intelligence analysis?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not enough at all, and 5 represents more than enough, do you feel you found enough adequate information to complete this analysis?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not enough time at all and 5 represents more than adequate amount of time, do you feel you had enough time to complete this process?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents no enjoyment at all and 5 represents thorough enjoyment, how much did you enjoy this process?

1 2 3 4 5

Approximately how much time did you spend completing this analysis?___________________

What do you think the purpose of this experiment was?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

What is your class rank?

FR SO JR SR G1 G2

Please provide any additional comments or feedback in the space provided:________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Appendix 8:

Multi-Criteria Decision Making Participation Debriefing

Thank you for participating in this research process. I appreciate your contribution and willingness to support the student research process.

The purpose of this study was to determine the effectiveness of using MCDM as an analytical methodology in the field of intelligence. Currently there has been little research done on this topic, and this study hopes to take the first of many steps in increasing one’s ability to accurately predict both what course of action one’s adversary is likely to choose, as well as forecast what is happening in the environment in which an analyst works in. My experiments today were designed to demonstrate the significance of using this methodology in solving intelligence related problems.

By shifting the process of MCDM from an internal focus to an external focus, it is keeping with the theme of intelligence. Additionally, utilizing methods that promote a more efficient method of structuring data may increase an analyst’s ability to learn what is happening with his/her adversary or the environment in which they are working in. I plan to use the results from this study to determine if the use of MCDM in solving intelligence related problems improves the analytic process and product.

Please note, you will be able to review the results of this experiment. The results will be included in a paper that I am preparing to present at the International Studies Association (ISA) Conference, and will be accessible online. Additionally, results and conclusions will be accessible via my completed thesis.

If you have any further question about MCDM or this research you can contact me at ljakub00@mercyhurst.edu

Appendix 9:

Multi-Criteria Decision Making Lecture

Everybody Makes Decisions On A Daily Basiso Some decisions can be made with our intuition or “gut feelings” as they

are not life altering and require little to no analysis. Examples: What will I eat for breakfast in the morning? What road

will I use to travel to work?o Other decisions require the balancing of multiple factors or criteria, and do

affect our life in some way. Examples: Where will I go to college? What kind of car will I

buy? What career will I choose to embark on? o In the field of intelligence, decision related to national security, law

enforcement, and business directly relate to the safety of a nation, the rise and fall of a country, and the stability of an organization. These decisions mandate appropriate analysis.

Examples: What is likely to happen in Country X in the next two years? What will likely happen to the oil fields in Country Y over the next 12-24 months?

Multi-Criteria Decision Making (MCDM)o The process of evaluating possible courses of action (COA) in a

systematic way against a set of fixed criteria.o Involves the breakdown of complex problems into smaller pieces in order

for more thorough analysis.o Often called “matrix analysis.”o MCDM is a generic term used to encompass a broad range of analytical

methodologies o In MCDM, emphasis is on placing value, or judgment of an item’s worth

and desirability, on pieces of criterion, a standard by which a judgment or decision may be made.

What Constitutes a MCDM problem?o MCDM is used when a decision needs to be made that has two or more

known COAs or two or more possible COAs.

o Either a “need to find best solution” or “we need to find a solution.”o Must have multiple criteria.

Process Of MCDM (8 Steps)o Requirement/Question and Collection

Good Essential Question Understanding INTENT of question, is essential to good analysis

o Establish Possible COA

Brainstorm Want to identify as many COAs as possible

o Establish “Screening Criteria” Designed to eliminate COAs “Must have/be” guidelines

o Screen COAo Establish “Evaluation Criteria”

Designed to rank COAs Can be same as screening criteria.

o Weight Evaluation Criteriao Evaluate COA

What scale? Best to worst, 1 to 3, 1 to 5, 1 to 10? Based on? Intuition? Standards? Even distribution?

o Make Recommendations/Estimate

Conventional Use Of MCDM Vs. Use Of MCDM In Intelligenceo Difference of perspective

The conventional MCDM process focuses on one’s own decision-making process, a situation where an individual has complete control.

The intelligence focused MCDM process focuses on the decision-making process of others, a situation in which an analyst has no control over.

o Purpose By shifting the process of MCDM from an internal focus to an

external focus, it is keeping with the theme of intelligence Therefore, instead of establishing the COAs about the “best”

decision for an individual, an analyst seeks to determine which COA one’s adversary or competitor is likely to choose.

Likely Benefits Of The MCDM Process In Intelligenceo Utilizing methods that promote a structured review of external information

increases an analyst’s awareness of his/her adversary, and the environment in which they are working in.

By continually assessing the social, political, economical and technological aspects of an adversary’s environment, an analyst might gain insight to factors outside of a decision maker’s immediate control, deterring them from decisions they would otherwise likely make.

o The use of an intelligence focused MCDM process may help to prevent analytic pitfalls, such as the satisficing strategy and groupthink.

o Should allow for a more complete analysis of the situation and account for possible COAs that would otherwise have not been considered.

Appendix 10:

Experiment Section #2 (Experimental Group)

You are a national security analyst specializing in Russian relations. Your director is interested in the outcome of the upcoming meeting of the Organization of Petroleum Exporting Countries (OPEC), taking place in Algeria on December 17th, 2008. Using Multi-Criteria Decision Analysis, please supply an answer to the following question:

How is Russia likely to seek to interact with OPEC after the December 17th, 2008 meeting in Algeria?

The following is designed to guide you through the process of MCDM. Please note, Steps 2, 3 and 4 are mandatory. Step 1 is optional, and is designed to assist you with completion of the matrix. *You may use any sources you would like.

STEP 1: This step is optional

Possible Course of Action: What courses of actions are possible for Russia? You may have as many courses of action as you would like

COA 1:COA 2:COA 3:

Screening Criteria: Designed to eliminate COA’soo

Remaining Courses of Action: What courses of actions are possible for Russia? (After screening applied) You may have as many courses of action as you would like

COA 1:COA 2:

Evaluation Criteria: Designed to rank COAs. Evaluation Criteria may be tangible items (things that can be measured) or intangible items (things things that cannot be measured). These criteria can be ranked by scale: best to worst 1 to 3, 1 to 5, etc. They can also be based on intuition, standards, even distribution, etc.

Eval 1:Eval 2:Eval 3:

Step 2: This step is mandatory

Matrix: Please complete the Matrix in Excel (format is on the shared drive). Below is to demonstrate what the matrix should look like.

  Criteria 1: Criteria 2: Criteria 3: Criteria 4: Total/Rank Order

COA 1:          

COA 2:          

COA 3:          

COA 4:          

COA 5:          

It is likely that Russia will seek to ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

STEP 3: This step is mandatory Please list your reason(s) for your analysis: *You do not need to provide your sources

STEP 4: This step is mandatory

Please rate your level of analytic confidence.

Important Information:Analytic Confidence:

Analytic Confidence reflects the level of confidence an analyst has in his or her estimates and analyses. It is not the same as using words of estimative probability, which indicate likelihood. It is possible for an analyst to suggest an event is virtually certain based on the available evidence, yet have a low amount of confidence in that forecast due to a variety of factors or vice versa.

As you are considering your level of analytic confidence, please consider these seven factors:

1. Use of Structured Method(s) in Analysis2. Overall Source Reliability3. Source Corroboration/Agreement4. Level of Expertise on Subject/Topic & Experience5. Amount of Collaboration6. Task Complexity7. Time Pressure.

Mark your analytic confidence on the scale below.

Low |-----------------------------------------------------------------------| High

Appendix 11:

Post-Test Questionnaire (Experimental Group)

Answer the following questions by circling the best response. Please answer all the questions honestly.

On a scale from 1 to 5, where 1 represents not at all familiar and 5 represents extremely familiar, how familiar are you with the process of Multi-Criteria Decision Making (MCDM)?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not at all familiar and 5 represents extremely familiar, how familiar are you with the process of Analysis of Competing Hypothesis (ACH)?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not at all and 5 represents extremely frequently, how frequently do you use analytic methods in solving intelligence related problems?

1 2 3 4 5

On a scale from 1 to 5, where 1 no improvement and 5 represents a significant improvement, how much did this using this methodology improve your intelligence analysis?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents no enjoyment at all and 5 represents thorough enjoyment, how much did you enjoy this process?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not enough at all, and 5 represents more than enough, do you feel you found enough adequate information to complete this analysis?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not enough time at all and 5 represents more than adequate amount of time, do you feel you had enough time to complete this process?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not enough instruction at all and 5 represents more than enough instruction, do you feel you had enough instruction to complete this process using MCDM?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents extremely difficult and 5 represents extremely easy, how was this methodology to use?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not at all interested and 5 represents extremely interested, how interested are you in learning about different analytic methods after completion of this exercise?

1 2 3 4 5

On a scale from 1 to 5, where 1 represents not at all likely and 5 represents extremely likely, how likely is it that you will use this methodology in the future?

1 2 3 4 5

Approximately how much time did you spend completing this analysis (Not including classroom instruction)?___________________

What do you think the purpose of this experiment was?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

What is your class rank?

FR SO JR SR G1 G2

Please provide any additional comments or feedback in the space provided:______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________-_____________________________________________________________________________

Tests of Normality

.221 20 .012 .924 20 .116

.212 20 .019 .935 20 .193

Group A (Control)

Group B (Experiment)

Statistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

Lilliefors Significance Correctiona.

Appendix 12:

Statistical Data

Length of Time to Complete Analysis (in minutes):

Null: Group B does not take less time to finish.

Alternative: Group B takes less time to finish than group A. – claim

Will be using t-test for independent samples.

Testing normality assumption as sample sizes are < than 30.

Shapiro-Wilk test gives p-values > ( = 0.05), thus normality assumption is satisfied for both samples.

110100908070605040

Observed Value

2

1

0

-1

-2

Exp

ecte

d N

orm

al

Normal Q-Q Plot of Group A (Control)

Most points are close to the line thus the assumption of normality is satisfied for the group A.

90807060504030

Observed Value

2

1

0

-1Exp

ecte

d N

orm

al

Normal Q-Q Plot of Group B(Experiment)

Group A (Control)

110

100

90

80

70

60

50

40

Most points are close to the line thus the assumption of normality is satisfied for the group B.

Box plot shows no outliers for group A.

Group Statistics

20 70.0000 17.09109 3.82168

24 58.9583 16.61450 3.39142

GroupGroup A - Control

Group B - Experiment

Length of time in minutesN Mean Std. Deviation

Std. ErrorMean

Independent Samples Test

.542 .466 2.167 42 .036 11.04167 5.09607 .75738 21.32596

2.161 40.143 .037 11.04167 5.10950 .71612 21.36721

Equal variancesassumed

Equal variancesnot assumed

Length of time in minutesF Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

Independent Samples Test

.542 .466 2.167 42 .036

2.161 40.143 .037

Equal variancesassumed

Equal variancesnot assumed

Length of time in minutesF Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Group B (Experiment)

90

80

70

60

50

40

30

Here need to check if the assumption of equal variances is satisfied.

According to Levene’s test (P-value = 0.466) > ( = 0.05), thus assumption of equal variances is satisfied.

Box plot shows no outliers for group B.

Tests of Normality

.165 21 .137 .945 21 .268

.137 24 .200* .939 24 .157

Groups for AnalyticConfidenceGroup A (Control)

Group B (Experiment)

Level of AnalyticConfidence

Statistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

This is a lower bound of the true significance.*.

Lilliefors Significance Correctiona.

According to above table, t-test value = 2.167, P-value = 0.036.

Since (P-value = 0.036) < ( = 0.05), null hypothesis is rejected.

Conclusion: At 5% level, Group B takes less time to finish than group A.

Level of Analytic Confidence:

Null: Group B does not have higher level of analytic confidence.

Alternative: Group B does have higher level of analytic confidence than group A. – claim

Will be using t-test for independent samples.

Testing normality assumption as sample sizes are < than 30.

Kolmogorov-Smirvov test gives p-values > ( = 0.05), thus normality assumption is satisfied for both samples.

1086420

Observed Value

2

1

0

-1

-2

Exp

ecte

d N

orm

alfor groupAC= Group A (Control)

Normal Q-Q Plot of Level of AnalyticConfidence

108642

Observed Value

2

1

0

-1

-2

Exp

ecte

d N

orm

al

for groupAC= Group B (Experiment)

Normal Q-Q Plot of Level of AnalyticConfidence

Most points are close to the line thus the assumption of normality is satisfied for the group A.

Most points are close to the line thus the assumption of normality is satisfied for the group B.

Group Statistics

21 5.9286 1.75560 .38310

24 6.3125 1.74961 .35714

Groups for AnalyticConfidenceGroup A (Control)

Group B (Experiment)

Level of AnalyticConfidence

N Mean Std. DeviationStd. Error

Mean

Independent Samples Test

.002 .967 -.733 43 .467 -.38393 .52363 -1.43993 .67207

-.733 42.171 .468 -.38393 .52375 -1.44078 .67292

Equal variancesassumed

Equal variancesnot assumed

Level of AnalyticConfidence

F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

Independent Samples Test

.002 .967 -.733 43 .467

-.733 42.171 .468

Equal variancesassumed

Equal variancesnot assumed

Level of AnalyticConfidence

F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Group B(Experiment)

Group A (Control)

Groups for Analytic Confidence

10.00

8.00

6.00

4.00

2.00

0.00

Lev

el o

f A

nal

ytic

Co

nfi

den

ce

314

Here need to check if the assumption of equal variances is satisfied.

According to Levene’s test (P-value = 0.967) > ( = 0.05), thus assumption of equal variances is satisfied.

Box plot shows outliers for both groups. Even with the presence of outliers, normality is satisfied. Also these values are important for the analysis. Thus the decision is not to remove the outliers.

Tests of Normality

.208 21 .019 .941 21 .229

.317 24 .000 .749 24 .000

Groups for BulletPoint ReasoningGroup A - Control

Group B - Experiment

Bullet Point ReasoningStatistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

Lilliefors Significance Correctiona.

According to above table, t-test value = -0.733, P-value = 0.467/2 = 0.2335 (need to divide the P-value by 2 as we have one-tailed test where as SPSS gives you 2-tailed test P-value.

Since (P-value = 0.2335) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, Group B does not have higher level of analytic confidence than group A.

Bullet Point Reasoning:

Null: Group B does not have less bullet points than group A.

Alternative: Group B does have less bullet points than group A. – claim

Will be using t-test for independent samples.

Testing normality assumption as sample sizes are < than 30.

Shapiro-Wilk test gives p-value > ( = 0.05), thus normality assumption is satisfied for Group A.

12108642

Observed Value

2

1

0

-1

-2

Exp

ecte

d N

orm

alfor groupBullet= Group A - Control

Normal Q-Q Plot of Bullet PointReasoning

Both Kolmogorov-Smirnov and Shapiro-Wilk test give p-value < ( = 0.05), thus normality assumption is not satisfied for Group B. Need to check Normal Probability plot for group B.

15129630

Observed Value

2

1

0

-1

Exp

ecte

d N

orm

al

for groupBullet= Group B - Experiment

Normal Q-Q Plot of Bullet PointReasoning

Most points are close to the line thus the assumption of normality is satisfied for the group A.

The graph shows curvature thus the assumption of normality is not satisfied for the group B.

Since for group B normality is not satisfied, cannot use t-test. Will go with nonparametric tests. Need to use Mann-Whitney test.

Descriptives

6.9524 .55410

6.8915

7.0000

6.448

2.53922

3.00

12.00

9.00

5.7917 .58353

4.5845

6.9988

5.4722

5.0000

8.172

2.85869

3.00

15.00

12.00

Mean

5% Trimmed Mean

Median

Variance

Std. Deviation

Minimum

Maximum

Range

Mean

5% Trimmed Mean

Median

Variance

Std. Deviation

Minimum

Maximum

Range

Groups for BulletPoint ReasoningGroup A - Control

Group B - Experiment

Bullet Point ReasoningStatistic Std. Error

Ranks

21 26.55 557.50

24 19.90 477.50

45

Groups for BulletPoint ReasoningGroup A - Control

Group B - Experiment

Total

Bullet Point ReasoningN Mean Rank Sum of Ranks

Test Statisticsa

177.500

477.500

-1.733

.083

Mann-Whitney U

Wilcoxon W

Z

Asymp. Sig. (2-tailed)

Bullet PointReasoning

Grouping Variable: Groups for Bullet Point Reasoninga.

According to above table, Mann-Whitney test value = -1.733, P-value = 0.083/2 = 0.0415 (need to divide the P-value by 2 as we have one-tailed test where as SPSS gives you 2-tailed test P-value.

Since (P-value = 0.0415) < ( = 0.05), null hypothesis is rejected.

Conclusion: At 5% level, Group B does have less bullet points than group A.

Tests of Normality

.258 21 .001 .873 21 .011

.276 24 .000 .741 24 .000

Groups for PretestGroup A - Control

Group B - Experiment

Pretest Q. 2Statistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

Lilliefors Significance Correctiona.

Comparison Question #2 for Pre-test: On a scale from 1 to 5, where 1 represents not knowledgeable at all and 5 represents extremely knowledgeable, how knowledgeable are you with this topic?

Null: For Q2 answers do not differ significantly for group A and B.

Alternative: For Q2 answers do differ significantly for group A and B. – claim

Will be using t-test for independent samples.

Testing normality assumption as sample sizes are < than 30.

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

Most points are close to the line thus the assumption of normality is satisfied for the group A.

4.03.53.02.52.01.51.0

Observed Value

1.5

1.0

0.5

0.0

-0.5

-1.0

-1.5

Exp

ecte

d N

orm

al

for groupPre= Group A - Control

Normal Q-Q Plot of Pretest Q. 2

Group Statistics

21 2.5238 .98077 .21402

24 1.7500 .98907 .20189

Groups for PretestGroup A - Control

Group B - Experiment

Pretest Q. 2N Mean Std. Deviation

Std. ErrorMeanIndependent Samples Test

.223 .639 2.629 43 .012 .77381 .29439 .18012 1.36750

2.630 42.303 .012 .77381 .29422 .18017 1.36745

Equal variancesassumed

Equal variancesnot assumed

Pretest Q. 2F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

54321

Observed Value

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

Exp

ecte

d N

orm

alfor groupPre= Group B - Experiment

Normal Q-Q Plot of Pretest Q. 2

Group B - ExperimentGroup A - Control

Groups for Pretest

5.00

4.00

3.00

2.00

1.00

Pre

test

Q. 2

42

Most points are close to the line except one thus the assumption of normality is satisfied for the group B.

Box plot shows outlier for group B. Even with the presence of outliers, normality is satisfied. Also this value is important for the analysis. Thus the decision is not to remove the outlier.

Independent Samples Test

.223 .639 2.629 43 .012

2.630 42.303 .012

Equal variancesassumed

Equal variancesnot assumed

Pretest Q. 2F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Tests of Normality

.218 21 .010 .904 21 .041

.277 24 .000 .810 24 .000

Groups for PretestGroup A - Control

Group B - Experiment

Pretest Q. 3Statistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

Lilliefors Significance Correctiona.

Here need to check if the assumption of equal variances is satisfied.

According to Levene’s test (P-value = 0.639) > ( = 0.05), thus assumption of equal variances is satisfied.

According to above table, t-test value = 2.629, P-value = 0.012.

Since (P-value = 0.012) < ( = 0.05), null hypothesis is rejected.

Conclusion: At 5% level, Q2 answers do differ significantly for group A and B.

Comparison Question #3 for Pre-test: On a scale from 1 to 5, where 1 represents not interested at all and 5 represents extremely interested, how interested are you in learning about analytic methodologies?

Null: For Q3 answers do not differ significantly for group A and B.

Alternative: For Q3 answers do differ significantly for group A and B. – claim

Will be using t-test for independent samples.

Testing normality assumption as sample sizes are < than 30.

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

54321

Observed Value

1

0

-1

-2

Exp

ecte

d N

orm

al

for groupPre= Group A - Control

Normal Q-Q Plot of Pretest Q. 3

Most points are close to the line thus the assumption of normality is satisfied for the group A.

5.04.54.03.53.02.52.0

Observed Value

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

Exp

ecte

d N

orm

al

for groupPre= Group B - Experiment

Normal Q-Q Plot of Pretest Q. 3

Most points are close to the line thus the assumption of normality is satisfied for the group B.

Group Statistics

21 3.4762 1.03049 .22487

24 2.8333 .91683 .18715

Groups for PretestGroup A - Control

Group B - Experiment

Pretest Q. 3N Mean Std. Deviation

Std. ErrorMeanIndependent Samples Test

.201 .656 2.215 43 .032

2.197 40.433 .034

Equal variancesassumed

Equal variancesnot assumed

Pretest Q. 3F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Group B - ExperimentGroup A - Control

Groups for Pretest

5.00

4.00

3.00

2.00

1.00

Pre

test

Q. 3

3

Box plot shows outlier for group A. Even with the presence of outliers, normality is satisfied. Also this value is important for the analysis. Thus the decision is not to remove the outlier.

Independent Samples Test

.201 .656 2.215 43 .032

2.197 40.433 .034

Equal variancesassumed

Equal variancesnot assumed

Pretest Q. 3F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Tests of Normality

.227 21 .006 .884 21 .018

.336 24 .000 .820 24 .001

Groups for PretestGroup A - Control

Group B - Experiment

Pretest Q. 1Statistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

Lilliefors Significance Correctiona.

Here need to check if the assumption of equal variances is satisfied.

According to Levene’s test (P-value = 0.656) > ( = 0.05), thus assumption of equal variances is satisfied.

According to above table, t-test value = 2.215, P-value = 0.032.

Since (P-value = 0.032) < ( = 0.05), null hypothesis is rejected.

Conclusion: At 5% level, Q3 answers do differ significantly for group A and B.

Comparison Question # 1 for Pre-test: On a scale from 1 to 5, where 1 represents not interested at all and 5 represents extremely interested, how interested are you in participating in this experiment?

Null: For Q1 answers do not differ significantly for group A and B.

Alternative: For Q1 answers do differ significantly for group A and B. – claim

Will be using t-test for independent samples.

Testing normality assumption as sample sizes are < than 30.

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

5.04.54.03.53.02.52.0

Observed Value

1.5

1.0

0.5

0.0

-0.5

-1.0

-1.5

Exp

ecte

d N

orm

al

for groupPre= Group A - Control

Normal Q-Q Plot of Pretest Q. 1

Most points are close to the line thus the assumption of normality is satisfied for the group A.

5.04.54.03.53.02.52.0

Observed Value

2

1

0

-1

-2

Exp

ecte

d N

orm

al

for groupPre= Group B - Experiment

Normal Q-Q Plot of Pretest Q. 1

Most points are close to the line thus the assumption of normality is satisfied for the group B.

Group Statistics

21 3.4762 .98077 .21402

24 3.3333 .76139 .15542

Groups for PretestGroup A - Control

Group B - Experiment

Pretest Q. 1N Mean Std. Deviation

Std. ErrorMeanIndependent Samples Test

2.674 .109 .549 43 .586

.540 37.570 .592

Equal variancesassumed

Equal variancesnot assumed

Pretest Q. 1F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Independent Samples Test

2.674 .109 .549 43 .586

.540 37.570 .592

Equal variancesassumed

Equal variancesnot assumed

Pretest Q. 1F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Group B - ExperimentGroup A - Control

Groups for Pretest

5.00

4.50

4.00

3.50

3.00

2.50

2.00

Pre

test

Q. 1

Here need to check if the assumption of equal variances is satisfied.

According to Levene’s test (P-value = 0.109) > ( = 0.05), thus assumption of equal variances is satisfied.

Box plot does not show outliers for either group. This is a desired result for normality.

Tests of Normality

.277 21 .000 .860 21 .006

.297 24 .000 .830 24 .001

Groups for PretestGroup A - Control

Group B - Experiment

Posttest Q. 3Statistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

Lilliefors Significance Correctiona.

According to above table, t-test value = 0.549, P-value = 0.586.

Since (P-value = 0.586) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, Q1 answers do not differ significantly for group A and B.

Comparison Question #3 for Post-test: On a scale from 1 to 5, where 1 represents not at all and 5 represents extremely frequently, how frequently do you use structured analytic methods in solving intelligence related problems?

Null: For Q3 answers do not differ significantly for group A and B.

Alternative: For Q3 answers do differ significantly for group A and B. – claim

Will be using t-test for independent samples.

Testing normality assumption as sample sizes are < than 30.

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

5.04.54.03.53.02.52.0

Observed Value

1.5

1.0

0.5

0.0

-0.5

-1.0

-1.5

Exp

ecte

d N

orm

alfor groupPre= Group A - Control

Normal Q-Q Plot of Posttest Q. 3

Most points are close to the line thus the assumption of normality is satisfied for the group A.

5.04.54.03.53.02.52.0

Observed Value

2

1

0

-1

-2

Exp

ecte

d N

orm

al

for groupPre= Group B - Experiment

Normal Q-Q Plot of Posttest Q. 3

Most points are close to the line thus the assumption of normality is satisfied for the group B.

Group Statistics

21 3.1429 .91026 .19863

24 3.5000 .72232 .14744

Groups for PretestGroup A - Control

Group B - Experiment

Posttest Q. 3N Mean Std. Deviation

Std. ErrorMean

Independent Samples Test

.155 .696 -1.466 43 .150

-1.444 38.063 .157

Equal variancesassumed

Equal variancesnot assumed

Posttest Q. 3F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Group B - ExperimentGroup A - Control

Groups for Pretest

5.00

4.50

4.00

3.50

3.00

2.50

2.00

Po

stte

st Q

. 3Box plot does not show outliers for either group. This is a desired result for normality.

Independent Samples Test

.155 .696 -1.466 43 .150

-1.444 38.063 .157

Equal variancesassumed

Equal variancesnot assumed

Posttest Q. 3F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Tests of Normality

.241 20 .003 .862 20 .009

.299 24 .000 .812 24 .000

Groups for PretestGroup A - Control

Group B - Experiment

Posttest Q. 4Statistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

Lilliefors Significance Correctiona.

Here need to check if the assumption of equal variances is satisfied.

According to Levene’s test (P-value = 0.696) > ( = 0.05), thus assumption of equal variances is satisfied.

According to above table, t-test value = -1.466, P-value = 0.150.

Since (P-value = 0.150) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, Q3 answers do not differ significantly for group A and B.

Comparison Question #4 for Post-test: On a scale from 1 to 5, where 1 represents not at all and 5 represents a significant improvement, how much do you feel analytic methods improve intelligence analysis?

Null: For Q4 answers do not differ significantly for group A and B.

Alternative: For Q4 answers do differ significantly for group A and B. – claim

Will be using t-test for independent samples.

Testing normality assumption as sample sizes are < than 30.

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

5.04.54.03.53.02.52.0

Observed Value

1.0

0.5

0.0

-0.5

-1.0

-1.5

Exp

ecte

d N

orm

al

for groupPre= Group A - Control

Normal Q-Q Plot of Posttest Q. 4

Most points are close to the line thus the assumption of normality is satisfied for the group A.

Group Statistics

20 3.9000 .96791 .21643

24 3.5417 .65801 .13431

Groups for PretestGroup A - Control

Group B - Experiment

Posttest Q. 4N Mean Std. Deviation

Std. ErrorMean

5.04.54.03.53.02.52.0

Observed Value

2

1

0

-1

-2

Exp

ecte

d N

orm

alfor groupPre= Group B - Experiment

Normal Q-Q Plot of Posttest Q. 4

Group B - ExperimentGroup A - Control

Groups for Pretest

5.00

4.50

4.00

3.50

3.00

2.50

2.00

Po

stte

st Q

. 4

Most points are close to the line thus the assumption of normality is satisfied for the group B.

Box plot does not show outliers for either group. This is a desired result for normality.

Tests of Normality

.349 21 .000 .727 21 .000

.510 24 .000 .401 24 .000

Groups for PretestGroup A - Control

Group B - Experiment

Posttest Q. 6Statistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

Lilliefors Significance Correctiona.

Independent Samples Test

1.345 .253 1.456 42 .153

1.407 32.474 .169

Equal variancesassumed

Equal variancesnot assumed

Posttest Q. 4F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Here need to check if the assumption of equal variances is satisfied.

According to Levene’s test (P-value = 0.253) > ( = 0.05), thus assumption of equal variances is satisfied.

Independent Samples Test

1.345 .253 1.456 42 .153

1.407 32.474 .169

Equal variancesassumed

Equal variancesnot assumed

Posttest Q. 4F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

According to above table, t-test value = 1.456, P-value = 0.153.

Since (P-value = 0.153) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, Q4 answers do not differ significantly for group A and B.

Comparison Question #6 for Post-test: On a scale from 1 to 5, where 1 represents not enough time at all and 5 represents more than adequate amount of time, do you feel you had enough time to complete this process?

Null: For Q6 answers do not differ significantly for group A and B.

Alternative: For Q6 answers do differ significantly for group A and B. – claim

Will be using t-test for independent samples.

Testing normality assumption as sample sizes are < than 30.

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

5.04.54.03.53.0

Observed Value

0.6

0.3

0.0

-0.3

-0.6

-0.9

-1.2

-1.5

Exp

ecte

d N

orm

al

for groupPre= Group A - Control

Normal Q-Q Plot of Posttest Q. 6

Most points are close to the line thus the assumption of normality is satisfied for the group A.

5.04.54.03.53.0

Observed Value

0.0

-0.5

-1.0

-1.5

Exp

ecte

d N

orm

al

for groupPre= Group B - Experiment

Normal Q-Q Plot of Posttest Q. 6

Descriptives

4.4286 .16288

.557

.74642

3.00

5.00

4.8333 .09829

.232

.48154

3.00

5.00

Mean

Variance

Std. Deviation

Minimum

Maximum

Mean

Variance

Std. Deviation

Minimum

Maximum

Groups for PretestGroup A - Control

Group B - Experiment

Posttest Q. 6Statistic Std. Error

Most points are not close to the line thus the assumption of normality is not satisfied for the group B.

We cannot use the t-test. Need to use nonparametric test, Mann-Whitney test.

Ranks

21 19.36 406.50

24 26.19 628.50

45

Groups for PretestGroup A - Control

Group B - Experiment

Total

Posttest Q. 6N Mean Rank Sum of Ranks

Tests of Normality

.223 20 .010 .809 20 .001

.261 24 .000 .872 24 .006

Groups for PretestGroup A - Control

Group B - Experiment

Posttest Q. 7Statistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

Lilliefors Significance Correctiona.

Test Statisticsa

175.500

406.500

-2.248

.025

Mann-Whitney U

Wilcoxon W

Z

Asymp. Sig. (2-tailed)

Posttest Q. 6

Grouping Variable: Groups for Pretesta.

According to above table, Mann-Whitney test value = 175.5, P-value = 0.025.

Since (P-value = 0.025) < ( = 0.05), null hypothesis is rejected.

Conclusion: At 5% level, Q6 answers do differ significantly for group A and B.

Comparison Question #7 for Post-test: On a scale from 1 to 5, where 1 represents no enjoyment at all and 5 represents thorough enjoyment, how much did you enjoy this process?

Null: For Q7 answers do not differ significantly for group A and B.

Alternative: For Q7 answers do differ significantly for group A and B. – claim

Will be using t-test for independent samples.

Testing normality assumption as sample sizes are < than 30.

Group Statistics

20 3.9000 .78807 .17622

24 3.4583 .93153 .19015

Groups for PretestGroup A - Control

Group B - Experiment

Posttest Q. 7N Mean Std. Deviation

Std. ErrorMean

Independent Samples Test

.685 .412 1.678 42 .101

1.704 41.984 .096

Equal variancesassumed

Equal variancesnot assumed

Posttest Q. 7F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

5.04.54.03.53.0

Observed Value

1.0

0.5

0.0

-0.5

-1.0

Exp

ecte

d N

orm

al

for groupPre= Group A - Control

Normal Q-Q Plot of Posttest Q. 7

54321

Observed Value

2

1

0

-1

-2

Exp

ecte

d N

orm

al

for groupPre= Group B - Experiment

Normal Q-Q Plot of Posttest Q. 7

Most points are close to the line thus the assumption of normality is satisfied for the group A.

Most points are close to the line thus the assumption of normality is satisfied for the group B.

Independent Samples Test

.685 .412 1.678 42 .101

1.704 41.984 .096

Equal variancesassumed

Equal variancesnot assumed

Posttest Q. 7F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)

t-test for Equality of Means

Here need to check if the assumption of equal variances is satisfied.

According to Levene’s test (P-value = 0.412) > ( = 0.05), thus assumption of equal variances is satisfied.

According to above table, t-test value = 1.678, P-value = 0.101.

Since (P-value = 0.101) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, Q7 answers do not differ significantly for group A and B.