ABS VV&A Framework Study Phase II Pythagoras COIN – Application of the Validation Framework

ABS VV&A Framework Study Phase IIPythagoras COIN – Application of the Validation

Framework

Lisa Jean MoyaWernerAnderson, Inc.

[email protected]

Phase II Workshop 39 July 2008

7/9/2008

1

P-COIN Validation Briefing Wkshp3 Moya

mailto:[email protected]

Scenario

7/9/2008 P-COIN Validation Briefing Wkshp3 Moya

2

Tsun

ami o

ccur

sM

EU/M

EB o

rder

ed to

AO

A

MEU

sai

ls fo

r AO

A

MEB

sai

ls fo

r AO

A

MEB

arri

ves

AOA

MEU

arri

ves

AOA

PB

day

PB

-10

PB

-19

PB

-24

PB

-25

PB

-26

MEU

Fw

d m

oves

ash

ore

& es

t lia

ison

PB

-18

MEU

con

duct

s in

itial

sur

vey

Analysis Context• Can Pythagoras be used to model population dynamics?

• In a Disaster Relief/Humanitarian Assistance mission for the stated scenario, is it better to base the MAGTF ashore or afloat?

• Alternative selection drivers– Do no harm: create no increase in insurgency activity.– Improve the political situation: create an improvement in GOVT and

Pro-GOVT sectors.• Measures

– Box & Whisker plot comparisons of the percent of population by population segment in each insurgency sector at end state (18 months)


3

Conceptual Model of Civilian Population4

FARC Pro-FARC Neutral Pro-GoC GoC

Insurgency Behavior Orientation

CivilianPopulation

PopulationSegments

FARC = Revolutionary Armed Forces of Colombia

GoC = Govt of Colombia

Natural DriftSalienceInfluencing events



5

Catholic Church

Displaced Persons

Illicit Organizations

Military

Old Money

Police

Urban Middle Class

Urban Poor

FARC Pro-FARC Neutral Pro-GOVT GOVT


6

Catholic Church

Displaced Persons


Military

Old Money

Police

Urban Middle Class

Urban Poor


3.03

2.63

3.76

3.89

3.21

4.23

3.62

3.39

Areas of Interest

• Core– MAGTF influence on

Insurgency orientation• Cases

– MAGTF/No MAGTF– Ashore/Afloat

• Dynamic Influences– Natural Drift– Salience

• Background– Population segments


7

Areas of Interest

• Core– MAGTF influence on

Insurgency orientation• Cases

– MAGTF/No MAGTF– Ashore/Afloat

• Dynamic Influences– Natural Drift– Salience

• Background– Population segments


8

First order assessment

Agent Allocation

7/8/2008 WG29MoyaLisa 13

Segment FARC Pro-FARC Neutral Pro-GOVT GOVT Catholic Church 0 0 38 0 62 Displaced Persons 7 37 45 10 1 Illicit Organizations 0 39 28 18 15 Military 2 10 0 88 0 Old Money 0 2 4 63 31 Police 0 5 0 95 0 Urban Middle Class 4 4 62 10 20 Urban Poor 6 9 66 15 4 Total 19 106 243 299 133

Pythagoras-COIN Building Blocks


9

Natural Drift: Incremental


FARC Pro-FARC Neutral Pro-GOVT

GOVT

FARC 99.8% 0.1% 0.0% 0.0% 0.0%

Pro-FARC 0.4% 99.5% 0.2% 0.0% 0.0%

Neutral 0.0% 0.7% 98.7% 0.5% 0.0%

Pro-GOVT 0.0% 0.0% 0.9% 99.1% 0.1%

GOVT 0.0% 0.1% 0.0% 0.1% 99.7%

Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5

+4 0 +2 0 0

Displaced Persons Pro-FARC Attribute Changes

Catholic Church

Displaced Persons


Military

Old Money

Police

Urban Middle Class

Urban Poor


Salience: Relative


Cath

olic

Chur

ch

Disp

lace

d Pe

rson

s

Illic

it O

rgs

Mili

tary

Old

M

oney

Polic

e

Urb

an

Mid

dle

Clas

s

Urb

an

Poor

Catholic Church 0.000 0.000 0.000 (0.078) 0.801 (0.078) 0.000 0.000Displaced Persons 0.706 (0.262) (0.052) (0.230) 0.178 (0.366) 0.480 (0.208)Illicit Organizations 0.207 0.000 0.425 (0.053) 0.420 (0.190) (0.467) 0.183Military 0.750 (0.351) (0.295) 0.492 0.262 0.685 0.000 (0.257)Old Money 0.474 (0.486) (0.713) 0.000 0.840 (0.506) 0.494 (0.250)Police 0.000 (0.843) (0.174) (0.039) (0.273) 0.767 0.000 (0.218)Urban Middle Class 0.816 (0.623) (0.387) 0.136 (0.269) (0.205) 0.392 0.132Urban Poor 0.202 (0.259) 0.101 0.225 (0.033) (0.134) 0.552 (0.079)

Catholic Church

Displaced Persons


Military

Old Money

Police

Urban Middle Class

Urban Poor


3.03

2.63

3.76

3.89

3.21

4.23

3.62

3.39

Influencing Events: Multiplier


Sea Based Shore Based

Segment Right Left Right Left Catholic Church 0.845 0.000 0.401 0.000 Displaced Persons 0.596 0.000 0.721 0.117 Illicit Organizations 0.000 0.397 0.000 0.447 Military 0.000 0.408 0.000 0.408 Old Money 0.631 0.000 0.631 0.000 Police 0.564 0.000 0.564 0.000 Urban Middle Class 0.780 0.000 0.184 0.210 Urban Poor 0.798 0.000 0.722 0.211

Salience as a Dynamic Influence


10

Cath

olic

Ch

urch

Disp

lace

d Pe

rson

s

Illic

it O

rgan

izatio

ns

Mili

tary

Old

Mon

ey

Polic

e

Urb

an M

iddl

e Cl

ass

Urb

an P

oor

Catholic Church 0.000 0.000 0.000 (0.078) 0.801 (0.078) 0.000 0.000 Displaced Persons 0.706 (0.262) (0.052) (0.230) 0.178 (0.366) 0.480 (0.208) Illicit Organizations 0.207 0.000 0.425 (0.053) 0.420 (0.190) (0.467) 0.183 Military 0.750 (0.351) (0.295) 0.492 0.262 0.685 0.000 (0.257) Old Money 0.474 (0.486) (0.713) 0.000 0.840 (0.506) 0.494 (0.250) Police 0.000 (0.843) (0.174) (0.039) (0.273) 0.767 0.000 (0.218) Urban Middle Class 0.816 (0.623) (0.387) 0.136 (0.269) (0.205) 0.392 0.132 Urban Poor 0.202 (0.259) 0.101 0.225 (0.033) (0.134) 0.552 (0.079)

Salience as a Dynamic Influence


11

Cath

olic

Ch

urch

Disp

lace

d Pe

rson

s

Illic

it O

rgan

izatio

ns

Mili

tary

Old

Mon

ey

Polic

e

Urb

an M

iddl

e Cl

ass

Urb

an P

oor


So "Initial" Values Displaced Persons Neutral "Initial"

Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5 0 7 988 5 0

Catholic Church ProCOIN "Initial" 0 0 0 1000 0

Catholic Church Influence

- - - 71% - Displaced Persons Change

0 7 988 711 0 Final Values (Normalization)

0 4 579 417 0

As the simulation might progress …


12

So "Initial" Values Displaced Persons Neutral Agent

Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5 0 7 988 5 0

Catholic Church ProCOIN "Initial" with attributes at a future timestep (currently Pro-FARC) 0 1000 0 0 0

Catholic Church Influence

- - - 71% -

Displaced Persons Change 0 7 988 1 0

Final Values (Normalization)

0 4 992 1 0

An extreme example to demonstrate the issue

Implications• Influence changers only applied w.r.t. initial state;

changes in orientation do not change the influence→ Dynamic effects of salience and natural drift are not

accounted for→ Secondary and tertiary effects of MAGTF arrival not

accounted for… Dampening on the insurgency orientation! Risk is that the simulation does not model the desired

population dynamics ! Risk is that the dynamics of the MAGTF arrival are not

adequately captured7/9/2008 P-COIN Validation Briefing Wkshp3 Moya

13

Data


14

"Humanitarian exchange." Wikipedia, The Free Encyclopedia. 3 Jul 2008, 10:26 UTC. Wikimedia Foundation, Inc. 9 Jul 2008 <http://en.wikipedia.org/w/index.php?title=Humanitarian_exchange&oldid=223273181>.

• Data imprecise– Mitigated by “tolerance” and

multiple runs• Data processing

– Need to verify that process results in expected directional & magnitude shifts

• Data is perishable– Natural drift data has an

embedded perishibility• Would an actual model use

require a “warm-up period” on the Markov Chain?

– Other influencing events might significantly change the data values

http://en.wikipedia.org/w/index.php?title=Humanitarian_exchange&oldid=223273181



http://en.wikipedia.org/wiki/Image:Jornadasecuestro1.jpg

Uses of the Model

• Markov assumptions in referent descriptions– No long term effects– May need a “warm-up”

period• Outside the salience or along

with salience?

• Data precision– No exact results; results in

the distribution• Data perishability

– Need to collect new data after significant events

– Including MAGTF departure

• Were the dynamics captured … – Could add other influencing

events• Could add additional

dynamics

• Q: Can we apply the influencers more robustly?

• Q: Would changing our initial starting agents help?


15

http://en.wikipedia.org/wiki/Image:Jornadasecuestro1.jpg

Analysis Results

• Given in box & whiskers plots at end state with data table

• No statistical comparisons

• No “hard” description of better

• Point estimate in time (18 months)


16

Percent Population Pro-COIN, COIN

40

50

60

70

80

90

100

NoM

AG

TF

Ash

ore

Aflo

at

NoM

AGTF

Ash

ore

Aflo

at

NoM

AGTF

Ash

ore

Aflo

at

NoM

AG

TF

Asho

re

Aflo

at

NoM

AGTF

Ash

ore

Aflo

at

NoM

AGTF

Ash

ore

Aflo

at

NoM

AGTF

Ash

ore

Aflo

at

NoM

AGTF

Ash

ore

Aflo

at

Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban MiddleClass

Urban Poor

Perc

ent P

opul

atio

n

Afloat has nearly equal or more Pro-COIN, COINPercent Population Pro-FARC, FARC

0

5

10

15

20

25

30

35

40

NoM

AGTF

Ash

ore

Aflo

at

NoM

AGTF

Ash

ore

Aflo

at

NoM

AGTF

Asho

re

Aflo

at

NoM

AGTF

Ash

ore

Aflo

at

NoM

AGTF

Ash

ore

Aflo

at

NoM

AGTF

Ash

ore

Aflo

at

NoM

AG

TF

Ash

ore

Aflo

at

NoM

AG

TF

Ash

ore

Aflo

at

Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor

Perc

ent P

opul

atio

n

Afloat has the same or fewer Pro-FARC, FARC

MAGTF Influence Data

• Expect: Sea better than shore – Urban Middle Class & Urban Poor “drive” result

• Catholic Church drive more right with Sea vs Shore• Displaced Persons “wash”?• Salience causes Urban Poor and Middle Class to be like Military;

Military to be like Catholic Church – in opposition to the direct MAGTF influence … what would we expect?


17





Cath

olic

Ch

urch

Disp

lace

d Pe

rson

s

Illic

it O

rgan

izatio

ns

Mili

tary

Old

Mon

ey

Polic

e

Urb

an M

iddl

e Cl

ass

Urb

an P

oor


Second Order Effects


18

What do we expect in the interactions

Earlier iteration (Military)


19

• Not clear from documentation what is being reported

• These multiple influences appear to be captured– Presuming no population

segment strays “too far” from initial state

– Except … Military does!

No MAGTF

AfloatAshore

0

10

20

30

40

50

60

70

80

90

100

0 2 4 6 8 10 12 14 16 18 20 22 24

Months

Popu

latio

n Se

gmen

t Per

cent

age

FARC Pro-FARC Neutral Pro-COIN COIN

0

10

20

30

40

50

60

70

80

90

100

0 2 4 6 8 10 12 14 16 18 20 22 24

Months

Popu

latio

n Se

gmen

t Per

cent

age

FARC Pro-FARC Neutral Pro-COIN COIN

20

Analysis Conclusions (IPR#5)• “Which COA is better?” cannot be answered with much

confidence• “What is the chance that ashore is better than afloat?” can be

answered with greater confidence– More pro-Government sentiment if Marines stay afloat– Lower pro-FARC sentiment if Marines stay afloat– Marine arrival has a polarizing effect (fewer neutrals)– Marine arrival in either case increases anti-Government sentiments of

the Illicit Organizations and the Military• Afloat seems to usually do less harm.

– There is no factor in our influence estimation that BOTH reduces the negative impact of Ashore AND increase the negative impact of Afloat

21

Analysis Conclusions (IPR#5) (cont)

• Because the current Markov chain will eventually return to the same steady state, regardless of MAGTF action, once the MAGTF leaves, we need to consider:– Does the MAGTF commander care about leaving a lasting

impression?– At what point in time do we measure ‘better’?– Pythagoras could change the final steady state as a

function of one or more population segments exceeding or falling below some target value. However, this data was not collected

Validation Conclusions1. The P-COIN simulation fails to capture the dynamic effects

intended in the conceptual model of the insurgency in Colombia provided to the P-COIN developer. That is, P-COIN does not capture the secondary and tertiary effects of the natural drift of population segments between insurgency sectors or the salience between population segments resulting from the influencing event of the MAGTF.

2. The data supporting the P-COIN model is perishable and of low precision. Care should be taken when using the data beyond its origination date; perhaps “warming-up” the Markov chains supporting the data used to build the P-COIN model. Further, the data cannot be deemed valid if an influencing event occurs that would cause the base data used in this simulation to change.


22

Validation Conclusions (cont)3. The P-COIN model should not be used to evaluate long term

effects on the population resulting from the influencing event of the MAGTF arrival.

4. This model and simulation cannot be deemed as predictive of the actual population distributions amongst insurgency sectors in the event that the scenario described in the scenario documentation actually occurs.

5. There is little risk in using the results of the analysis since the analysis does not advocate a change in current Marine Corps procedure. However, item 1 implies that P-COIN also provides little insight into the ashore or afloat question in its current implementation.


23

Recommend: Applying influencers more robustly

What Would Be Useful• Better documentation on the P-COIN instantiation• Time series data• A descriptive walk-thru of results charts (meaning &

implications)• Verification cases (isolated effects) to ensure dynamics have

expected direction (first derivative) and order of magnitude– Descriptions of why we believe it is correct

• Referent– Better explanations of expected resulting effects from data

values• Most had to be inferred• Order of magnitude differences unknown

• Expected interaction effects would be “spectacular”7/9/2008 P-COIN Validation Briefing Wkshp3 Moya

24

Can the Results Be Trusted?

• Without trusting the dynamics– Take caution but …– Recommendation is

innocuous• Under current political

circumstances– No … new data is required

• Can Pythagoras model population dynamics– Probably … more care is required in the instantiation


25

Impact

High

HighLow

Low

Likelihood of Failure

Unacceptable

Very High

High

Some

Acceptable

Low

Negligible

Risk Of Using The ABS

Impact

High

HighLow

Low

Likelihood of Failure

Unacceptable

Very High

High

Some

Acceptable

Low

Negligible

Risk Of Using The ABS

Levels of Validation Process


26

Adapted from Harmon & Youngblood (2005) p. 186

Subjective validation

Objective requirements

Objective results

Objective referent

Automated validation

Initial (level 0)

Subjective (level 1)

Complete(level 2)

Accurate(level 3)

Confident(level 4)

Automated(level 5)

Tier 0, “I have no idea.”

Tier 1, “It works; trust me.”

Tier 3, “It does the right things; its repns are complete enough.”

Tier 4, “For what it does; its repns are accurate enough.”

Tier 5, “I’m confident that this simn is valid.”

SME

SME

Indnt Observer

Indnt Observer

Formal Proof

None SME opinion Single source Multiple sources

Rigorously derived

Correlated with statistical estimates of

uncertaintiesReferent

None

Valid

ated

Analyz

ed

Conce

ptual

Model

ABS VV&A Framework Study Phase II Pythagoras COIN – Application of the Validation Framework

Documents

Transcript of ABS VV&A Framework Study Phase II Pythagoras COIN – Application of the Validation Framework