Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex...
-
Upload
clyde-douglas -
Category
Documents
-
view
217 -
download
0
Transcript of Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex...
![Page 1: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/1.jpg)
Umbrella PresentationTheme C: Cognitive Science of Cyber SA
ASU (Cooke) Cyber Security as a Complex Cognitive
SystemPSU (McNeese & Hall) Computer-aided Computer-Aided Human Centric Cyber
Situation Awareness
1
![Page 2: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/2.jpg)
System Analysts
Computer network
SoftwareSensors, probes• Hyper
Sentry• Cruiser
Mul
ti-S
enso
ry H
um
an
Com
put
er
Inte
ract
ion
• Enterprise Model
• Activity Logs • IDS reports
• Vulnerabilities
Cognitive Models & Decision Aids• Instance Based Learning Models
• Simulation• Measures of SA & Shared SA
• • • D
ata
Co
nd
itio
nin
gA
sso
cia
tion
& C
orr
ela
tion
Automated Reasoning Tools• R-CAST• Plan-based
narratives• Graphical
models• Uncertainty
analysis
Information Aggregation
& Fusion• Transaction
Graph methods
• Damage assessment
Computer network
• •
•
Real World
Test-bed
2
![Page 3: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/3.jpg)
Situation Awareness
Endsley’s Definition:the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future
Perception Comprehension Projection
![Page 4: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/4.jpg)
Cyber Situation Awareness is Inherently Human
4
SA is not in the technology (e.g., visualization); it is in the interface between humans and technology
![Page 5: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/5.jpg)
Team Situation Awareness
A team’s coordinated perception and action in response to a change in the environment
Contrary to view that all
team members need to “be on the same page”
5
![Page 6: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/6.jpg)
Detector Responder Threat Analyst
Perception Comprehension Projection
Cyber SA is Distributed and Emergent
![Page 7: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/7.jpg)
Detector Responder Threat Analyst
Perception Comprehension Projection
Cyber SA is Distributed and Emergent
![Page 8: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/8.jpg)
Cyber Security as a Complex Cognitive SystemN.Cooke, P. Rajivan, M. Champion, G. Dube, V. Buchanan, S. Jariwala
Cognitive ScienceTheoretical Foundations
Top-down
Bottom-Up
Distributed Research Simulations
Metrics &Measures
Cognitive Systems Engineering
Observe
Observation
Fields of Practice
Cyber Defense
CyberCog & DEXTAR
Communication & CoordinationTeam Situation Awareness
Agent-Based & EAST Modeling
Interactive Team Cognition/ Sociotechnical Systems Theory
Tools & Methods
![Page 9: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/9.jpg)
Theoretical FoundationsHuman-Centered
Distributed Research Simulations
CyberCog & DEXTARInteractive Team Cognition/Sociotechnical Systems Theory Workload
Specialization
Teams vs Groups
Team and Organization Models
Actual ExperimentalStudiesConducted
Cyber Security as a Complex Cognitive SystemN.Cooke, P. Rajivan, M. Champion, G. Dube, V. Buchanan, S. Jariwala
![Page 10: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/10.jpg)
Computer-Aided Human Centric CyberSituation AwarenessM. McNeese, D. Hall, N. Giacobe,
V. Mancuso, D. Minotra, and E. McMillan
Cognitive ScienceTheoretical Foundations
Top-down
Bottom-Up
Distributed Research Simulations
Metrics &Measures
Cognitive Systems Engineering
Observe
Observation
Fields of Practice
Cyber Defense
teamNETS
Visual Analytics Testbench Complex Event Processing
Situated Cognition
Tools & Methods
![Page 11: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/11.jpg)
Theoretical FoundationsHuman-Centered
Distributed Research Simulations
teamNETSSituated Cognition
Attention/Disruption
Memory / Access Awareness
Team Cognition
Embedded Model of the Threat
Actual ExperimentalStudiesConducted
Computer-Aided Human Centric CyberSituation Awareness
M. McNeese, D. Hall, N. Giacobe, V. Mancuso, D. Minotra, and E. McMillan
![Page 12: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/12.jpg)
ASU/PSU ObjectivesPSU Objectives
• To understand Individual and Team cognition of Situation Awareness in Cyber-Security domains
• Refine and implement evaluation environment to support evaluation of new analysis models, cognitive tools, and adversarial team cognition via hidden knowledge profiles
• Develop new tools for practice based on field- and laboratory-based findings
ASU Objectives
• To develop theory of team-based SA to inform assessment metrics and improve interventions (training and decision aids)
• Iterative Refinement of Cyber Testbeds based on cognitive analysis of the domain
– Cybercog– DEXTAR
• Conduct experiments on Cyber TSA in the testbed to develop theory and metrics
• Extend empirical data through modeling
![Page 13: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/13.jpg)
Cyber Security as a Complex Cognitive System
Cyber Security as a Complex Cognitive System
Nancy J. Cooke, PhD
Prashanth Rajivan, MS
Michael Champion, MSShree JariwalaGeneviève Dubé, Université Laval, Québec Verica Buchanan
Arizona State UniversityOctober 29, 2013
13This work has been supported by the Army Research Office under MURI Grant W911NF-09-1-0525.
![Page 14: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/14.jpg)
Outline•Overview of Project •Definitions and Theoretical Drivers •Empirical Study on Teams vs. Groups •Agent-Based Modeling •Two Case Studies and EAST Models •Next Steps
14
![Page 15: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/15.jpg)
Overview of Project
15
![Page 16: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/16.jpg)
ASU Project OverviewObjectives:
Understand and Improve Team Cyber Situation Awareness via • Understanding cognitive /teamwork elements of situation awareness
in cyber-security domains• Implementing a synthetic task environment to support team in the
loop experiments for evaluation of new algorithms, tools and cognitive models
• Developing new theories, metrics, and models to extend our understanding of cyber situation awareness
Department of Defense Benefit:• Metrics, models, & testbeds for assessing human effectiveness and
team situation awareness (TSA) in cyber domain• Testbed for training cyber analysts and testing (V&V) algorithms and
tools for improving cyber TSA
Scientific/Technical Approach - Year 4• Explore the role of teamwork in cyber defense
through:• Empirical work in CyberCog testbed• Agent-Based Modeling• Case Studies and EAST Modeling
• Further refine team metrics and testbeds
Year 4 Accomplishments• Found an empirical benefit of cyber teaming• Replicated this benefit in an agent-based model• Compared two cyber defense organizations• Refined team metrics and cybercog testbed
ChallengeStruggle to maintain realism in testbed scenarios while
allowing for novice participation and team interaction – now addressing with CyberCog and Dextar
![Page 17: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/17.jpg)
17
Summary of FY 13 ASU Accomplishments
PUBLICATIONSCooke, N. J., Champion, M., Rajivan, P., & Jariwala, S. (2013). Cyber Situation Awareness and Teamwork. EAI Endorsed Transactions on Security and Safety. Special Section on: The Cognitive Science of Cyber Defense, 13. Cooke, N. J. & McNeese, M. (2013). Preface to special issue on the cognitive science of cyber defence analysis. EAI Endorsed Transactions on Security and Safety. Special Section on: The Cognitive Science of Cyber Defense, 13
Rajivan, P., Champion, M., Cooke, N. J., Jariwala, S., Dube, G., & Buchanan, V. (2013). Effects of teamwork versus group work on signal detection in cyber defense teams. In D. D. Schmorrow and C.M. Fidopiastis (Eds.), AC/HCII, LNAI 8027, pp. 172-180., Berlin: Springer-Verlag. Rajivan, P., Janssen, M. A., & Cooke, N. J., (2013). Agent-based model of a cyber security defense analyst team. Proceedings of the 57th Annual Conference of the Human Factors and Ergonomics Society, Santa Monica, CA: Human Factors and Ergonomics Society.
Champion, M., Rajivan, R., Jariwala, S., Cooke, N. J., & Buchanan, V. Understanding the cyber security task. Poster presented at ASU's Sixth Annual Workshop on Information Assurance, May 1, 2013, Tempe, AZ.
STUDENTS SUPPORTED• Prashanth Rajivan (PhD)• Verica Buchanan (UG)
PROJECTS SUPPORTED FY 13• CyberCog and metrics development• CyberCog study• Agent-based models of cyber teaming• Agent-based models of cyber warfare• Case Studies and EAST models
COLLABORATION• Coty Gonzalez – IBLT and Agent-Based Modeling• Sushil Jajodia – DEXTAR• Several MURI partners on an ARL proposal
TECH TRANSFER• Working with Charles River Analytics and AFRL on team
measures of cyber defense• Working with SA Technologies on cyber visualization• Presentation to ASU Information Assurance • Presentation to General Dynamics – The Edge
AWARDPrashanth Rajivan wins HFES 2013 Alphonse Chapanis Award for best student paper!!!
![Page 18: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/18.jpg)
Definitions and Theoretical Drivers
18
![Page 19: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/19.jpg)
Theoretical Drivers
• Interactive Team Cognition• Sociotechnical Systems Theory/
Human Systems Integration
19
![Page 20: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/20.jpg)
Interactive Team CognitionTeam is unit of analysis = Heterogeneous and interdependent group of individuals (human or synthetic) who plan, decide,
perceive, design, solve problems, and act as an integrated system.
Cognitive activity at the team level= Team Cognition
Improved team cognition Improved team/system effectiveness
Heterogeneous = differing backgrounds, differing perspectives on situation
(surgery, basketball)20
![Page 21: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/21.jpg)
Interactive Team CognitionTeam interactions often in the form of explicit
communications are the foundation of team cognition
ASSUMPTIONS
1) Team cognition is an activity; not a property or product
2) Team cognition is inextricably tied to context
3) Team cognition is best measured and studied when the team is the unit of analysis
21
![Page 22: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/22.jpg)
Implications of Interactive Team Cognition
• Focus cognitive task analysis on team interactions
• Focus metrics on team interactions (team SA)
• Intervene to affect team interactions
22
![Page 23: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/23.jpg)
Cyber Defense as a Sociotechnical System• Cyber defense functions involve cognitive processes allocated to
– Human Operators – Tools/Algorithms
• Human Operators– Different roles and levels in hierarchy– Heterogeneity (Information, skills and knowledge)
• Tools– For different kinds of data analysis and visualization– For different levels of decision making
• Together, human operators and tools are a sociotechnical system– Human System Integration is required
![Page 24: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/24.jpg)
Scaling Up Complexity
![Page 25: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/25.jpg)
25
Findings: Cyber Security Defense Analyst Teaming
• Cyber analysts work as a group – Not as a team– Collaboration among cyber operators is minimal– Little role differentiation– Bottom-up information flow
• Possible Reasons– Cognitive overload– Organizational reward structures– “Knowledge is Power”– Lack of effective collaboration tools
![Page 26: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/26.jpg)
Empirical Study on Teams vs. Groups
26
![Page 27: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/27.jpg)
27
Hypotheses• Reward structures conducive to team
work in cyber defense analyst groups performing triage level analysis will lead to higher signal detection performance.
• Improving interactions between analysts (micro level) can improve overall cyber defense performance (macro level emergence)
![Page 28: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/28.jpg)
28
CyberCog -Synthetic Task Environment
• Task: team based triage analysis using the CyberCog simulation.
• Synthetic Task Environment– Simulation environment– Recreate team and
cognitive aspects of the task
![Page 29: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/29.jpg)
29
CyberCog STE
![Page 30: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/30.jpg)
30
The Experiment
• 3-person teams/groups in which each individual is trained to specialize in types of alerts
• 2 conditions:– Team Work (Primed & Rewarded for team work)–Group Work (Primed & Rewarded for group work)
• 6 individuals at a time– Team Work - Competition between the 2 teams– Group Work - Competition between the 6 individuals
• Experimental scenarios:– 225 alerts– Feedback on number of alerts correctly classified - constantly
displayed on big screen along with other team or individual scores• Simulates knowledge is power for individuals group condition• Measures
Signal Detection Analysis of Alert ProcessingAmount of Communication Team situation awarenessTransactive MemoryNASA TLX – workload measure
Training Practice Scenario 1 TLX Scenario2 TLX Questionnaire
![Page 31: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/31.jpg)
31
Results
![Page 32: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/32.jpg)
32
Cyber Teaming is Beneficial for Analyzing Novel and Difficult Alerts
• Working as team helps when alerts are novel and involves multi step analysis, not otherwise.
• Signal Detection Measure: A' as performance measure
• A' ranges from values 0.5 and 1 with 0.5 indicating lowest performance possible and 1 indicating highest performance possible.
![Page 33: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/33.jpg)
33
Cyber Teaming Helps When the Going Gets Rough
F(1,18) = 5.662, p = .029** (Significant effect of condition)
Sens
itivi
ty to
true
ale
rts
![Page 34: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/34.jpg)
34
Groups that Share Less Information Perceive More Temporal Demands than High Sharers
• NASA TLX Workload Measure: Temporal Demand• Measures perception of time pressure• Higher the value higher the task demand
Statistically significant across scenarios and conditions (p-value = 0.020)
![Page 35: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/35.jpg)
35
Groups that Share Less Information Perceive Work to be More Difficult than High Sharers
• NASA TLX Workload Measure: Mental Effort• Measures perception of mental effort• Higher the value, more mental effort required
Statistically significant across scenarios and conditions (p-value = 0.013)
![Page 36: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/36.jpg)
36
Conclusion• Break the “Silos”• Use the power of human teams to tackle
information overload problems in cyber defense.
• Simply encouraging and training analysts to work as teams and providing team level rewards can lead to better triage performance
• Need collaboration tools and group decision making systems.
![Page 37: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/37.jpg)
Agent-Based Modeling
37
![Page 38: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/38.jpg)
38
Introduction• Human-in-loop experiment
– Traditional method to study team cognition• Agent based model
– Macro emergence– A complimentary approach
• Modeling computational agents with – Individual behavioral characteristics – Team interaction patterns
• Extend Lab Based Experiments
![Page 39: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/39.jpg)
39
Model Description• Agents: Triage analysts• Task: Classify alerts• Rewards for classification• Cognitive characteristics:
– Knowledge and Expertise– Working memory limit– Memory Decay
![Page 40: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/40.jpg)
40
Model Description• Learning Process: Simplified – Probability based
– 75% chance to learn– Cost: 200 points– Payoff: 100 points
• Collaboration: Two strategies to identify partners– Conservative or Progressive– Cost: 100 points for each– Payoff: 50 points for each
• Attrition
![Page 41: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/41.jpg)
41
Model ProcessRecruit if needed
Assign alerts
Collaborate with Agents
Team?
Get Rewards
Add Knowledge
Learn? Know?
Yes
NoNo
YesYes
AdjustExpertise
AndRemoveAnalysts
No
![Page 42: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/42.jpg)
42
Model in Netlogo Software
![Page 43: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/43.jpg)
Agents in the Progressive/Teamwork Condition Classified More Alerts
(replicates experiment)
43
p<0.001
![Page 44: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/44.jpg)
Agents in Team of Six Classified More Alerts
44
p = 0.004
![Page 45: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/45.jpg)
45
Irrespective of Team Size Agents in Progressive Condition Classified More Alerts
![Page 46: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/46.jpg)
Agents in Progressive Condition Accrued Least Rewards
46
p<0.001
![Page 47: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/47.jpg)
Agents in Small Teams Accrued Most Rewards
47
p<0.001
![Page 48: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/48.jpg)
48
Agents in Large Progressive Teams Accrued Least Rewards
![Page 49: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/49.jpg)
49
Conclusion• Large progressive teams classified most alerts• Large progressive teams accrued least
rewards• Big progressive teams
– Lot of collaboration – Less learning – Constant knowledge swapping– More net rewards of 50 points
• However small progressive teams accrued rewards on-par
![Page 50: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/50.jpg)
50
Conclusions
• Small heterogeneous teams of triage analysts could be beneficial.
• Agent based modeling – Can extend lab based experiments– Can be used to ask more questions quickly– Can raise new questions and identify gaps
![Page 51: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/51.jpg)
Two Case Studies and EAST Models
51
![Page 52: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/52.jpg)
EASTEvent Analysis of Systemic Teamwork framework
(Stanton, Baber, & Harris, 2012) • Integrated suite of methods allowing the effects of one set of constructs
on other sets of constructs to be considered– Make the complexity of socio-technical systems more explicit– Interactions between sub-system boundaries may be examined– Reduce the complexity to a manageable level
• Social Network– Organization of the system (i.e., communications structure)– Communications taking place between the actors working in the team.
• Task Network– Relationships between tasks– Sequence and interdependences of tasks
• Information Network– Information that the different actors use and communicate during task
performance With Neville Stanton, University of Southampton, UK
![Page 53: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/53.jpg)
Approach• Interviews with cyber network defense leads
from two organizations on social structure, task structure, and information needs
• Hypothetical EAST models created• Surveys specific to organization for cyber
defense analysts developed• Surveys administered to analysts in each
organization to refine models
53
![Page 54: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/54.jpg)
Social Network Diagramsof Incident Response/Network Defense Teams
Detector (6)
Responder (6)
Threat Analyst
(1)
OpTeam
Analyst 2
Analyst 1
Analyst 3
Analyst 4
Cyber Command
Customer
Industry Military
![Page 55: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/55.jpg)
Sequential Task Network DiagramIndustry Incident Response Team
Threat Analyst
(1)
Modeling
TrainingHosting Accounts
RootCertificate
Detector(6)
CreditCard
ClassifyAlerts
Un-known
Responder(6)
DeeperClassification
Alerts
Training
From:CreditCard
From: Root
Certificate
From:Hosting Accounts
From:Un-
knownOp
Team
Update Servers
Training
Network maintenance
![Page 56: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/56.jpg)
Sequential Task Network DiagramMilitary Network Defense Team
Customer
Gather Batch of Reports
Review Alerts
HandoffReview Events
Customer Assignment
Dispatch
Cyber Command
![Page 57: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/57.jpg)
Information Network Diagramof Incident Response/Network Defense Teams
Responder
DDOS Tools
IDS
In-house software
Detector
Antivirus IDSAudio
Alerts
Analyst
Workflow
System
ReportingBatches of Alerts
Shift Change Meeting
Dictionary
On-Line Help
MilitaryIndustry
Reports
Web Sites
Incident Reports
ShiftChangeMeeting
Shift Change Meeting
Incident Reports
IDS
![Page 58: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/58.jpg)
58
EAST Conclusions• A descriptive form of modeling that facilitates
understanding of sociotechnical system• Can apply social network analysis parameters to each
of these networks and combinations• Can better understand system bottlenecks,
inefficiencies, overload• Can better compare systems• Combined with empirical studies and agent-based
modeling can allow us to scale up to very complex systems
![Page 59: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/59.jpg)
Next Steps
59
![Page 60: Umbrella Presentation Theme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649e3f5503460f94b3086f/html5/thumbnails/60.jpg)
Plan for FY 14
Cognitive Task
Analyses and Theory
Development
Testbed and Scenario
Development
Experimentation Models and Metrics
FY 14Refine theory and models of cyber situation awareness
DEXTAR: Known vs. Unknown vulnerabilities & attack patterns; Systematic increase of data and difficulty
Metric testing and validation in DEXTAR
Explore teaming possibilities and structures in cyber defense analysis
Develop models from empirical data and extend to larger and more complex teaming
60