Umbrella PresentationTheme C: Cognitive Science of Cyber SA
ASU (Cooke) Cyber Security as a Complex Cognitive
SystemPSU (McNeese & Hall) Computer-aided Computer-Aided Human Centric Cyber
Situation Awareness
1
System Analysts
Computer network
SoftwareSensors, probes• Hyper
Sentry• Cruiser
Mul
ti-S
enso
ry H
um
an
Com
put
er
Inte
ract
ion
• Enterprise Model
• Activity Logs • IDS reports
• Vulnerabilities
Cognitive Models & Decision Aids• Instance Based Learning Models
• Simulation• Measures of SA & Shared SA
• • • D
ata
Co
nd
itio
nin
gA
sso
cia
tion
& C
orr
ela
tion
Automated Reasoning Tools• R-CAST• Plan-based
narratives• Graphical
models• Uncertainty
analysis
Information Aggregation
& Fusion• Transaction
Graph methods
• Damage assessment
Computer network
• •
•
Real World
Test-bed
2
Situation Awareness
Endsley’s Definition:the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future
Perception Comprehension Projection
Cyber Situation Awareness is Inherently Human
4
SA is not in the technology (e.g., visualization); it is in the interface between humans and technology
Team Situation Awareness
A team’s coordinated perception and action in response to a change in the environment
Contrary to view that all
team members need to “be on the same page”
5
Detector Responder Threat Analyst
Perception Comprehension Projection
Cyber SA is Distributed and Emergent
Detector Responder Threat Analyst
Perception Comprehension Projection
Cyber SA is Distributed and Emergent
Cyber Security as a Complex Cognitive SystemN.Cooke, P. Rajivan, M. Champion, G. Dube, V. Buchanan, S. Jariwala
Cognitive ScienceTheoretical Foundations
Top-down
Bottom-Up
Distributed Research Simulations
Metrics &Measures
Cognitive Systems Engineering
Observe
Observation
Fields of Practice
Cyber Defense
CyberCog & DEXTAR
Communication & CoordinationTeam Situation Awareness
Agent-Based & EAST Modeling
Interactive Team Cognition/ Sociotechnical Systems Theory
Tools & Methods
Theoretical FoundationsHuman-Centered
Distributed Research Simulations
CyberCog & DEXTARInteractive Team Cognition/Sociotechnical Systems Theory Workload
Specialization
Teams vs Groups
Team and Organization Models
Actual ExperimentalStudiesConducted
Cyber Security as a Complex Cognitive SystemN.Cooke, P. Rajivan, M. Champion, G. Dube, V. Buchanan, S. Jariwala
Computer-Aided Human Centric CyberSituation AwarenessM. McNeese, D. Hall, N. Giacobe,
V. Mancuso, D. Minotra, and E. McMillan
Cognitive ScienceTheoretical Foundations
Top-down
Bottom-Up
Distributed Research Simulations
Metrics &Measures
Cognitive Systems Engineering
Observe
Observation
Fields of Practice
Cyber Defense
teamNETS
Visual Analytics Testbench Complex Event Processing
Situated Cognition
Tools & Methods
Theoretical FoundationsHuman-Centered
Distributed Research Simulations
teamNETSSituated Cognition
Attention/Disruption
Memory / Access Awareness
Team Cognition
Embedded Model of the Threat
Actual ExperimentalStudiesConducted
Computer-Aided Human Centric CyberSituation Awareness
M. McNeese, D. Hall, N. Giacobe, V. Mancuso, D. Minotra, and E. McMillan
ASU/PSU ObjectivesPSU Objectives
• To understand Individual and Team cognition of Situation Awareness in Cyber-Security domains
• Refine and implement evaluation environment to support evaluation of new analysis models, cognitive tools, and adversarial team cognition via hidden knowledge profiles
• Develop new tools for practice based on field- and laboratory-based findings
ASU Objectives
• To develop theory of team-based SA to inform assessment metrics and improve interventions (training and decision aids)
• Iterative Refinement of Cyber Testbeds based on cognitive analysis of the domain
– Cybercog– DEXTAR
• Conduct experiments on Cyber TSA in the testbed to develop theory and metrics
• Extend empirical data through modeling
Cyber Security as a Complex Cognitive System
Cyber Security as a Complex Cognitive System
Nancy J. Cooke, PhD
Prashanth Rajivan, MS
Michael Champion, MSShree JariwalaGeneviève Dubé, Université Laval, Québec Verica Buchanan
Arizona State UniversityOctober 29, 2013
13This work has been supported by the Army Research Office under MURI Grant W911NF-09-1-0525.
Outline•Overview of Project •Definitions and Theoretical Drivers •Empirical Study on Teams vs. Groups •Agent-Based Modeling •Two Case Studies and EAST Models •Next Steps
14
Overview of Project
15
ASU Project OverviewObjectives:
Understand and Improve Team Cyber Situation Awareness via • Understanding cognitive /teamwork elements of situation awareness
in cyber-security domains• Implementing a synthetic task environment to support team in the
loop experiments for evaluation of new algorithms, tools and cognitive models
• Developing new theories, metrics, and models to extend our understanding of cyber situation awareness
Department of Defense Benefit:• Metrics, models, & testbeds for assessing human effectiveness and
team situation awareness (TSA) in cyber domain• Testbed for training cyber analysts and testing (V&V) algorithms and
tools for improving cyber TSA
Scientific/Technical Approach - Year 4• Explore the role of teamwork in cyber defense
through:• Empirical work in CyberCog testbed• Agent-Based Modeling• Case Studies and EAST Modeling
• Further refine team metrics and testbeds
Year 4 Accomplishments• Found an empirical benefit of cyber teaming• Replicated this benefit in an agent-based model• Compared two cyber defense organizations• Refined team metrics and cybercog testbed
ChallengeStruggle to maintain realism in testbed scenarios while
allowing for novice participation and team interaction – now addressing with CyberCog and Dextar
17
Summary of FY 13 ASU Accomplishments
PUBLICATIONSCooke, N. J., Champion, M., Rajivan, P., & Jariwala, S. (2013). Cyber Situation Awareness and Teamwork. EAI Endorsed Transactions on Security and Safety. Special Section on: The Cognitive Science of Cyber Defense, 13. Cooke, N. J. & McNeese, M. (2013). Preface to special issue on the cognitive science of cyber defence analysis. EAI Endorsed Transactions on Security and Safety. Special Section on: The Cognitive Science of Cyber Defense, 13
Rajivan, P., Champion, M., Cooke, N. J., Jariwala, S., Dube, G., & Buchanan, V. (2013). Effects of teamwork versus group work on signal detection in cyber defense teams. In D. D. Schmorrow and C.M. Fidopiastis (Eds.), AC/HCII, LNAI 8027, pp. 172-180., Berlin: Springer-Verlag. Rajivan, P., Janssen, M. A., & Cooke, N. J., (2013). Agent-based model of a cyber security defense analyst team. Proceedings of the 57th Annual Conference of the Human Factors and Ergonomics Society, Santa Monica, CA: Human Factors and Ergonomics Society.
Champion, M., Rajivan, R., Jariwala, S., Cooke, N. J., & Buchanan, V. Understanding the cyber security task. Poster presented at ASU's Sixth Annual Workshop on Information Assurance, May 1, 2013, Tempe, AZ.
STUDENTS SUPPORTED• Prashanth Rajivan (PhD)• Verica Buchanan (UG)
PROJECTS SUPPORTED FY 13• CyberCog and metrics development• CyberCog study• Agent-based models of cyber teaming• Agent-based models of cyber warfare• Case Studies and EAST models
COLLABORATION• Coty Gonzalez – IBLT and Agent-Based Modeling• Sushil Jajodia – DEXTAR• Several MURI partners on an ARL proposal
TECH TRANSFER• Working with Charles River Analytics and AFRL on team
measures of cyber defense• Working with SA Technologies on cyber visualization• Presentation to ASU Information Assurance • Presentation to General Dynamics – The Edge
AWARDPrashanth Rajivan wins HFES 2013 Alphonse Chapanis Award for best student paper!!!
Definitions and Theoretical Drivers
18
Theoretical Drivers
• Interactive Team Cognition• Sociotechnical Systems Theory/
Human Systems Integration
19
Interactive Team CognitionTeam is unit of analysis = Heterogeneous and interdependent group of individuals (human or synthetic) who plan, decide,
perceive, design, solve problems, and act as an integrated system.
Cognitive activity at the team level= Team Cognition
Improved team cognition Improved team/system effectiveness
Heterogeneous = differing backgrounds, differing perspectives on situation
(surgery, basketball)20
Interactive Team CognitionTeam interactions often in the form of explicit
communications are the foundation of team cognition
ASSUMPTIONS
1) Team cognition is an activity; not a property or product
2) Team cognition is inextricably tied to context
3) Team cognition is best measured and studied when the team is the unit of analysis
21
Implications of Interactive Team Cognition
• Focus cognitive task analysis on team interactions
• Focus metrics on team interactions (team SA)
• Intervene to affect team interactions
22
Cyber Defense as a Sociotechnical System• Cyber defense functions involve cognitive processes allocated to
– Human Operators – Tools/Algorithms
• Human Operators– Different roles and levels in hierarchy– Heterogeneity (Information, skills and knowledge)
• Tools– For different kinds of data analysis and visualization– For different levels of decision making
• Together, human operators and tools are a sociotechnical system– Human System Integration is required
Scaling Up Complexity
25
Findings: Cyber Security Defense Analyst Teaming
• Cyber analysts work as a group – Not as a team– Collaboration among cyber operators is minimal– Little role differentiation– Bottom-up information flow
• Possible Reasons– Cognitive overload– Organizational reward structures– “Knowledge is Power”– Lack of effective collaboration tools
Empirical Study on Teams vs. Groups
26
27
Hypotheses• Reward structures conducive to team
work in cyber defense analyst groups performing triage level analysis will lead to higher signal detection performance.
• Improving interactions between analysts (micro level) can improve overall cyber defense performance (macro level emergence)
28
CyberCog -Synthetic Task Environment
• Task: team based triage analysis using the CyberCog simulation.
• Synthetic Task Environment– Simulation environment– Recreate team and
cognitive aspects of the task
29
CyberCog STE
30
The Experiment
• 3-person teams/groups in which each individual is trained to specialize in types of alerts
• 2 conditions:– Team Work (Primed & Rewarded for team work)–Group Work (Primed & Rewarded for group work)
• 6 individuals at a time– Team Work - Competition between the 2 teams– Group Work - Competition between the 6 individuals
• Experimental scenarios:– 225 alerts– Feedback on number of alerts correctly classified - constantly
displayed on big screen along with other team or individual scores• Simulates knowledge is power for individuals group condition• Measures
Signal Detection Analysis of Alert ProcessingAmount of Communication Team situation awarenessTransactive MemoryNASA TLX – workload measure
Training Practice Scenario 1 TLX Scenario2 TLX Questionnaire
31
Results
32
Cyber Teaming is Beneficial for Analyzing Novel and Difficult Alerts
• Working as team helps when alerts are novel and involves multi step analysis, not otherwise.
• Signal Detection Measure: A' as performance measure
• A' ranges from values 0.5 and 1 with 0.5 indicating lowest performance possible and 1 indicating highest performance possible.
33
Cyber Teaming Helps When the Going Gets Rough
F(1,18) = 5.662, p = .029** (Significant effect of condition)
Sens
itivi
ty to
true
ale
rts
34
Groups that Share Less Information Perceive More Temporal Demands than High Sharers
• NASA TLX Workload Measure: Temporal Demand• Measures perception of time pressure• Higher the value higher the task demand
Statistically significant across scenarios and conditions (p-value = 0.020)
35
Groups that Share Less Information Perceive Work to be More Difficult than High Sharers
• NASA TLX Workload Measure: Mental Effort• Measures perception of mental effort• Higher the value, more mental effort required
Statistically significant across scenarios and conditions (p-value = 0.013)
36
Conclusion• Break the “Silos”• Use the power of human teams to tackle
information overload problems in cyber defense.
• Simply encouraging and training analysts to work as teams and providing team level rewards can lead to better triage performance
• Need collaboration tools and group decision making systems.
Agent-Based Modeling
37
38
Introduction• Human-in-loop experiment
– Traditional method to study team cognition• Agent based model
– Macro emergence– A complimentary approach
• Modeling computational agents with – Individual behavioral characteristics – Team interaction patterns
• Extend Lab Based Experiments
39
Model Description• Agents: Triage analysts• Task: Classify alerts• Rewards for classification• Cognitive characteristics:
– Knowledge and Expertise– Working memory limit– Memory Decay
40
Model Description• Learning Process: Simplified – Probability based
– 75% chance to learn– Cost: 200 points– Payoff: 100 points
• Collaboration: Two strategies to identify partners– Conservative or Progressive– Cost: 100 points for each– Payoff: 50 points for each
• Attrition
41
Model ProcessRecruit if needed
Assign alerts
Collaborate with Agents
Team?
Get Rewards
Add Knowledge
Learn? Know?
Yes
NoNo
YesYes
AdjustExpertise
AndRemoveAnalysts
No
42
Model in Netlogo Software
Agents in the Progressive/Teamwork Condition Classified More Alerts
(replicates experiment)
43
p<0.001
Agents in Team of Six Classified More Alerts
44
p = 0.004
45
Irrespective of Team Size Agents in Progressive Condition Classified More Alerts
Agents in Progressive Condition Accrued Least Rewards
46
p<0.001
Agents in Small Teams Accrued Most Rewards
47
p<0.001
48
Agents in Large Progressive Teams Accrued Least Rewards
49
Conclusion• Large progressive teams classified most alerts• Large progressive teams accrued least
rewards• Big progressive teams
– Lot of collaboration – Less learning – Constant knowledge swapping– More net rewards of 50 points
• However small progressive teams accrued rewards on-par
50
Conclusions
• Small heterogeneous teams of triage analysts could be beneficial.
• Agent based modeling – Can extend lab based experiments– Can be used to ask more questions quickly– Can raise new questions and identify gaps
Two Case Studies and EAST Models
51
EASTEvent Analysis of Systemic Teamwork framework
(Stanton, Baber, & Harris, 2012) • Integrated suite of methods allowing the effects of one set of constructs
on other sets of constructs to be considered– Make the complexity of socio-technical systems more explicit– Interactions between sub-system boundaries may be examined– Reduce the complexity to a manageable level
• Social Network– Organization of the system (i.e., communications structure)– Communications taking place between the actors working in the team.
• Task Network– Relationships between tasks– Sequence and interdependences of tasks
• Information Network– Information that the different actors use and communicate during task
performance With Neville Stanton, University of Southampton, UK
Approach• Interviews with cyber network defense leads
from two organizations on social structure, task structure, and information needs
• Hypothetical EAST models created• Surveys specific to organization for cyber
defense analysts developed• Surveys administered to analysts in each
organization to refine models
53
Social Network Diagramsof Incident Response/Network Defense Teams
Detector (6)
Responder (6)
Threat Analyst
(1)
OpTeam
Analyst 2
Analyst 1
Analyst 3
Analyst 4
Cyber Command
Customer
Industry Military
Sequential Task Network DiagramIndustry Incident Response Team
Threat Analyst
(1)
Modeling
TrainingHosting Accounts
RootCertificate
Detector(6)
CreditCard
ClassifyAlerts
Un-known
Responder(6)
DeeperClassification
Alerts
Training
From:CreditCard
From: Root
Certificate
From:Hosting Accounts
From:Un-
knownOp
Team
Update Servers
Training
Network maintenance
Sequential Task Network DiagramMilitary Network Defense Team
Customer
Gather Batch of Reports
Review Alerts
HandoffReview Events
Customer Assignment
Dispatch
Cyber Command
Information Network Diagramof Incident Response/Network Defense Teams
Responder
DDOS Tools
IDS
In-house software
Detector
Antivirus IDSAudio
Alerts
Analyst
Workflow
System
ReportingBatches of Alerts
Shift Change Meeting
Dictionary
On-Line Help
MilitaryIndustry
Reports
Web Sites
Incident Reports
ShiftChangeMeeting
Shift Change Meeting
Incident Reports
IDS
58
EAST Conclusions• A descriptive form of modeling that facilitates
understanding of sociotechnical system• Can apply social network analysis parameters to each
of these networks and combinations• Can better understand system bottlenecks,
inefficiencies, overload• Can better compare systems• Combined with empirical studies and agent-based
modeling can allow us to scale up to very complex systems
Next Steps
59
Plan for FY 14
Cognitive Task
Analyses and Theory
Development
Testbed and Scenario
Development
Experimentation Models and Metrics
FY 14Refine theory and models of cyber situation awareness
DEXTAR: Known vs. Unknown vulnerabilities & attack patterns; Systematic increase of data and difficulty
Metric testing and validation in DEXTAR
Explore teaming possibilities and structures in cyber defense analysis
Develop models from empirical data and extend to larger and more complex teaming
60
Top Related