A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth...
-
Upload
patrick-hoffman -
Category
Documents
-
view
218 -
download
0
Transcript of A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth...
![Page 1: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/1.jpg)
A Testbed for Studies of Team A Testbed for Studies of Team Cognition in the Cyber Security Cognition in the Cyber Security
DomainDomain
A Testbed for Studies of Team A Testbed for Studies of Team Cognition in the Cyber Security Cognition in the Cyber Security
DomainDomain
Nancy J. CookePrashanth RajivanShankaranarayanan Venkatanarayanan
Arizona State University5 May 2010
![Page 2: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/2.jpg)
Background• Education: Cognitive Psychology/Human Factors
George Mason University, B.A. New Mexico State University, M.A., Ph.D.
• Positions Rice University New Mexico State University Arizona State University & Cognitive Engineering Research Institute
• Applied Experience: U.S Air Force, Navy, Army, NASA, NTSB, VA
• Section Editor, Human Factors• USAF Scientific Advisory Board• National Research Council Committee on Human
Systems Integration
Relevant Research
Team Cognition Military, Cyber, and Medical Applications
Communication Analysis
Metrics for Coordination and CollaborationSponsors
• Air Force Office of Scientific Research• Air Force Research Laboratory• Office of Naval Research• Army Research Office • Leonard Wood Institute• Veteran’s Administration – MWM VERC
Cooke’s Background
![Page 3: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/3.jpg)
Overview
• MURI and ASU Team• Team Cognition and Team Situation
Awareness• Other Team Testbeds• CyberCog – New Testbed
![Page 4: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/4.jpg)
MURI: Computer-aided Human Centric Cyber Situation Awareness
DoD Multidisciplinary University Research Initiative (MURI) program project, funded through Army Research Office
Two fundamental limitations of Cyber Situation Awareness (C-SA)• Gap: human cognition < -- > C-SA tools
– Situation data exceeds “cognitive throughput” of human analysts• “Blind spots” in views of cyber situation for existing C-SA tools (including
auditing, vulnerability scanners, attack graph tools, intrusion detection systems, damage assessment tools, and forensics tools)
Cyber-SA Vision• Build data < -- > human decision links through innovations
– knowledge fusion– cognitive automation– artificial intelligence– visual analytics
• Awareness-driven cyber defense vs. malware behavior dependent defense • Automatic blind spot identification and monitoring techniques
![Page 5: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/5.jpg)
MURI Partners• Professor Peng Liu, Penn State University, Overall PI • Professor Nancy Cooke, Arizona State University • Professor Coty González, Carnegie Mellon University • Professor Dave Hall, Penn State University • Professor Sushil Jajodia, George Mason University • Professor Mike McNeese, Penn State University • Professor Peng Ning, NC State University • Professor VS Subrahmanian, Univ. of Maryland • Professor John Yen, Penn State University • Professor Michael Young, NC State University
![Page 6: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/6.jpg)
ASU MURI TeamNancy J. CookeProfessor, Cognitive Science & EngineeringCollege of Technology and Innovation
Prashanth RajivanGraduate Student Master’s in Computing StudiesCollege of Technology Innovation
Shankaranarayanan VenkatanarayananGraduate Student Master’s in Computing StudiesCollege of Technology and Innovation
![Page 7: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/7.jpg)
Teams and Cognitive TasksTeam is unit of analysis = Heterogeneous and interdependent group of individuals (human or synthetic) who plan, decide,
perceive, design, solve problems, and act as an integrated system.
Cognitive activity at the team level= Team Cognition
Improved team cognition Improved team/system effectiveness
Heterogeneous = differing backgrounds, differing perspectives on situation
(surgery, basketball)
![Page 8: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/8.jpg)
• Unmanned Aerial Vehicles • USS Vincennes shoots down
Iranian airbus (1988)• Challenger/Columbia accidents
tied to poor organizational decision making (1986/2003)
• Response to 9/11 reveals communication breakdowns (2001)
• Katrina response lacked coordination (2005)
• Sago Mine disaster report cites poor command-and-control (2006)
• VA Tech communications substandard (2007)
• Friendly fire incidents• Various health care mishaps
attributed to poor teamwork
Some Instances of Failures of Team
Cognition
![Page 9: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/9.jpg)
Miracle on the Hudson
And some successes…
Response to Fargo flooding
![Page 10: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/10.jpg)
Interactive Team Cognition in a Nutshell
Team interactions often in the form of explicit communications are the foundation of team cognition
ASSUMPTIONS
1) Team cognition is an activity; not a property or product
2) Team cognition is inextricably tied to context
3) Team cognition is best measured and studied when the team is the unit of analysis
![Page 11: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/11.jpg)
US 2004 Olympic Basketball Team
"We still have a couple of days, but I don't know where we are," replied USA head coach Larry Brown to a question Wednesday on where his team was in its preparations. "We have good moments and bad, but I've got a pretty good understanding of who needs to play. Now the job is to get an understanding of how we have to play."
A team of experts does NOT make an expert team
Collaborative skill is not additive
![Page 12: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/12.jpg)
US 1980 Olympic Ice Hockey Team
Herb Brooks and 20 young “no-names” won the 1980 Olympic Gold Medal in Ice Hockey
An expert team made up of no-names…
![Page 13: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/13.jpg)
In our UAV STE three operators must coordinate over headsets in order to
maneuver their UAV to take pictures of ground targets
Our UAV Testbed
UAV-STE: Uninhabited Air Vehicle (ground control station) Synthetic Task Environment for research on team cognition (DURIP 1997; USAF funded)
![Page 14: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/14.jpg)
Air Vehicle Operator controls UAV airspeed, heading, and altitude and monitors air vehicle systems
Payload Operator controls camera settings, takes photos, and monitors camera systems
DEMPC navigator, mission planner, plans route from target to target under constraints
Interdependence requires interaction, communication, & coordination
Three team members with inter- dependent tasks
![Page 15: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/15.jpg)
Our MacroCog (Macro-Cognition Testbed)
MacroCog Testbed
Navy-funded lab for strategic planning and decision- making in the context of noncombatant evacuation operations
![Page 16: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/16.jpg)
MacroCog Roles in Current Experiment
Information Warfare Specialist
Personnel Specialist: Military
Equipment Specialist:
Land/Sea Vehicles
Personnel Specialist:
Humanitarian
Equipment Specialist: Air
Vehicles
Experimenter 1
Experimenter 2
![Page 17: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/17.jpg)
Example of Empirical Results on Team Cognition
As teams acquire experience, performance improves, interactions improve, but not individual or collective knowledge
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10
Mission
Team
Per
form
ance
Tm 1
Tm 2
Tm 3
Tm 4
Tm 5
Tm 6
Tm 7
Tm 8
Tm 9
Tm 10
Tm 11
• Individuals are trained to criterion prior to M1• Asymptotic team performance after 4 40-min missions (robust finding)• Knowledge changes tend to occur in early learning (M1) and stabilize• Process improves and communication becomes more standard over time
40-min missionsSpring Break
![Page 18: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/18.jpg)
Team Situation Awareness
A team’s coordinated perception and action in response to a change in the environment
How can we exercise team SA in a testbed?
How can we measure it?
How can we intervene to improve it?
Contrary to view that all team
members need to “be on the same page”
![Page 19: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/19.jpg)
What is Meant by Coordinated Perception and Action?
![Page 20: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/20.jpg)
Measure of Team Situation Awareness• Change is introduced (communication breakdown, enemy in area,
storm) that will impact mission
• 2-3 team members are presented cues regarding change
• Team members need to perceive cues in a coordinated way (i.e., connect the dots) to identify the change
• Team members coordinate to take action relevant to the change (e.g., change altitude, communicate indirectly)
• Measure in terms of outcome and process – who on team was involved?
![Page 21: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/21.jpg)
CyberCog Simulator
Web based Simulator application for measuring individual interaction and team collaboration (e.g., team situation awareness) in a Cyber security analysis situation
![Page 22: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/22.jpg)
CyberCogSimulator – System Overview
![Page 23: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/23.jpg)
CyberCogSimulator – Components
• Cyber Security Analyst (User)– Assigned a specific role such as Denial of Service
(Dos) specialist, Malware specialist and Phishing specialist
– Understands the scenario given, use events and attack symptoms, collaborates with other participants to identify a potential attack or a combination of attacks
– The team reaches a common consensus on the type of attack and its corresponding events
![Page 24: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/24.jpg)
CyberCogSimulator – Components
• Master controller and Evaluator– Queries attack scenarios, events and symptoms
from the database– Distributes the events and symptoms to the
participants– Logs the interaction between participants at real
time– Evaluates and scores the participants findings
with the expected results
![Page 25: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/25.jpg)
CyberCogSimulator – Components• Database server
– MySQL database server stores :-• Attack Scenarios• Events corresponding to attack scenarios
including some false positives & noise events• Attack Symptoms for each specialization (E.g.,
Dos, Malware , Phishing) identified• The expected results, interaction (between
participants ) logs and attack conclusion arrived at by each team for each session
![Page 26: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/26.jpg)
User and Team Views
Legends
FunctionsFunctions
Data
![Page 27: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/27.jpg)
CyberCog Simulator- Interaction
![Page 28: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/28.jpg)
CyberCogSimulator- Architecture
Microsoft IIS
Database
Intra/Internet Malware Specialist
Phishing Specialist
Dos Specialist
Client Tier
Controller & View Tier
POCO’s
ADO.net
Web Services
Model Tier
![Page 29: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/29.jpg)
Conclusion
• There are current gaps and limitations in Cyber Situation Awareness
• Cyber situation awareness by teams involves the coordinated perception and action in the face of a change in the cyber situation
• CyberCog will allow the MURI team and others to better understand team-based cyber SA and to test algorithms and tools developed for improving it
![Page 30: A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.](https://reader037.fdocuments.us/reader037/viewer/2022103015/5514262c550346d8488b5a64/html5/thumbnails/30.jpg)
Team Cognition Research Program
Testbeds: 1) UAS C22) Navy Strategic Planning
Empirical Studies in Testbed
UAS Field Data
Theory Development
ACT-R Model of Synthetic Teammate
Dynamical Systems Modeling
Measures
3540 3560 3580 3600 3620 3640 3660 3680 3700
2940
2950
2960
2970
2980
2990
Time (s)
Cum
ulat
ive
Spe
akin
g (s
)