Team Cognition During a Sumulated Close Air Support Exercise Cognition... · 2008. 6. 24. ·...
Transcript of Team Cognition During a Sumulated Close Air Support Exercise Cognition... · 2008. 6. 24. ·...
-
Defence Research andDevelopment Canada
Recherche et développementpour la défense Canada Canada
Team Cognition During a Simulated Close Air Support Exercise:Results From a New Behavioral Rating Instrument
Jerzy Jarmasz1, Richard Zobarich2, Lora Bruyn-Martin2, andTab Lamoureux2
1Defence Research & Development Canada, Toronto,Canada
2HumanSystems Inc., Guelph, Canada
-
Defence R&D Canada • R & D pour la défense Canada
Close Air Support (CAS): Overview
US DoD Joint Pub 3-09.3:CAS is air action by fixed- and rotary-wingaircraft against hostile targets that are inclose proximity to friendly forces and thatrequire detailed integration of each airmission with the fire and movement ofthose forces.
-
Defence R&D Canada • R & D pour la défense Canada
Teamwork in CAS
• Distributed
– Coordinated rather thancollective tasks
• Joint
– Always land and air atleast
• Can be multination/coalition
– Common language ≠common procedures
• This is not easy!!
-
Defence R&D Canada • R & D pour la défense Canada
Importance of Combat ID in CAS
• Friendly forces and adversary are close to each other,and close to civilians and infrastructure
• Targets are small, nimble and mobile: individuals,private vehicles, huts, instead of tanks and bunkers
• Coalition missions:
– Allies can be hard to distinguish from adversary
– Pilots and terminal controllers might not havecommon procedures
• Risk of Combat ID failure leading to fratricide orcivilian casualties is very real in CAS (just read thenews!)
-
Defence R&D Canada • R & D pour la défense Canada
Training approach: Distributed MissionTraining (DMT)
• USAF has extensive experience with DMT (a.k.a DMO)
• Canadian Forces Air Warfare Centre standing up its DMTcapability
• Will allow Canadian Forward Air Controllers (FACs) topractice CAS skills more often and with a wider range of(coalition) airframes than currently possible
– Note that FACs tend to be Army personnel in the CF
– FAC is a secondary duty in the CF
• Requirement to assess effectiveness of DMT inimproving team performance in CAS missions
-
Defence R&D Canada • R & D pour la défense Canada
Study: Assessment of Team Performance in CAS
• Goal: develop behaviourally-anchored rating scale(BARS) for team performance in CAS
• DMT event: Exercise Northern Goshawk, August 2007(CAN-US-UK, involving DRDC Toronto, the Air ForceResearch Laboratory & Defence Science & TechnologyLaboratory)
• Focus on Canadian component of event: CAS support ofa Canadian Forces (CF) convoy mission, controlled by aCF Forward Air Controller
-
Defence R&D Canada • R & D pour la défense Canada
BARS instrument development(Jarmasz et al., 2008)
1. Hierarchical Task Analysis of generic CASmissions
• Focus on FAC perspective
• Validated with CF CAS subject matter experts
2. Identification of key team interaction points basedon HTA
3. Development of behavioural ratings for interactionpoints using behavioural markers of teamcognition breakdown (Wilson, Salas, Priest &Andrews, 2007)
-
Defence R&D Canada • R & D pour la défense Canada
Main measurement points
Perform Battle Damage AssessmentCommunicate Options with PilotSituation Updates
5. Post-attack
Transmit CAS brief and communicate remarksTransmit talk-onDeconflict Target AreaDesignate TargetAbort Mission (as needed)
4. Target prosecution
Communicate Options with PilotSituation Updates
3. Non-kinetic CAS support
Receive pilot’s check-in (includes situation brief)2. Pilot Check-in
Understand Situation UpdatesTransmit Immediate CAS Request (as needed)
1. Pre-check in
TasksCAS mission phase
-
Defence R&D Canada • R & D pour la défense Canada
From team cognition to rating scales(adapted from Wilson et al., 2007)
• To what extent were team members working toward the sameends?
• How effective were FAC/others as a team?Cooperation
• How well were team members' knowledge requirementsmanaged?
• How well did team members monitor each other's performance?• How effective was back-up behaviour?• How adaptable were team members to the changing demands of
the situation?
Coordination
• How effective was information exchange?• Was information exchange economical?• Did closed looped communication go as expected?
Communication
-
Defence R&D Canada • R & D pour la défense Canada
Sample BARS: Communication (CAS Brief)
FAC failed to provide sufficient brief & remarks for pilot to complete mission.1
FAC omitted important brief & remarks items that were available to him.2
FAC passed all items of brief & remarks available to him but had to communicate withothers to obtain missing info requested by pilot (e.g. MAXORD, ACA measures)3
FAC passed all items of brief and most key remarks but provided some remarks onlywhen prompted (e.g. danger close initials)4
FAC passed complete and accurate brief (following theatre standard), & provided allkey remarks info (e.g., weapons effects, attack geometry, ACA measures, number ofattempts, level of risk for blue and white forces, danger close initials).
5
Communication: How effective was information exchange?
-
Defence R&D Canada • R & D pour la défense Canada
Sample BARS: Coordination (CAS Brief)
FAC/pilot failed to display a common understanding of the SOPs &brief/remarks.1
FAC/pilot made explicit attempts at coordinating knowledge, and achieved anincomplete common understanding of the SOPs & briefs/remarks.2
FAC/pilot made explicit attempts at coordinating knowledge (e.g., asked eachother questions), and at great effort (e.g., much time spent in discussion) achievedcommon understanding of the SOPs & brief/remarks.
3
FAC/pilot coordinated explicitly and effectively (e.g., discussed coordinatesformats), and achieved a common understanding of the CAS brief SOPs &brief/remarks with little effort.
4
FAC/pilot implicitly coordinated in an effective manner (e.g., did not requirespecial coordination or discussion beyond standard turn-taking), and displayed acommon understanding of CAS brief SOPs and of the brief/remarks.
5
Coordination: How well were team members’ knowledge requirements managed?
-
Defence R&D Canada • R & D pour la défense Canada
Sample BARS: Cooperation (CAS Brief)
FAC/Pilot could not agree on brief SOPs and did not collaborate to ensure pilotreceived & understood brief & remarks; pilot did not have any useable information toproceed with talk-on & attack.
1
FAC/Pilot collaborated poorly to achieve joint understanding of brief & remarks, anddisagreed significantly about brief SOPs; pilot had only a fraction of the requiredinformation for the talk-on and attack.
2
FAC/Pilot collaborated to understand all mandatory CAS brief info, but did notcooperate fully on understanding remarks or had a significant disagreement aboutbrief SOPs; pilot has most but not all info required for talk-on and attack.
3
FAC/Pilot collaborated to understand all relevant CAS brief & remarks info but bothhad slightly different priorities on brief SOPs, which were easily resolved or accepted.4
FAC/Pilot collaborated to ensure CAS brief SOPs (mandatory CAS brief, or thatcollective SA did not require brief) were adhered to and that all required info waspassed & understood for the attack.
5
Cooperation: To what extent were FAC/Pilot working towards the same ends?
-
Defence R&D Canada • R & D pour la défense Canada
Results
• Two raters performed ratings during Ex NorthernGoshawk
• Minor changes on Day 1; data analyzed for Days 2,3, 4 only
• Due to sheer number of ratings, each rater focusedon slightly different ratings subsets
– Thus not all possible ratings collected
• Only qualitative patterns are presented
-
Defence R&D Canada • R & D pour la défense Canada
Summary statistics for subset of ratings
5.00 (0.00)NR5.00 (NA)Cooperation
4.75 (0.50)NR4.67 (0.58)Coordination
5.00 (0.00)NR5.00 (0.00)Communication4.5 Abort mission
NRNRNRCooperation
4.43 (0.55)4.50 (0.58)NRCoordination
4.50 (0.84)4.33 (0.58)4.14 (1.07)Communication4.4 Designate target
4.00 (0.00)4.00 (0.00)5.00 (0.00)Cooperation
4.33 (0.52)4.00 (0.00)4.13 (0.75)Coordination
4.33 (1.12)4.00 (1.00)4.33 (1.00)Communication4.3 Deconflict target area
5.00 (0.00)5.00 (0.00)5.00 (0.00)Cooperation
4.08 (0.90)4.13 (0.83)3.83 (0.83)Coordination
4.29 (0.69)3.83 (1.17)4.22 (0.97)Communication4.2 Transmit talk-on
5.00 (0.00)5.00 (0.00)5.00 (0.00)Cooperation
4.50 (0.55)4.75 (0.50)4.50 (0.76)Coordination
4.50 (0.84)4.33 (1.03)4.22 (1.20)Communication4.1 Transmit CAS brief andcommunicate remarks
4.50 (0.58)4.00 (1.15)4.67 (0.58)Cooperation
4.17 (0.39)3.17 (0.41)4.00 (0.00)Coordination
4.13 (0.52)4.33 (1.03)3.89 (0.93)Communication3.2 Understand situation updates
4.33 (0.52)4.00 (0.00)3.50 (0.84)Coordination
4.44 (0.73)3.67 (0.98)3.83 (1.17)Communication3.1 Communicate options withpilot
Day 4Day 3Day 2Team cognition categoryTask
-
Defence R&D Canada • R & D pour la défense Canada
Event 1: Talk on for unauthorized drop
5How effective were FAC/others as a team?
5To what extent were team members working toward the same ends?Cooperation
4How adaptable were team members to the changing demands of thesituation?
5How effective was back-up behavior?
2How well did team members monitor each other's performance?
3How well were team members' knowledge requirements managed?Coordination
4Did closed looped communication go as expected?
3Was communication economical?
5How effective was information exchange?Communication
RatingBARSTeam cognitioncategory
-
Defence R&D Canada • R & D pour la défense Canada
Event 2: Day 4 aborted attack
5To what extent did team members display mutual trust?Cooperation
5How well did team members monitor each other's performance?
4How well were team members' knowledge requirements managed?Coordination
5Was communication economical?
5How effective was information exchange?CommunicationAbort mission
5How effective were FAC/others as a team?
5To what extent were team members working toward the same ends?Cooperation
3How adaptable were team members to the changing demands of thesituation?
5How effective was back-up behavior?
3How well did team members monitor each other's performance?
3How well were team members' knowledge requirements managed?Coordination
5Did closed looped communication go as expected?
3Was communication economical?
5How effective was information exchange?CommunicationTransmittalk-on
Rating BARSTeam cognitioncategory
Task
-
Defence R&D Canada • R & D pour la défense Canada
Event 3: Talk on for difficult attack
5How effective were FAC/others as a team?
5To what extent were team members working toward the same ends?
4How adaptable were team members to the changing demands of thesituation?
Cooperation
5How effective was back-up behaviour?
4How well did team members monitor each other's performance?
4How well were team members' knowledge requirements managed?Coordination
5Did closed looped communication go as expected?
3Was communication economical?
5How effective was information exchange?Communication
RatingBARSTeam cognitioncategory
-
Defence R&D Canada • R & D pour la défense Canada
Discussion
BARS instrument captured general patterns consistentwith subjective impressions collected from observers(willingness of participants to cooperate,communications problems)
Team processes improved over the duration of theexercise for the key CAS tasks
BARS ratings captured team cognition patterns forcertain key events
Difficulty in applying BARS (number, experience)
Can’t assess interrater reliability yet
Participant skill & scenario difficulty levels madeassessing discriminant validity difficult
-
Defence R&D Canada • R & D pour la défense Canada
Take-home message
• The BARS instrument captured both micro- and macro-level team cognition patterns during the exercise
• Some of the observed patterns are directly relevant toCombat ID and fratricide avoidance
• An improved BARS instrument could support thedevelopment and assessment of training for teamcognition in CAS, thereby improving Combat ID
-
Defence R&D Canada • R & D pour la défense Canada
Next steps
• Assess inter-rater reliability, improve if needed
• Improve instrument’s ease of use
• Validate & refine BARS instrument content with SMEs
• Determine which tasks should be focused on
• Apply to more DMT CAS events with wider range oftask difficulties and participant experience
• Apply the HTA/Team cognition approach to othercollective tasks (e.g., IED detection)
-
Defence R&D Canada • R & D pour la défense Canada