Modelling Synergies in Large Human-Machine Networked Systems

Research Team: CMU: K. Sycara, C. Lebiere, P. Scerri

Cornell: M. Campbell

George Mason: R. Parasuraman

MIT: J. How, M. Cummings

U of Pittsburgh: M. Lewis

Research Goals

• Develop validated theories and techniques to predict behavior of large-scale, networked human-machine systems involving unmanned vehicles

• Model human decision making efficiency in such networked systems

• Investigate the efficacy of adaptive automation to enhance human-system performance

State of the Art Limitations

• No effective models of large scale interactions of human-agent systems, especially intelligent information processing agents and decision-making humans. – Human-in-the-loop experiments to gather data– Abstract models to capture important effects but still remain

manageable

• High fidelity cognitive models for single human, not interacting with other humans. – These models must be extended to account for impacts of

interaction.

• No established methods for effective model abstraction– A process of abstraction and validation needs to be designed to

promote confidence in the abstracted models.

Overall Approach

To cut the Gordian knot of tractability vs. fidelity, our modeling methodology is multi-level (increasing abstraction):

1. human-in-the-loop

2. high-fidelity cognitive models

3. large scale simulation

4. high level abstractions

Approach

• Human-in-the-loop: perform controlled experiments with human decision-makers in small group settings.

– The resulting data will be used to develop, constrain and validate high-fidelity cognitive models of the various operational and decision-making roles.

• High-fidelity models: will be validated in the same small-group settings as the human experiments, then used to generate predictions in larger but still tractable simulations (on the order of dozens of entities).

– Results from those simulation runs will be used to develop intelligent agents with similar characteristics as the high-fidelity cognitive models but more computationally tractable.

• Large scale simulations: will be validated against both human experimental data and cognitive model simulations and will be used in larger (on the order of hundreds of entities), settings to generate new predictions.

• The dynamics of the level 3 simulations will be abstracted in models of high abstractions (e.g. statistical mechanics models) that can be used to describe the dynamics of systems involving many thousands of entities.

– We will insert one or two humans or a small number of high fidelity models in larger simulations to provide additional cross-validation.

Self-correcting Process

Research Strategy

• Human interaction with unmanned vehicles (UVs) – this domain incorporates the aspects of information fusion, interaction with automation, coordination, planning, and command

• In our proposed studies we will systematically examine the effects of complexity on system performance, by varying such factors as the number and types of UVs, the amount of sensor information, and the number of nodes in the network.

Tasks

• Information processing and fusion– Change in model performance as function of change in

network topology – Tradeoffs between delivering information and

overwhelming attentional capacity (shallow hierarchies)

• Planning and scheduling tasks– Robust scaling in human performance but deviations from

optimum – Robustness to environmental change

• Management and control of human assets– Integration of general and specialized skills

Issues

• How to use human data to develop, parameterize and validate high-fidelity cognitive models without the level of experimental control typically achieved in the laboratory?

• How to account for large expected variability in the data resulting from the combination of complex dynamic tasks and individual differences in ability, knowledge and strategy?

• What aspects of the high-fidelity cognitive models and machine models are abstractable in computational agents and/or mathematical formulas?

• What characteristics of the cognitive model must be retained?– adaptivity and learning are essential characteristics of human

performance that should be preserved in abstract agents or formulations.

• What machine (e.g. UAV) characteristics must be retained?

Issues

• Understand scaling properties of cognitive performance– Most experiments look at a single performance point rather

than as a function of problem complexity, time pressure, etc– Key component in abstracting performance at higher levels

• Understand interaction between humans and machines– Most experiments study and model human performance under

a fixed scenario that misses key dynamics of interaction– Key aspect of both system robustness and vulnerabilities

• Understand generality and composability of behavior– Almost all models are developed for specific tasks rather than

assembling larger pieces of functionality from basic pieces

– Key enabler of scaling models and abstracting their properties

Issue 1: Scaling Properties

• Study human performance at multiple complexity points– Vary problem complexity (e.g. number of targets)– Vary information complexity (e.g. target characteristics)– Vary network topology (e.g. number of related partners)– Vary rate of change of environment (e.g. appearance or disappearance

of targets, weather, network topology)

• Quantify impact on all measures of performance– Direct performance (number of targets handled, etc)– Situation awareness (various levels, memory-based measures)– Workload (both self-reporting and physiological measures)

Issue 2: Dynamic Interaction

• Main problem is degrees of freedom between multiple Decision Makers

• Methodology has been developed to model multi-agent interactions in games, logistics (supply chain) problems

– Develop baseline model to capture first-order behavior– Replace most HITL with baseline model(s)– Refine model based on greater data accuracy

• Methodology can be extended to multiple levels of our approach, each time abstracting to next level in hierarchy

• Impact of fixed, flexible or emergent network organization– Vary degree of task integration (e.g. planning across boundaries)– Vary level of information sharing (e.g. through interface screens)

• Accurate cognitive models for human-machine interaction– Adaptive interfaces (e.g. to predicted model workload)– Model-based autonomy (eg. handle monitoring, routine decision-making)

Issue 3: Behavior Abstraction

• First two issues build solutions toward this one

• Study of scaling properties helps capture response function for all aspects of target behavior

• Abstraction methodology helps iterate and test models at various levels of abstraction to maximize retention

• Issues:– Grain scale of components (generic tasks, unit tasks?)– Attainable degree of fidelity at each level?– Capture individual differences or average, normative behavior?

• Latter may miss key interaction aspects outliers• Individual differences as architectural parameters (WM, speed)• Use cognitive model to generate data to train machine learning agent

Research Issues for Humans-UV Teams

• Humans coupled to UVs– Too many problem parameters – need a more natural language for specifying the

desired goals, which can then be re-interpreted into an optimization problem to be solved

• Humans coupled to information consensus– Used to ensure consistency in the situational awareness of the team

• Computational intensive filtering operation– Human operator has important information to provide in the form of target

classifications or data from other sources– How best to convey to operator the uncertainty in the current SA

• Combined with potential target value, it gives some sense of what should be addressed

– How can operator codify uncertainty in new information inserted in the algorithm.

Controlling Robot Teams

• How can teams of people control teams of UVs of increasing size?– What is the density of UVs a human(s) can control?– What kinds of command are possible for a particular

density?• Issue: As the complexity of the networked system increases,

e,g, scaling up in numbers, system performance may degrade because the cognitive capacities of individual human supervisors will be exceeded.

• Simply increasing the number of operators may not solve the problem due to increased coordination demands between hybrid team members

Real & Simulated Testbeds

Evaluation Measures

Develop predictive measures for human-UV teams, which include: • Efficiency with which operators and teams of operators allocate

their attention to the UVs; Efficiency with which an operator (and the system in general) allocates tasks;

• Costs associated with one or more operators switching between UV tasks;

• Impact of constraints such as information flow rates and decision-making rates;

• Models of how well operators collaborate with the UVs and other operators, planners, humans within the overall command structure and battlefield;

• Characteristics and limitations of system robustness in terms of unexpected failures and unanticipated inputs;

• Mechanisms for providing conformance monitoring feedback (e.g., indicate potential of decision making errors by a human operator to a higher-level commander).

• These data will be used by cognitive modelers in the project to develop models capturing the coupling between the human operators and automated planners and its effect on performance.

Research Evaluation

• Evaluation will be done at each level – Varying the task– Varying the number of actors in the system– Varying the proportion of humans and robots– Varying the level of cognitive and machine agent fidelity

• Evaluation will be done across levels– Devise and test abstractions that robustly map across models

by retaining key characteristics– Take small number of entities from lower (high fidelity level) and

test higher level for compliance of input/output behavior

Example: Evaluation Across Levels 2 and 3

•Infeasible to use the cognitive models in any large scale tests. •Small subset of the nodes will be extracted from the large scale simulation with the input and output from those nodes generated by the simulation. •The behavior of this subset can be compared against the results of experiments with cognitive models for verification.

Project Plan (1)

• Develop different scenarios (e.g. information dissemination; information fusion; plan generation) that allow – hybrid interactions of interest at each level and – testing of aggregate system properties of interest (e.g.

vulnerability; stability of overall system behavior) at each level

• Develop algorithms and techniques (e.g. appropriate simulation methods) that characterize aggregate system behavior at each level

• Develop methods for mapping fundamental characteristics of system components and interactions across levels

• Perform tests under different assumptions of system functionality (e.g. X% of sensor nodes fail; network capacity has decreased by Y%)

Project Plan (2)

• Design experiments where humans will interact with automation in various scenarios, e.g. supervisory control of assets, and with one another in various decision and information sharing tasks. These experiments will attempt to quantify types of human capabilities (e.g. learning, adaptation, planning) and their information processing and perceptual limitations.

• Develop high fidelity cognitive models to capture limits of information processing, constraints on attention, decision making bottlenecks but also incorporate pattern recognition and other capabilities such as learning and adaptivity that might remediate those shortcomings.

• Investigate the interaction properties of cognitive models as well as their computational tractability as the number of models that interact with each other increases.

• Discover appropriate abstractions of cognitive models that efficiently capture the basic information processing capability of humans. The models will capture the input and output characteristics of the node, without modeling the details of the cognitive processes by which the information is processed.

Project Plan (3)

• Test and validate the machine models, cognitive models and hybrid models at each level

• Develop techniques for validating the models across levels.

• Use validated models for producing predictions of overall human-machine systems and recommendations for the design of future systems and organizations.

• Develop strategies for mitigating performance degradation which focus on the use of different forms of adaptive or flexible automation

• Contrast the different automation approaches so as to achieve an optimal balance between too much flexibility, which can increase cognitive workload, and too little, which can increase system unpredictability, decrease situation awareness

Deliverables

• Validated methodology, based primarily on feedback between experiments and simulations at different level of granularity, for the study of complex human-machine systems

• Analytical results, for smaller scale problems that can serve as “islands of validation” for the larger scale simulations

• Results on system performance for various phenomena of interest, e.g. unpredictable system behavior and vulnerability

• Techniques for principled abstraction of human models and their behavior properties and effect on system performance.

Scientific Significance

• A key research question in this work is to develop a multi-level process that grounds and validates the models while allowing broad questions to be answered with abstract models. The process by which this model is developed will be a key research contribution that will improve a range of other research programs that are of importance to the Air Force

• The overall four level approach will be a publicly available, empirically grounded model of large scale human-agent interaction. Thus, the overall model will serve as a invaluable resource for many other researchers working in this area

• The work will make specific contributions in areas ranging from human-agent interaction to statistical mechanics. These contributions will be of benefit to a wide audience

Potential Applications to DoD

• NCW offers both new opportunities and the possibility of unintended consequences or unanticipated changes to human roles. The proposed research explores these issues through:– mathematical models,

– simulation

– cognitive models

• The research will provide– Identification of potential bottlenecks, challenges to stability, and

obstacles to human control

– Solutions to these problems before they are encountered in the field

Outreach and Research Training of Students

• 10 graduate students • 4 post doctoral fellows • Joint seminars at CMU and U. of Pittsburgh will be an ongoing

activity throughout the duration of the project. • Results of this research will be incorporated in courses taught

by the participating PIs.

• Broader contributions to education will occur through scientific meetings, publications, and by maintaining a public project web

site containing products of the research (e.g. code, data).

Roles of Team Members

• Sycara: PI, overall technical and management responsibility; main focus large scale models, levels 3 and 4, secondary focus and collaborations at level 2

• Campbell: decision models in levels 2 and 3• Cummings: human-UV modeling in level 1• How: human-UV automation in level 1, analysis and synthesis of

hybrid controllers, level 4• Lebiere: high fidelity cognitive models, levels 1 and 2• Lewis: scaling span of control, levels 1 and 2• Parasuraman: adaptive human-machine automation, levels 1 and 2• Scerri: large scale coordination, levels 3 and 4, collaborations at

level 2