The Cognitive Ergonomics Laboratory

NSF Information TechnologyResearch (ITR) Annual Review

David B. Kaber, Mo-Yuen Chow, Robert A. St. AmantDepartments of Industrial Engineering, Electrical and

Computer Engineering and Computer Science

North Carolina State University

This research was supported by the National Science Foundation (NSF), ITR Program (Grant No. 0426852).The opinions presented are those of the author and do not necessarily reflect the views of the NSF.

“Intelligent Human-Machine Interface &Control for Highly Automated Biological

Screening Processes”

Senior Personnel

w Kaber (PI) - Cognitive work analysis (CWA) anduser interface design for usability and function.w Chow (co-PI) - High-throughput screening (HTS)

process simulation and model of network controlof multiple screening lines.w St. Amant (co-PI) - Structured knowledge

representation, cognitive model development, anddesign for human-robot interaction (HRI).w R. Stoll (co-PI) - Liaison to URO and PI on parallel

grant from German SMV.

2

The Cognitive Ergonomics Laboratory

Research Assistants

w Becca Green (IE) - Cognitive task analysis (CTA); abstractionhierarchy (AH) modeling; user interface design (UID)evaluation and recommendations.

w Noa Segall (IE) - CTA; AH modeling; UID evaluation.

w Zheng Li (ECE) - Real-time resource allocation and schedulingfor network-based control; Adaptive bandwidth allocation.

w Rangsarit (Jae) Vanijjirattikhan (ECE) - Petri Net and simulationmodeling of HTS processes (e.g., sample plate preparation).

w Thomas Horton (CS) - Translation of structured knowledgerepresentation results to computational cognitive model code.

w Lloyd Williams (CS) - Graphical knowledge representationmethods as basis for cognitive modeling.

3

Organizational Partners

w URO (funded by SMV):n Complimentary research -

“Physio-ergonomicOptimized Human-machineInterfaces for Life ScienceAutomation.”

1. Measure physiological load inperformance of automatedlife science processes.

2. Develop web-based accessand interfaces for remotescreening data analysis.

3. Optimize human-machineinteraction for HTS systems.

n NCSU to provide HTSUID recommendationsto URO for evaluation in“home-grown” controlsoftware and for webinterface development.

n NCSU to sharecomputational cognitivemodel of screeningsupervisor performancewith URO to defineoptimal HRI approach.

4

Other Collaborators

w Consultation on cognitive modeling work:n Enhancement of ACT-R (adaptive control of

thought-rational) cognitive architecture.n Develop visual processing routines for models

for effective application of architectures to HRI.l Mike Byrne (Rice University)

l Mike Fotta (DNAmerican, Inc.)

l Mark St. John (Pacific Science & Engineering)

l Frank E. Ritter (Pennsylvania State University)

The Cognitive Ergonomics Laboratory

5

Activities of IE Subteam

w Review of CWA literature.w Goal-directed task analysis (GDTA) of

supervisory control of HTS lines (structuredinterviews with expert biopharmacologist).

w AH modeling to describe configuration andfunction of automated devices on HTS line.w Comparison of CTA method results for UID

recommendation formulation.

The Cognitive Ergonomics Laboratory

6

Literature Survey

w No comprehensive method for work-domain analysis(WDA), including problem and constraint ID.

w CTA can be used to support work-centered design but noapplication to interactive systems.

w CTA useful for identifying conditions for which auto may beused by operators.

w WDA method can also be used to ID auto needs.w AH modeling has been applied to auto.

w Research supports use of multiple CTA methods:n AH models used to ID limitations of auto to support operator.n ID gaps in operator understanding of systems and training needs.

w Across studies, no systematic approach for translation ofCWA results to system/UID guidelines for performance.

Mazaeva & Bisantz, in review.

Naikar et al., in press

Szczepkowski et al., in press

Szczepkowski et al., in press

Xu et al., in press

Xu et al., in press

7

GDTA

w Method:n Conduct procedural task analysis on basic HTS

method development using COTS.l ID steps of auto enzyme-based assay on HTS line.

n Interview expert biopharmacologist to describeknowledge structures in use of HTS automation.

n Goal, subgoal, task, decision, info needs:1.1. Adapt “bench-top” method to HTS line for rapid identification of useful compounds:

1.1.3. Establish plate configuration to achieve statistically valid results:T1 Ensure no edge effects in assay process (e.g., thermal effects on wells at edgesof plates).

Do I need to include empty wells at perimeter of plate?Need - Number of sample concentrations to be included on plate.Need - Cost of leaving plate wells empty.

Example subgoal,task, decisionsand informationneeds..

8

Overarching Goal and Major Subgoals

The Cognitive Ergonomics Laboratory

9

Subgoal Diagram and Info Needs

Data in lowest blockcan be used to defineinformation displaycontent for screeningprocess planningsoftware.

Content of GDTA tobe used as basis forstructured knowledgerepresentation method(SHAKEN) and to begincognitive modeldevelopment.

10

AH Modeling

w Abstract representation of physical device(e.g., bar coder, incubator, line robot, platereader, pipetter) and auto to control device.(Proprietary device software or action/ config.

dialog in overall method editor (e.g., SAMI).)n Define purpose of automation, general functions,

and ID physical components/interface options.

n Show how purpose is implemented and whycomponents are needed for functions.

w Supports operator learning of auto functionsand how to access through interface

11

AH Model for Bar Coder Software

States, operational conditions, and specific settings of bar coder control software

Manual string entry

Read strings from file

Manual entry of labelprefix and suffix withauto indexing of string

Select among labelformat options (Code128, Code 39, text only) Select “Update”

option

Select amongoptions for platesites (A, B, C, D)

Define globaloperation of device

Printing control

Label and applycontrol

Reading control

Select (global)action Select type of bar code

Store type ofbar code Define content of bar code

Select label positionon microplate Set offsets for label height

(relative to top/bottom of plate)Set frequency of bar codeverification (or none)

Select plate positionfor reading Set parameters for

addressing reading errors Set parameters for verification errors(reading previously labeled plates)

Label and read micro-plates

Controlmechanisms ofbar coder

Print and applydevice software Elect use of internal

SILAS transport ID

Select among actionoptions (Automation,Read, Manual Use)

Manual entry ofposition of codeon label (in mm)Select among “Read” or

“No-Read” options

Manual entry and storenumber of read periods Select among reading error

handling options (log error,flag error and stop) Select among verification error handling

options (i.e., errors during verificationof bar code on plate against SILAS ID -Ignore, or log and stop)

• Columns - Decomposesoftware system (as whole) tospecific interface features.

• Rows - Decompose purposeand general functions tospecific functions options.

Interface DecompositionFu

nctio

n D

ecom

posi

tion

12

Comparison of CTA Results

w Review decisions and info needs for goals inGDTA and compare with auto functions andinterface content reflected in AH model.n ID components of software not supportive of

operator goals (reduce unnecessary options).n ID components inadequate to meet info needs.

w Make UID recommendations:n Interface format suggestions organized based on

taxonomy of usability heuristics.n Interface content/auto capability recomms.

organized based on taxonomy of auto.

Nielsen, 1993

Parasuraman et al., 2000

13

Example UID Recommendations

1.1.9.4 Facilitate test plate incubation

Task 1 steps:Preparation (heating) of plates:• Input/select minimum time, Input/select

maximum time– Input time estimate (manual entry)

Apply micro-environment to test plates:• Process of setting and verifying heating

system state/temperature– Use touch-keypad to set temperature

Control of plate incubation processes:• Setting and activating plate chamber heat

– Manual start-up or auto activation

Integrate incubator into method

What is the incubation duration for the plate?

What is the temperature of the incubator microenvironment required by the assay?

Knowledge of “bench top” assay, as documented in literature The Cognitive Ergonomics Laboratory

Recommendations:• Allow temp. and

duration setting withone dialog.

• Eliminate redundancyin config. methods -use keypad orsoftware not both.

• Suggest temp. andduration based onLIMS database.

(Target user controland freedom anderror prevention.)

Decisions identified byexpert. Tasks stepsdictated by existingsoftware.

14

Activities of ECE Subteam

w Model IP-based remote real-time systemsupervisory control of multiple HTS lines.w Develop MATLAB/SIMULINK based

simulator of robots on HTS lines.

w Investigate adaptive bandwidth allocationalgorithm for control of multiple HTS lines.w Plan for integration of simulation with the

computational cognitive model and prototypeprocess control interface.

15

Petri Net Modeling of HTSLines

w High-level Petri Nets, including timed andcolored, appropriate for HTS line modeling.n Can incorporate process time data on “healthy”

and faulty conditions and resource attributes.

n Nets can capture process synchronization,asynchronous events, parallel operations andresource conflicts (unlike queuing networks).

The Cognitive Ergonomics Laboratory

Chen et al., 2005 Zuberek, 2004

Zurawski & Zhou, 1994

16

Overall Supervisory Control Model

Model includes fourHTS lines undercontrol of singlesupervisor. Centralserver relays usercontrol commandsand system statefeedback. Each linemonitored fordevice errors andreal-time processquality assurance.

17

Subsystem Model

Each line includesmultiple stationsequipped with multiplesensors and can acceptmultiple controlcommands. Processquality parameters aremonitored in real-timeat each step in method.

18

Target Subsystem Process:Sample Plate Preparation

Stock solution,deep well plates.

Biomek2000work surfaceconsider to bestorage buffer.

Each plate inprocess isconsideredtoken orentity in PetriNet model.

ReaderGross processtime estimatesfrom Entzian andJunginger.

Create sample plateswith differentconcentrations ofcompound.

19

Petri Net and Simulation Models

l pn - Places on process line.

l tn - Transitions among nodes (definerobot actions on token from pi to pi+1).

l Tokens are manipulated by robots (Rn)(e.g., ORCA (R1), Biomek2000 (R2)).

Focus on sample platepreparation process.

20

Network-Based Control ofMultiple HTS Line

w Relevant scheduling methods for addressingadaptive bandwidth allocation problem exist:n Real-rate scheduling models.n Feedback elastic scheduling models.

w Current work focused on mathematicalformulation of multiple line control problem.n Definitions of structure of physical subsystems

on HTS line at CELISCA are critical.n Using sampling rate scheduling approach to

model bandwidth allocation in network control.

e.g., Goel et al., 2004

e.g., Chen & Dai, 2003

21

Basic Resource Allocation Concept

• Complexity of subsystem signals (ui) indicator ofpotential process errors (i.e., process device orquality parameter errors).

• Sampling frequency (fi) for specific subsystemadapted by server based on system signal state.

• Greater bandwidth (Bi) allocation to subsystemsexhibiting faulty states for increased sampling.

Multiple HTS linesunder control ofsingle supervisor

Adaptive bandwidth allocationalgorithm automatically adjustfrequency of sampling forsubsystem with error condition.

22

Activities of CS Subteam

w Survey tools for automated translation amongcognitive modeling formalisms (e.g., SHAKEN).w Formalize translation between AI planning language

(PDDL - Planning Domain Defn. Language) andcode of cognitive modeling architecture (ACT-R).w Extend capabilities of ACT-R architecture in area

of visual processing to support HRI design.w Survey interaction mechanisms/presentation

techniques of existing approaches to HRI.

w Plan integration of tools with interface designactivity and process simulation work.

23

Building Tools for ModelingHRI in HTS Processes:

w Translate between natural language of CTA(GDTA) and structured/programming language forcognitive modeling (e.g., ACT-R).n GDTA to PDDL (via SHAKEN) - CTA result translated

to AI formalisms (planning language) including:l Conditionals on environmental states for action.

l Outcomes of action on environment.

l Representation of knowledge about UI feature dependencies.

n PDDL to ACT-R (via G2A extension):l Translate from AI planning formalisms to ACT-R (cognitive

model) primitives (using Lisp code).

Ghallab et al., 1998

The Cognitive Ergonomics Laboratory

St. Amant et al., 2004

24

Step 1: Develop SHAKEN Model forTranslation of CTA Result to PDDL

l SHAKEN - Web-based interactive tool based on mixture(“Martini”) of knowledge representation methods.

Hierarchy oftasks forensuring assayquality. Linksrepresentprecedence andcompositionrelationships.

Each node isindependent entitywith own processes.

Node are combinedor linked torepresent tasks.

25

Step 2: Encode CTA model andOutput in PDDL Form

GDTA hierarchicaloutline

SHAKEN knowledgeformulation

Output Lisp-based code

Translation to PDDLsyntax

CTA result as planningrepresentation

;;; THIS SAVE FROM CMAP OF: _Ensure-assay-quality5223 ;;; KM REPRESENTATION OF THIS "PROTOTYPE"

(disable-classification)

(Ensure-assay-quality has (superclasses (Event)))

(Ensure-assay-quality now-has (prototypes (_Ensure-assay-quality5290)))

(_Ensure-assay-quality5290 has (prototype-of (Ensure-assay-quality)) (prototype-scope (Ensure-assay-quality)) (prototype-participants (_Ensure-assay-quality5290 _ID-quality-criteria5291 _Assess-run-quality5292 _Cacluate-quality-stats5293 _Compare-test-stats5294 _Time-Interval5295 _Establish-CV5296 _Establish-IC50-criterion5297 _Establish-Z-factor-criterion5298 _Place5299 _Time-Interval5301 _Determine-Z-factor5302 _Generate-avg-IC50s5303 _Determine-sample-CV5304 _Place5305 _Time-Interval5306 _Compare-assay-Z-to-criterion5307 _Compare-avg-IC50s-to-data-for-compound5308 _Compare-sample-CVs-to-criterion5309 _Place5310 _Time-Interval5311 _Place5312 _Establish-CV-limit5313 _Establish-CV-out-of-limit-allowed5314 _CV-criterion5315 _Sample-CV5316)))

26

Extending Scope of Models forHRI Domains

w Development of extensions to ACT-R architectureto enhance visual routines:n Include component for focus of visual attention to

model task requiring real-time monitoring.

n Allows for cognitive model to interact with sameinterfaces (process screen images) actual user sees andpromotes validity of evaluation.

w Combine modeling extensions with high-levelassessment of impact of human-robot interfacecontent and format on behavior.n Critique of conventional HCI metaphors for HRI design.

27

Year 2 Planning

w Major tasks:n Prototype usable interface alternatives for interactive

screening process control tasks (Summer/Fall 2005).

n Develop computational cognitive model of supervisorycontroller behavior in HTS process control and dataanalysis (“ensure quality assay”) (Fall 2005/Spring 2006).

n Prototype of MATLAB/SIMULINK-based simulator ofHTS line in plate preparation (Fall 2005/Spring 2006).

n Apply cognitive model to control HTS line simulationthrough interface prototypes (Summer/Fall 2006).

w Follow integrated approach to research tasks toincrease pace of knowledge discovery.

28

Project Planning

w Conceptual diagram of project analyses andmodels, and information flows:

PDDL

Cognitive modeling to focus on goa- ensuring assay results meet qualitycriteria (See 1.2 in GDTA).

• No custom software for task.• Task can be performed

iteratively to refine process.

MATLAB/SIMULINK

29

Training Opportunities30

w IE subteam activities:n Graduate student learning of CTA methods.

n URO biochemist/process engineer familiarization with CTA methods.

n URO use of GDTA/AH models from developing training materials.

n Training of students to use CTA results as basis for UID.

w ECE subteam activities:n MATLAB/SIMULINK simulation can be used by process engineers in

developing/testing new automation ideas for HTS lines.

n Integration of simulation with control interface prototypes supportsstudent training on iterative UID process.

n Presentation of research results in grad. courses (ECE 556; ECE 716).

w CS subteam activities:n Grad. student learning of knowledge acquisition and planning tech.

n Training of graduate student in basic cognitive model construction.

Publications and Website31

w Journal Manuscripts:n Kaber, D. B., Segall, N. & Green, R. (in preparation). Using multiple cognitive task analysis methods for

supervisory control interface design in high-throughput biological screening processes. To be submitted toTheoretical Issues in Ergonomics Science.

n Ritter, F. E., Van Rooy, D., St. Amant, R., & Simpson, K. (in review). Providing user models with direct access tocomputer interfaces: An exploratory study of a simple human-robot interface. Submitted to the Journal ofCognitive Engineering and Decision Making.

n St. Amant, R., Horton, T. E., & Ritter, F. E. (in press). Model-based evaluation of expert cell phone menuinteraction. ACM Transactions on Computer-Human Interaction.

n St. Amant, R., Kukreja, U., Ritter, F. E., Boyce, C. W., & Janssen, C. (in review). Cognitive Modeling and Human-Robot Interaction: Remote Navigation. Submitted to IEEE Transactions on Systems, Man, and Cybernetics.

w Conference Proceedings Papers:n Horton, T. E., & St. Amant, R. (in preparation). A taxonomy of presentation and manipulation techniques for

human-robot interfaces. To be submitted to the International Conference on Human-Robot Interaction.

n Kaber, D. B. (2005). Human-centered design for human-robot interaction. In J. Sinay, P. Mondelo, K. L. Saarela,W. Karwowski & M. Mattila (Eds.), CAES ’2005 Proceedings (CD-ROM). Kosice, Slovak Republic, May 25-28, 2005.

n Li, Z., Vanijjirattikhan, R., Chow, M-Y. & Kaber, D. B. (in preparation). Adaptive bandwidth allocation andsampling rates in control of multiple high throughput biological screening lines. To be submitted to IEEE ICIT05,IEEE IECON06, or IEEE ICRA06.

n St. Amant, R., & Wood, A. B. (in press). Tool use for autonomous agents. In National Conference on ArtificialIntelligence (AAAI), Pittsburgh, PA. AAAI Press.

n St. Amant, R., Riedl, M. O., Ritter, F. E., & Reifers, A. (in press). Image processing in cognitive models withSegMan. Proceedings of HCI International. Las Vegas, NV.

n Vanijjirattikhan, R., Li, Z., Chow, M-Y. & Kaber, D. B. (in preparation). Timed Petri Net modeling and simulationof high throughput biological screening lines. To be submitted to IEEE ICIT05, IEEE IECON06, or IEEE ICRA06.

http://people.engr.ncsu.edu/dbkaber/NSF_ITR/

Contributionsw Across disciplines - Highly

specialized student trainingin research methods andlife sciences auto leadingto rare base of technicalprofessionals for industry.

w CS subteam:n Novel integration of CTA

and AI research methods tosupport cognitive modeling.

n Bridge between AI planningarea and cognitive modelingnot explored - approach totranslations betweenplanning and cognitive modellanguages is novel.

w ECE subteam:n Novel HTS process simulation

supporting human-centereddesign of interfaces and adaptivenetwork control technology.

n Novel quality of control conceptfor human interaction with lifesciences automation.

n Novel IP-based time-sensitivesystem monitoring and faultdetection control.

w IE subteam:n No prior research has applied

CTA methods to life scienceprocesses.

n Novel combination of GDTA andAH modeling to ID shortcomingsof existing auto and to supportUID recommendations.

32