11/29/2017

1

Designing Human-Automation (or Robot) Relations

Joachim MeyerDept. of Industrial Engineering

Fleischman School of Engineering

Verification and Validation of Autonomous Systems Network Workshop, 23rd-24th Nov. 2017Cumberland Lodge, Windsor Great Park, Berkshire, UK

System Designer

Operator

Regulators

Clients

11/29/2017

2

Cognitive Engineering

Human System World

Human-Machine System

Optimizing the design of human-machine systems

2

2

)()

()()

()(

)(

211

11

11

2111111

11

11

c

c

dxxSMHMMHHHMMHHH dxxNFCFFCCCFFCCC

cN

cS

Model

Controlled Experiments

Observations in the Field

11/29/2017

3

Operator Clients

System Designer

Regulators

Technology is for people

Metropolis (1927)

Unless …

6

11/29/2017

4

Needs (Current and Future)

7

Automation and Basic Needs

8

11/29/2017

5

Automation in Buildings

9

What does automation do?The availability of a resource (water, food, etc.)in different conditions

Time

Ava

ilab

ility

No Limitations (e.g., Younger Person)

Some Limitations (e.g., Older Person)

With Automation (for everybody)

No Automation

10

11/29/2017

6

When Something Goes Wrong

• Natural disasters• Earthquakes

• Flooding

• Hurricanes

• Etc.

• Intentional harm• Military attacks

• Terrorism

• Cyberattack

• Etc.

“Irma”, downtown Miami, Sept. 10, 2017

Earthquake damage, Beichuancounty, China, May 2008

11

Earthquake Damage to High-Rise Buildings

Wang et al. (2015)

12

11/29/2017

7

What does automation do?The availability of a resource (water, food, etc.)in different conditions

Time

Ava

ilab

ility

No Limitations (e.g., Younger Person)

Some Limitations (e.g., Older Person)

No Automation

With Automation

No Limitations

With Limitations

Automation Failure

13

Automation and basic needs

• Automation can help provide resources• Automation can stabilize resource supply (if it

functions)• If automation fails, resource supply may be

dramatically disrupted

In general terms:• In normal conditions, automation lowers the

variability of outcomes• With automation failures the variability of

outcomes increases greatly (more than without automation)

14

11/29/2017

8

iDrive interface

15

Performance TimeLavie, T., & Meyer, J. (2010). Benefits and costs of adaptive user interfaces. International Journal of Human-Computer Studies, 68, 508-524.

Routine drives Non-Routine drive

Routine drives Non-Routine drive

During routine drives, higher levels of adaptivity help performance, especially for older drivers

During non-routine drives, higher levels of adaptivity hurt performance, especially for older drivers 16

11/29/2017

9

Interactions with systems increasingly become Supervisory Control

Systems mainly require and support cognition and especially decision making

Boeing 314 Flying Boat (Clipper)

Introduced 1939 for very long (transatlantic)

flights. In service until 1946.

Flight crew of 5: captain, first officer,

navigator, flight engineer, wireless operator

18

11/29/2017

10

There are only two things required to fly a modern airliner: A pilot and a dog. It’s the pilot’s job to feed the dog. It’s the dog’s job to bite the pilot if he touches anything in the cockpit. Anonymous pilot joke

19

Stages of Automation (Parasuraman, Sheridan & Wickens, 2000)

High

Low

Decision

Selection

High

Low

Information

Analysis

High

Low

Information

Acquisition

High

Low

Action

Implementation

System

B

System

A

11/29/2017

11

Should we involve people in decision making?

• Binary categorizations as the basic type of decisions• Malignancy in medical image or not?

• Email attachment malicious or not?

• Credit card transaction legitimate or fraudulent?

• Machine malfunctioning or not?

• Child mistreated or not?

• Etc., etc.

• Categorizations can be analyzed with Signal Detection Theory (e.g. Green & Swets, 1966)

Principles of Signal Detection Theory (SDT)

Macmillan, N. A., & Creelman, C. D. (2004). Detection theory: A user's guide. Psychology press.

Stanislaw, H., & Todorov, N. (1999). Calculation of signal detection theory measures. Behavior research methods, instruments, & computers, 31(1), 137-149

Reject

C

𝑬𝒏

𝑬𝑺

Accept

• 𝐸𝑛, 𝐸𝑠 ~ Gaussian distribution

• 𝐸𝑛~𝑁 0,1 𝐸𝑠~𝑁 𝜇𝑠, 1

• 𝑑′ = 𝜇𝑠 − 𝜇𝑛 = 𝜇𝑠 (Sensitivity)

• 𝛽 =𝑓𝑠 𝑐

𝑓𝑛 𝑐(Response Criterion)

• 𝛽∗ =1−𝑃𝑠

𝑃𝑠

𝑉𝐶𝑅−𝑉𝐹𝐴

𝑉𝐻𝑖𝑡−𝑉𝑀𝑖𝑠𝑠(For Max. EV)

• 𝑐 =𝑙𝑛𝛽

𝑑′+

1

2𝑑′ (Cutoff point)

11/29/2017

12

A Simple Decision Support System

𝑋 = ቊ1 𝐴𝑐𝑐𝑒𝑝𝑡0 𝑅𝑒𝑗𝑒𝑐𝑡

𝑌 = ቊ1 𝐴𝑙𝑎𝑟𝑚0 𝑁𝑜 𝐴𝑙𝑎𝑟𝑚

The Aim is to Identify & Reject Signals

The System SThe Environment

Ala

rm M

od

ule

Hu

man

Use

r

Signalps

Noise1-ps

e

Y

X

Observable Parameter

d’A

d’h

d’H – HO sensitivity, d’A- warning sensitivity

Lines indicate d’eff as predicted by the expression d’eff=(d’H2+d’A

2-.3d’Hd’A).5

Maximal possible combined sensitivity values (d’eff*)

11/29/2017

13

d’eff / d’H for warning systems with different levels of validity, as

expressed by the ratio d’A / d’H, for different levels of trust in the

system.

d’eff/d’h as a function of the level of trust in the system

Underweighting Overweighting

So should we combine humans and automation in decisions?• We can do so, but the benefits are relatively small, and they are

maximal when the sensitivities are similar

• Benefits depend on humans assigning correct weights to the information from automation

• Do they assign correct weights?

11/29/2017

14

Result:

People often do not optimally utilize information from cues, alarms, alerts, decision aids

Predicted and observed sensitivity (d'eq) values in the four blocks for the lower sensitivity system (d'Cue=1.93) and the higher sensitivity system (d'Cue=3.03). The arrows indicate the distance between the sensitivity achieved by the combined human-machine system and the sensitivity of the cueing system alone.

Meyer, J., Wiczorek, R., & Günzler, T. (2014). Measures of reliance and compliance in aided visual scanning. Human Factors, 56, 840-849.

If people don’t cooperate optimally with automation, perhaps we should simply let them control it?

11/29/2017

15

Operator Clients

System Designer

Regulators

Letting people make choices

30

11/29/2017

16

Automation and Control

31

So should we let people adjust system settings?

People do not set optimal values for parameters they can control

Threshold values for different probabilities when misses were costly (left panel) and when false alarms were costly (right panel) for the three information conditions. The dashed lines represent optimal threshold values.

Botzer, A., Meyer, J., Bak, P., & Parmet, Y. (2010). Cue threshold settings for binary categorization decision. Journal of Experimental Psychology: Applied, 16, 1-15.

32

11/29/2017

17

Actually, very often they can’t possible adjust the system correctly …

Number of observations needed to detect a change in a proportion(Meyer & Sheridan, 2017)

pType1Error = .05pType2Error = .1

33

Placebo buttons

34

11/29/2017

18

But human control may not be just for people’s feeling in control

• Advanced automation creates systems that may potentially harm people

• We may want to keep people in control

Users' responsibility when interacting with automated systemsWork with Nir Douer

What should be human involvement?

Who should be responsible?

To what extent?

Nir Douer Advisor: Prof. Joachim Meyer

11/29/2017

19

Stages of Automation (Parasuraman, Sheridan & Wickens, 2000)

High

Low

Decision

Selection

High

Low

Information

Analysis

High

Low

Information

Acquisition

High

Low

Action

Implementation

System

B

System

A

Conant (1976): Application of Information Theory to Systems

Conant, R. C. (1991). Laws of information which govern systems. In Facets of Systems Science (pp. 419-448). Springer US.

N−Dimensional Information Theory

𝐻 𝑆 = −

𝑆

𝑝(𝑠)𝑙𝑜𝑔2𝑝(𝑠)

𝑆 = 𝑋1, 𝑋2,…𝑋𝑛

𝐹 = σ𝑗=1𝑛 𝐻 𝑋𝑗 =𝐹𝑡+𝐹𝑏 + 𝐹𝑐+𝐹𝑛

𝐼 𝑋1 : 𝑋2 : … . 𝑋𝑛 =

𝑗=1

𝑛

𝐻 𝑋𝑗 −𝐻 𝑋1, 𝑋2 …𝑋𝑛

Throughput Blockage Coordination Noise

Partition Law Of Information

Total Information Flow

11/29/2017

20

Combining the 2 Models to Construct Responsibility Measures

State

Au

tom

atio

nH

um

an U

ser

Information acquisition Information analysis Action selection Action Implementation

Y

The Environment

Observable Parameters

The System

Output to the Environment (Outcome)

Z

𝑅𝑒𝑠𝑝𝑜𝑛𝑠𝑖𝑏𝑖𝑙𝑖𝑡𝑦 ≝𝐻 𝑍/𝑌1, 𝑌2

𝐻 𝑍(the proportion of the outcome that does not result from automation)

Y1

X1

Y2

X2

An Example – A Simple Alarm System (cont.)

0 ≤ 𝑅𝑒𝑠𝑝𝑜𝑛𝑠𝑖𝑏𝑖𝑙𝑖𝑡𝑦 ≤ 1

𝑅𝑒𝑠𝑝𝑜𝑛𝑠𝑖𝑏𝑖𝑙𝑖𝑡𝑦 ≝𝐻 𝑋/𝑌

𝐻 𝑋=

𝐻 𝑋,𝑌 −𝐻(𝑌)

𝐻 𝑋

If H(X/Y)=H(X)X & Y are independent

If H(X/Y)=0 Y determines X

11/29/2017

21

Douer & Meyer (2017)

Responsibility As a Factor of Sensitivity Ratio d’h/ d’a

d’h/ d’a

d'h/d'a < 0.2 Responsibility

~0%

d'h/d'a > 5 Responsibility

~100%

Responsibility dispersion increases

as d'h/d'a 1

d'h/d'a > 2 Responsibility > 80%

11/29/2017

22

Humans as a crumple zoneM.C. Elish (2016). Moral crumple zones: Cautionary tales in Human-Robot Interaction. We Robot 2016 working paper 3/20/2016.

• “Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions.”

43

Operator Clients

System Designer

RegulatorsSome Conclusions

• People often cannot adequately adjust system settings

• Aiding people may be problematic –perhaps get rid of automation or of people

• People can benefit from automation

• But there are also risks

• Putting people into the loop is not simple (and often very problematic)

11/29/2017

23

Operator Clients

System Designer

RegulatorsWhat then?We need predictive models and quantitative methods to provide the basis for informed decision making

We have to do so cautiously• Considering human

capabilities• Considering technological

possibilities• Considering our values

We want to keep automation and automating functions

Comments welcome at jmeyer@tau.ac.il

46