Conversation, Vision, and Driving:

Experimental Predictions vs. Real-World Crashes

Richard A. YoungGeneral Motors Engineering

Wayne State University School of Medicine

2

Acknowledgements Chris Schreiner, Arizona State University

Li Hsieh, Wayne State University

Richard J. Genik II, Christopher C. Green, Wayne State University Medical School

Susan M. Bowyer, John E. Moran, Norman Tepley, Henry Ford Hospital

Linda Angell, GM Safety Center

Jon Hankey, Luke Neuratter, Virginia Tech Transportation Institute

3

Background Although often casually discussed as though it were a single task, driving is

actually several tasks which play out over different levels of functional hierarchy, on different timescales, and often concurrently.

An important issue is the design of driver-vehicle interfaces that support safe and efficient vehicle operation under such multi-tasking conditions, particularly with the addition of secondary in-vehicle tasks (e.g., cell phone conversations).

Little is known about the brain mechanisms underlying primary driving tasks, much less secondary tasks -- although it is likely that there is some functional overlap, and perhaps consequent interference between primary and secondary tasks.

While interference between the manual requirements of driving and of secondary tasks is obvious—dialing a telephone and steering a vehicle both compete for the same hands—less obvious cognitive demands of secondary tasks are becoming a growing concern.

Some simulator studies claim the distracting effects of, for example, a cell phone conversation may be significant.

Such claims require careful investigation as to what might cause such effects in the laboratory, and whether such effects can produce detectable increases in crash rates in real-world driving compared to normal baseline driving.

4

Society Costs and Health Risks

Motor vehicle crashes cost $231 billion per year in U.S. alone

# 8 leading cause of death (Anderson & Smith, 2003)

# 4 largest public health problem in U.S. Exceeded only by heart disease, depression

and stroke

Driver error is the principal cause in 45% to 75% of crashes (Wierwille et al., 2002) 38% of all crashes are related to driver

distraction (Virginia Tech Transportation Institute, 2005)

5

Objectives(1) Compare the behavioral and neural

correlates of conversation effects on visual event detection during driving using the same visual event detection paradigm in brain imaging, behavioral testing, and closed-road driving experiments

(2) Evaluate hypotheses that may help explain the discrepancy between the predicted crash rates from such experimental studies and the observed real-world airbag-deployment crash rates during conversation on a mobile phone embedded in a vehicle.

6

Methods Previous laboratory and closed-road experimental studies, and some

epidemiological real-world studies, predict an increase in crashes arising from phone conversations during real-world driving compared to baseline driving.

One proposed underlying mechanism is the possible effect of conversation on detection of visual events.

To investigate this mechanism, the "load" paradigm (Angell et al., 2002; Young et al., 2005b) assessed the effects of conversation on visual event detection during simulated driving in behavioral labs, fMRI and MEG imaging centers, and actual driving on a closed road.

The primary task was to depress a foot pedal in response to a small red light presented to the left or below the driving scene at unpredictable times. The secondary task was to engage in a conversation.

The participant pressed a button to answer a ring tone, and then answered simple auditory questions such as “What is your birthdate?”. fMRI (Young, 2005a) and MEG data (Bowyer et al., 2006) were analyzed to examine the neural substrates of driving with and without conversation, and the behavioral results validated in the lab and closed-load studies.

The predictions of these and other experimental investigations were then compared with a large and complete body of real-world data associating conversation usage on mobile phones embedded in vehicles and crashes severe enough to deploy an airbag (Young, 2001).

Data and analyses are presented to evaluate hypotheses that may help explain the differences between the real-world crash data and experimental predictions.

7

Methods – Six Sites1. Wayne State Medical School fMRI Lab

(behavior + brain)2. Henry Ford Hospital MEG Lab (behavior

+ brain)3. Wayne State University Speech-

Language Lab (static behavior)4. GM Milford “Usability Lab” (static

behavior)5. Virginia Tech Smart Road (closed-road)6. Real-World Driving (OnStar)

8

1. WSU School of Medicine: fMRI Research Facility

1.5 T & 4T Siemens scanner

Headphones

9

2. Henry Ford Hospital: MEG Driving Study

MEG: A technique for localizing sources of electrical activity within the human brain by non-invasively measuring the magnetic fields arising from such activity.

10

2. MEG Experimental Set-Up

Peripheral light event

Central light event

Mirror arrangement

Subject in MEG

Subject View

11

Wayne State University: Speech-Language Neuroscience

Laboratory Static

Behavioral Driving Test Lab

12

3. WSU: Speech-Language Neuroscience Laboratory

Static Behavioral Driving Test Lab

13

4. Milford Usability Lab: Front Event Light

14

4. Milford Usability Lab: Side Event

15

4. Milford Usability Lab: Wizard

16

1-4. Driving Video Scene

17

4. Milford Usability Lab:

18

5. Virginia Tech Smart Road

19

5. VTTI Lights

20

5. VTTI Road

21

6. Real-World Driving (OnStar)

22

6. Methods: Real-World Driving (OnStar) Predictions from these experimental findings were

validated against a crash database to see if conversation increases the rate of crashes in the real world when compared to baseline driving.

We examined real-world data from a database of 91 million personal calls from three million drivers over a period of 2.5 years.

We counted the number of cases in which a hands-free cellular call was in progress when an air bag crash occurred.

We then calculated a crash Incident Rate Ratio (IRR) comparing the Incident Rate of an air bag crash while engaged in a call to the Incident Rate of an air bag crash during baseline driving for the same population of drivers.

23

RESULTS

24

1. fMRI Results: Dual vs. View

Increase in brain activation (colored regions) to DUAL task vs. brain activation from VIEW task. This image shows the brain activation regions specific to the EVENT task when performed in a driving-like context.

Key: PM (medial pre-motor), AC (anterior cingulate). PC (precuneus), IPL (inferior parietal lobule), PM (medial pre-motor), SPL (superior parietal lobule), PCG (postcentral gyrus), PM (medial pre-motor), Cb (cerebellum).

Saggital (side) view of one “slice” of the brain

Top views of the brain at different slices of a horizontal plane.

Color scale for t-score for comparison in brain activation

25

2. MEG Results: Brain Movie During Light and Brake Response (screen only)

26

2. MEG Results: Brain Movie During Light and Brake Response

27

2. MEG Results: Right Superior Parietal Lobe

28

1-2. Summary Results - Brain Modulations in the right superior parietal

region and visual cortices specifically mediated the baseline reaction time to visual light events.

Conversation in these settings appears to contribute to increased reaction times by reducing brain modulation to visual events in these specific brain regions.

29

3. Results – Wayne State Speech Lab

30

4. Results - Milford

TBD

31

5. Results – Smart Road

0.0

0.5

1.0

1.5

2.0

2.5

Han

d-H

eld

Pho

ne, N

oC

on

Spe

aker

phon

e, N

oC

on

OnS

tar,

No

Con

Han

d-H

eld

Pho

ne,

Con

Spe

aker

phon

e, C

on

OnS

tar,

Con

10-d

igit

hand

-hel

ddi

alin

g

Bas

elin

e dr

ivin

g

Missed Events were assigned a 10 second value

Short Conversation

Long Conversation

10 Digit Dial

BaseLine

Handheld Handsfree On*

Handheld Handsfree On*

32

1-5 Overall Experimental Results - Behavioral Conversation slightly increased

visual event reaction times in laboratory and closed-road driving experiments compared to a no conversation baseline, with little or no effect on miss rates.

The short conversation condition had a longer visual reaction time than the long conversation.

33

6. Results: Real-World Driving (OnStar) –ORAL ONLY – NOT FOR DISTRIBUTION

Slide deleted

34

6. Results: Real-World Driving (OnStar) ORAL ONLY – NOT FOR DISTRIBUTION

Slide Deleted

35

6 Results – Real World (2) The absolute frequency of airbag

deployment crashes occurring during conversation on a wireless phone embedded in a vehicle is lower than predicted by experimental data.

The relative airbag deployment crash rate is no greater than baseline driving, also contrary to prediction from experimental data.

36

DiscussionFive hypotheses help explain the discrepancy between

the experimental predictions and real crash rates: 1. The visual reaction time increase arising from conversations

in experimental studies may be too small to give rise to a detectable increase in real-world crash rates;

2. Lab-based event detection does not account fully for similar types of event detection in real driving, let alone for crashes (Angell et al., 2006; Curry et al., 2005; Young et al., 2005);

3. Conversation may at times mitigate risks such as fatigue; 4. Portable and embedded cell phones may have substantially

different human factors properties; 5. Drivers may tend to change behavior during calls in ways

that reduce net risk: e.g., placing calls in relatively benign driving conditions, reducing risky driving maneuvers, increasing headway, glancing longer to the forward roadway, or increasing glances to mirrors after detecting an event.

37

Conclusions The real-world airbag-deployment crash rate during

conversations on an a hands-free wireless device embedded in a vehicle is substantially lower than what is predicted by brain-imaging, simulator, closed-road, or epidemiological studies.

Several hypotheses may help resolve the discrepancy between the predicted and observed real-world crash rates.

This study concludes that the claim that the effects of conversation on visual event detection as observed in experiments can accurately predict real-world crash rates in all cases requires careful investigation.

38

THANK YOU!

39

Funding Acknowledgments• The baseline research at Wayne State Medical

School was supported by an unrestricted gift from the GM Foundation.

• The baseline research at Henry Ford Hospital was supported by NIH/NINDS Grant RO1-NS30914.

• The work adding secondary tasks is supported by grants from the Crash Avoidance Metrics Partnership (CAMP) funded by GM and Ford Motor Company to Wayne State Medical School and Henry Ford Hospital.

• Current research is funded by Michigan Technology Tri-County Corridor.

40

References Angell, L. S., Young, R. A., Hankey, J. M. & Dingus, T. A. (2002) “An evaluation of alternative methods for

assessing driver workload in the early development of in-vehicle information systems,” SAE Proceedings, 2002-01-1981, Joint Industry/Government SAE Conference, Wash., D.C.

Angell, L. S., Auflick, L. L., Austria, P. A., Kochhar, D. S., Tijerina, Biever, L. W., Diptiman, T. , Hogsett, J. & Kiger, S. (CAMP) (2006) Driver Workload Metrics Project: Final Report Sponsored by National Highway Traffic Safety Administration, Washington, D.C., November. DOT HS 810 635.

Angell, L. S., Auflick, L. L., Austria, P. A., Kochhar, D. S., Tijerina, Biever, L. W., Diptiman, T. , Hogsett, J. & Kiger, S. (CAMP) (2006) Driver Workload Metrics Project: Final Report - Appendices Sponsored by National Highway Traffic Safety Administration, Washington, D.C., November. DOT HS 810 635.

Curry, R.C., Greenberg, J.A., & Kiefer, R.J. (2005). "NADS versus CAMP closed-course comparison examining 'last second' braking and steering maneuvers under various kinematic conditions", Crash Avoidance Metrics Partnership (CAMP), Contract DTFH61-01-X-00014, Wash., DC, August, DOT HS 809 925.

Bowyer, S., Moran, J., Hsieh, L., Manoharan, A., Young R.A., Malladi, K., Yu, Y-J., Chiang, Y-R., Hersberger, R., Genik, R., & Tepley, N. (2006). "MEG localization of neural mechanisms underlying reaction time to visual events while watching a driving video: Effects of conversation," International Congress Series: New Frontiers in Biomagnetism: Proc. of the 15th International Conference on Biomagnetism, Vancouver, BC Canada, August 21-25, 2006. D. Cheyne, B. Ross, G. Stroink and H. Weinberg (Editors).

Young, R. (2001) "Association between embedded cellular phone calls and vehicle crashes involving air bag deployment," Proc. of Driving Assessment 2001: International Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Snowmass, Utah, August.

Young, R. A., Hsieh, L., Graydon, F. X., Genik II, R., Benton, M. D., Green, C. C., Bowyer, S. M., Moran, J. E., & Tepley, N. (2005a). "Mind-on-the-Drive: Real-time functional neuroimaging of cognitive brain mechanisms underlying driver performance and distraction," Human Factors in Driving, Telematics and Seating Comfort 2005, SP-1934, Society of Automotive Engineering, Warrendale, PA, April.

Young, R. A., Aryal, B., Muresan, M., Ding, X., Oja, S. & Simpson, S. (2005b), “Road-to-lab: Validation of the static load test for predicting on-road driving performance while using advanced in-vehicle information and communication devices,” Proc. of the Third International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Rockport, Maine, July.

41

BACKGROUND

42

Additional Data and Studies

1. Crash-Avoidance Metrics Partnership (static, closed-road, and on-road)

43

6. CAMP Static, Closed-Road, Open-Road

44

Mind on the Drive• Metrics for the major aspects of driver visual-

manual distractions are well established - Measure eyes-off-road time, hands-off-wheel time, etc.

• Cognitive distraction is not as easy to measure - Cognitive activity may not be reflected in observable

behavior- Driver reports of their own distraction are insufficient

o Drivers are not always aware that they are distracted

• Problem: Can Mind on the Drive be directly measured?

45

Solution: Brain Imaging• Modern brain imaging techniques may allow direct

measurement of cognitive driving distraction • Benefits:

- Gives a sound, scientific basis to understanding cognitive aspects of driver distraction

- Saves time and expense in designing secondary systems to minimize cognitive distraction

o By understanding the fundamental underlying mechanisms of cognitive distraction, testing of in-vehicle systems or principles can be hypothesis-driven, requiring less time and expense to conduct such tests.

- Improves public, private, and governmental understanding in this important area

- Knowledge of the underlying mechanism may lead to guidelines that reduce driver error

46

Objectives of Study Reported Today• To establish a baseline foundation for the

basic brain events underlying the detection of a light event on a roadway, to the formation of a braking response by the foot.- This baseline foundation is reported here today

• This study lays the foundation to understand the effect of the addition of secondary tasks in the vehicle, such as cell phone conversations, on the basic event detection and braking responses.

47

fMRI at Wayne State Medical School• fMRI = “Functional Magnetic Resonance

Imaging”• Allows the researcher to see the activation of

different regions of the brain to a high degree of spatial accuracy.

• fMRI measures molecular changes associated with blood flow that are in turn associated with neural activity - Pulses a magnetic field into the intact awake brain- Measures the return magnetic response of the brain

to that pulsed field.

48

Benefits of fMRI• Safe and non-invasive • No risk to subjects• High spatial resolution

- Identifies the activated brain structures exactly• Can image deep structures in the brain (limbic

system) • Supports other imaging techniques

- fMRI knowledge can be imported into other imaging solution sets, tying down the brain regions involved

- fMRI enhances the findings of other imaging methods, by improving the accuracy and robustness of these other results.

49

fMRI Experiment: MethodsStimulus Conditions

EVENT Task Alone

VIEW Task Alone (No Events)

DUAL Task: VIEW + EVENTS

4

+Baseline Fixation

50

fMRI Boxcar Stimulus Presentation

Boxcar design, or alternating blocks of baseline fixation (40 seconds) followed by task condition 1, 2, or 3 (40 seconds), followed by baseline, etc. for four blocks (360 seconds). The timing of the event lights in conditions 1 and 3 is illustrated in the line below.

Video

Event Lights

Time (sec)

51

fMRI Methods• Six subjects (three male three female) age

range 22-28 yrs.• fMRI brain data collected in 180 repeating

“volume” acquisitions during the four blocks (360 sec) for each task condition

• Brain data translated to a common brain to compensate for individual anatomical variation

• Translated data is then averaged• Made statistical comparisons between brain

activations for different conditions of interest

52

fMRI Results: DUAL Task vs. Baseline

PCb

PCb

Z=-24Brain response to DUAL task vs. brain response to baseline. There is widespread activation from the DUAL task as would be expected. Key: PM (medial pre-motor and also some motor), PC (precuneus), AC

(anterior cingulate). SPL (superior parietal lobule), FEF (frontal eye fields and some pre-motor), P (pulvinar), CH (caudate head), Cb (cerebellum).

Saggital (side) view of two “slices” of the brain.

Top views of the brain at different “slices” of a horizontal plane.

Color scale for t-score for comparison in brain activation

53

Discussion of fMRI Experiment• Provides strong evidence that fMRI

experimental methods can be applied to study primary driving tasks

• Success for addition of secondary tasks in ongoing experiments is likely.

• The hot spots identified in fMRI experiment at Wayne State Medical School provided the anatomical locations/references for the following experiments at Henry Ford Hospital.

54

Neuromagnetism Lab at Henry Ford Hospital

• Magneto-encephalography (MEG)- A safe and non-invasive method to measure

activation of neurons in the conscious human brain in real-time

• Henry Ford Hospital has only MEG facility in Southeast Michigan- One of only several in the world

55

Magneto-encephalography (MEG)A technique for

localizing sources of electrical activity within the human brain by non-invasively measuring the magnetic fields arising from such activity.

56

Strengths of MEG• Measures activity of neurons

in the brain (particularly cortical neurons)

• Millisecond temporal resolution

• Millimeter spatial resolution• Correlates function and

anatomy• Safe and non-invasive

technique to image neural function – no risk to subjects

57

Variability of MEG• Within subjects: To control, all trials are

repeated in two separate runs.• Amplitude tends to be consistently weaker in the

second run in the same subject by about 30%.• Between subjects:

• Temporal latency differences less than 8 msec or 7% of temporal variation.

• Spatial location differences on the order of 1 cm when all brains are placed on a common image

• In general, amplitudes may vary widely, but temporal latencies and spatial locations are consistent within and between subjects

58

MEG Baseline Experiment• Similar to fMRI experiment• Measured brain mechanisms underlying simple

event detection and braking (no secondary tasks) • Subject watches driving video of real driving

scene• Light events presented in central or peripheral

visual field, randomly presented in time• Subject taps foot pedal to simulate braking in

response to lights• No “blocking” necessary as in fMRI, continuous

presentation of driving scene and light events.

59

MEG Results: Average of 40 Light and Brake Events for Subject #1

Light interval brake

light off,click

60

MEG Results and Discussion• Baseline MEG experiment was successful

- Baseline brain responses to light and foot pedal presses were easily observable with low noise.

- Main brain regions identified, associated with fMRI results

- Activation of multiple brain regions when the subject had to respond to the light event plus watch the driving video, as contrasted with just watching the driving video alone (same as fMRI).

- Exact timing relations of brain events from light to brake response are now specified

61

Current Work in Progress (Wayne State and Henry Ford)

• Add secondary phone answering task• Ring phone, press button to pick up • Engage in (silent) conversation to questions

- Conditions: No, short, and long conversation• Determine what delays in reaction time and

misses occur to light events during phone answering and conversation

• Determine the underlying brain mechanisms of cognitive distraction by subtracting baseline brain responses to events alone, to events + phone answering

62

Overall Conclusions• The baseline foundation experiments reported

here today indicate a high probability of success for later experiments adding in secondary tasks.- The basic brain structures and the timing between

them, as activated by a light stimulus and a brake response, were easily identified.

- The variability between and within subjects is small enough that the effects anticipated from secondary tasks will be easily discriminated from normal variability.

• This brain imaging approach may prove of benefit in extending the current state-of-the-art in improving driver awareness, keeping “Mind on the Drive,” and reducing driver errors.