Deliverable 7.1 Evaluation Frameworkmade HMI under automation This project has received funding from...

Adaptive ADAS to support incapacitated drivers Mitigate Effectively risks through tailor

made HMI under automation

This project has received funding from the European Union’s Horizon 2020

research and innovation programme under grant agreement No 688900

Deliverable 7.1 – Evaluation Framework

Deliverable Identity

Work Package No. WP7

Work Package Title Evaluation

Activity No. A7.1

Activity Title Evaluation framework and data gathering tools

Dissemination level PU = Public

Main Author(s)

Marta Pereira Cocron, TUC

Alex Vallejo, IDIADA

Maria Beatriz Delgado, IDIADA

Marc Wilbrink, DLR

Anna Anund, VTI

Stas Krupenia, Scania

Luca Zanovello, Ducati

File Name D7.1 Evaluation Framework

Online resource http://www.adasandme.com

Ref. Ares(2018)1093426 - 27/02/2018

http://www.adasandme.com/link-to-deliverable

ADAS&ME (688900) ADASANDME_Deliverable_7.1.doc

Month Year (e.g. November 2016) of 147 Version 0.X

Version History

Date Version Comments

06/07/17 First version made by Marta Pereira Cocron

28/7/2017 Changes introduced by Marta Pereira Cocron

29/9/2017 Changes introduced by Beatriz Delgado and Alex

Vallejo

3/11/2017 Changes introduced by Alicia Lotz, Marcel

Mathissen and Marc Wilbrink

01/2018 New details added by Marta Pereira Cocron, Alex

Vallejo and Beatriz Delgado

01/2018

Text added by Marc Wilbrink (UC C & D), Anna Anund (UC G) and Stas Krupenia (UC A), Ioannis Symeonidis (UC E&F), Stella Nikolaou (UC E&F) and Luca Zanovello (UC E&F), Elenora Meta (CTL)

02/2018

Corrections made by Marc Wilbrink, Anna Anund,

Luca Zanovello, Alicia Lotz, Marcel Mathissen,

Alex Vallejo and Beatriz Delgado, Marco Manuzzi

Authors (full list)

Ioannis Symeonidis, CERTH

Stella Nikolaou, CERTH

Marco Manuzzi, DIANESE

Elenora Meta, CTL

Alicia Lotz, OVGU

Marcel Mathissen, Ford

Project Coordinator

Dr. Anna Anund

Research Director / Associate Professor

VTI - Olaus Magnus väg 35 / S-581 95 Linköping / Sweden

Tel: +46-13-20 40 00 / Direct: +46-13-204327 / Mobile: +46-709 218287

E-mail: [email protected]

Legal Disclaimer

The information in this document is provided “as is”, and no guarantee or warranty is given that the information

is fit for any particular purpose. The above referenced authors shall have no liability for damages of any kind

including without limitation direct, special, indirect, or consequential damages that may result from the use of

these materials subject to any liability which is mandatory due to applicable law.

The present document is a draft. The sole responsibility for the content of this publication lies with the authors. It

does not necessarily reflect the opinion of the European Union. Neither the INEA nor the European Commission

is responsible for any use that may be made of the information contained therein.

© 2016 by ADAS&ME Consortium

mailto:[email protected]

ADAS&ME (688900) Del.No 7.1

February 2018 of 147 Version 1

Table of Contents

TABLE OF CONTENTS ...................................................................................................................................... 3

INDEX OF FIGURES .......................................................................................................................................... 6

INDEX OF TABLES ............................................................................................................................................ 6

GLOSSARY ........................................................................................................................................................... 8

EXECUTIVE SUMMARY ................................................................................................................................... 9

1. INTRODUCTION ......................................................................................................................................10

2. EVALUATION AIMS ................................................................................................................................10

3. RELATION AND INTERCONNECTIONS WITH OTHER WPS .......................................................11

4. GENERAL PLANNING ............................................................................................................................11

5. EVALUATION PREPARATIONS ...........................................................................................................14

6. EVALUATION SITES ...............................................................................................................................16

6.1 IDIADA’S PROVING GROUND ...................................................................................................................16 6.2 VTI’S DRIVING SIMULATOR .....................................................................................................................19

7. USE CASE A – ATTENTIVE LONG-HAUL TRUCKING ...................................................................20

7.1 FUNCTION SPECIFICATION ........................................................................................................................20 7.2 ASSESSED DRIVER STATES ........................................................................................................................22 7.3 EVALUATION OBJECTIVES ........................................................................................................................22 7.4 EVALUATION SITE ....................................................................................................................................22 7.5 TIME PLAN ................................................................................................................................................22 7.6 DESIGN AND CONDITIONS .........................................................................................................................23 7.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ....................................................................................24 7.8 PROCEDURE ..............................................................................................................................................24 7.9 TASKS TO INDUCE STATES ........................................................................................................................27 7.10 MEASUREMENTS ..................................................................................................................................28

8. USE CASE B – ELECTRIC VEHICLE RANGE ANXIETY ................................................................31

8.1 FUNCTION SPECIFICATION ........................................................................................................................31 8.2 ASSESSED DRIVER STATES ........................................................................................................................33 8.3 EVALUATION OBJECTIVES ........................................................................................................................33 8.4 EVALUATION SITE ....................................................................................................................................33 8.5 TIME PLAN ................................................................................................................................................34 8.6 DESIGN AND CONDITIONS .........................................................................................................................35 8.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ....................................................................................35 8.8 PROCEDURE ..............................................................................................................................................35 8.9 TASKS TO INDUCE STATES ........................................................................................................................37 8.10 MEASUREMENTS ..................................................................................................................................37

9. USE CASE C – DRIVER STATE-BASED SMOOTH & SAFE AUTOMATION TRANSITIONS ...40

9.1 FUNCTION SPECIFICATION ........................................................................................................................40 9.2 ASSESSED DRIVER STATES ........................................................................................................................42 9.3 EVALUATION OBJECTIVES ........................................................................................................................42 9.4 EVALUATION SITE ....................................................................................................................................42 9.5 TIME PLAN ................................................................................................................................................42 9.6 DESIGN AND CONDITIONS .........................................................................................................................43 9.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ....................................................................................44 9.8 PROCEDURE ..............................................................................................................................................44 9.9 TASKS TO INDUCE STATES ........................................................................................................................46



9.10 MEASUREMENTS ..................................................................................................................................47 10. USE CASE D – NON-REACTING DRIVER EMERGENCY MANOEUVRE ....................................50

10.1 FUNCTION SPECIFICATION ....................................................................................................................50 10.2 ASSESSED DRIVER STATES ...................................................................................................................50 10.3 EVALUATION OBJECTIVES ...................................................................................................................50 10.4 EVALUATION SITE ................................................................................................................................50 10.5 TIME PLAN ...........................................................................................................................................50 10.6 DESIGN AND CONDITIONS ....................................................................................................................52 10.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ...............................................................................52 10.8 PROCEDURE .........................................................................................................................................52 10.9 TASKS TO INDUCE STATES....................................................................................................................53 10.10 MEASUREMENTS ..................................................................................................................................53

11. USE CASE E – LONG RANGE ATTENTIVE TOURING WITH MOTORBIKE .............................56

11.1 FUNCTION SPECIFICATION ....................................................................................................................56 11.2 ASSESSED DRIVER STATES ...................................................................................................................59 11.3 EVALUATION OBJECTIVES ....................................................................................................................59 11.4 EVALUATION SITE ................................................................................................................................59 11.5 TIME PLAN ...........................................................................................................................................59 11.6 DESIGN AND CONDITIONS ....................................................................................................................60 11.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ...............................................................................60 11.8 PROCEDURE .........................................................................................................................................60 11.9 TASKS TO INDUCE STATES....................................................................................................................62 11.10 MEASUREMENTS ..................................................................................................................................62

12. USE CASE F – RIDER FAINT .................................................................................................................66

12.1 FUNCTION SPECIFICATION ....................................................................................................................66 12.2 ASSESSED DRIVER STATES ...................................................................................................................66 12.3 EVALUATION OBJECTIVES ...................................................................................................................66 12.4 EVALUATION SITE ................................................................................................................................66 12.5 TIME PLAN ...........................................................................................................................................68 12.6 DESIGN AND CONDITIONS ....................................................................................................................68 12.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ...............................................................................68 12.8 PROCEDURE .........................................................................................................................................68 12.9 TASKS TO INDUCE STATES....................................................................................................................68 12.10 MEASUREMENTS ..................................................................................................................................68

13. USE CASE G – PASSENGER PICK UP/DROP OFF AUTOMATION FOR BUSES ........................69

13.1 ASSESSED DRIVER STATES ...................................................................................................................71 13.2 EVALUATION OBJECTIVES ....................................................................................................................71 13.3 EVALUATION SITE ................................................................................................................................71 13.4 TIME PLAN ...........................................................................................................................................71 13.5 DESIGN AND CONDITIONS ....................................................................................................................72 13.6 SAMPLE SELECTION CRITERIA AND RECRUITMENT ...............................................................................72 13.7 PROCEDURE .........................................................................................................................................73 13.8 TASKS TO INDUCE STATES....................................................................................................................74 13.9 MEASUREMENTS ..................................................................................................................................74

14. METHODOLOGICAL CONSIDERATIONS AND DATA ANALYSIS ..............................................77

15. ETHICAL PROCESSES ...........................................................................................................................79

15.1 ETHICAL PROCESSES IN ALL STAGES OF METHODOLOGY: HUMAN PARTICIPANTS ........................80 16. REFERENCE DOCUMENTS ...................................................................................................................81

ANNEXES ............................................................................................................................................................83

ANNEX 1 . ORDER OF SESSIONS ..........................................................................................................................84 ANNEX 2. RECRUITMENT QUESTIONNAIRE .........................................................................................................87



ANNEX 3. DEMOGRAPHICS QUESTIONNAIRE ......................................................................................................91 ANNEX 4. GENERAL QUESTIONNAIRE ON AUTOMATED DRIVING/RIDING ............................................................95 ANNEX 5. ACCEPTANCE SCALE ........................................................................................................................101 ANNEX 6. TRUST SCALE ...................................................................................................................................103 ANNEX 7. SYSTEM USABILITY SCALE (SUS) ...................................................................................................106 ANNEX 8. KAROLINSKA SLEEPINESS SCALE ....................................................................................................108 ANNEX 9. STRESS SCALE .................................................................................................................................110 ANNEX 10. QUESTIONNAIRE ON POTENTIAL SYSTEM USAGE AND ACQUISITION ...............................................112 ANNEX 11. DEBRIEFING INTERVIEW.................................................................................................................116 ANNEX 12. TASKS TO INDUCE STATES ..............................................................................................................121 ANNEX 13. IDIADA’S PROVING GROUND .......................................................................................................124 ANNEX 14. TEST CASE TEMPLATE ...................................................................................................................131



Index of Figures

FIGURE 1. GENERIC MAP OF IDIADA’S PROVING GROUND. ....................................................................................16 FIGURE 2. GENERAL ROAD OF IDIADA’S PROVING GROUND. .................................................................................17 FIGURE 3. HIGH-SPEED TRACK AT IDIADA’S PROVING GROUND. ...........................................................................18 FIGURE 4. VTI’S DRIVING SIMULATOR II TO BE USED IN USE CASE G. .....................................................................19 FIGURE 5. EVENT CHART FOR DRIVER STATE RECOGNITION AND MITIGATION IN USE CASE A .................................21 FIGURE 6. IDIADA’S HIGH SPEED TRACK FOR USE CASE A EVALUATION AND DETAILED ROAD WORKS LAYOUT.

ROAD WORKS WILL BE PLACED WHERE THE RED STAR IS. ..............................................................................26 FIGURE 7. EVENT CHART FOR RANGE ANXIETY AND RANGE INCIDENT MITIGATION WITH COMPLIANCE IN USE CASE

B ...................................................................................................................................................................32 FIGURE 8. ROUTE FOR THE USE CASE B EVALUATION (GOOGLE MAPS, 2018) .........................................................34 FIGURE 9. IDIADA’S HIGH-SPEED TRACK FOR USE CASE B AND RESPECTIVE SECTIONS. ........................................37 FIGURE 10. USE CASE B TEST PROCEEDING ............................................................................................................37 FIGURE 11. EVENT CHART FOR DRIVER STATE RECOGNITION AND MITIGATION IN USE CASE C. ..............................41 FIGURE 12. IDIADA’S HIGH-SPEED TRACK FOR USE CASE C EVALUATION AND DETAIL FOR ROAD WORKS LAYOUT.

......................................................................................................................................................................45 FIGURE 13. EVENT CHART FOR NON-REACTING DRIVER EMERGENCY MANOEUVRE IN USE CASE D. ........................51 FIGURE 14. IDIADA’S HIGH SPEED TRACK FOR USE CASE D AND RESPECTIVE SECTIONS.......................................53 FIGURE 15. MOTORCYCLE USED FOR THE UC E EVALUATIONS ...............................................................................56 FIGURE 16. PPE USED FOR UC E/F EVALUATIONS .................................................................................................57 FIGURE 17. EVENT CHART FOR DRIVER STATE RECOGNITION AND MITIGATION IN USE CASE E. ..............................58 FIGURE 18. HIGH-SPEED TRACK FOR THE EVALUATION OF UC E ............................................................................62 FIGURE 19. EVENT CHART FOR DRIVER STATE RECOGNITION AND MITIGATION IN USE CASE F. ..............................67 FIGURE 20. EVENT CHART FOR PASSENGER PICK UP/DROP OFF AUTOMATED SYSTEM. ............................................70 FIGURE 21. THE CITY BUS SCENARIO OF USE CASE G. .............................................................................................73 FIGURE 22. SCHEME OF USE CASE G EXPERIMENT DESIGN ......................................................................................74 FIGURE 23. PLAN B AND C FOR THE APPLICATION OF THE METHODOLOGY .............................................................78

Index of Tables

TABLE 1. EVALUATIONS’ TIME PLAN OVERVIEW.....................................................................................................12 TABLE 2. OVERVIEW OF COMMON METHODOLOGICAL ASPECTS .............................................................................13 TABLE 3. CHARACTERISTICS OF IDIADA’S GENERAL ROAD. .................................................................................17 TABLE 4. CHARACTERISTICS OF IDIADA’S HIGH SPEED TRACK. ............................................................................18 TABLE 5. FOUR WEEKS ACTIVITY DESCRIPTION FOR USE CASE A ............................................................................23 TABLE 6 . USE CASE A SESSIONS .............................................................................................................................23 TABLE 7. USE CASE A METRICS ..............................................................................................................................28 TABLE 8. FOUR WEEKS ACTIVITY DESCRIPTION FOR USE CASE B ............................................................................34 TABLE 9. USE CASE B SESSION PROCEDURE ............................................................................................................35 TABLE 10. USE CASE B METRICS .............................................................................................................................38 TABLE 11. FOUR WEEKS ACTIVITY DESCRIPTION FOR USE CASE C ..........................................................................43 TABLE 12. USE CASE C AND D SESSIONS ................................................................................................................43 TABLE 13. USE CASE C METRICS .............................................................................................................................47 TABLE 14. USE CASE D METRICS ............................................................................................................................54 TABLE 15. FOUR WEEKS ACTIVITY DESCRIPTION FOR USE CASE E ..........................................................................60 TABLE 16. USE CASE E PROCEDURE ........................................................................................................................61 TABLE 17. USE CASE E METRICS .............................................................................................................................63 TABLE 18. FOUR WEEK ACTIVITY DESCRIPTION FOR USE CASE G ............................................................................71 TABLE 19. USE CASE G SESSION PROCEDURE ..........................................................................................................72 TABLE 20. USE CASE G METRICS ............................................................................................................................75



Glossary

ADAS ADVANCED DRIVER ASSISTANCE SYSTEMS

DSM DECISION SUPPORT MODULE

HMI HUMAN MACHINE INTERFACE

IVIS IN-VEHICLE INFORMATION SYSTEMS

LCD LIQUID-CRYSTAL DISPLAY

PPE PERSONAL PROTECTIVE EQUIPMENT

RF RADIO FREQUENCY

UC USE CASE

UI USER INTERFACE

V2X ITS-G5 EUROPEAN STANDARD VEHICULAR COMMUNICATIONS PROTOCOL

WP WORK PACKAGE



Executive summary

This document presents the first version of the plan that will be used to conduct the evaluation

in ADAS&ME. It describes the main aspects that will be taken into consideration for testing

the systems being developed in the project.

The general aim of this evaluation plan is to prepare tests that enable verifying if the systems

are working in accordance to what was expected, i.e. if systems can detect when the driver/rider

is not capacitated to drive/ride and are capable to provide adequate support by sending

unambiguous mitigation strategies and/or suggest automation. Furthermore, it is also intended

to collect the opinion of real drivers/riders about the developed functions.

At the moment of writing of this deliverable, the systems are still being developed. Thus, these

evaluation plans were defined based on information provided by WP1 (the refinement of the

use cases), and by the use case teams (update on system developments). It is foreseen that the

content presented in this report will be further specified in a near future (deadline of the

Deliverable’s updated version due in M24).

This document is structured in the following manner: the first sections describe the aims of the

evaluations, highlight the aspects that need to be considered for the preparation of these tests,

and give an overview of the time schedule. A detailed description of the evaluation follows,

and each use case has a dedicated chapter. At the end, considerations are made taking into

account the global methodology applied.



1. Introduction

Evaluations are crucial when developing in-vehicle systems. In order for these evaluations to

be as complete as possible, not only the technical aspects of the system should be verified. The

interaction between the system and the drivers/riders, together with their opinion, should also

be taken into account. Only when considering both technical and human factors aspects it is

possible to have a complete characterisation of the system’s impact on road safety and how

systems should be further improved.

Despite the name of the deliverable, the document does not provide an exhaustive theoretical

evaluation framework. The authors considered that a more practical and detailed document

containing each use case evaluation plan would better serve the project. Some decisions

concerning the systems developments are also dependent on where and how systems will be

evaluated. Thus, detailed plans are presented in this document.

At the moment of writing of this report, the system algorithms are being developed, which

means that it is not possible to foresee with great detail all systems’ functionalities,

characteristics and limitations. Consequently, the evaluation plans were defined based on

information provided by WP1 (the refinement of the use cases), and by the use case teams

concerning the updates on system developments. In order to better specify what type of

functions can be evaluated at the end of the project and how, a test case template was filled in

by each use case leader (annex 14). The collected information was then transferred and adapted

to the content of this deliverable.

In spite of the close communications between WP7 and the use case team leaders, the evaluation

team bore in mind the necessary independency to define the final tests as it is defined in the

description of work. This will lead to a higher quality in the application of the methodology and

in the data collection process. Apart from independency, homogeneity framed also the

evaluation plans. Each use case final test has a similar design and procedure, and also uses the

application of common tools. This is not only important to define a coherency among the

different tests being conducted in the project, but also allows the comparison of results between

system functions which will give a broader idea of what the general public thinks about driver’s

state detection and vehicle automation.

The content presented in this deliverable can and should be detailed in a near future (deadline

of the Deliverable’s updated version due in M24). Specifically, the system’s HMI

characteristics should be considered for further specification of the methodology. Other

adjustments are also expected for the latter version of this Deliverable as unforeseen constraints

from the development might slightly alter the function’s characteristics and limitations. The

results coming from the pilot testing done in WP4 will considerably contribute to the

improvement of these plans. Additionally, the updated version of D7.1 will also contain a

general plan about what will be done during the demonstration day, planned to happen in M42.

2. Evaluation aims

To assure that the developed systems are able to accurately detect when the driver/rider is not

capacitated to drive, and consequently mitigate these states and avoid dangerous situations

through the use of adapted HMI and automation modes, the ADAS&ME evaluations aim at:



• Verifying the effectiveness of the systems to recognize the driver’s state

• Attesting the capacity of the HMI to display clear and unambiguous information

• Evaluating the driver behaviour following a system warning/suggestion

• Collecting the driver’s opinion on the system’s usability

• Knowing the driver’s trust and acceptance levels regarding the ADAS&ME functions

3. Relation and interconnections with other WPs

The evaluation plan is developed in WP7. This work package gathers all the evaluation

activities, which follow the systems development done in previous WPs. Activity 7.1 is in

charge of such a task. Exchanges with activities of other WPs happen with the aim of knowing

more about the different system implementations, which will constitute the ADAS&ME final

system. The plan is created based on these conversations in order to ensure an adequate and

efficient evaluation, in line with the project objectives.

A work package with major interaction with WP7 is WP4, which develops the driver/rider state

monitoring algorithms. Such algorithms will be evaluated at the final phase of the project. Thus,

WP7 strongly needs to follow up all the related activities in order to design the best evaluation

methodology to determine the performance, and later impact, of the different detection

strategies. Moreover, activity 7.2 is supporting different data collections that will be a key input

to WP4. Information from preliminary sensor setups and algorithms is being gathered

(including professional and non-professional drivers in the loop) and will be used to better

understand the different monitoring situations and thus to improve the detection strategies.

At the same time, WP7 is indirectly related to WP3 “Environmental sensing” as the outputs of

this work package will have an influence on the decisions and results of WP4.

Another work package with a very relevant linkage is WP5, where the whole HMI strategy is

defined. All the different interactions with the driver/rider in each use case will also be

evaluated. Thus, it is very important to stay up to date with all the related activities in order to

design the correct evaluation methodology.

WP6 will ensure that all the systems have been properly integrated in the vehicles and are ready

for the evaluations. The OK of this WP is thus necessary before starting A7.3.

WP8 will use the evaluation results to calculate and estimate the impact of ADAS&ME on

safety. Communication with this WP allows the completion of the questionnaire on potential

system usage and purchase, which will deliver the content for posterior analysis.

Finally, it is worth saying that the evaluation is defined around the different use cases, which

were designed and further defined in WP1. The complete understanding of all UCs and their

singularities allows the definition of proper evaluation scenarios to get the most relevant results.

4. General planning

Table 1 presents the ADAS&ME evaluation plan. Evaluations will be conducted from M36 to

M39 in two distinct places, dependent on the use case. Use cases A, B, C, D, E and F will be

evaluated on IDIADA’s test tracks. Use case G will be tested in a simulator. Four weeks are



foreseen for the evaluation of the automated functions belonging to each use case. These four

weeks include, in general terms, the installation of the data acquisition system inside the

vehicles, the conduction of the evaluation with real users and also the disassemble of all

equipment. A precise description of the four weeks plan is presented ahead in each section.

All developed functions will be evaluated with the help of real drivers/riders. The selection and

recruitment of participants will be mainly conducted before each four weeks plan starts.

Table 1. Evaluations’ time plan overview.

UC M36 M37 M38 M39

W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4

A

B

C&D

E&F

G

As already mentioned in the introduction, while defining the evaluations, homogeneity was an

important aspect. The general ADAS&ME evaluation plan should be more than a compilation

of distinct use case evaluation plans. A common framework should guide the work to allow

comparisons among results. Due to the nature of the use cases and the distinct driver states in

focus, not all aspects of the methodology were possible to reproduce among plans. Thus, the

evaluation plan proposes: 1) a sample where drivers/riders are between 25 and 55 years-old and

have a minimum of 5 years of driving/riding experience (all UCs); 2) the utilisation of the same

tasks to induce the same driver states (UC A and C); 3) the application of the same pre-driving

and post-driving measurements (all UCs).



Table 2. Overview of common methodological aspects

Use case

A

Use case

B

Use case

C

Use

case

D

Use case

E

Use case

F

Use case

G

Site IDIADA’s high-speed track Simulator

Driver states Fatigue

Stress

Distraction

Emotion

Stress

(anxiety)

Fatigue

Stress

Distraction

Emotion

- Physical

fatigue

Distraction

Physical

fatigue

Fatigue

Stress

Distraction

Sample’s age

and

experience

Truck drivers

between 25

and 40 years

old

drivers between 25 and 40 years-

old

riders between 30 and 55

years old

Bus drivers

between 35

and 50

years-old

> 5 years of driving/riding experience with such a vehicle

Tasks to

induce states

Fatigue: one

night awake

Stress: n-back

task

Emotion:

badly working

speech system

Distraction:

SuRT

Fatigue: one

night awake

Stress: n-back

task

Emotion:

badly working

speech system

Distraction:

SuRT

- Physical

fatigue:

Long ride

Distraction:

to be

decided

Physical

fatigue:

Long ride

Fatigue:

night

awake

Inattention:

ticketing

task

Measurements Pre-driving

Demographic Questionnaire

General questionnaire on automated driving

Post-driving

Acceptance scale

Trust scale

System Usability Scale

Questionnaire on potential system usage and acquisition



5. Evaluation preparations

Taking into account the present document and the framework in which evaluations were

initially planned, the teams located at the evaluation sites will refine and operationalise the

procedures leading to the final tests.

In order for evaluations to be conducted smoothly and without delays, preparations will go

beyond what is described in this deliverable, not only during activity 7.1 but also 7.3. Apart

from the necessary technical equipment, the following aspects will be considered while

preparing the evaluations:

a) Sample selection and recruitment

To guarantee homogeneity among the samples recruited for the ADAS&ME final tests,

determined selection criteria will be considered such as gender, age, driver/rider experience,

ADAS/IVIS experience and attitude towards technology. The sample selection and recruitment

will be done before the evaluation takes place and will be conducted by the respective

evaluation team on site. The evaluations will be advertised in the media (e.g. website, local

newspapers, email messages), locally (distribution of prospects and information sheets in the

facilities), and via direct contact of potential participants. The recruitment questionnaire (Annex

2) will be available on the internet, paper, or applied during a phone conversation. This

questionnaire will be translated to the participant’s mother tongue. Surrogate participants will

also be recruited in case of drop-out. Appointments will be scheduled with the participants (one

or more) and, to assure that drivers/riders do not forget an appointment, a member of the

evaluation team will call the driver/rider a day before reminding him/her about the scheduled

session’s time.

b) Information sheets, consent forms and questionnaires

All the material that need to be read or filled out by participants will be translated to his/her

mother tongue. Information sheets and consent forms will be printed out. Questionnaires and

scales will be digitally prepared. Participants will complete the questionnaires on a Pad, which

will ease storing information and reduce the amount of work prior to data analysis.

c) Technical protocol and sample schedule

The technical protocol consists of a check-list to ensure that all equipment is in place and

working. This will be done for each UC. It helps reviewing that all vehicle systems are working

as intended, that the vehicles have enough fuel/energy and that all other potential interruptions

have been removed. Before starting the session, two members of the staff should go thoroughly

through this protocol. A schedule of the sample should be attached to the protocol. This

information sheet contains a list of all participants with a time plan for each session and, in case

participants have to visit the evaluation facilities more than once, the order of the sessions. As

different sessions might have a slightly different experiment protocol, the technical protocol

might as well differ.



d) Test guide and instructions

This is a detailed protocol showing, step by step, which actions the experimenter should take to

run the study, which materials are needed, where he/she should take the participant. It includes

everything that must be said to the participant. Certain information, like evaluation goals,

evaluation procedure and system description, must be read (verbatim) in order to assure that all

participants receive exactly the same instructions.

e) Pre-test and dataset verification

A pre-test must be prepared and conducted at least two days prior to the visit of the first

participant. The pre-test subject can be a member of the working group that is not directly

involved in the preparation of the evaluations. This will assure a higher independency of the

feedback given regarding failures and improvements. The pre-test should be conducted exactly

as if it was a session with a real participant (information sheets, technical protocol, test guide

and instructions should be used). This serves to verify if all equipment is working properly and

if the procedure is efficient. During the pre-test data must be recorded as this allows to confirm

if the output dataset can be used to perform the planned analysis. In case evaluations are

composed of several sessions with distinct characteristics, it is suggested to perform a pre-test

per session, or at least go through the moments that are distinct (e.g. a drive with concurrent

performance of a distinct additional task).



6. Evaluation sites

Two sites will be used to perform the evaluations of the functions being developed in

ADAS&ME. The following sections provide further details on each of them.

6.1 IDIADA’s proving ground

Inaugurated in 1994, Applus IDIADA proving ground1 is the most comprehensive independent

proving ground in Europe. Excellent climatic conditions allow for year-round testing. The

allocation of a dedicated customer service representative for each testing team ensures that the

testing program runs smoothly and that objectives are achieved on time and within budget. The

proving ground is designed to easily monitor all movements. The tracks of the Applus IDIADA

proving ground are equipped with a total of 10 Road-Side Unites providing V2X ITS-G5 full

coverage over the whole area. Figure 1 Figure 1 presents a generic map of the facilities. Figure

2 and Table 3 detail the general road that allows access to the different test areas.

Figure 1. Generic map of IDIADA’s proving ground.

1 http://www.applusidiada.com/en/activity/Proving_ground-1328274726564

Caption:

0-General Road

1-High Speed Track

2- Noise Track

3-Fatigue Track/Comfort Track

4-Dynamic Platform A

5-Dry Handling Circuit/ Dynamic Platform

C

7-Straight Line Braking

Surfaces/

Comfort Track and SIM city

8-Dynamic Platform B

9-Off road/Forest Track

10-Wet Circle



Figure 2. General road of IDIADA’s proving ground.

Table 3. Characteristics of IDIADA’s general road.

Direction of travel Clockwise

Total length 5.333 m

Length of south straight 1.620 m

Longitudinal gradient (south straight and braking area) 0 %

Braking area (length) 300 m

Braking area (width) 20 m

The High Speed Track will be the most used for the ADAS&ME evaluations (use cases A, B,

C, D, E and F) This track consists of an oval track with 4 lanes, 2 km straight and 200 km/h

neutral speed (Figure 3). Table 4 gives more details on the test-track characteristics.

Use case E will also make use of other tracks (e.g. fatigue track). These and other facility details

can be seen in a more comprehensive document located in Annex 13.



Figure 3. High-speed track at IDIADA’s proving ground.

Table 4. Characteristics of IDIADA’s high speed track.


Length lane 1 7.493 m




Length of straights 2.000 m

Neutral steer speed 200 Km/h

Maximum banking bend 80% (38.66º)

Radius of the bends 471 m

Longitudinal gradient (straights) 0.3%

Transverse gradient (straights) 1.0%



6.2 VTI’s Driving Simulator

VTI’s advanced driving simulator Sim II2 will be used to perform the evaluation of use case G.

This simulator has a visual system consisting of six SXRD projectors that give a 120-degrees

forward field of vision. Each projector has a resolution of 1 920 x 1 080 pixels, which gives the

simulator very high visual acuity. The images are warped and blended together with help of a

software solution for automatic calibration. Rear-view mirrors are simulated with two LCDs.

The simulator has a vibration table and a motion system that can provide both linear and tilt

motion. The vibration table allows road unevenness to be simulated at higher frequencies and

the tilt motion is used among other things to simulate the long accelerations that occur when

driving through bends and during longitudinal acceleration or braking. Since the simulator’s tilt

motion affects both the compartment/cab and the projection screen while the vibration table

only affects the compartment/cab in relation to the screen, it is possible to create a realistic road

feel, for example in experiments with rumble strips.

Figure 4 shows the simulator environment.

Figure 4. VTI’s driving simulator II to be used in use case G.

2 https://www.vti.se/en/research-areas/vtis-driving-simulators/



7. Use case A – Attentive long-haul trucking

7.1 Function specification

Use Case A has been created to support the development of the sleep and rest algorithms to

increased European competitiveness, enhance sensory and data analysis robustness, enhance

HMIs and advise strategies via unobtrusive measures, and improve efficiency of transport

networks by the increased uptake of automated driving features. Although all these expected

impacts are important, not all can be evaluated via a final driving test. The purpose of the use

case A final evaluation is to determine to what extent the expected impacts can be achieved by

the ADAS&ME system.

The function developed by this use case is able to detect the driver’s state, verifying if he/she

is capacitated to drive. The driver state recognition can be performed both while driving

manually and in automation mode. When driving manually, and in case the driver’s state

compromises safety or efficiency, the driver receives information that increasingly encourages

him/her to handover driving to the automation. This possibility to make a transition to

automation is always available, even if the driver is capacitated to drive.

When driving automated, the system continues monitoring the driver’s state. In case a takeover

is needed (from automated to manual driving) and the driver is not capacitated to drive, the

system is able to send messages with the aim of mitigating the state. For the event that these

state mitigation strategies do not work, the system can also perform a safe stop of the vehicle.

Figure 5 shows the events for an example of driver state recognition and mitigation before a

transition occurs.

The automated driving functionality is limited to driving on high quality roads with clear lane

markings and in clear and dry weather conditions only. No other vehicles are needed for this

testing, though simulated roadworks will be required.

For the purpose of ADAS&ME evaluations, this function will be implemented in a SCANIA

Truck.



Figure 5. Event chart for driver state recognition and mitigation in use case A

Driver

User Interface

Personalization

Driver monitoring

Environmental monitoring

Decision system

Vehicle

State X

State X detection

Warning to be sent

Warns the driver

Chose UI modality

Manual driving Automated driving

Hands over State X

State X detection

Warning to be sent

Warning + state mitigation

Chose UI modality

Capacitated driver

Road works detection

Mitigated state detection

Warning to be sent

Take over allowance

Chose UI modality

Takes over



7.2 Assessed driver states

Five driver states are considered for this use case. Fatigue, stress, emotion (anger /frustration)

and distraction will be induced states during the evaluation (considered as independent

variables). Furthermore, the state rest (i.e. periods of rest) will be created by the usage of

automation and will be measured as a dependent variable.

7.3 Evaluation Objectives

An assumption exists that automated driving, under at least normal driving conditions, will be

safer and more efficient than human controlled driving. The objective of this use case is thus to

encourage drivers to use automated driving when such a possibility exists. When automated

driving is possible, and the driver is in a fit state to drive, he/she will be encouraged to handover

driving control to the automation. When the driver state changes to the extent that either safety

or efficiency could be compromised (due to distraction, stress, or fatigue), the driver will

receive information that increasingly encourages him/her to handover driving to the automation.

The goal of the use case is to encourage drivers to handover control to the automated driving,

and to be even more likely to do so when their driver state has the potential to compromise

safety or efficiency. Of interest is also whether the HMIs developed within ADAS&ME

encourage greater use of automated driving compared to driving without the ADAS&ME

HMIs.

An outcome of ADAS&ME is that when the ADAS&ME system detects a driver state that has

a potential negative effect on driving performance or safety (and automated driving is possible),

then warnings and information given to the driver should encourage handover to the

automation.

Thus, this test case will assess the following:

1. To what extent will drivers spontaneously handover control to the vehicle automation

when such automated driving becomes available on that section of road.

2. To what extent will drivers hand over the control to the vehicle automation when

prompted to do so based on a driver state warning.

3. To what extent will drivers accept and trust the ADAS&ME HMI as a potential

product.

4. To what extent do drivers understand the handover and takeover processes and how

many errors (if any) occur during these processes.

7.4 Evaluation site

This evaluation will be conducted on IDIADAS’s proving ground, more specifically on the

high-speed track.

7.5 Time plan

This evaluation will be carried out during M39. The four weeks dedicated to this test will be



structured in the following way (Table 5):

Table 5. Four weeks activity description for use case A

Day Activity description

1 to 5 Technical preparations and finalisation of participant recruitment

6 Pre-tests

7 Final technical and methodological adjustments

8 to 19 Evaluation with real drivers

20 Equipment disassemble

During the 12 days dedicated to performing data acquisition (“Evaluation with real drivers” in

the table above), real drivers will be invited to test the ADAS&ME system. The length of the

entire evaluation procedure (Welcome/Briefing and Questionnaires; Familiarisation; Driving

Test; Debriefing) differs among the sessions.

7.6 Design and conditions

The evaluations will have a within-subjects design. The independent variable Driver state will

have five conditions: fatigue, stress, emotion, distraction and capacitated to drive (baseline)

each of which is experienced by all participants. In order to avoid bias due to learning effect,

the order driver state conditions are induced to participants will be balanced across the sample.

Annex 1 presents this schema.

Due to the nature of the evaluations, drivers will not be induced to all states in one unique

session, which means that participants will be invited to visit IDIADA’s test facilities three

times, as it can be seen in Table 6.

Table 6 . Use case A sessions

Session A – evaluation under the

influence of fatigue (120 to 140

min.)

Note: These evaluations will

have to be scheduled for the same

period of the day (e.g. beginning

of the afternoon) due to the

influence of the circadian

rhythm.

• Welcome + Briefing – 5 to 20 min3

• Familiarisation trial (drive) – 15 min

• Pause (to intensify state) – 10 min

• Induced state trial – 60 min

• Debriefing – 30 min

3 Duration changes if session is the first one or not. Participants’ first session will be longer, as

more questionnaires will have to be filled in – demographics and general about automated

driving. For the following sessions these will not be needed and participants can start with the

familiarization trials after a short welcoming.



Session B – evaluation under the

influence of distraction +

emotion (125 to 140 min.)

• Welcome + Briefing – 5 to 20 min

• Familiarization trial (drive + distraction task) – 20

min

• Distraction trial – 20 min


• Familiarization (emotion task) – 5 minutes

• Emotion trial – 20 min


Session C – evaluation for

baseline and under the influence

of stress (115 to 130 min.)


• Familiarization trial (drive) – 15 min

• Baseline trial – 30 min

• Debriefing for baseline – 20 min

• Familiarization (stress task) – 5 minutes

• Stress trial – 20 min


7.7 Sample selection criteria and recruitment

A total of 12 participants will integrate the sample. Due to the event of drop-out, more

participants will be recruited (estimated 16) in order to assure that 12 data sets for each driver

state are available for analysis. Surrogate participants recruited to compensate a drop-out should

perform the conditions in the same order the dropped-out participant did. This will assure a

balanced distribution of the conditions over the sample. A detailed plan for which and when

participants visit IDIADA’s facilities will be set closer to month 39 and in accordance with the

participants availability.

The sample will be composed of truck drivers between 25 and 40 years old that have more than

five years of driving experience. Drivers must have interacted already with an advanced driver

assistance system (ADAS) or an in-vehicle information system (IVIS).

7.8 Procedure

Upon arrival, participants will be welcomed and informed about the main aim of the evaluation.

A consent form, containing details on the privacy of the data collected and how it will be used,

will be presented. Then, participants will be asked to fill in a demographic questionnaire (annex

3) and a general questionnaire on automated driving (Annex 4).

A training trial will follow, in which participants get acquainted with the vehicle in manual and

automated mode. During this phase, the driver is accompanied by a member of the staff and

drives the complete high-speed track circuit at a recommended speed (between 50 and 80 km/h).

Transitions are trained, and the driver gets to know the interface that displays information about

the system status. During the training trial, the driver has the opportunity to ask questions to the

staff member about the interface, the system, and the track. While driving, the driver states will



not be induced (e.g. distraction, stress, emotion) nor detected, and the road will be free from

road works.

During the driving test, conditions (independent variables) will be implemented. This means

that driver states will be induced and both behavioural and driving parameters will be measured.

Before start driving, the type of task the drivers have to perform along the test (if any) will be

explained. These tasks will help inducing the state that will lately be detected by the system.

Drivers will not be informed about the purpose of the task. Specific instructions on how and

when this information should be conveyed to the drivers will be in written form. This will be

prepared prior to the evaluations and passed to the drivers always in the same manner. A system

of incentives (the more tasks the higher the monetary compensation) will motivate drivers to

perform the tasks while driving.

During the driving test a test engineer will be inside the vehicle in case something unexpected

happens and the driver requires help. Drivers will be informed about this and also that they

should not communicate with the test engineer. A driving speed between 50 and 80 km/h will

have to be maintained during the driving test (in both manual and automated conditions). For

three of the five conditions (distraction, emotion and stress), a secondary task will be performed

as state induction task. The baseline and the fatigue conditions will not require the performance

of an additional task. Below is a detailed description of what participants will do during the

driving test in each condition.

Baseline condition:

The baseline drive will take place with the same automated truck, however, the ADAS&ME

system will not be used. Instead, participants will receive a simple baseline HMI (without an

accompanying driver monitoring system) that includes an Instrument Cluster symbol plus

auditory alert to indicate the availability of automated driving.

The following are the key events. Figure 6 presents a schema of what will happen on the test

track.

1. On the high-speed test track, the participant drives manually from the starting point.

2. The first curve section is done manually and when entering the south straight section

the participant receives notification that automated driving is possible.

3. The participant can then drive manually or handover control to the vehicle (no action

will be imposed).

4. If participants retain manual control, they will receive a standard “automated driving

possible” at regular time intervals.

5. In case participants handover control to the vehicle, and after a period of driving, they

will be informed that automated driving should be deactivated (due to road works) and

that they must retake control.

6. Participants takeover control of the vehicle before entering the road works zone.

Road works will be placed on the test track for the last lap by staff members. These will be

“movable” road works mounted while the driver is completing the lap (the driver will not see

the road works being mounted). After passing the road works, participants will drive to the

starting point manually. When arriving, the test drive will end.



Figure 6. IDIADA’s high speed track for use case A evaluation and detailed road works

layout. Road works will be placed where the red star is.

Fatigue condition:

Of interest in this condition is to what extent the ADAS&ME system detects fatigue and rest.

Participants that have worked overnight and arrive for testing early in the morning will be

recruited. There are two main areas of interest in this study. First, does the system accurately

detect that the driver is tired and encourage the driver to handover control to the automated

driving functionality? Second, can the system detect that the driver is resting and deliver

appropriate feedback to the driver acknowledging their resting state?

Before initiating the driving test (after the training trial) the drivers will have to wait inside the

vehicle for about 10 minutes (cover story: the researcher will have to calibrate a system and it

will take some minutes). During this waiting time the driver will be asked to be seated inside

the vehicle and not to communicate or perform any task. The environment inside the vehicle

will be quiet and comfortable. After this, the test will start. There will be no performance of

additional tasks while driving.

The following are the key events:

1. Participants drive manually from the starting point

2. Manual drive continues over a period of time necessary for the drivers to get fatigued

and allow the system to detect this state

3. The system detects the state and sends dedicated sleepiness prompt together with the

possibility to change to automated driving.

4. If the participant hands over control to the automation, s/he will drive a set number of

circuits using the automated driving functionality.

5. During the period of automated driving, the ADAS&ME system will be used to detect

if the participant is resting. If the participant does experience periods of rest, the

system will communicate this to him/her in a subtle way (so as to not disturb them

from their rest), and the participant will be informed that their tachograph (device for

digitally recording their legal driving state) will change from Driving mode to Resting

mode.

S

South straight section

1st

curve 2nd curve

North straight section



6. After this time, they will receive notification that automated driving will soon be no

longer available due to road works (in the meanwhile the road work obstacles will be

moved onto the test track).

7. The driver will be asked to takeover control before the road works.

8. After passing the road works, participants will drive to the starting point manually.

When arriving, the drive will end.

Stress, emotion and distraction conditions:

The procedure for all other driver states will not differ much from the previous, with the

exception that secondary tasks will be used to trigger the driver state of interest. The events for

these conditions will be the following:

1. Participants drive manually from the starting point.

2. When entering the south straight section a request will be sent to the driver to start a

secondary task.

9. The performance of the secondary task lasts until the required driver state is detected.

When this happens, participants will receive a dedicated driver state warning/prompt

and the stimuli for secondary task performance (visual/auditory) will stop.

3. In case participants hand over control to the automation, they will drive some laps in

automated mode.

4. After this time, they will receive notification that automated driving will soon be no

longer available due to road works (in the meanwhile the road work obstacles will be

moved onto the test track).

5. The driver will be required to takeover control before the road works.

6. After passing the road works, participants will drive to the starting point manually.

When arriving, the drive will end.

7.9 Tasks to induce states

In order to induce the desired states while driving, the following tasks will be used:

Fatigue: As described before, participants invited to perform the evaluation will be asked not

to sleep the night before, so their fatigue level is high. Before initiating the driving test (after

the training trial) the driver will have to wait inside the vehicle for about 10 minutes with the

aim of increasing the state.

Stress: the Verbal response delayed digit recall task (n-back) will be performed by the drivers.

For purposes of ADAS&ME evaluations, the 2-back task level will be utilised (Mehler, Reimer

and Dusek, 2011). The task and its instructions are explained with further detail in annex 12.

Emotion (frustration): Drivers will have to programme a badly working speech system. The

driver will be given one or several goals to achieve (e.g. call a number in the address book,

dictate a message). The driver will be frequently misunderstood by the system and the task/sub-

task completion will be possible only after several trials.

During the training trial (while standing still), drivers will be instructed on how to communicate

with the system, more specifically, which words should be used and with which timing. Manual



interaction will not be possible. The system will work properly during the familiarisation

period.

Distraction: The Surrogate reference task (SuRT) will be performed by participants while

driving (ISO14198, 2012). During the use case A evaluation only one level of the task will be

used to induce distraction. Further details can be read in annex 12.

7.10 Measurements

Measurements will be utilised in three distinct moments during the evaluations: before, during

and after the driving test.

Pre-driving test measurements

During the briefing and questionnaires phase, pre-test measurements will access the

demographic information of participants, their general opinion on the use of automated

driving/riding functions and also their general acceptance and trust of automation. The

following tools will be utilised:

• Demographic questionnaire (Annex 3) – should be filled in only in the first session/visit.

For the subsequent ones there is no need for drivers to fill in this questionnaire.

• General questionnaire on automated driving (Annex 4) – should be filled in only in the

first session/visit. For the subsequent ones there is no need for drivers to fill in this

questionnaire.

Driving test measurements

These measurements will happen while the participant is performing the test drives on the test-

track. Table 7 makes a presentation of all the metrics.

Table 7. Use case A metrics

Type Name Rate Format Description

Vehicle Data Time stamp Integer

Vehicle Data Speed Integer

Vehicle Data Automation

status

(active/inactive)

Integer

Vehicle Data Steering wheel

angel

Integer



Vehicle Data Brake pedal

position

Integer

Vehicle Data X position of

vehicle

Integer

Vehicle Data Y position of

vehicle

Integer

Driver

Behaviour

Visual

Behaviour

Eye movements and

fixations towards the road

environment and the control

panel.

Transition

timing

Moment drivers make the

handover and the takeover

(alone or in relation with the

prompts/warnings)

Time to

complete

transition

Time between initial action

to start transition until the

last action

Actions Video Hesitations, verbal

communications that can

give a better understanding

about the transition

difficulty

Transition errors Video Errors during the handover

or takeover manoeuvre

Driver State State level Sensors +

Video

To verify if the driver is

under the influence of the

desired state or not. May

detect differences when

state appears (after starting

the task) and when state

disappears (after receiving

mitigation strategy warning)

State Task

Performance

Task activation To verify if the driver

performs the secondary task

(control measurement)

Timing Task Audio/video When is the state task



Standby interrupted in relation with

the warning

Post- driving test measurements

The following measurements will be applied after the drive on the test-track.

• Debriefing interview (Annex 11)

• Acceptance scale (Annex 5)

• Trust scale (Annex 6)

• System Usability Scale (Annex 7)

• Questionnaire on potential system usage and acquisition (Annex 10)



8. Use case B – Electric vehicle range anxiety


The function developed for use case B aims at supporting a better management of the driving

range information for fully electric vehicles. The system enables detecting the driver’s range

anxiety. By comparing this state information with the range module (that calculates if the range

is in fact enough to reach the destination entered in the beginning of the trip) the system decides

which strategy is adequate to mitigate this anxiety state. In case anxiety is detected but range is

enough to reach the destination, the system will activate the range anxiety mitigation strategy.

Information will be sent to the driver informing him/her that calculated range is enough to reach

the destination entered. However, if the range module detects that range is not enough a range

incident mitigation strategy is chosen. In this case, the driver will be given the possibility of re-

routing to stop at a charging station. In case this is accepted, the driver will also have the

possibility of activating automation. This automated and energy efficient driving mode will take

the vehicle to the charging point.

In case the driver does not comply with re-routing to charging, the system imposes a 5 Km

protection. This means that 5 km before the vehicle is out of range, the vehicle searches for a

safe place to park and forces a safe stop.

Figure 7 shows the events in case of range anxiety and incident scenario followed by

compliance to re-routing and activation of automation.

For the purpose of ADAS&ME evaluations, this function will be implemented in a Vedecom

Electric Vehicle, based on a Renault Zoe platform, modified for Suburban Automated Driving.

In automated mode, the vehicle will be stealthily driven by a Wizard of Oz co-pilot, who will

be seated at the back seat. This control will be done via a joystick and participants will not be

aware that this automated mode is being controlled by the staff member.



Figure 7. Event chart for range anxiety and range incident mitigation with compliance in use case B

Driver

User Interface

Personalization

Driver monitoring


Decision system

Vehicle

Anxious driver

Anxiety detection

Send warning #1

Advise driver (re-routing)

Chose UI modality

Re-routing accepted

New mission

Send info + automation suggestion

Inform driver + propose hand over

Chose UI modality

Manual driving Automated driving

Activates automation

Chose UI modality

Automation status

Range Module Not enough to

reach destination

Anxious driver

Anxiety detection

Send warning #1

Inform driver

Chose UI modality

Enough to reach destination




The driver state covered in use case B is emotion, more specifically anxiety.

8.3 Evaluation objectives

To evaluate the functions being developed in use case B three test cases will be implemented.

Test case 1: will evaluate how accurately the system is able to detect and mitigate range

anxiety while driving on a public road;

Test case 2: will evaluate how accurately the system is able to detect and mitigate driver’s

anxiety in case of a range incident while driving on a public road, by means of re-routing

and automation driving;

Test case 3: will evaluate the opinion of drivers in case a forced safe stop is activated due

to non-compliance with the range incident mitigation strategies.

The evaluation of the functions developed in use case B has the aim of investigating:

1. the effectiveness of the system to recognize the driver’s state

2. the capacity of the system to mitigate anxiety in case the range is enough to reach the

destination (through clear and unambiguous messages)

3. the capacity of the system to mitigate anxiety in case the range is not enough (through

the suggestion of re-routing and automation)

4. if and when drivers comply with the range incident strategies (re-routing and

automation)

5. drivers’ opinion regarding the system’s interface in respect to usability

6. drivers’ trust and acceptance of the system

8.4 Evaluation site

This evaluation will be conducted on public roads in Santa Oliva, and also on IDIADA’s

proving ground high speed track. Figure 8 shows a map of the route that will be used to perform

this evaluation. The entered address will be Pacs del Penedès via B-212, a destination 24.1 km

from IDIADA’s proving ground that takes about 32 min. to reach. The evaluation test will start

and finish at IDIADA’s proving ground.



Figure 8. Route for the use case B evaluation (Google maps, 2018)

8.5 Time plan


structured in the following way:

Table 8. Four weeks activity description for use case B



6 Pre-tests




Twelve days will be dedicated to performing data acquisition. Real drivers will be invited to

test the function. The entire procedure is foreseen to last no longer than 2 hours and 30 minutes

(Table 9). A maximum of two participants will evaluate the system per day, one in the morning

and another in the afternoon. In the meantime the vehicle will have to be charged.



Table 9. Use case B session procedure

• Welcome + Briefing – 20 min


• Public road test drive – 45 min

• Debriefing for public road test drive – 30 min

• Explanation test-track trial – 5 min

• Test-track drive – 15 min

• Debriefing test-track drive – 15 min


A within-subjects design will be implemented as all participants will test the system under the

same conditions. The independent variable Mitigation strategy will be tested by participants

under three conditions: range anxiety, range incident with compliance, range incident without

compliance. All three conditions will be tested in one session.


A total of 24 participants will integrate the sample. The sample will be composed by drivers

between 25 and 40 years-old that have more than five years of driving experience. The selected

participants will not have any experience with an electric vehicle. Drivers must have interacted

already with an advanced driver assistance system (ADAS) or an in-vehicle information system

(IVIS). Participants must speak French fluently, as functions will be primarily developed for

French speaking drivers.

8.8 Procedure

Upon arrival, participants will be welcomed and informed about the main aim of the test. They

will be given a consent form to sign, containing details on the privacy of the data collected and

how it will be used. Following, they will be asked to fill in some questionnaires.

A training trial will be performed at the beginning on the high-speed track, in order for

participants to get acquainted with the vehicle (in manual and automated driving mode). After

an explanation on how the system works, the driver will be asked to drive on the high-speed

track at a speed not higher than 50 km/h. For safety reasons and also to control the vehicle in

automated mode, a member of the staff will be inside the vehicle the entire time (besides the

driver). Participants will not be aware that the staff member is controlling the vehicle in

automation mode (via a joystick). Transitions from manual to automated driving will be tested

and the interface that displays the information will also be presented. Participants will be

informed that automated driving is also a very energy efficient driving mode. During this

familiarisation moment, the driver states will neither be induced nor detected. The vehicle is

fully charged at the beginning of the familiarization trial.

A driving test will follow. The staff member will be inside the vehicle (at the back seat) in case



something unexpected happens. Unless something serious happens, the driver should not

communicate with the staff member. Thus, if the driver poses a question while conducting the

evaluation related to an uncritical situation, the staff member will not answer not to influence

the evaluation of the system. At least 70% of range should be available to start the driving test.

Below follows a detailed description of what will happen during the driving test:

1st test case:

(Inside IDIADA’s facilities)

. The destination is entered by a member of the staff and drivers will be informed of how many

km they will have to drive to reach the destination and return. In case they perform the complete

course (going to Parés Baltà Winery and return) their monetary compensation will have an

increase of 20€.

. Drivers will be informed that the available range should be enough. In any case, as traffic

conditions are unpredictable, drivers should verify if range does not decrease too drastically.

Advises will be given on how to save energy in case it is needed, like not turning on the air

conditioning too long, or drive too fast.

(The vehicle leaves IDIADA’s facilities)

. After about 10 minutes of driving (and independent of the driving style), the range starts

decreasing dramatically (the displayed range is not real, but a simulated one, that will be

manipulated by the staff member traveling along – about 1% every 30 sec.).

. Upon the detection of anxiety, the system sends a first message with the aim of mitigating

range anxiety (1st test case – anxiety although range is enough). This scenario should be

simulated until the range reaches a plausible limit (e.g. 40% of range left).

2nd test case:

. When 40% of simulated range left is reached, and before reaching the original destination,

range incident mitigation should start: drivers will be advised of re-routing, which will be to

return immediately to IDIADA’s facilities. The possibility of turning on automation will also

be given once the drivers are inside IDIADA’s facilities.

3rd test case:

On the high speed test track (Figure 9), drivers will be informed that they will evaluate another

function of the system (the safe stop): The new function is activated in case the driver ignores

the warnings and does not comply with re-routing and automation activation. Under these

circumstances and in case range is very low, the vehicle will take over control, search a safe

place to park and perform a safe stop. The procedure for this test case will be as follows:

. Driver starts driving the vehicle manually on the test track at a speed not higher than 50 km/h;

. When advised by the staff member travelling inside the vehicle, the driver hands over control

of the vehicle (manual to automated driving).



. The system starts sending range incident mitigation messages that the driver will be instructed

to ignore by not performing an action/touching the vehicle controls;

. The system performs a safe stop

For a complete overview of the procedure consult Figure 10.

Figure 9. IDIADA’s high-speed track for use case B and respective sections.

Figure 10. Use case B test proceeding


The driver will not have to perform additional tasks to be under the influence of the desired

state (anxiety). The state will be induced by the simulation of a dramatic range decrease while

on the road.

8.10 Measurements

To evaluate all test cases belonging to use case B, measurements will be applied in three distinct

moments: before; during and after the driving test is completed.


Their application will happen during the briefing and questionnaires phase. Two questionnaires

will be filled in by the participants: A demographic questionnaire (Annex 3) to collect the

driving experience and experience with information and driver assistance systems; and a

general questionnaire on automated driving (Annex 4) to gather the opinion of drivers

concerning automated functions in vehicles.




These measurements will be recorder while participants are performing the driving test, both

on the public road and also on the test-track. The following table (Table 10) gives more details

on each metric.

Table 10. Use case B metrics

Environment Type Name Rate Format Description

Public road Vehicle Data

Public road

and

Test track

Vehicle Data Time stamp

Public road Vehicle Data Speed Instant speed chosen by the driver while driving manually

Public road Vehicle Data Re-routing activation

Verify if re-routing option was activated by the driver

Public road

and

Test track

Vehicle Data Automation status

(active inactive)

Status of the automation while driving on the public road

Public road Vehicle Data Timing initiation automation

Timing for the activation of the automation after the driver has received the warning

Vehicle Data Steering wheel angle

Vehicle data Energy recuperation

Energy recovered from a particular driving behaviour

Public road and

Test track

Driver Behaviour

Visual Behaviour

Eye movements and fixations towards the road environment and the control panel.



Public road and

Test track

Driver Behaviour

Actions and communications

Video Hesitations, verbal communications that can give a better understanding about the interaction with the system









9. Use case C – Driver state-based smooth & safe automation transitions


The function developed for use case C aims at supporting smooth and safe transitions between

automation levels. This function uses tailored interaction strategies for specific driver states and

characteristics by using different timing, modalities and HMI-components for transitions.

Once driving automated, the system will be capable of detecting situations that cannot be

managed by automation, i.e. environmental conditions where independent and safe automated

driving is no longer possible to be performed, such as road works. When this happens, a

takeover is needed. The system will also detect the driver’s state and check if he/she is

capacitated to drive. In case the driver is capacitated to drive, the system will allow the

transition. When this situation does not verify due to driver’s fatigue, stress, inadequate

emotional state or distraction, the system is able to send warnings/information to mitigate the

driver’s state, helping him/her to be again capacitated to drive and perform the needed takeover.

Figure 11 shows the events for a driver state recognition and mitigation before a transition

occurs.

For the purpose of ADAS&ME evaluations, this function will be implemented in DLR’s test

vehicle (FASCarII), a modified VW Passat which is equipped with multiple computers and

sensors enabling the vehicle to drive in highly automated mode (SAE3). The FASCar II will be

equipped with project specific sensors and computers from project partners to detect driver

states and elements of the driving environment. It includes also prototypes of software

(automation and environmental sensing) as well as hardware (actuators at steering wheels and

pedals to induce haptic feedback). Further, a second braking pedal is installed at the passenger

seat. When the FASCar II is driven in highly automated mode, a safety driver is required in the

passenger seat to bring the vehicle to a safe stop in case of an unlikely event of a system

failure. Further information regarding technical and hardware architecture of the FASCar II can

be found in ADAS&ME internal Deliverable D2.1.



Figure 11. Event chart for driver state recognition and mitigation in use case C.

Driver

User Interface

Personalization

Driver monitoring


Decision system

Vehicle


Takeover needed (automatic to manual)

State X detection State X mitigated

Send notification #2 + Takeover request

Vehicle ready to takeover

Chose UI modality

Automated driving Manual driving

Takeover control State X

Chose UI modality

Notifies driver + Strategy to mitigate state

Send notification #1 + Choses strategy to mitigate state

Capacitated driver




The driver states covered in use case C are the following: Fatigue, stress, emotion (frustration)

and distraction.


To evaluate the function being developed in use case C three test cases will be implemented.

Test case 1: will evaluate how accurately the ADAS&ME Driver monitoring system is

able to detect certain driver states while driving in a SAE level 3 automation in the DLR

test vehicle on a test track at a concrete speed of 30 Km/h;

Test case 2: will evaluate how accurately the ADAS&ME Driver monitoring system is

able to detect certain driver states while driving in SAE level 0 (no automation) in a test

vehicle on a test track at a concrete speed of 30 Km/h;

Test case 3: will evaluate if the (in test case one and two) detected driver state could be

used to trigger a driver state specific HMI version for a transition from automated to

manual driving. Further it will be investigated if the HMI version is able to initiate a safe

and well accepted transition.

Thus, the objectives of the tests conducted for this use case are to evaluate the following aspects:

1. effectiveness of the system to recognize the driver’s state

2. effectiveness of the tailored interaction strategies for the transition (automated to

manual)

3. driver behaviour following a system warning

4. drivers’ opinion regarding the system’s interface in respect to usability

5. drivers’ trust and acceptance of the system

9.4 Evaluation site

This evaluation will be conducted on IDIADA’s proving ground, more specifically on the high-

speed track.

9.5 Time plan


structured in the following way (Table 11):



Table 11. Four weeks activity description for use case C


1 to 5 Technical preparations and finalisation of subject recruitment

6 Pre-tests




Twelve days will be dedicated to performing data acquisition. Real drivers will be invited to

test the function. The entire procedure is foreseen to last no longer than 2 hours. Table 12 gives

more details on the duration of the sessions. For more details on the complete procedure, use

case D section of this deliverable must be considered as some sessions will evaluate both C and

D use cases.


A within-subjects design will be implemented. The independent variable Driver state will have

four conditions: fatigue, stress, emotion, and distraction each of which is experienced by all

participants. In order to avoid bias due to learning effect, the order the states are induced to

participants will be balanced across the sample. Annex 1 presents this schema.

Due to the nature of these evaluations, drivers will not be induced to all states in one unique

day, which means that participants will be invited to visit IDIADA’s test facilities more than

once. Table 12 presents a brief procedure of each visit.

Table 12. Use case C and D sessions

Session A – evaluation under the

influence of fatigue (80 to 95

min.)


have to be scheduled for the same

period of the day (e.g. beginning

of the afternoon) due to the

influence of the circadian

rhythm.

• Welcome + Briefing – 5 to 20 min4


• Pause (to intensify state) – 10 min


• Debriefing – 30

4 Duration changes if session is the first one or not. Participants’ first session will be longer, as

more questionnaires will have to be filled in – demographics and general about automated

driving. For the following sessions these will not be needed and participants can start with the

familiarization trials after a short welcoming.




influence of emotions (100 to

115 min.)


• Familiarization trial (drive) – 15 min

• UC D evaluation – 20 min

• Familiarization (state induct. task) – 10 min



Session C – evaluation under the

influence of distraction and

stress (110 to 125 min.)


• Familiarization trial (drive + distraction task) – 20

min

• Distraction trial – 20 min


• Familiarization (stress task) – 5 minutes

• Stress trial – 20 min




participants will be recruited (estimated 16) in order to assure that 12 data sets (and only 12)

for each state are available for analysis. Surrogate participants recruited to compensate a drop-

out should perform the conditions in the same order the dropped-out participant did. This will

assure a balanced distribution of the conditions over the sample. A detailed plan of which and

when participants visit IDIADA’s facilities will be set closer to month 36 and in accordance

with the drivers’ availability.

The sample will be composed of drivers between 25 and 40 years old that have more than five

years of driving experience. Drivers must have interacted already with an advanced driver

assistance system (ADAS) or an in-vehicle information system (IVIS). The participants must

speak German fluently, as functions will be primarily developed for German speaking drivers

(especially important under the influence of inadequate emotional states, as voice will be used

as trigger).

9.8 Procedure



how it will be used. Following, participants will be asked to fill in a demographic questionnaire

(Annex 3).

A training trial will be performed at the beginning, in order for participants to get acquainted

with the vehicle in manual and automation mode. The driver is accompanied by a member of

the staff and drives the complete high-speed track circuit at a speed of about 30 km/h.

Transitions are trained and the driver gets to know the interface that displays information about

the system status. During the training trial, the driver has the opportunity to ask questions to the



staff member about the interface, the system, and the track. During this familiarisation moment,

the driver states will neither be induced nor detected.

During the driving test, conditions (independent variables) will be implemented. This means

that driver states will be induced and both behavioural and driving parameters will be collected.

This phase starts by explaining the type of task the driver has to perform along the test. This

task will induce the desired state. Drivers will not be informed about the purpose of the task.

Specific instructions on how and when this information should be conveyed to the drivers will

be in written form. This will be prepared prior to the evaluations and passed to the drivers

always in the same manner. A system of incentives (the more tasks the higher the monetary

compensation) will motivate drivers to perform the tasks while driving.

During the driving test (Figure 12) a member of the staff will be inside the vehicle in case

something unexpected happens and the driver requires help. Drivers will be informed about this

and also that he/she should not communicate with this person. A constant driving speed of 30

km/h will have to be maintained during the driving test (in both manual and automated

conditions). For three of the four conditions secondary tasks will have to be performed. These

tasks intend to induce the driver states. Below follows a detailed description of what participants

will do during the driving test.

Figure 12. IDIADA’s high-speed track for use case C evaluation and detail for road works

layout.

1st Test case

. Participants start driving the vehicle manually;

. They drive on the 1st curve section (initial part of this section has only one lane due to road

works);

. The task should start about 30 seconds after the drive starts (after leaving the road works) –

stimulus will be given to the driver in the adequate moment via audio system of the vehicle;

. System detects state and sends information about the need of a transition;



. Participants hand over control at the beginning of south straight section.

2nd Test case

. Participant enters the south straight section by activating the automation;

. Driver is informed to re-start task;

. System detects the need of a transition and the driver's state and sends a warning.

. Participant takes over control before the east curve section (before entering the construction

zone)

Repeat 1st and 2nd test case. Test will end after transition to manual driving).


In order to induce the desired states while driving, the following tasks will be used:

Fatigue: Participants invited to perform the evaluation will be asked not to sleep the night

before, so their fatigue level is high. Before initiating the driving test (after the training trial)

the drivers will have to wait inside the vehicle for about 10 minutes (cover story: the researcher

will have to calibrate a system and it will take some minutes). During this waiting time the

driver will be asked to be seated inside the vehicle and not to communicate or perform any task.

The environment inside the vehicle will be quiet and comfortable. After this, the test will start.

There will be no performance of a secondary task while driving.

Stress: The Verbal response delayed digit recall task (n-back) will be performed by the drivers.

For purposes of ADAS&ME evaluations, the 2-back task level will be utilised (Mehler, Reimer

and Dusek, 2011). MIT. The task and its instructions are explained in further detail in annex

12.

Emotion (frustration): Drivers will have to programme a badly working speech system. The

driver will be given one or several goals to achieve (e.g. call a number in the address book,

dictate a message). The driver will be frequently misunderstood by the system and the task/sub-

task completion will be possible only after several trials.

During the training trial (while standing still), drivers will be instructed on how to communicate

with the system, more specifically, which words should be used and with which timing. Manual

interaction will not be possible. The system will work properly during the familiarisation

period.

Distraction: The Surrogate reference task (SuRT) will be performed by participants while

driving (ISO14198, 2012). During the use case C evaluation only one level of the task will be

used to induce distraction. Further details can be read in Annex 12.



9.10 Measurements

Several measurements will be applied in three distinct moments during the evaluations: before

the driving test happens; during the driving test and after this is completed.



demographic information of participants, their general opinion on the use of automated

driving/riding functions and also their general acceptance and trust of automation. The

following tools will be utilised:

• Demographic questionnaire (Annex 3) – should be filled in only in the first session/visit.

For the subsequent ones there is no need for drivers to fill in this questionnaire.

• General questionnaire on automated driving (Annex 4) – should be filled in only in the

first session/visit. For the subsequent ones there is no need for drivers to fill in this

questionnaire.


These measurements will be collected while the participant is performing the test on the test-

track. They have different nature and are distributed over three clusters: 1) procedure data (this

data identifies the active “condition” and makes it possible to connect gathered data to the

condition); 2) vehicle data (data coming from the vehicle parameters or vehicle position); 3)

driver behaviour (data belonging to the behaviour of the driver and his/her direct actions related

with the driving); 4) driver state (data regarding the induced state); 5) state task performance

(data originating from the performance of the additional task used to induce the state). Table 13

gives more information about each metric.

Table 13. Use case C metrics


Procedure snippet Number 50hz Integer To which condition is the data related to

Procedure Sub-snippet Number

50hz Integer To which condition is the data related to

Procedure Active scenario 50hz Integer To which condition is the data related to

Vehicle Data Speed 50hz Integer Velocity of the vehicle

Vehicle Data Time stamp 50hz Integer Time stamp of the data



Vehicle Data Automation status (active/inactive)

50hz Integer Activation of the automation

Vehicle Data Steering wheel angel

50hz Integer Current angle of the steering wheel

Vehicle Data Brake pedal position

50hz Integer Position of the brake pedal

Vehicle Data Automation level

50hz Integer Active automation level

Vehicle Data X position of vehicle

50hz Integer X Coordinates of the vehicle

Vehicle Data Y position of vehicle

50hz Integer Y Coordinates of the vehicle

Driver Behaviour

Visual Behaviour


Driver Behaviour

Transition timing

Moment drivers make the transition in relation with the warnings

Driver Behaviour

Actions Video Hesitations, verbal communications that can give a better understanding about the transition difficulty

Driver State State activation Video Control measurement for confirmation (and comparison with sensor data) of each state level.

State Task Performance

Task activation To verify if the driver performs the secondary task (control measurement)


Timing Task Standby

Audio/video When is the state task interrupted in relation with the warning





10. Use case D – Non-reacting driver emergency manoeuvre


The function developed for use case D is an escalation of use case C. Similar to use case C,

once driving on a highway in SAE level 3, when approaching a system limit a driver state based

transition request (from automated to manual) is sent. However, the driver does not react to the

request and fails to takeover control of the vehicle in time. Due to the non-reacting driver, the

automation needs to intervene and starts a safe stop manoeuvre. The safe stop manoeuvre results

in an emergency brake to a full stop.

Figure 13 shows the events that lead to a safe stop manoeuvre.

This function will be implemented in the same vehicle as use case C, i.e. the DLR’s test vehicle

(FASCarII), a modified VW Passat equipped with multiple computers and sensors enabling the

vehicle to drive in highly automated mode (SAE3). More information can be found in section

9.1 of this Deliverable or in Deliverable D2.1.


Use case D function is being developed for the same driver states as use case C. However, for

purposes of the evaluations, this test case will be conducted independent of the driver state, as

the driver’s non-reaction will not really happen but has to be forced/simulated.


To evaluate the function being developed in use case D, a test case will investigate:

1. if the chosen interaction strategy for a safe stop manoeuvre is communicated in a clear

and unambiguous way

2. the drivers’ trust and acceptance on such a function

10.4 Evaluation site

Like for use case C, this evaluation will be conducted on IDIADA’s proving ground, more

specifically on the high speed track.

10.5 Time plan

This evaluation will be carried out during M38, together with use case C. All the technical and

methodological preparations will be done simultaneously for all test cases belonging to both

use cases.



Figure 13. Event chart for non-reacting driver emergency manoeuvre in use case D.

Driver

User Interface

Personalization

Driver monitoring


Decision system

Vehicle


Takeover needed (automatic to manual)

State X detection Inadequate State X level

Send notification #2 + Transition information

Informs driver

Chose UI modality

Automated driving

State X

Chose UI modality

Notifies driver + Strategy to reduce state level

Send notification #1 + Strategy to reduce state level

Safe stop




The drivers will not be under the influence of a state, which means that they will be fully

capacitated to drive. Thus, there will be only one condition: safe stop emergency manoeuvre.


Both use cases C and D will share the same sample. All 12 participants will evaluate use case

D function. For more details on the sample characteristics consult section 9.7 of this deliverable.

10.8 Procedure

Use case D evaluation will be integrated in the evaluation of use case C together with the test

drive under the influence of inadequate emotional states. In order to avoid that emotional states

will have an influence over the evaluation of use case D, this non-reacting driver emergency

manoeuvre will be performed at first place.

A training trial before use case D driving test will be conducted to allow participants getting

used to the vehicle and the test track. Even if this session is not the first on, i.e. the driver has

visited IDIADA to perform the evaluation of this system under the influence of another driver

state, it is suggested to perform this training trial to recall drivers about the system and it’s

interface. During the training the driver will receive a warning before the transition moments

in order to perform the handovers and takeovers in the correct moment. At the end of this

training trial the tasks for the use case D evaluation will be explained. The driver will be

informed that the notifications to make transitions from automated to manual driving should be

ignored (but the transitions from manual to automated driving should be done). This means that,

when driving automated, and after receiving a request to takeover control of the vehicle, the

driver should do nothing: neither touching the pedals nor the steering wheel. The vehicle will

then perform an emergency manoeuvre that consists of bringing the vehicle to a full stop on the

lane’s shoulder.

During the driving test a member of the staff will be inside the vehicle in case something

unexpected happens and the driver required help. Drivers will be informed about this and also

that he/she should not communicate with this person. A constant driving speed of 30 Km/h will

have to be maintained during the driving test (in both manual and automated conditions).

The detailed evaluation is planned to happen as follows:

. The driving test begins at the starting point (as it can be seen in Figure 14).

. The driver will initiate manual driving at a speed of 30 km/h on the first curve section of the

test track.

. Moments before leaving this section, the driver receives a notification to handover control of

the vehicle (from manual to automated driving).

. The driver follows this notification and the vehicle starts automated driving when already on

the south straight section.

. Before reaching the end of this section the driver will again receive a notification, this time to



take back control of the vehicle (from automated to manual driving)..

. The driver ignores this notification.

. After the 2nd notification the vehicle performs a safe stop by parking on the lane’s shoulder at

the end of the south straight section.

The driver will then be asked to re-start driving manually at a speed of 30 km/h. The procedure

should be repeated for the 2nd curve and north straight section of the track. The driver will then

experience the emergency manoeuvre a second time. The evaluation will end after the second

safe stop (near the starting point).

Figure 14. IDIADA’s high speed track for use case D and respective sections.


No tasks will be performed to induce driver states while evaluating use case D. The driver will

perform this test fully capacitated to drive.

10.10Measurements

To evaluate the emergency manoeuvre, measurements will be applied in three distinct moments

during the evaluations: before the driving test happens; during the driving test and after this is

completed.


Pre-test measurements for this use case will be integrated in use case C sessions.


Like for use case C, the following measurements will happen while the participant is performing

the test on the test-track. Table 14 gives more information about each metric.



Table 14. Use case D metrics


Procedure snippet Number 50hz Integer To which condition is the data related to


50hz Integer To which condition is the data related to

Procedure Active scenario 50hz Integer To which condition is the data related to

Vehicle Data Speed 50hz Integer Velocity of the vehicle

Vehicle Data Time stamp 50hz Integer Time stamp of the data


50hz Integer Activation of the automation


50hz Integer Current angle of the steering wheel


50hz Integer Position of the brake pedal


50hz Integer Active automation level


50hz Integer X Coordinates of the vehicle

Vehicle Data Y position of vehicle

50hz Integer Y Coordinates of the vehicle

Driver Behaviour

Visual Behaviour

Eye movements and fixations towards the road environment and control panel while the vehicle is performing the emergency manoeuvre.

Actions Video Arm and feet movement towards the vehicle controls (complete movements or hesitations) and verbal communications that can give a better understanding about the driver’s comfort while safe stopping.



11. Use case E – Long range attentive touring with motorbike


The function developed for use case E supports riders during long-range motorbike touring.

Once driving manually, the system takes into consideration the information collected via

environmental sensing (e.g. air temperature, atmospheric pressure) and rider state monitoring

(e.g. Skin temperature in different body regions, heart rate). In case the system detects signs of

fatigue (related to the muscular effort and the environment temperature), distraction or stress, a

warning will be sent to inform about this situation. Strategies to mitigate the rider state will also

be applied, i.e. suggestion to make a pause. The aim is to anticipate high risky situations (e.g.

rider dehydration and faint) through offering effective rider support. In case of non-compliance

with the suggested strategies and maintenance of the rider’s state, the system informs about the

activation of a recovery mode. The recovery mode limits the motorcycles performances without

stopping it, i.e., while riding in recovery mode - the motorcycle will have a limited power,

torque or rpm (the strategy is still under discussion). To deactivate this mode and have again

the full motorcycle potential, the rider has to take a break for a certain period of time. Figure

17 displays the events for state recognition and mitigation of use case E.

For the purpose of ADAS&ME evaluations, this function will be implemented in a Ducati

Multistrada 1260, a motorcycle designed for touring (Figure 15). The motorcycle will be

adapted to integrate a control unit, which will run the DSM (Decision Support Module) and the

rider states detection algorithms, an RF (Radio Frequency) communication unit, to

communicate with the Personal Protective Equipment (PPE), and will contain some

modifications at software level to implement the HMI and automated function.

Figure 15. Motorcycle used for the UC E evaluations

It is important not to forget that the PPE includes all the main rider monitoring sensors. In detail,

the following garments will be used, specifically modified for this research project (Figure 16):

• Dainese Super Rider D-Dry Jacket, used as a shell for the Back Protector, it also

provides rider protection on shoulders and arms



• AGV Corsa R E2205 Helmet, adapted with an IMU and 8 temperature sensors

• Dainese Druids D1 Long Gloves, adapted with IMU, PPG, GSR, Temperature,

Humidity and UV sensors

• Dainese D-Core Dry Underwear Tee, adapted with Temperature, Humidity, ECG, and

Respiration sensors

• Dainese D-Air ProArmor Back Protector G2, adapted with IMU, pressure sensor and

GPS. The unit also includes the RF communication unit and the system battery used to

power up all the PPE units

Figure 16. PPE Used for UC E/F evaluations



Figure 17. Event chart for driver state recognition and mitigation in use case E.

State X Driver

User Interface

Personalization

Driver monitoring


Decision system

Vehicle

State X detection

Warning #1

Choose UI modality Chose UI

modality

Activate

recovery

mode

Activates recovery mode Info state + warning to stop #1

Manual riding

Warning #2

Chose UI modality

Warning to stop + Info recovery mode

State X detection State X detection

Recovery mode active




The rider states covered in use case E are physical fatigue, distraction and stress. Physical

fatigue (with its extreme level, i.e. the thermal faint) and distraction will have priority over

stress for the final evaluations of the system. For this reason the plan being presented will

consider these two states.


Use case E aims at developing dedicated functions that deduce physical fatigue (till faint),

distraction and stress, combined with effective HMI and first examples of automation. These

functions aim at increasing riders’ safety and comfort, with a beneficial effect on riders’ fatality

figures which, in contrast to drivers’, are not decreasing rapidly enough.

The purpose of UC E final assessments is to verify:

• if the system is able to predict physical fatigue (including faint) and visual distraction

• if riders accept to be monitored

• if riders accept to be assisted (including the automated function) during their ride and

trust the system

• if the HMI, displaying information and state mitigation strategies, is clear and

unambiguous

• how riders behave following a system warning (following the recovery mode) and if

they comply with the suggested strategies to mitigate their state

In a nutshell, the main goal is to encourage riders to be assisted, so they can enjoy their ride

comfortably and avoid safety-critical situations that could lead to crashes.


This evaluation will be conducted in IDIADA’s Proving Ground and on the public roads near

these facilities. More details about the sites are given below, in the procedure section.

11.5 Time plan

This evaluation will be conducted on IDIADA’s proving ground and on the public roads near

these facilities. More details about the sites are given below, in the procedure section (Table

15).



Table 15. Four weeks activity description for use case E



6 Pre-tests


8 to 19 Evaluation with real riders


Like for other use cases about 12 days will be dedicated to performing data acquisition. Real

riders will be invited to test the function.


A within-subjects design will be implemented. The independent variable Driver state will have

two conditions: physical fatigue, and distraction each of which is experienced by all

participants.


A total of 12 participants will integrate the sample. The sample will be composed of riders

between 30 and 55 years old that have more than five years of driving experience and ride more

than 4000 km per year. Preferably riders should have already interacted with motorcycle

advanced functions like traction control, motorcycle stability control, cornering ABS, vehicle

hold control or, in alternative, with car advanced driver assistance systems (ADAS) and/or in-

vehicle information systems (IVIS). Riders should speak an average English, since the functions

will be developed in this language.

11.8 Procedure

Table 16 present an overview of the procedure. Upon arrival, participants will be welcomed

and informed about the main aim of the test. They will be given a consent form to sign,

containing details on the privacy of the data collected and how it will be used. Following,

participants will be asked to fill in a demographic questionnaire (Annex 3) and a general

questionnaire on automated driving/riding (Annex 4).

Participants will be shown the motorcycle and the PPE will be set up. A first training trial will

be done on the public roads near IDIADA. The aim is to get the riders acquainted with the

motorcycle. During this phase, no rider state will be induced nor detected. The training trial will

last about one hour. After arriving from this ride, participants will be directed to the high-speed

track. There the system functionalities and HMI will be explained, followed by a 10 minutes

ride to try it.



Table 16. Use case E procedure


• Explanation of public ride and set up of PPE – 15 min

• Public road familiarization ride – 60 min

• System explanation – 10 min

• Distraction ride – 20 min

• Debriefing for distraction ride – 20 min

• Explanation fatigue track – 10 min

• Fatigue ride – 40 min (max)

• Recovery mode test – 15 min

• Use case F evaluation – 30 min

• Debriefing for fatigue test and UC F – 30 min

The distraction riding test will follow. During this test, several visual distraction triggers will

be used to force the rider to look down (motorcycle control panel), to the left and right side

(unexpected event happening in the road environment). This distraction ride will last about 20

minutes. A debriefing interview for this test will follow.

The next ride will aim at verifying if the system is able to recognise physical fatigue. The

motorcycle will be fully loaded (10 kg in each side pannier and in top case) in order to accelerate

the occurrence of muscular fatigue.

The participant will start by riding on IDIADA’s fatigue test-track, a track with certain

obstacles, specially conceived for this purpose (more details on this track can be seen in Annex

13). The system will be active during this period and will detect physical fatigue in case the

rider shows signs of it. Messages for the mitigation of this state will also be presented. Rider

will then decide whether to stop to do a break or continue riding. In case he/she continue riding,

a maximum time limit will be set to stop this test.

Participants will then be asked to ride on the high-speed track (Figure 18) to test what happens

when mitigation messages are not complied. The evaluation will start after warning#1 (see

Figure 17). Participants will be asked to ignore the system warnings and continue riding; this

can be based on a specific task (e.g. the rider will be paid based on the km ridden on the bike).

The recovery mode will be activated and the rider will experience the motorcycle’s performance

limitation. This test will be short in time (maximum 15 minutes).

Before debriefing of this test, the use case F function will be evaluated. More details can be

found in Chapter 12.



Figure 18. High-speed track for the evaluation of UC E


Distraction: The nature of the distraction tasks will be detailed for the second version of this

deliverable as they much depend on the results of the pre-pilot tests being conducted at the

moment. Moreover, due to the reduced literature on such type of tasks dedicated to motorcycles,

suggestions will not be included (as it happened for the other use cases) as the success of these

tasks to induce distraction are not yet clear. As more than one distraction task will be induced,

the order with which they appear will be randomised among the sample to avoid bias of the

results. More details on this matter will also be given for the updated version of this deliverable.

Fatigue: The rider will perform an initial ride on the public road and a subsequent one on a

fatigue test track. It is expected that these two tests, together with the weather condition (reason

why the evaluations will be conducted in the summer) will contribute to the appearance of

muscular fatigue. During the evaluations a medical doctor will be present.

11.10 Measurements

Several measurements will be applied in three distinct moments during the evaluations: before

the driving test happens; during the driving test and after this is completed.

Pre-riding test measurements


demographic information of participants (demographic questionnaire, Annex 3) and their

general opinion on the use of automated driving/riding functions (general questionnaire on

automated driving, Annex 4).

Start



1st

curve section 2nd

curve section



Riding test measurements

These measurements will happen while the participant is performing the test on the test-track

(fatigue and high-speed). They have different nature and are distributed over three clusters: 1)

vehicle data (data related to the riding performance); 2) rider’s signals (data belonging to the

physiology and behaviour of the rider and his/her direct actions related with the riding); 3)

environmental data (data related to the surrounding environment). Table 17 details these

metrics.

Table 17. Use case E metrics


Vehicle Data Speed Up to 100 Hz

Integer

Vehicle Data Longitudinal Acceleration

Up to 100 Hz

Vehicle Data Lean angle Up to 100 Hz

Vehicle Data Brake use (wheel pressure)

Up to 100 Hz

This is the pressure applied to the brake disk

Vehicle Data Brake use (master pressure)

Up to 100 Hz

This is the measure of the braking power asked by the rider. The ABS system can decide to modulate/reduce. This is why wheel and master pressure may differ.

Vehicle Data Accelerator position

Up to 100 Hz

Rider’s signals Hands vibrations

Two accelerometers are integrated in the gloves to measures the vibrations induced on the upper extremities by the motorcycle

Rider’s signals Skin temperatures

10 Hz on Hand and chest

4Hz on Head

Several temperature sensors are integrated in the wearables, to measure skin temperatures in different body regions (torso, head, hands)



Rider’s signals Heart Rate PPG sampled at 20 Hz

ECG sampled at 400 Hz

Based on a PPG and an ECG sensor

Rider’s signals Heart Rate variability

Based on a ECG sensor

Rider’s signals Respiratory Rate

Respiration sampled at 20 Hz

This is measured by a chest strap

Rider’s signals Blood pressure This is an indirect measure based on a PPG sensor (integrated in the left glove) and the ECG sensor

Rider’s signals Electrodermal activity

50 Hz One EDA sensor is integrated in the right glove

Rider’s signals Relative position of the head

This is the main signal for distraction detection and is based on the measurements of two IMUs, one placed in the helmet and one on the jacket

Rider’s signal Riding time 5 Hz Based on GPS time, a GPS module is integrated in the wearables

Environmental data

Air Temperature

10 Hz on wearables

Environmental parameter, it can come either from a sensor placed on the motorcycle or on a sensor placed on the wearables

Environmental data

Relative Humidity

10 Hz Two relative humidity sensors are integrated in the gloves.

Environmental data

UV index 10 Hz This is useful to understand environmental conditions. Two UV sensor is integrated in the gloves



Environmental data

Atmospheric pressure

25 Hz The jacket unit has a barometer. This signal, along with air relative humidity, UV index and air temperature can be used to roughly estimate the local (i.e. in the area surrounding the vehicle) weather conditions

Environmental data

Traffic information

These two signals will come from the navigation unit

Environmental data

Incident information

Post- riding test measurements








12. Use case F – Rider faint


The function developed for use case F supports the riders in case physical fatigue reaches an

extreme state (faint). This use case can be seen as an escalation of use case E. In case sensors

detect extreme temperatures, together with signs of muscular fatigue, the system sends

mitigation strategy messages (suggestion to stop) and activates the recovery mode (which limits

the motorcycle’s power). If the rider continues to ride and his/her physical condition degrades,

the system performs an emergency manoeuvre – stabilises the motorcycle and reduces its speed,

possibly till stopping it. Figure 19 shows the events that lead to the activation of this function.

For the purpose of ADAS&ME evaluations, this function will also be implemented in a Ducati

Multistrada 1260, like for use case E. This will be a different motorcycle, with the same

characteristics and an additional stabilisation system. To evaluate use case F, an active capsize

control unit (made up of dedicated actuators) will also be integrated. The PPE will be the same

as in UC E.


The rider state covered in use case F is physical fatigue (incl. thermal faint).


Use case F aims at:

• Verifying if the system is able to predict extreme physical fatigue (including faint)

• Knowing if riders accept to be assisted when such critical situations occur, trust such

a system and would be willing to buy it/use it


This evaluation will be conducted on IDIADA’s proving ground together with use case E. In

case it is not possible to test this function on IDIADAS’s test track (in case the weather

conditions are not extreme enough or riders are not sufficiently fatigued), a backup plan will be

prepared to evaluate this use case in CERTH’s simulator. More details about this plan will be

given in the updated version of this deliverable (due in M24).



Figure 19. Event chart for driver state recognition and mitigation in use case F.

Driver

User Interface

Personalization

Driver monitoring


Decision system

Vehicle

State X detection

Warning #1

Choose UI modality

Info. state + warning to stop #1

Manual riding

Safe stop

decision

State X

Chose UI modality

Info. about active

stabilisation

State X detection

Active stabilisation + possibly stop



12.5 Time plan

This evaluation will be carried out during M37 along with the evaluations conducted for use

case E.


A unique driver state condition will be imposed to evaluate this system: extreme physical

fatigue.


The sample will be the same as for use case E.

12.8 Procedure

The evaluation of use case F function will be performed at the end of use case E test. After

testing the motorcycle’s performance limitation, the rider will be asked to make a short pause

to change bike (the motorcycle with stabilisation system, considered the prototypal equipment,

cannot be used on the fatigue test track). The rider will be asked to continue riding on the high-

speed track. Since it is not possible, for ethical reasons, to induce faint, the evaluation of the

system will be performed triggering externally the system.

The complete procedure of both use cases can be seen in Annex 1.


Fatigue: No additional task will be performed to induce extreme physical fatigue. The rider

will continue riding until this state is reached.

12.10Measurements

The measurement for this use case will be the same as for use case E.



13. Use case G – Passenger pick up/drop off automation for buses

The function developed for use case G supports bus drivers performing the passenger pick

up/drop off tasks. This function is activated when approaching the bus stop area, the so-called

Safety Zone Area. The system asks the driver if he/she is ready to hand over control to the

system. The driver responds by tapping the steering wheel in a specific way for a yes or a no.

If the driver agrees, the bus takes over the control and the light source at the driver’s seat shift

to automated driving. The bus is now, with help of automation, slowly stopped at the bus stop.

The driver is released from his task and can use his/her time to interact and communicate with

the passengers, handle ticketing, support passengers or rest. When all passengers have boarded

the bus asks the driver if it should depart. The bus driver agrees by tapping the steering wheel.

The light at the driver’s seat shifts to automation. Getting close to the exiting area of the Safety

Zone, the bus informs the driver that it is getting close to take over (transition back to driver).

The bus driver turns his/her head and gaze to the road and the system checks that he/she is alert

and attentive enough. Then, the system asks the driver to take over control. The driver accepts

by tapping the steering wheel, the light source turns to manual mode. If the driver is not detected

to be alert and attentive, the bus will slow down, and the driver is reminded to look ahead in

order to be able to take over the control. If no change is seen the bus stops. Figure 20 shows the

described events.

This function will be implemented in VTI’s driving simulator.

ADAS&ME (688900) Del No 7.1


Figure 20. Event chart for Passenger pick up/drop off automated system.

Driver

User

Interface

Personalization

Driver

monitoring

Environmental

monitoring

Decision system

Vehicle

Bus stop detected

Suggest hand over

(automatic to manual)

Choose

UI

modality

Capacitated driver Driver under

state X

Chose UI

modality

Send

Info

Automated driving

Warn driver +

State mitigation strategy

Passengers

ready

Info handover

possibility

Confirms + hand over

Manual driving

Vehicle ready

to start

departure

State X

Ready to takeover

Chose UI

modality

Allows takeover

Takeover

Manual

driving

Chose UI

modality

Informs

start departure

Confirms

departure




The use case G function is being developed for the detection of fatigue and distraction.

Furthermore, the levels of stress will also be measured. The following sections will give more

details about how these states will be induced and/or measured during this use case evaluation.


The evaluation focuses on the drivers interaction with the docking system both when giving the

control to the bus and when taking it back. Focus is on safe transitions and environmental

friendly driving, but also on less stress and fatigue among the bus drivers from a more generic

perspective. More specifically this evaluation aims at evaluating:

• The effect and reliability of the system for automated docking at bus stop with focus

on the effect on drivers states (fatigue and stress) and performance (acceleration,

deceleration and braking).

• The drivers’ reactions (eye gaze, HRV, GSR, acceleration, deceleration, braking)

during the transition both when they are detected to be ready to takeover the control

and when they are not.

• The driver’s opinion (acceptance, trust, system usability) about the automated docking

system based on questionnaires and interviews.


Use case G will be performed in VTI’s simulator (Sim II).

13.4 Time plan

This evaluation will be carried out during M39. All necessary preparations will be done

beforehand. A pre-pilot (stage 2 testing) will take place 2 months before in order to get time

enough to improve and correct the scenarios. During month 37 the technical preparation will

be finalized, and all participants will be invited to evaluate the system. The four weeks dedicated

to this test will be structured as follows (Table 18):

Table 18. Four week activity description for use case G


1 to 5 Finalisation of technical preparations and participants recruitment; Pre-test

6 - 19 Evaluation with real drivers

20 Equipment disassemble and Wrap up

Fourteen days will be dedicated to performing data acquisition. Real bus drivers will be invited



to test the function. The entire use case G procedure (Welcome/Briefing and Questionnaires;

Familiarisation; Driving Test; Debriefing) is foreseen to last no longer than 2 hours. Annex 1

gives more details on the duration of the sessions.


The use case G evaluation will have a within-subjects design. Two factors (each with two

conditions) will be used: 1) Driver state (alert vs fatigued); 2) System presence (with system vs

without system/baseline). The scenario will cover city bus driving including driving between

bus stops (free) and the bus stops. At a specific number of bus stops, drivers will use the ticket

machine as a 3) secondary task performance (not distracted vs distracted). The evaluation will

be composed of two sessions, performed in distinct days. In order to avoid bias due to learning

effect, sessions will be balanced across the sample. Table 19 presents a brief procedure of each

visit and a schema for the order of sessions.

Table 19. Use case G session procedure

Session A – the driver is fully

capacitated to drive (Driver state

condition: alert; about 120 min)

• Welcome + Briefing –20 min

• Familiarisation trial – 10 min

• 1st Test drive (one of the conditions System

presence) - 30 min

• 2nd Test drive (one of the conditions System

presence) – 30 min



influence of fatigue (about 120

min)


have to be scheduled for the

same period of the day (e.g.

beginning of the afternoon) due

to the influence of the circadian

rhythm.


• Familiarisation trial – 10 min

• 1st Test drive (one of the conditions System

presence) - 30 min

• 2nd Test drive (one of the conditions System

presence) – 30 min




participants will be recruited (estimated 20) in order to assure that exactly 16 data sets for each

state are available for analysis. Surrogate participants recruited to compensate a drop-out should

perform the conditions in the same order the dropped-out participant did. This will assure a

balanced distribution of the conditions over the sample.

The sample will be composed of drivers between 35 and 50 years-old that have been working

as bus drivers for at least five years. The sample will have an equal distribution of males and

females.



13.7 Procedure



how it will be used. Following, participants will be asked to fill in a demographic questionnaire

(Annex 3).

A training trial will be performed at the beginning, in order for participants to get acquainted

with the driving simulator while driving at bus stops and outside them. Both manual and

automation modes will be tested when approaching and leaving bus stops. During this phase

they will have the possibility to ask questions to the test leader.

Two distinct test drives will follow: one without the system (baseline) and another with the

system. The driving scenario simulates the city of Lund (Figure 21). Participants will drive a

bus route with about 10 bus stops to dock, with simulated passengers to drop off. There will be

passengers waiting at the bus stop, but not boarding. Participants will drive this simulated route

twice: one with the automated docking system activated and the other one without.

In order to evaluate the reliability of the system to detect if the driver is attentive enough to be

able to take back the control in a safe way, the driver will be asked to perform a secondary task

(simulating a ticketing task) on half of the bus stops (5). A detailed description of this task is

given in section 13.8 of this deliverable.

A scheme of the entire experiment design can be seen in Figure 22.

Figure 21. The city bus scenario of use case G.



Figure 22. Scheme of use case G experiment design


Distraction: The ticketing task will impose the driver to be distracted. This task will start after

the handover and will finish when the driver receives an information of the system to focus

his/her attention on the road (prior to the Annex - from automated to manual). The participants

will be informed about in which bus stops the task should be conducted.

Fatigue: Participants invited to perform the evaluation will come twice, both in the afternoon.

Once when they have been driving a morning part of a split shift schedule and once when they

have been of duty during the morning. The sounds in the simulator is as in a real bus, but

without passengers talking.

Stress: No task will be performed to induce stress. Stress is used as a dependent variable and

measured as a consequence of the conditions imposed to the experiment.

13.9 Measurements

The following section presents the use case G measurements that will be applied in three distinct

moments along the tests.


Upon arrival drivers will get information on the overall evaluation goal. During this briefing

moment they will fill in a demographic questionnaire (Annex 3) and a general questionnaire on

automated driving (Annex 4).




Table 20 presents the measurements that will be done while the driver is driving the simulator.

Table 20. Use case G metrics


Vehicle Data Time stamp Integer

Vehicle Data Speed Integer


Integer


Integer

Vehicle Data Automation status

Integer Verifies if automation is active or inactive

Vehicle Data Automation level Integer Inside the safety zone

Vehicle Data Transitions activation timing

Integer Timing for activation of the automation and manual driving

Driver Behaviour

Visual Behaviour

Integer SmartEye system for detection of the eye gaze and blink duration

Driver Behaviour

Heart Rate Integer Measured via steering wheel sensor

Driver Behaviour

Galvanic Skin Response

Integer Measured via steering wheel sensor

Driver Behaviour

Heart rate variability and Galvanic skin response

Integer Empatica wrist Watch

Driver Behaviour

Actions and communications

Video Hesitations, verbal communications that can give a better understanding about the transition difficulty

Driver State Fatigue status - Karolinska Sleepiness Scale

Every 5 min.

Electronic questionnaire

Measures the sleepiness level (Annex 8)



(KSS)

Driver State Stress status – Stress Scale

Every 5 min.

Electronic questionnaire

Measures the stress level (Annex 9)


Secondary Task Video To verify how many task units are performed (control measurement)


Timing Task Standby

Audio/video When is the state task interrupted in relation with the warning (control measurement)


For each of the sessions, after both drives are performed (with and without the system) the

driver will be debriefed. The following instruments will be used:






ADAS&ME (688900) QMR No [X]


14. Methodological considerations and data analysis

The analysis of the data collected in the ADAS&ME evaluations will be performed separately

for each use case. In accordance with the goals set for the evaluations, the data collected will

belong to three distinct general groups:

a) Technical data: data coming from the vehicle and in-vehicle system, which allow to

verify how well the system is able to recognise the environmental and driver state, and

also if the adequate warnings/instructions to mitigate the driver’s state are transmitted.

b) Behavioural data: coming from the way drivers/riders behave and react to the system’s

actions and messages. These can be the visual behaviour but also the timing chosen to

end a particular task or activate automation.

c) Subjective data: these data contains the users’ opinions regarding the usability of the

system and also the general acceptance and trust on the tested systems.

A preference is given to a within-subject design for all use cases, which means that participants

will experience all conditions imposed to testing the systems. The need of testing the system

under the influence of several conditions (states) and the difficulty of testing all these condition

in one unique day, led to the option of performing the evaluations of some use cases in several

sessions. However, this represents an increased difficulty not only for the selection and

recruitment of participants, but also because the probability of drop-out is high. In case the

following evaluation preparations show us that these and other unforeseen constraints will not

allow to apply the methodology presented in this document two alternative plans are presented.

These plans contain some changes in the experiment design and performance of conditions for

three use cases (A, C and D). They will allow, in case of unavoidable restrictions to apply the

initial evaluation plan, evaluations to be done in the planned timeframe and with the expected

quality.

Plan B. Replacement of within-subject design by mixed-subject design for use case A and

C

The three sessions planned to be done by all participants will be separated among two sample

groups. One group performs the fatigue sessions and the other is invited to come twice and

evaluate the system in the other two sessions. This scheme is presented in Figure 23.

Plan C. Driver state conditions distributed among use cases

Each use case will not test all different conditions, but just a part of them. In any case, no

conditions will be left to test, as all will be integrated in the evaluations (but in different use

cases). The scheme presented in Figure 23 clarifies this plan. A further simplification could be

the integration of UC D test into the session A, which would mean that participants would only

have to visit IDIADA once to evaluate UC C and D functions.



Figure 23. Plan B and C for the application of the methodology

Data collection and analysis will be conducted separately for each use case. Even though, due

to similar methodological base, specific comparisons among use cases will be possible. The

amount of data collected, complexity and time constraints will not allow a transversal

comparison of all data types among use cases. Differences in the application of certain methods

would also make such comparisons unsuitable to make general conclusions. However,

subjective data, collected through the application of the exact same scales, can be put together

to give a general overview of participants` acceptance, trust, potential system usage and

acquisition, and also considerations about the systems usability.

Session A

(fatigue)

Session B

(distraction + emotion)

Session C

(baseline + stress)

Sam

e sa

mp

le

Use Case A Use Case C & D

Session A

(fatigue)

Session B

(UC D + emotion)

Session C

(distraction + stress)

Sam

e sa

mp

le

Ori

gin

al p

lan

Session A

(fatigue)

Session B


Session C

(baseline + stress)

Sam

ple

A Session A

(fatigue)

Session B

(UC D + emotion)

Session C

(distraction + stress)

Pla

n B

Sam

ple

B

Sam

ple

A

Sam

ple

B

Sam

e sa

mp

le

Sam

e sa

mp

le Session A

(fatigue)

Session B

(baseline + stress)

Session B

(UC D)

Session A


Pla

n C



15. Ethical processes

This section is meant to give a general overview of ethical and legal principles so that a research

in accordance with best practices would be conducted. For all evaluations the framework

described in D10.3 ADAS&ME Ethics Manual needs to be followed, together with what is

written in D11.1-D11.3.

“Ethics: Norms for conduct that distinguish between acceptable and unacceptable behaviour”

This is the most common way of defining "ethics". As researchers we need to pursue an

acceptable behaviour for our colleagues and the group of participants involved.

In following paragraph, with the intention of contextualise this sensitive issue, it is gathered a

rough and general summary of some ethical principles that address various codes (Shamoo &

Resnik, 2015).

During the assessment project we have attempted to keep these principles in mind and enforce

them for the good of the project.

Honesty: Strive for honesty in all scientific communications. Honestly report data, results,

methods and procedures, and publication status. Do not fabricate, falsify, or misrepresent data.

Do not deceive colleagues, research sponsors, or the public.

Objectivity: Strive to avoid bias in experimental design, data analysis, data interpretation, peer

review, personnel decisions, grant writing, expert testimony, and other aspects of research

where objectivity is expected or required. Avoid or minimize bias or self-deception. Disclose

personal or financial interests that may affect research.

Integrity: Keep your promises and agreements; act with sincerity; strive for consistency of

thought and action.

Carefulness: Avoid careless errors and negligence; carefully and critically examine your own

work and the work of your peers. Keep good records of research activities, such as data

collection, research design, and correspondence with agencies or journals.

Openness: Share data, results, ideas, tools, resources. Be open to criticism and new ideas.

Respect for Intellectual Property: Honour patents, copyrights, and other forms of intellectual

property. Do not use unpublished data, methods, or results without permission. Give proper

acknowledgement or credit for all contributions to research. Never plagiarise.

Confidentiality: Protect confidential communications, such as papers or grants submitted for

publication, personnel records, trade or military secrets, and patient records.

Responsible Publication: Publish in order to advance research and scholarship, not to advance

just your own career. Avoid wasteful and duplicative publication.

Responsible Mentoring: Help to educate, mentor, and advise students. Promote their welfare

and allow them to make their own decisions.

Respect for colleagues: Respect your colleagues and treat them fairly.



Social Responsibility: Strive to promote social good and prevent or mitigate social harms

through research, public education, and advocacy.

Non-Discrimination: Avoid discrimination against colleagues or students on the basis of sex,

race, ethnicity, or other factors not related to scientific competence and integrity.

Competence: Maintain and improve your own professional competence and expertise through

lifelong education and learning; take steps to promote competence in science as a whole.

Legality: Know and obey relevant laws and institutional and governmental policies.

Human Subjects Protection: When conducting research on human subjects minimize harms

and risks and maximize benefits; respect human dignity, privacy, and autonomy; take special

precautions with vulnerable populations; and strive to distribute the benefits and burdens of

research fairly.

15.1 Ethical processes in all stages of methodology: HUMAN PARTICIPANTS

The main role of human participants in research is to serve as sources of data. Researchers have

a duty to ‘protect the life, health, dignity, integrity, right to self-determination, privacy and

confidentiality of personal information of research subjects’. In essence, respect for person,

beneficence and justice5.

In ADAS&ME a consensus procedure will be followed to get the expected participation.

Through a job advertisement as the unique way to get involve as a participant in our project, it

is possible to get involved people with the needed profile respecting the voluntary participation

principle.

In the same way, informed consent is essential so participants must be fully informed about

the procedures and risks involved in research and must give their consent to participate.

Ethical standards also require that researchers do not put participants in a situation where they

might be at risk of harm as a result of their participation. Harm can be defined as both physical

and psychological. That is why we provide a brief talk and documentation regarding

occupational risk issues with very good scores.

There are two standards that are applied in order to help protect the privacy of research

participants. Almost all research guarantees the participants’ confidentiality -- they are assured

that identifying information will not be made available to anyone who is not directly involved

in the study.

5 The tree ethical principles for using any human subjects for research. Belmont Report.

30/09/1978



16. Reference documents

Åkerstedt, T. & Gillberg, M. (1990) Subjective and Objective Sleepiness in the Active

Individual, International Journal of Neuroscience, 52:1-2, 29-37,

DOI:10.3109/00207459008994241

Åkerstedt, T., Anund, A., Axelsson, J., Kecklund, G. (2014). Subjective sleepiness is a

sensitive indicator of insufficient sleep and impaired waking function. Journal of Sleep

Research, 23 (3), 242-254

Anund, A., Ihlström, J., Fors, C., Kecklund , G., & Filtness, A. (2016). Factors associated

with self-reported driver sleepiness and incidents in city bus drivers. INDUSTRIAL

HEALTH, 54, 1-10.

Brooke, J. (1996). SUS: A quick and dirty usability scale. In Jordan, P.W., Thomas, B.,

Weerdmeester, B. A., McClelland, I. L. (Eds.), Usability Evaluation in Industry (pp. 189-

194). London: Taylor & Francis.

Dahlgren A, Kecklund G, Åkerstedt T. Different levels of work-related stress and the

effects on sleep, fatigue and cortisol. Scandinavian Journal of Work, Environment &

Health, 2005, 31(4), 277-285

Damböck, D.; Bengler, K.; Farid, M.; Tönert, L.: Übernahmezeiten beim

hochautomatisierten Fahren. Tagungsband der VDI-Tagung Fahrerassistenz in München,

Jahrgang 15, Seite 16ff, 2012

Franke, T., Attig, C., Wessel, D. (2017). Affinity for technology interaction - a personal-

resource perspective. Paper presented at the Human Factors and Ergonomics Society

Europe Chapter 2016 Annual Conference. Rome, Italy.

Gold, C.; Damböck, D.; Lorenz, L.; Bengler, K. (2013): “Take over!” How long does it

take to get the driver back into the loop? In Proceedings of the Human Factors and

Ergonomics Society Annual Meeting, Vol. 57, No. 1, pp. 1938–1942

Ihlström, J., Kecklund , G., & Anund, A. (2017). Split-shift work in relation to stress,

health and psychosocial work factors among bus drivers. Work, 56, 531–538.

doi:DOI:10.3233/WOR-172520;

ISO14198. (2012). Road vehicles - Ergonomic aspects of transport information and

control systems - Calibration tasks for methods which asses driver demand due to the use

of in-vehicle systems. ISO/TS 14198:2012(en).

Jian, Jiun-Yin, Bisantz, Ann M. and Drury, Colin G. (2000). Foundations for an

Empirically Determined Scale of Trust in Automated Systems. International Journal of

Cognitive Ergonomics, 4(1), 53-71.



Mehler, B., Reimer, B., Dusek, J.A. (2011). MIT AgeLab Delayed Digit Recall Task (n-

back). MIT AgeLab white paper 2011-3B.

Philips, R., & Björnskau, T. (2013). Health, safety and bus driveres (1279-2014).

Shamoo, A. & Resnik, D. (2015). Responsible Conduct of Research. New York : Oxford

University Press.

Taylor, R. M. (1990). Situation awareness rating technique (SART): the development of

a tool for aircrew systems design. In: Situational Awareness in Aerospace Operations

(Chapter 3). France: Neuilly sur-Seine, NATO-AGARD-CP-478.

Tse, J. L. M., Flin, R., & Mearns, K. (2006). Bus driver well-being review: 50 years of

research. Transportation Research Part F: Psychology and Behaviour, 9(2), 89-114.

doi:10.1016/j.trf.2005.10.002.

Van Der Laan, J. D., Heino, A., & De Waard, D. (1997). A simple procedure for the

assessment of acceptance of advanced transport telematics. Transportation Research Part

C: Emerging Technologies, 5(1), 1-10.

Annexes

List of Annexes

Annex 1. Order of sessions

Annex 2. Recruitment questionnaire

Annex 3. Demographics questionnaire

Annex 4. General questionnaire on automated driving/riding

Annex 5. Acceptance scale

Annex 6. Trust scale

Annex 7. System Usability Scale (SUS)

Annex 8. Karolinska Sleepiness Scale

Annex 9. Stress Scale

Annex 10. Questionnaire on potential system usage and acquisition

Annex 11. Debriefing interview

Annex 12. Tasks to induce states

Annex 13. IDIADA’s Proving Ground

Annex 14. Test Case Template

Annex 1 . Order of sessions

Use case A

Order of sessions

Participant Nr. Order of session

(1st visit) > (2nd visit) > (3rd visit)

1 Session A > Session B > Session C


3 Session A > Session C > Session B


5 Session B > Session A > Session C


7 Session B > Session C > Session A


9 Session C > Session A > Session B


11 Session C > Session B > Session A


Use case C and D

Order of sessions

Participant Nr. Order of session

(1st visit) > (2nd visit) > (3rd visit)













Use case G

Order of sessions

Participant Nr. Order of sessions

(1st visit) > (2nd visit)

1 Session A (Baseline-System) – Session B (System- Baseline)




5 Session A (System - Baseline) – Session B (Baseline - System)




9 Session B (Baseline-System) – Session A (System- Baseline)




13 Session B (System - Baseline) – Session A (Baseline - System)




Annex 2. Recruitment questionnaire

[Partner LOGO]

Recruitment questionnaire

Evaluation name: ___________________________________ Project UC: ___________

Responsible: _______________________________________ Participant ID:_________

Thank you so much for filling in this questionnaire. In order to perform the evaluation of the

ADAS&ME system we need to select participants with certain characteristics. The information

provided by you will be important to perform this recruitment. Please, in case you have any

question, do not hesitate to ask.

Please, fill in the following fields:

1. Gender

Female Male

2. Age

_____________ Years

3. Do you take any medication?

Yes No

4. If yes, please specify which.

___________________________________________________________________________

5. For how long do you hold a VEHICLE 6driving license?

_____________ Years

6. How frequently do you drive a VEHICLE

Every day On weekdays On weekends Every month Seldom

7. How many kilometres have you driven a VEHICLE in the past 12 month?

_____________ Km

8. Have you ever used an adaptive driver assistance system in a VEHICLE or a passenger

car? (E.g. cruise control, speed limit, night vision system, lane keeping system, etc.)

Yes No

6 VEHICLE should be replaced by the vehicle’s name in focus for the evaluations, i.e. “a

passenger car”/ “motorcycle”/”truck”/”bus”

[Partner LOGO]

9. If yes, please indicate which one(s). (Please write the name of the system(s) or make a

description of what the system(s) was able to do)

__________________________________________________________________________

10. Have you used the system more than once?

Yes No

11. How often do you use this system(s)?

Every day On weekdays On weekends Every month Seldom

12. To finalize, we would like to ask you to complete the following table, which contains

statements about your interaction with technical systems. The term ‘technical systems’

refers to apps and other software applications, as well as entire digital devices (e.g. mobile

phone, computer, TV, car navigation).7

Please indicate the degree to which you agree/disagree with the

following statements. co

mple

tely

dis

agre

e

larg

ely

dis

agre

e

slig

htl

y

dis

agre

e

slig

htl

y

agre

e

larg

ely

agre

e

com

ple

tely

ag

ree

01 I like to occupy myself in greater detail with technical

systems.

02 I like testing the functions of new technical systems.

03 I predominantly deal with technical systems because I have

to.

04 When I have a new technical system in front of me, I try it

out intensively.

05 I enjoy spending time becoming acquainted with a new

technical system.

06 It is enough for me that a technical system works; I don’t

care how or why.

07 I try to understand how a technical system exactly works.

08 It is enough for me to know the basic functions of a

technical system.

09 I try to make full use of the capabilities of a technical

system.

7 Franke, T., Attig, C., Wessel, D. (2017). Affinity for technology interaction - a personal-

resource perspective. Paper presented at the Human Factors and Ergonomics Society Europe

Chapter 2016 Annual Conference. Rome, Italy.

[Partner LOGO]

In case you are selected to the test, we will contact you. Please, indicate fill in the following

fields:

Name:_______________________________________________________________

Telefone:_____________________________________________________________

E-Mail:______________________________________________________________

If you have questions please contact: [email protected]

Thank you for your time!

Annex 3. Demographics questionnaire

[Partner LOGO]

Demographics questionnaire


Responsible: _______________________________________ Participant ID:_________

Thank you so much for participating in the ADAS&ME system evaluation. The following

questionnaire will allow us to collect important information about you and your experience

with the driving/riding task and with other in-vehicle systems. In case you have any question,

do not hesitate to ask.

Please, fill in the following fields:

1. Gender

Female Male

2. Age

_____________ Years

3. What is your highest education level? (E.g. University degree, High school degree …)

___________________________________________________________________________

4. For how long do you hold a VEHICLE8 driving license?

_____________ Years

5. What type of category does your driving license apply for?

(A) Motorcycle (B) Passenger car (C) Lorry (D) Bus (E) Special trailer

6. Please indicate in the following table (and each type of vehicle) how often you drive it.

Everyday Few times

a week

Few times

a month Seasonally Rarely Once Never

Car

Small

Mid-size

Large

Luxury

Sport

Electric

Other:

Motorcycle

Naked

Sport

Touring

Cruiser

Off road

8 VEHICLE should be replaced by the vehicle’s name in focus for the evaluations, i.e. “a

passenger car”/ “motorcycle”/”truck”/”bus”

[Partner LOGO]

Other:

Truck

Long-

haulage

Distribution

Construction

Other:

Bus

City bus

Coach

Other:

7. How many kilometres have you driven with your VEHICLE in the past 12 month?

_____________ Km

8. Have you ever used an in-vehicle assistance system in a VEHICLE?

(E.g. cruise control, speed limit, night vision system, lane keeping system, etc.)

Yes No

9.Please indicate in the following table the systems you have used in the past, and respective

frequency of use.

Systems Never Seldom Often Always

Cruise Control

(maintain selected speed)

Adaptive Cruise Control (ACC)

(keep both speed and distance to previous vehicle)

Intelligent Speed Adaptation (ISA)

(warns when driving too fast)

Lane Departure Warning (LDW)

(warns when passing the middle line)

Lane Keeping Assist (LKA)

(like LDW but can also steer your vehicle back into

your lane if you do not react to the warning)

Forward Collision Warning (FCW)

(warns when the collision risk is close with car in

fronts)

Blind Spot Information System (BLIS)

(informs about the presence of a vehicle in the blind

sport area)

Driver Alert – sleepiness warning

(warns the driver when falling asleep)

Night vision

[Partner LOGO]

(helps the driver to detect VRUs or animals on the road

when it is dark)

Parking assist

(helps the driver to park the car)

Traction/wheelie control

(only for motorcycle)

Motorcycle stability control

(only for motorcycle)

Other:

Thank you for filling in the questionnaire.

Annex 4. General questionnaire on automated driving/riding

[Partner LOGO]

General questionnaire on automated driving/riding


Responsible: _______________________________________________________________

Participant ID:_____________ Session Nr. __________

1. What is your general opinion about having in the vehicle functions that can automate parts

or the entire driving/riding task? (tick just one)

Very positive Somewhat

positive Neutral

Somewhat

negative Very negative

2. Indicate in which environment or traffic situations you would like to use automated

driving/riding functions? (tick as much as you which)

Urban/city centre

Rural roads

Motorways

Other types of roads: ___________________________________________________

Unfamiliar roads

Familiar roads

Daily driven roads

Sunny weather conditions

Raining

Snowing

Traffic jams

Overtaking manoeuvres

Lane changing manoeuvres

Car following situations

Parking manoeuvres

Stop at traffic signs

Driving at night

Interpretation of road signs

Abrupt braking

Collision avoidance

Heavy traffic

Light traffic

Manage conflict situations (with pedestrians/other vehicles)

Other manoeuvres: _____________________________________________________

(Options continue next page)

[Partner LOGO]

When feeling tired/sleepy

When feeling frustrated/irritated

In case additional tasks, other than driving, need to be performed

When feeling fully capacitated to drive

Other states: __________________________________________________________

3. Please indicate how much do you agree with each of the following statements.

com

ple

tely

dis

agre

e

larg

ely

dis

agre

e

slig

htl

y

dis

agre

e

slig

htl

y

agre

e

larg

ely

agre

e

com

ple

tely

agre

e

Automated functions can help me in case I am tired

Automated functions can help me in case I am sleepy.

Automated functions can help me in case I am

distracted.

Automated functions can help me in case I am irritated.

Automated functions can help me in case I am angry.

Automated functions can help me in case I am

frustrated.

Automated functions can help me in case I am fully

capacitated to drive.

4. Several aspects are listed in the following table. Please indicate how these aspects can

change in case automated functions are used.

Decrease Stays the

same Increase

Safety

Number of accidents

Accidents’ severity

Comfort

Attention towards the road

Boredom

Happiness

Stress

[Partner LOGO]

Fatigue

Possibility to drive longer

Possibility to relax/rest

Possibility to do other tasks

Possibility to drive when driver is not fully capacitated

Possibility to drive longer (more years)

Other:_______________________________________

5. Please, evaluate automated functions in general by placing a cross per line, i.e., between

each of dimension pairs9.

Useful Useless

Pleasant Unpleasant

Bad Good

Nice Annoying

Effective Superfluous

Irritating Likeable

Assisting Worthless

Undesirable

Desirable

Raising Alertness Sleep-inducing

9 Source: Van der Laan, J.D., Heino, A., & De Waard, D. (1997). A simple procedure for the

assessment of acceptance of advanced transport telematics. Transportation Research - Part C:

Emerging Technologies, 5, 1-10.

(http://www.hfes-europe.org/accept/accept.htm)

[Partner LOGO]

6. Below is a list of statements for evaluating your general impression on systems that

automate the driving/riding task. Please, mark an “X” on each line at the point which

best describes your feeling10.

The systems are deceptive

1

Not at all

2 3 4 5 6 7

Extremely

The systems behave in an underhanded manner

1

Not at all

2 3 4 5 6 7

Extremely

I am suspicious of the systems’ intent, action, or outputs

1

Not at all

2 3 4 5 6 7

Extremely

I am wary of the systems

1

Not at all

2 3 4 5 6 7

Extremely

The systems’ actions will have a harmful or injurious outcome

1

Not at all

2 3 4 5 6 7

Extremely

I am confident in the systems

1

Not at all

2 3 4 5 6 7

Extremely

The systems provide security

1

Not at all

2 3 4 5 6 7

Extremely

The systems have integrity

10 Source: Jian, Jiun-Yin, Bisantz, Ann M. and Drury, Colin G. (2000). Foundations for an



[Partner LOGO]

1

Not at all

2 3 4 5 6 7

Extremely

The systems are dependable

1

Not at all

2 3 4 5 6 7

Extremely

The systems are reliable

1

Not at all

2 3 4 5 6 7

Extremely

I can trust the systems

1

Not at all

2 3 4 5 6 7

Extremely

I am familiar with the systems

1

Not at all

2 3 4 5 6 7

Extremely

Thank you for filling in the questionnaire.

Annex 5. Acceptance scale

[Partner LOGO]

Acceptance scale11

Please, evaluate the system you have just tested by placing a cross per line, i.e., between each

of dimension pairs.

Useful Useless

Pleasant Unpleasant

Bad Good

Nice Annoying

Effective Superfluous

Irritating Likeable

Assisting Worthless

Undesirable

Desirable

Raising Alertness Sleep-inducing

11 Source: Van der Laan, J.D., Heino, A., & De Waard, D. (1997). A simple procedure for the

assessment of acceptance of advanced transport telematics. Transportation Research - Part C:

Emerging Technologies, 5, 1-10.

(http://www.hfes-europe.org/accept/accept.htm)

Annex 6. Trust scale

[Partner LOGO]

Trust in Automation12

Below is a list of statements for evaluating your general impression on the system you have

just tested. Please, mark an “X” on each line at the point which best describes your feeling.

The system is deceptive

1

Not at all

2 3 4 5 6 7

Extremely

The system behaves in an underhanded manner

1

Not at all

2 3 4 5 6 7

Extremely

I am suspicious of the system’s intent, action, or outputs

1

Not at all

2 3 4 5 6 7

Extremely

I am wary of the system

1

Not at all

2 3 4 5 6 7

Extremely

The system’s actions will have a harmful or injurious outcome

1

Not at all

2 3 4 5 6 7

Extremely

I am confident in the system

1

Not at all

2 3 4 5 6 7

Extremely

The system provides security

1

Not at all

2 3 4 5 6 7

Extremely

The system has integrity

1

Not at all

2 3 4 5 6 7

Extremely

12 Source: Jian, Jiun-Yin, Bisantz, Ann M. and Drury, Colin G. (2000). Foundations for an



[Partner LOGO]

The system is dependable

1

Not at all

2 3 4 5 6 7

Extremely

The system is reliable

1

Not at all

2 3 4 5 6 7

Extremely

I can trust the system

1

Not at all

2 3 4 5 6 7

Extremely

I am familiar with the system

1

Not at all

2 3 4 5 6 7

Extremely

Annex 7. System Usability Scale (SUS)

[Partner LOGO]

System Usability Scale (SUS)13

Please check the box that reflects your immediate response to each statement. Make sure you

respond to every statement.

Strongly

disagree

Strongly

agree

I think that I would like to use this system

frequently 1 2 3 4 5

I found the system unnecessarily complex 1 2 3 4 5

I thought the system was easy to use 1 2 3 4 5

I think that I would need the support of a

technical person to be able to use this system 1 2 3 4 5

I found the various functions in this system

were well integrated 1 2 3 4 5

I thought there was too much inconsistency

in this system 1 2 3 4 5

I would imagine that most people would

learn to use this system very quickly 1 2 3 4 5

I found the system very cumbersome to use 1 2 3 4 5

I felt very confident using the system 1 2 3 4 5

I needed to learn a lot of things before I could

get going with this system 1 2 3 4 5

13 Brooke, J. (1996). SUS: a „quick and dirty‟ usability scale. In P.W.Jordan, B. Thomas,

B.A. Weerdmeester, and I.L. McClelland (Eds.) Usability Evaluation in Industry (189-194).

London: Taylor and Francis.

Annex 8. Karolinska Sleepiness Scale

Karolinska Sleepiness Scale14

14 Åkerstedt, T. & Gillberg, M. (1990) Subjective and Objective Sleepiness in the Active

Individual, International Journal of Neuroscience, 52:1-2, 29-37,

DOI:10.3109/00207459008994241

Åkerstedt, T., Anund, A., Axelsson, J., Kecklund, G. (2014). Subjective sleepiness is a sensitive

indicator of insufficient sleep and impaired waking function. Journal of Sleep Research, 23 (3),

242-254

Please, circle the number that represents your sleepiness level during the last 5 minutes.

1 extremely alert

2 very alert

3 alert

4 rather alert

5 neither alert nor sleepy

6 some signs of sleepiness

7 sleepy, but no effort to keep alert

8 sleepy, some effort to keep alert

9 very sleepy, great effort to keep alert, fighting sleep

Annex 9. Stress Scale

Stress Scale15

15 Dahlgren A, Kecklund G, Åkerstedt T. Different levels of work-related stress and the effects

on sleep, fatigue and cortisol. Scandinavian Journal of Work, Environment & Health, 2005,

31(4), 277-285.

How do you rate you stress level during the last minutes?

Please, circle the respective number. You may also use the intermediate steps.

1 Very low stress (very calm and relaxed)

2

3 Low stress (calm and relaxed)

4

5 Neither low or high stress

6

7 High stress (high tension and pressure)

8

9 Very high stress (very high tension and pressure – on the verge what I can tolerate)

Annex 10. Questionnaire on potential system usage and acquisition

[Partner LOGO]

Questionnaire on potential system usage and purchase

1. What is your general opinion about the system you have just tested? (tick just one)

Very positive

Somewhat

positive

Neutral Somewhat

negative Very negative

2. Indicate in which environment or traffic situations you would use this system (tick as

much as you which)

Urban/city centre

Rural roads

Motorways

Other types of roads: ___________________________________________________

Unfamiliar roads

Familiar roads

Daily driven roads

Sunny weather conditions

Raining

Snowing

Traffic jams

Overtaking manoeuvres

Lane changing manoeuvres

Car following situations

Parking manoeuvres

Stop at traffic signs

Driving at night

Interpretation of road signs

Abrupt braking

Collision avoidance

Heavy traffic

Light traffic

Manage conflict situations (with pedestrians/other vehicles)

Other manoeuvres: _____________________________________________________

When feeling tired/sleepy

When feeling frustrated/irritated

In case additional tasks, other than driving, need to be performed

When feeling fully capacitated to drive

Other states: __________________________________________________________

[Partner LOGO]

3. How frequently would you activate this system? Tick one box.

Never Always

4. As you may know from other systems, sometimes false alarms/warnings are sent. For

which of the following percentages of false alarms would you buy this system?

0 % false alarms

0 - 1 % false alarms







5. Several aspects are listed in the following table. Please indicate how these aspects can

change in case drivers use the system you have just tested.

Decrease

Stays the

same Increase

Safety

Number of accidents

Severity of the accidents

Comfort

Attention towards the road

Boredom

Happiness

Stress

Fatigue

Possibility to drive longer (longer trips)

Possibility to drive longer (more years)

Other:____________________________________

[Partner LOGO]

7. Would you like to have this system in your vehicle?

Yes

No

Only if _____________________________________________________________

8. Would you buy a vehicle with such a system?

Yes

No

Only if ________________________________________________________

9. Using a scale of 1 to 5, where 1 is ‘Not at all’ and 5 is ‘Completely’, how much would

the following aspects may affect your overall intention to purchase the system?

The cost ______

More freedom to do other things while on my trips ______

Increase of personal safety ______

Time savings ______

Comfort ______

High level of technology ______

Others: ______________________________________________________________

10. Would the convenience justify a big increase in the cost of this system?

Yes

No

I don’t know

11. How much would you be willing to pay to have this system?

__________ , _____ €

Annex 11. Debriefing interview

Debriefing interview

Use case A Questions

1. How do you think the evaluation went?

2. What is your opinion now about automated driving?

3. Do you have any questions about the system and how it works?

4. Do you know if something caused the system to send those messages?

5. Note: In case the driver is not informed that the system is able to recognise the driver

states

6. What do you think about the ability of the system to recognize the driver’s state?

7. Note: If driver has to perform more than one evaluation, consider doing this question

only in the last session.

8. Were the messages sent by the system clear?

9. Note: If needed, specify per message or interface element

10. What was, in your opinion, unclear/difficult to understand?

11. Have you noticed any false alarms/incongruences of the system?

12. What went wrong/ what could have been better to improve this experience?

13. How can the system be improved?

14. Do you have any further comments on how the system works or on how the

information should be presented?

Use case B Questions

1st and 2nd test cases (public road driving test)





Note: If needed, specify per message or interface element

5. Were you worried/anxious during the trip about having enough range? If yes, where

exactly?

Note: present to the driver a scheme of the route and the moments where the

messages/warnings were displayed.

6. Did the messages/information on the display help to reduce your anxiety



9. What could have been better to improve this experience?




3rd test case (test-track)

1. How did you feel during the evaluation?


3. Do you have any questions about this function and how it works?

4. Was the intention of the system to perform an emergency stop clear?








Use case C Questions





Note: In case the driver is not informed that the system is able to recognise the

driver states


Note: If driver has to perform more than one evaluation, consider doing this

question only in the last session.









Use case D Questions





a. Note: If needed, specify per message or interface element







Use case E Questions


2. What is your opinion now about automated riding?



Note: In case the rider is not informed that the system is able to recognise the

driver states

5. What do you think about the ability of the system to recognize the rider’s state?

Note: If driver has to perform more than one evaluation, consider doing this










Use case F Questions












Use case G Questions




a. Note: If driver has to perform more than one evaluation, consider doing this










Annex 12. Tasks to induce states

Verbal response delayed digit recall task (n-back)

This task will be used to induce stress during the ADAS&ME evaluations. This task consists of

simple auditory stimuli that the driver listens to and repeats back following specific rules. This

task can have several levels of demand, which go from very mild (0-back) to high level (2-

back). For purposes of ADAS&ME evaluations the 2-back task will be used. This task will be

explained to the participants while the vehicle is standing still. The driving test will only start

when the driver has perfectly understood the auditory task.

Based the guidelines by Mehler, Reimer, & Dusek, (2011)16, this task will be explained using

the following instructions:

“While driving, we will ask you to perform an auditory task. You will listen to a list of single

digit numbers from 0 to 9. As this list is being read, you are to repeat out lout the number that

was read two numbers ago. For example, if I were to say the number 3, you would say nothing,

then if I say the number 2, you would say nothing, then if I say 6, you would say 2, and so on”

A scheme of the task can also be showed for a faster understanding.

Experimenter Driver

3 -

2 -

6 3

4 2

8 6

9 4

0 8

5 9

7 0

Note: If the experimenter verifies that the driver has difficulties in understanding the 2-back

task, the 0-back and the 1-back task can be primarily explained.

16 Mehler, B., Reimer, B., Dusek, J.A. (2011). MIT AgeLab Delayed Digit Recall Task (n-

back). MIT AgeLab white paper 2011-3B.

The Surrogate reference task (SuRT)

The Surrogate reference task (SuRT), described in the ISO/TS 14198:201217, is a visual search

and response task. Participants are to interact with a screen that displays several circles and

asked to look for a larger circle among other smaller distractor circles. After detection of the

target circle, participants have to press the right or left key of a numeric keypad thus inducing

a visual cursor moving to the target circle. This task can have distinct difficulty levels: from

easy (large difference in size between target and distractor circles) to hard (small difference in

size between target and distractor circles). The figures below show two examples of SuRT

displays. The first figure presents an easier level then the second one.

17 ISO14198. (2012). Road vehicles - Ergonomic aspects of transport information and control

systems - Calibration tasks for methods which asses driver demand due to the use of in-vehicle

systems. ISO/TS 14198:2012(en).

Annex 13. IDIADA’s Proving Ground

IDIADA’s Proving Ground

Inaugurated in 1994, Applus IDIADA proving ground is the most comprehensive independent

proving ground in Europe. It offers the highest level of customer support combined with first-

class test tracks and fully-equipped confidential workshops. Excellent climatic conditions allow

for year-round testing. The allocation of a dedicated customer service representative for each

testing team ensures that the testing program runs smoothly and that objectives are achieved on

time and within budget. The complex is designed to easily monitor all movement.

GENERIC MAP (Caption)

0. General Road

1. High speed Track

2. Noise Track

3. Fatigue Track/Comfort Track

4. Dynamic Platform A

5. Dry Handling Circuit/ Dynamic Platform C

6. Test Hills

7. Straight Line Braking Surfaces/Comfort Track and SIM city

9. Dynamic Platform B

10. Off road/Forest Track

11. Wet Circle

12. Wet Handling Circuit

IDIADA’s proving ground is recognized as one of the best facilities in the world, and is

renowned for the quality of its customer service. As a multi-user facility, safety and

confidentiality are of the highest priority.

The proving grounds of Applus IDIADA are equipped with a total of 10 Road-Side Unites

providing V2X ITS-G5 full coverage over the whole area.

PROVING GROUND and ADAS&ME

ADAS&ME evaluation test sites will require test tracks for highway and semi-urban

environments and public roads.

The test tracks used for the execution of the Test Cases will be at IDIADA’s proving ground.

As a multiuser facility, safety and confidentiality are of the highest priority.

The demonstrators will be carried out both in public roads and test tracks.

The highway scenarios will be implemented in the High Speed Track, consisting of an oval track

with 4 lanes, 2 km straight and 200 km/h neutral speed.

The semi-urban scenarios will be implemented in the dynamic platforms that consist of

completely flat asphalt surfaces, the High Speed Track and public roads (if needed).

The use of these facilities will ensure the safe testing of the performance of the technologies

with lower TRLs (Technology Readiness levels).

Such scenarios will be configured with all the elements needed for the right execution of the

Test Cases.

Below are detailed the tracks, platforms and their main features concerning dimensions and

other singular aspects:

GENERAL ROAD




Longitudinal gradient (south straight and braking

area) 0 %


Braking area (with) 20 m

Main use:

1. Access to other tracks and platforms

2. Check sensors and instrumentation tools

3. TEST (UCA, UC B, UC C+D, UC E)

HIGH-SPEED TRACK






Length of straights 2.000 m

Neutral steer speed 200 Km/h

Maximum banking bend 80% (38.66º)

Radius of the bends 471 m

Longitudinal gradient (straights) 0.3%

Transverse gradient (straights) 1.0%

Main use:

1. TEST (UCA, UC B, UC C+D, UC A, UC E)

DYNAMIC PLATFORM A




Longitudinal gradient (south straight and braking area) 0 %


Braking area (with) 20 m

DYNAMIC PLATFORM C

Direction of travel Anti-clockwise


Length of optional circuits 1.770 m

Width 7 m

Adherence coefficient 0.8

Lane changes section 300 m x 40 m

FATIGUE TRACK A

Direction of travel Clockwise / Anti-clockwise

Undulating concrete 145 m

Potholes with water 220 m

Gravel area 455 m

Undulating concrete with stones 420 m

Water wade (h: adjustable) 50 m x 4 m x (0 to 50 cm)

Salt water wade (h: adjustable) 20 m x 4 m x (0 to 50 cm)

Forest track 1.950 m

Block pavé road I 827 m

Block pavé II 922 m

Repaired asphalt 149 m

Stop and go area 300 m

Curbs 250 mm - 200 mm - 150 mm - 100

mm

Main use:

1. TEST (UC E+F)

PARKINGS/SAFETY AREA/REST ZONE

All around the proving ground there are several AREAS assigned for checking results, rest,

prepare conditions, take notes, etc.

In High Speed Track there are two Stop Zones in curves (12 x 100 m), two safety Stop Zones

in the south straight line (3 x 50 m), one Safety Stop Zones in the North straight line and a Stop

Zone in the main entrance (15 x 120 m).

Furthermore, there are two canopies for confidentiality, one in main entrance and another one

in the East curve.

Main use:

1. In case of emergency, changes, incidences, etc.

2. Entrance or exit to the Highway

3. Test Cases needs: It is possible to use for MRM or as a step of the story-board

CONTROL TOWER

The Proving Ground Controller is based in the Control Tower and is the principle contact for

Proving Ground users.

The Controller is responsible for managing and controlling traffic within the Proving Ground

and ensuring compliance with the Proving Ground Driving and Safety Regulations.

The Controller is responsible for coordinating internal and external emergency teams in any

emergency situation that may take place within the Proving Ground.

The Controller is the maximum authority of all traffic management within the Proving Ground:

his or her instructions must be followed at all times.

Main use:

1. Meeting point to: share material, give instructions, trainings, sample and expert roles

distribution, etc.

SERVICE STATION

Main use:

Supplying regular fuels

CHARGING FACILITY

Main use:

Supplying Electricity

WORKSHOPS

Main use:

1. Main working area

2. Vehicle instrumentation and preparation

ITINERARIES and LOGISTICS

SPECIAL COMMUTES:

1. IMMERSION ITINERARY/Briefing

2. Manual driving and contextualization route

3. Debriefing route

Annex 14. Test Case Template

Adaptive ADAS to support incapacitated drivers Mitigate Effectively risks through tailor

made HMI under automation

This project has received funding from the European Union’s Horizon 2020

research and innovation programme under grant agreement No 688900

Test Case Template

Work Package No. WP7

Work Package Title

Activity No. A7.1

Activity Title Test Cases Templates

Dissemination level PU = Public, CO=Confidential;..

Main Author(s) IDIADA

File Name ADASANDME_Deliverable_xx.x_dd-mm_y.doc

Online resource http://www.adasandme.com

http://www.adasandme.com/link-to-deliverable

ADAS&ME (688900) DX.Y – Title of the document


Version History

Date Version Comments

30/08/17 v1 First draft

20/12/17 v2 Sections review after feedback

25/01/18 v3 Full structure reviewed and updated

Authors (full list)

Beatriz Delgado, IDIADA

Alex Vallejo, IDIADA

Marta Cocron, TUC

Project Coordinator

Dr. Anna Anund

Research Director / Associate Professor

VTI - Olaus Magnus väg 35 / S-581 95 Linköping / Sweden

Tel: +46-13-20 40 00 / Direct: +46-13-204327 / Mobile: +46-709 218287

E-mail: [email protected]

Legal Disclaimer

The information in this document is provided “as is”, and no guarantee or warranty is given that the information is fit for any particular purpose. The above referenced authors shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials subject to any liability which is mandatory due to applicable law.

The present document is a draft. The sole responsibility for the content of this publication lies with the authors. It does not necessarily reflect the opinion of the European Union. Neither the INEA nor the European Commission is responsible for any use that may be made of the information contained therein.

© 2016 by ADAS&ME Consortium

mailto:[email protected]


Table of Contents

TABLE OF CONTENTS ................................................................................................................................ 3

INDEX OF FIGURES..................................................................................................................................... 6

INDEX OF TABLES ...................................................................................................................................... 6

GLOSSARY .................................................................................................................................................... 8

EXECUTIVE SUMMARY ............................................................................................................................. 9

1. INTRODUCTION ............................................................................................................................... 10

2. EVALUATION AIMS ......................................................................................................................... 10

3. RELATION AND INTERCONNECTIONS WITH OTHER WPS .................................................... 11

4. GENERAL PLANNING ...................................................................................................................... 11

5. EVALUATION PREPARATIONS ..................................................................................................... 14

6. EVALUATION SITES ........................................................................................................................ 16

6.1 IDIADA’S PROVING GROUND ......................................................................................................................... 16 6.2 VTI’S DRIVING SIMULATOR............................................................................................................................ 19

7. USE CASE A – ATTENTIVE LONG-HAUL TRUCKING ............................................................... 20

7.1 FUNCTION SPECIFICATION .............................................................................................................................. 20 7.2 ASSESSED DRIVER STATES ............................................................................................................................... 22 7.3 EVALUATION OBJECTIVES .............................................................................................................................. 22 7.4 EVALUATION SITE ......................................................................................................................................... 22 7.5 TIME PLAN .................................................................................................................................................. 22 7.6 DESIGN AND CONDITIONS .............................................................................................................................. 23 7.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ................................................................................................ 24 7.8 PROCEDURE ................................................................................................................................................ 24 7.9 TASKS TO INDUCE STATES............................................................................................................................... 27 7.10 MEASUREMENTS..................................................................................................................................... 28

8. USE CASE B – ELECTRIC VEHICLE RANGE ANXIETY ............................................................. 31

8.1 FUNCTION SPECIFICATION .............................................................................................................................. 31 8.2 ASSESSED DRIVER STATES ............................................................................................................................... 33 8.3 EVALUATION OBJECTIVES ............................................................................................................................... 33 8.4 EVALUATION SITE ......................................................................................................................................... 33 8.5 TIME PLAN .................................................................................................................................................. 34 8.6 DESIGN AND CONDITIONS .............................................................................................................................. 35 8.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ................................................................................................ 35 8.8 PROCEDURE ................................................................................................................................................ 35 8.9 TASKS TO INDUCE STATES............................................................................................................................... 37 8.10 MEASUREMENTS..................................................................................................................................... 37

9. USE CASE C – DRIVER STATE-BASED SMOOTH & SAFE AUTOMATION TRANSITIONS .. 40

9.1 FUNCTION SPECIFICATION .............................................................................................................................. 40 9.2 ASSESSED DRIVER STATES ............................................................................................................................... 42 9.3 EVALUATION OBJECTIVES .............................................................................................................................. 42 9.4 EVALUATION SITE ......................................................................................................................................... 42 9.5 TIME PLAN .................................................................................................................................................. 42 9.6 DESIGN AND CONDITIONS .............................................................................................................................. 43 9.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ................................................................................................ 44 9.8 PROCEDURE ................................................................................................................................................ 44 9.9 TASKS TO INDUCE STATES............................................................................................................................... 46 9.10 MEASUREMENTS..................................................................................................................................... 47

10. USE CASE D – NON-REACTING DRIVER EMERGENCY MANOEUVRE.................................. 50

10.1 FUNCTION SPECIFICATION ......................................................................................................................... 50 10.2 ASSESSED DRIVER STATES .......................................................................................................................... 50


10.3 EVALUATION OBJECTIVES .......................................................................................................................... 50 10.4 EVALUATION SITE .................................................................................................................................... 50 10.5 TIME PLAN ............................................................................................................................................. 50 10.6 DESIGN AND CONDITIONS ......................................................................................................................... 52 10.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ........................................................................................... 52 10.8 PROCEDURE ........................................................................................................................................... 52 10.9 TASKS TO INDUCE STATES .......................................................................................................................... 53 10.10 MEASUREMENTS..................................................................................................................................... 53

11. USE CASE E – LONG RANGE ATTENTIVE TOURING WITH MOTORBIKE ........................... 56

11.1 FUNCTION SPECIFICATION ......................................................................................................................... 56 11.2 ASSESSED DRIVER STATES .......................................................................................................................... 59 11.3 EVALUATION OBJECTIVES .......................................................................................................................... 59 11.4 EVALUATION SITE .................................................................................................................................... 59 11.5 TIME PLAN ............................................................................................................................................. 59 11.6 DESIGN AND CONDITIONS ......................................................................................................................... 60 11.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ........................................................................................... 60 11.8 PROCEDURE ........................................................................................................................................... 60 11.9 TASKS TO INDUCE STATES .......................................................................................................................... 62 11.10 MEASUREMENTS..................................................................................................................................... 62

12. USE CASE F – RIDER FAINT ........................................................................................................... 66

12.1 FUNCTION SPECIFICATION ......................................................................................................................... 66 12.2 ASSESSED DRIVER STATES .......................................................................................................................... 66 12.3 EVALUATION OBJECTIVES .......................................................................................................................... 66 12.4 EVALUATION SITE .................................................................................................................................... 66 12.5 TIME PLAN ............................................................................................................................................. 68 12.6 DESIGN AND CONDITIONS ......................................................................................................................... 68 12.7 SAMPLE SELECTION CRITERIA AND RECRUITMENT ........................................................................................... 68 12.8 PROCEDURE ........................................................................................................................................... 68 12.9 TASKS TO INDUCE STATES .......................................................................................................................... 68 12.10 MEASUREMENTS..................................................................................................................................... 68

13. USE CASE G – PASSENGER PICK UP/DROP OFF AUTOMATION FOR BUSES ...................... 69

13.1 ASSESSED DRIVER STATES .......................................................................................................................... 71 13.2 EVALUATION OBJECTIVES .......................................................................................................................... 71 13.3 EVALUATION SITE .................................................................................................................................... 71 13.4 TIME PLAN ............................................................................................................................................. 71 13.5 DESIGN AND CONDITIONS ......................................................................................................................... 72 13.6 SAMPLE SELECTION CRITERIA AND RECRUITMENT ........................................................................................... 72 13.7 PROCEDURE ........................................................................................................................................... 73 13.8 TASKS TO INDUCE STATES .......................................................................................................................... 74 13.9 MEASUREMENTS..................................................................................................................................... 74

14. METHODOLOGICAL CONSIDERATIONS AND DATA ANALYSIS ........................................... 77

15. ETHICAL PROCESSES ..................................................................................................................... 79

15.1 ETHICAL PROCESSES IN ALL STAGES OF METHODOLOGY: HUMAN PARTICIPANTS ............................................. 80 16. REFERENCE DOCUMENTS ............................................................................................................. 81

ANNEXES ..................................................................................................................................................... 83

ANNEX 1 . ORDER OF SESSIONS ................................................................................................................................ 84 ANNEX 2. RECRUITMENT QUESTIONNAIRE .................................................................................................................. 87 ANNEX 3. DEMOGRAPHICS QUESTIONNAIRE ............................................................................................................... 91 ANNEX 4. GENERAL QUESTIONNAIRE ON AUTOMATED DRIVING/RIDING ........................................................................... 95 ANNEX 5. ACCEPTANCE SCALE ............................................................................................................................... 101 ANNEX 6. TRUST SCALE ........................................................................................................................................ 103 ANNEX 7. SYSTEM USABILITY SCALE (SUS) .............................................................................................................. 106 ANNEX 8. KAROLINSKA SLEEPINESS SCALE ................................................................................................................ 108 ANNEX 9. STRESS SCALE ....................................................................................................................................... 110 ANNEX 10. QUESTIONNAIRE ON POTENTIAL SYSTEM USAGE AND ACQUISITION ................................................................ 112


ANNEX 11. DEBRIEFING INTERVIEW ........................................................................................................................ 116 ANNEX 12. TASKS TO INDUCE STATES ...................................................................................................................... 121 ANNEX 13. IDIADA’S PROVING GROUND ............................................................................................................... 124 ANNEX 14. TEST CASE TEMPLATE ........................................................................................................................... 131

TABLE OF CONTENTS .................................................................................................................................... 134

INDEX OF FIGURES ....................................................................................................................................... 137

INDEX OF TABLES ......................................................................................................................................... 137

GLOSSARY .................................................................................................................................................... 138

EXECUTIVE SUMMARY ................................................................................................................................. 139

1. INTRODUCTION ................................................................................................................................... 140

1.1 STORYBOARD ............................................................................................................................................ 140 1.2 TIME PLAN ................................................................................................................................................ 140

2. TEST CASE CONTENTS .......................................................................................................................... 141

2.1 OBJECTIVE ................................................................................................................................................ 141 2.2 DEFINITIONS ............................................................................................................................................. 141 2.3 TEST EQUIPMENT AND MATERIALS ................................................................................................................. 141 2.4 MEASUREMENT REQUIREMENTS ................................................................................................................... 142 2.5 PERSONNEL, ROLES AND SKILLS ..................................................................................................................... 143 2.6 SAFETY .................................................................................................................................................... 144

3. TEST SCENARIO ................................................................................................................................... 144

4. TEST EXECUTION ................................................................................................................................. 146


Index of Figures

FIGURE 1. GENERIC MAP OF IDIADA’S PROVING GROUND. ................................................................................... 16 FIGURE 2. GENERAL ROAD OF IDIADA’S PROVING GROUND. ................................................................................ 17 FIGURE 3. HIGH-SPEED TRACK AT IDIADA’S PROVING GROUND. .......................................................................... 18 FIGURE 4. VTI’S DRIVING SIMULATOR II TO BE USED IN USE CASE G. .................................................................... 19 FIGURE 5. EVENT CHART FOR DRIVER STATE RECOGNITION AND MITIGATION IN USE CASE A ................................ 21 FIGURE 6. IDIADA’S HIGH SPEED TRACK FOR USE CASE A EVALUATION AND DETAILED ROAD WORKS LAYOUT.

ROAD WORKS WILL BE PLACED WHERE THE RED STAR IS. ............................................................................. 26 FIGURE 7. EVENT CHART FOR RANGE ANXIETY AND RANGE INCIDENT MITIGATION WITH COMPLIANCE IN USE CASE B

..................................................................................................................................................................... 32 FIGURE 8. ROUTE FOR THE USE CASE B EVALUATION (GOOGLE MAPS, 2018) .......................................................... 34 FIGURE 9. IDIADA’S HIGH-SPEED TRACK FOR USE CASE B AND RESPECTIVE SECTIONS. ......................................... 37 FIGURE 10. USE CASE B TEST PROCEEDING ........................................................................................................... 37 FIGURE 11. EVENT CHART FOR DRIVER STATE RECOGNITION AND MITIGATION IN USE CASE C. ............................... 41 FIGURE 12. IDIADA’S HIGH-SPEED TRACK FOR USE CASE C EVALUATION AND DETAIL FOR ROAD WORKS LAYOUT.

..................................................................................................................................................................... 45 FIGURE 13. EVENT CHART FOR NON-REACTING DRIVER EMERGENCY MANOEUVRE IN USE CASE D. ....................... 51 FIGURE 14. IDIADA’S HIGH SPEED TRACK FOR USE CASE D AND RESPECTIVE SECTIONS. ....................................... 53 FIGURE 15. MOTORCYCLE USED FOR THE UC E EVALUATIONS ............................................................................................. 56 FIGURE 16. PPE USED FOR UC E/F EVALUATIONS ........................................................................................................... 57 FIGURE 17. EVENT CHART FOR DRIVER STATE RECOGNITION AND MITIGATION IN USE CASE E. ............................... 58 FIGURE 18. HIGH-SPEED TRACK FOR THE EVALUATION OF UC E........................................................................................... 62 FIGURE 19. EVENT CHART FOR DRIVER STATE RECOGNITION AND MITIGATION IN USE CASE F................................. 67 FIGURE 20. EVENT CHART FOR PASSENGER PICK UP/DROP OFF AUTOMATED SYSTEM. ............................................. 70 FIGURE 21. THE CITY BUS SCENARIO OF USE CASE G. .............................................................................................. 73 FIGURE 22. SCHEME OF USE CASE G EXPERIMENT DESIGN .................................................................................................. 74 FIGURE 23. PLAN B AND C FOR THE APPLICATION OF THE METHODOLOGY ............................................................................. 78

Index of Tables

TABLE 1. EVALUATIONS’ TIME PLAN OVERVIEW. ................................................................................................... 12 TABLE 2. OVERVIEW OF COMMON METHODOLOGICAL ASPECTS ........................................................................................... 13 TABLE 3. CHARACTERISTICS OF IDIADA’S GENERAL ROAD. ................................................................................ 17 TABLE 4. CHARACTERISTICS OF IDIADA’S HIGH SPEED TRACK. ........................................................................... 18 TABLE 5. FOUR WEEKS ACTIVITY DESCRIPTION FOR USE CASE A ............................................................................................ 23 TABLE 6 . USE CASE A SESSIONS .................................................................................................................................... 23 TABLE 7. USE CASE A METRICS ................................................................................................................................ 28 TABLE 8. FOUR WEEKS ACTIVITY DESCRIPTION FOR USE CASE B ............................................................................................ 34 TABLE 9. USE CASE B SESSION PROCEDURE ...................................................................................................................... 35 TABLE 10. USE CASE B METRICS .............................................................................................................................. 38 TABLE 11. FOUR WEEKS ACTIVITY DESCRIPTION FOR USE CASE C .......................................................................................... 43 TABLE 12. USE CASE C AND D SESSIONS ......................................................................................................................... 43 TABLE 13. USE CASE C METRICS .............................................................................................................................. 47 TABLE 14. USE CASE D METRICS .............................................................................................................................. 54 TABLE 15. FOUR WEEKS ACTIVITY DESCRIPTION FOR USE CASE E .......................................................................................... 60 TABLE 16. USE CASE E PROCEDURE ............................................................................................................................... 61 TABLE 17. USE CASE E METRICS .............................................................................................................................. 63 TABLE 18. FOUR WEEK ACTIVITY DESCRIPTION FOR USE CASE G ........................................................................................... 71 TABLE 19. USE CASE G SESSION PROCEDURE ................................................................................................................... 72 TABLE 20. USE CASE G METRICS .............................................................................................................................. 75


Glossary

ACRONYM WHAT IT MEANS



… …


Executive Summary

One/two page(s) executive summary.

This template is created with the intention of clarify all the subsections and subsections needed to tackle Test Plans.

We will try to explain what is beyond the scope of this document. Thus, the introduction section it is planned to include the reasons why this information is essential to create the wright scenarios for testing and validation.

Then, all the chapters considered are focus on the aspects to be completed for the Test Plan regarding all the different issues involved.


1. INTRODUCTION

1.1 Storyboard

NOTE: No need to complete this section. It is coming directly from the Evaluation Framework – Function specification.

A storyboard, understood as a graphic organizer in the form of illustrations or images displayed in sequence for the purpose of pre-visualizing a motion picture, animation, motion graphic, etc., is thought as an evocative tool for testers.

For the purpose of not forgetting one of all the steps that will provide the specific required scenario considered to achieve the goals described, it will be necessary to “draw” a storyboard where every critical point can be illustrated.

The storyboard can be used also as a tool to guarantee the possibility of being repeated with almost the same conditions.

1.2 Time plan

NOTE: No need to complete this section. It is coming directly from the Evaluation Framework – Time plan.

Explain when the evaluation will be carried out (month) and for how long. Fill out the table below and add text explaining the different stages and extra information if needed.

Example:


1 to 5 Technical preparations and finalisation of subject recruitment

6 Pre-tests





2. TEST CASE CONTENTS

2.1 Objective

NOTE: No need to complete this section. It is coming directly from the Evaluation Framework – Evaluation objectives + assessed driver state.

With a couple of sentences, the main objective and the driver states under test must be described.

Reading the objective subsection it is expected to understand the goal of this concrete test case that is being suggested. The goal that it is being working towards and hopes to achieve by the end of a course of action must be also understood.

For Example (a test case related to UC D):

“This test case is conceived to find out the possibilities of return a suitable driver state from a stress driver state detection. On this account, a sample of xx users/drivers will cover a special trajectory built of xx minutes and a special HMI version (strategy) called XYZ will be provided/deliver in order to return a suitable driver state.”

2.2 Definitions

In this subsection it would be important to gather all the terms with their definitions in order to clarify if necessary during the execution.

In case of dismiss or in case of disagreement as to the meaning or scope or any concept, this list could contribute to make clear little suspicions.

For example, ADAS&ME systems:

“AEB&ME”: This system developed in order to provide an unexpected and prompt brake during the test case with a xx minutes cadence.

Other involved systems:

VUT: Vehicle Under Test;

TV: Target Vehicle;

SIMULATION components…”

2.3 Test equipment and materials

In this subsection the test equipment and the involved materials must be identified and defined. This subsection is to identify all the instrumentation that will be needed for the test (inside and outside the vehicle).

It is expected to provide a detailed list regarding equipment and materials. This contribution is required in order to control the equipment set-up and behaviour as well as the booking of the support material.

For example regarding Equipment required:

“This test requires the installation of cones simulating a roadworks area. A change in the equipment at a given time must be done. When and where this change will take place will be informed prior to the beginning of the tests.”

Instrumentation for vehicles, sensors, actuators, etc.; Brake robot; Accelerator robot; Support cameras; Acquisition tools; etc.

Others examples:


“In this test case a target vehicle (rabbit) will be necessary in order to consider if the activation of ACC influence the time of manipulation of driver state with concrete HMI strategy. Likewise an extra Walkie-talkies will be used in order reinforce the information captured by the sensors prepared.”

2.4 Measurement requirements

NOTE: No need to complete this section. It is coming directly from the Evaluation Framework – Driving test measurements.

In this subsection everything regarding measurement criteria and format should be identified. It is expected to know about all the requirements associated with the use case in order to confirm and ensure the success of the test case indeed.

This subsection must be completed with aspects related to dependent variables features (Measurements and formats):

For example:

“This test case has special measurement requirements considering the aim we are pursuing. The stress levels measurement is going to be measured through special sensors focused on Galvanic Skin Response. This value will be provided in Hertz and an acceptance rage will be provided. During the test case the measurement of GSR will be continuous and a trigger will be set in order to emplace the moments (time, location, etc.) different peaks are met.”

Example table to be completed:


Procedure snippet Number 50hz Integer “description”


50hz Integer “description”

Procedure Active scenario 50hz Integer “description”

Vehicle Data Speed 50hz Integer “description”

Vehicle Data Time stamp 50hz Integer “description”











Vehicle Data Y position of 50hz Integer “description”


vehicle

Driver Behaviour

Visual Behaviour


Transition timing

Moment drivers make the transition in relation with the warnings

Actions Video Hesitations, verbal communications that can give a better understanding about the transition difficulty

Driver State State activation Video Control measurement for confirmation (and comparison with sensor data) of each state level.


Task activation To verify if the driver performs the secondary task (control measurement)

Timing Task Standby

Audio/video When is the state task interrupted in relation with the warning

2.5 Personnel, roles and skills

It is crucial to identify the skills and roles of the participants (sample + experts/support) and if extra training is needed.

In this subsection it is expected to clarify these special skills and roles. Furthermore, the explanation about this need should be very useful in order to understand inherent aspects of the test case.

For example:

“In order to achieve our aims regarding expert users, a sample consisting of drivers with special licence that have proven collision avoidance skills is needed. A minimum driving experience in years and also ADAS experience should be considered.”

“In this test case it is important to select a human factors expert to carry out the observer tasks for subjective measurement. These skills are related to the past experience in similar roles or tasks such as, ADAS functions assessment, passive observation and impartial evaluations.”

A summary list is recommended.

For example:

- 1 HMI expert - 1 vehicle dynamics expert - 1 technical support - …


2.6 Safety

When running experiments where human intervention and vehicles are involved and there is a possibility of hazard, it is a big responsibility to consider and estimate the risks involved. In fact, a risk assessment is the best way to achieve this approach. It is important to be aware about these hazards and try to prevent and/or avoid them and if it not possible, reduce the effects of the impact in human integrity.

In this subsection an explanation about these risks, their impact and the strategies planned to avoid them is expected. Also, presenting them briefly in a table will make it easier to understand and is recommended.

It is also expected to mention the extra safety equipment to be used further than the conventional and already expected restraint systems, e.g. helmet (for the cars or trucks), three-point harness, outriggers (for the bike), etc.

A safety check and a special proving ground risk assessment will be required and should be included in this document after having them complete.

For example:

“In this test case (motorbike) a driver harness shall be mounted in case of an extreme driver state implying loss of consciousness is induced. The same risk forces to the use of safety outriggers.”

Risk Impact Mitigation actions

3. TEST SCENARIO

This section is dedicated to describe all the different features and conditions regarding the evaluation scenario (both real and simulated) where the test case will be conducted.

Special road conditions should be included if they are relevant for the test execution and evaluation, e.g. dry/wet surface, real road simulation with other vehicles, daylight/night-time, etc.

Figures to support the explanation are expected.

For example:

“The evaluation scenario will be prepared on the high-speed track and 3 lanes will be used. It

consists in different road sections where specific actions are required in each of them.

First section is the south straight lane used to set-up/initialize the systems, get prepared…

Second section is the corner before entering the main section. This is used to check the systems

keep behaving as expected…

Third section is the evaluation area, the north straight where a road works zone is simulated in

the 2nd lane of the track…

…”


(Note that the figure does not match with the description above. It is only used as an example)

S


1st

curve 2nd curve



4. TEST EXECUTION

NOTE: No need to complete this section. It is coming directly from the Evaluation Framework – Procedure + Tasks to induce driver states.

This last section gathers all the test procedure in detail, i.e. actions, manoeuvres, measurements, driving conditions (speeds, lane to drive through…), driver interactions with the system, vehicle interactions with the scenario, driver states inducement moments (while driving), etc. happening throughout the whole test case.

This way the test responsible/coordinator will have all the information needed at any time to ensure the proper execution of the evaluation.

For example:

The test begins at 40km/h on the first lane, in manual mode…

…

When entering the evaluation area, the driver shall ignore the warnings to make the transition

from automated to manual driving. This means that, when driving automated, and after

receiving a warning to take over control of the vehicle, the driver should do nothing: neither

touching pedals nor the steering wheel. The vehicle will then should perform an emergency

manoeuvre that consists of bringing to vehicle to a full stop on the lane’s shoulder.

During the driving test a member of the staff will be inside the vehicle in case something

unexpected happens and the driver required help. Drivers will be informed about this and also

that he/she should not communicate with this person. A constant driving speed of 30 Km/h will

have to be maintained during the driving test (in both manual and automated conditions).

The detailed evaluation is planned to happen as follows:

. The driving test begins at the starting point (as it can be seen in Figure xx).

. The driver will initiate driving manually at a speed of 30 km/h on the first curve section of the

test track.

. Moments before leaving this section, the driver receives a warning to take over control of the

vehicle (from manual to automated driving).

. The driver follows this warning and the vehicle starts automated driving when already on the

south straight section.

. Before reaching the end of this section the driver will again receive a warning.

. The driver ignores this warning.

. After the 2nd warning the vehicle performs a safe stop by parking on the lane’s shoulder at the

end of the south straight section.

The driver will then be asked to re-start driving manually at a speed of 30 km/h. The procedure

should be repeated for the 2nd curve and north straight section of the track. The driver will then

experience the emergency manoeuvre a second time. The evaluation will end after the second

safe stop (near the starting point).


Figure xx. IDIADA’s high speed track for use case D and respective sections.

Deliverable 7.1 Evaluation Frameworkmade HMI under automation This project has received funding from...

Documents

Transcript of Deliverable 7.1 Evaluation Frameworkmade HMI under automation This project has received funding from...