A methodology for testing intersection related Vehicle-2-X applications.pdf

8/10/2019 A methodology for testing intersection related Vehicle-2-X applications.pdf

http://slidepdf.com/reader/full/a-methodology-for-testing-intersection-related-vehicle-2-x-applicationspdf 1/15



3.1. Information flow

Intersection related applications comprise a special

type of V2X applications which are perfectly convenient

for the introduction phase of V2X communication systems.

That is the case because of the simplicity of the informa-

tion flow. If DSRC is used, the information flow consists

of a unidirectional broadcast of single-hop transmissions

from the intersection’s IRSs to vehicle’s IVSs (see Fig. 1).

If cellular mobile communication is used, IVSs build up cel-

lular mobile communication connections and download

the required information about near intersections from

an ICS (see Fig. 2). However, this does not change the infor-

mation flow within vehicles substantially.

The relevant information to realize intersection related

V2X applications comprise the intersection topology per

intersection (for example see Fig. 12) and the signal state

timing per each lane of each intersection (for example see

Fig. 11) [18–20]. Both are represented by messages which

are introduced in Section 2.3. The intersection topologies

are modeled by an intersection ID, vectorized driving lanes,

driving lane numbers, suitable connection points, stop

lines, and more. The signal state timing data are modeled

by an intersection ID, lane numbers, minimum time to

change, maximum time to change, likely time to change,

and more. Both message types are broadcasted periodically

by intersection’s IRSs or could be downloaded from ICSs.

Although the vehicle’s Electronic Control Units (ECUs)

provide relevant information to realize intersection related

V2X applications which is currently already exchanged by

vehicle internal bus systems like the CAN-Bus. Typical rel-

evant vehicle internal information are among others: vehi-

cle speed, single wheel speed, engine speed, yaw rate,

direction indicator status, or fog light status. So, the access

of those data could easily be realized by vehicle manufac-

tures and, thus, be used by V2X implementations.

Finally, an IVS collects all those information from its

vehicle and its environment and uses it to realize the appli-

cation’s functionalities. In addition, the applications could

use additional facilities of vehicles (e.g. HMI or signaling

tones) to interact with the drivers. Examples of those

applications are described in the following Section 3.2.

3.2. Examples of applications

According to [3,9,21], typical intersection related appli-

cations are among other:

Green-light Optimal Speed Advisory (GLOSA) ‘‘Drivers

receive a recommendation in order to hit the next traf-

fic lights in green phase and to avoid waste accelera-

tion.’’ [9]

Red-light Violation Warning (RVW) ‘‘Warns drivers

when they are going to violate a red traffic light signal.’’

[9]

Typically, the warnings occur in a double-staged

approach. E.g., the first stage realizes a visual or audible

signaling; whereas, the second stage realizes a short

jerky braking maneuver if the driver shows no reaction

on the first signaling.

Traffic Light Assistant (TLA) If the vehicle has to stop

because of the traffic light signaling, the vehicle shows

the remaining waiting time of the corresponding signal-

ing group to the driver [21].

4. Test needs

Testing software intensive systems is a complex and

permanent task throughout the software life cycle and

should therefore be done very efficiently and effectively.

Moreover, the test process’ quality has significant influence

on the resulting products’ quality. Since no V2X applica-

tions have already been brought into market so far, no suf-

ficient experiences in testing of those novel systems are

available. Therefore, the test needs of intersection related

V2X applications are analyzed from the literature’s point

of view. Finally, based on these analysis requirements on

the test system are derived.

4.1. Test levels and test types

According to [22], test levels ordinary comprise compo-

nent testing (unit testing), integration testing, system test-

ing, and acceptance testing; whereas, the transitions are

smooth (see Fig. 3). As component testing and low-level

integration testing targets low-level functional testing

and low-level non-functional testing close to software

implementations, test methodologies for these parts of test

levels are already available (see [22–25]). So, testing

activities closely related to software implementations of V2X applications should be organized analog to common

IRS

IVS ECU

<<flow>> <<flow>>IVS ECUIRS 1..*110..*

vehicle internDSRC

Fig. 1. information flow of DSRC based intersection related V2Xapplications.

IRS

IVS ECU

<<flow>> <<flow>>IVS ECUIRS 1..*10..*0..*ICS

<<flow>> 11..*

vehicle interncellular mobilebackbone

ICS

Fig. 2. Information flow of cellular mobile based intersection related V2X

applications.

3156 S. Röglinger / Computer Networks 55 (2011) 3154–3168

http://-/?-

http://-/?-

http://-/?-

http://-/?-



software tests. To avoid contradictions, the test levels and

test types are defined as follows:

component testing ‘‘Testing of individual hardware

or software components or

groups of related components.’’

[26];

integration testing ‘‘Testing performed to expose

defects in the interfaces and in

the interactions between inte-

grated components or systems.

See also component integration

testing, system integration test-

ing.’’ [27];

system testing ‘‘The process of testing an inte-

grated system to verify that it

meets specified requirements.’’

[28];

acceptance testing ‘‘Formal testing with respect to

user needs, requirements, and

business processes conducted

to determine whether or not a

system satisfies the acceptance

criteria and to enable the user,

customers or other authorized

entity to determine whether or

not to accept the system.’’ [27]

after [26];

functional testing ‘‘Testing based on an analysis of

the specification of the function-

ality of a component or system.’’

[27];

non-functional testing ‘‘Testing the attributes of a com-

ponent or system that do not

relate to functionality, e.g. reli-

ability, efficiency, usability,

maintainability and portability.’’

[27].

In contrast, testing activities for high-level integration

testing and system testing, namely, high-level functional

testing and non-functional testing could not be handled

adequately with available test technologies and methodol-

ogies. Except for the hardware equipment required to

emulate the V2X communication partners and vehicle’s

ECUs, a test system has to be able to handle diverse

requirements in order to address the specific needs of

V2X applications. Section 4.3 establishes a proposal for

those requirements.

Additionally, acceptance testing is another wide area of

testing software intensive systems; especially, for cooper-

ative V2X applications. For intersection related V2X appli-

cations, acceptance testing has to target among others: the

driver acceptance, influences on a road network’s traffic

throughput, or the traffic flow smoothness. So, this area

of testing should be considered separately.

This also applies for testing safety, security, and privacy

aspects of ITS systems in general and, thus, also for inter-

section related V2X applications. Since ITS systems include

various communication interfaces, there might be much

potential for security violation attacks. Moreover, vehicle

take part in road traffic and, thus, security violation attacks

might induce serious road accidents. Therefore, also for

safety, security, and privacy, separate test methodologies

should be explored and applied.

In summary, testing high-level functional and non-

functional aspects of intersection related V2X applications

opens a differentiated and unsolved area. So, this paper fo-

cuses on these two aspects of testing.

4.2. Testing of system characteristics

As described in Section 2.2, the software architecture of

V2X systems will most probably consist of a stack architec-

ture. Here, multiple applications and underlaying run-time

platform components (e.g. middleware or the operating

system) will run on one system simultaneously. So, the

workload of those systems varies over time and thus the

execution time of algorithms and the response-time to

interrupts is non-deterministic. The relevant effects which

lead to non-deterministic systems (e.g., scheduling algo-

rithms, concurrency) are described in [29]. Moreover, the

whole system might be distributed over multiple ECUs

which are connected over communication networks with

non-deterministic timing properties (e.g. CAN-Bus). So,

the timing characteristic of intersection related V2X sys-

tems should be targeted as non-deterministic. For that rea-

son, functional and non-functional testing which targets

the timing of the systems should be repeated several times

and evaluation statistically.

Moreover, different environmental and vehicle internal

information sources and information flows of IVSs might

suffer from communication channel disturbances, mea-

surement and intermediate calculation inaccuracies, or

other negative effects. Since it is impossible to resolve

the effects of input signal variations to the application’s

outputs, statistic test methodologies have to be applied.

By using randomized algorithms as described in [30] for

test input signal modifications and statistical evaluations

of resulting application’s reactions and outputs, the former

problem could be addressed.

Finally, for testing activities targeted to non-functional

aspects of the system, at least performance testing, load

testing, stress testing, and reliability testing should there-fore be taken into account [22].

component testing

integration testing

system testing

acceptance testingcheck system behavior

against consumer’s

requirements

functional and

non-functional testing

low-level

high-level

Fig. 3. Test levels with test types.

S. Röglinger/ Computer Networks 55 (2011) 3154–3168 3157

http://-/?-

http://-/?-

http://-/?-

http://-/?-



4.3. Requirements on the test systems

In this subsection, a proposal for requirements on test

strategies and test architectures for intersection related

V2X systems are presented. These requirements are de-

rived from the intersection related V2X’s architecture (Sec-

tion 2) and the test needs (Section 4).

Requirement 1 Emulation of the specified behavior of

cooperation partners who are con-

nected via DSRC.

Requirement 2 Emulation of influences of wireless

communication on the DSRC communi-

cation. Suitable influences are among

others radio interferences, package col-

lisions, communication channel conges-

tion, fading, or shading [31,32]. Those

influences show according to [33,34]

effects on package loss rates and statis-

tical distributions of package transmis-

sion delays and might be caused by

communication of other vehicles or by

environmental objects like buildings or

trees.

Requirement 3 Emulation of the specified behavior of

cooperation partners who are connected

via cellular mobile communication.

Requirement 4 Emulation of influences of cellular

mobile networks on the communication

between ICSs and IVSs. Suitable influ-

ences are among other: network cover-

age, and available local field strength.

Those influences show effects on: prob-

ability of connections loss, round trip

time, channel throughput, and others.

Requirement 5 Emulation of the specified behaviors of

other ECUs which are also located

inside the vehicle.

Requirement 6 Emulation of the vehicle’s self localiza-

tion facilities. In most cases, GPS and

odometer based techniques are used.

Requirement 7 Test runs have to be reproducible

exactly which addresses synchroniza-

tion problems. Those problems may

appear by reapplying test runs on dis-

tributed systems.

Requirement 8 It must be possible to modify the test

input data in order to adjust their qual-

ity (e.g., add noise or reject messages).

Requirement 9 It must be possible to arbitrary combine

various test input data to stress differ-

ent test cases.

Requirement 10 It must be possible to build test verdicts

based on statistic evaluation of test

runs.

To avoid contradictions, the term emulation is defined

by ‘‘technique of one machine obtaining the same results

as another’’ [35] whereas simulation is defined by ‘‘the

technique of representing the real world by a computerprogram; a simulation should imitate the internal pro-

cesses and not merely the results of the thing being simu-

lated’’ [35].

Summing up, test strategies and test architectures for

intersection related V2X applications have to fulfill all or

at least a subset of the above established requirements.

In case that just a subset is fulfilled, not all test needs could

be satisfied. E.g., the test approach, presented in Section 5,

uses a two-layered strategy to use real-world traces for

test input stimulation, whereas, the first layer realizes

not all requirements.

5. Approach

In this section, an approach on how to test intersection

related V2X applications is presented. The approach

emphasizes the points identified in Section 4:

test levels: high-level integration testing and system

testing,

test types: high-level functional and non-functional

testing, system characteristics: test input modification and sta-

tistical test output evaluation.

Moreover, the approach’s design fulfill the require-

ments established in Section 4.3 by adapting a channel

and stream concept.

5.1. Identification of system under test

For testing functional and non-functional aspects of dis-

tributed systems, like intersection related V2X applica-

tions, all involved components have to be considered.However, as described in Section 3, most of the complexity

of intersection related V2X applications is located in IVSs.

Hence, IRSs are also completely new products; those types

of stations should also be tested. However, the complexity

of the IRS’s application implementations does not reach

the difficulty of the IVS’s application implementations. Fur-

thermore, ECUs are already well tested and not modified

because they are only information sources. So, they do

not need extra testing in terms of testing intersection

related V2X applications. For all those reasons, most

testing activities have to take IVSs into account. Therefore,

IVSs are treated as System Under Test (SUT) in the test

methodology.

5.2. Test strategy

The test strategy comprises the following two points:

stimulations of SUTs to stress test cases, and evaluation

of test results. The realization of these is shown for inter-

section related V2X applications in the following subsec-

tions. Since both points are closely related to the channel

and stream concept, the next subsection describes how

this concept could be applied for intersection related V2X

applications. The following subsection treats stimulation

and evaluation techniques, and the last subsection handlesdefinitions of assessment strategies.


http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-



Fig. 5 depicts how input streams and their modifica-

tions could be applied to SUTs. The additional timing mod-

ifications T cause positive or negative delays of the

modified input streams to create new test cases by using

the same input streams (requirement 9 of Section 4.3).

Thus, a set of T s could be seen as a configuration of differ-

ent test cases using the same input streams. However, this

construct is not in scope of this section but will be pickedup in Section 5.3.

Additionally, it could be seen in Fig. 5 and (3) that there

is a distinction between conventional and system-internal

channels. On the one hand, conventional channels are also

part of the final product. So, a is set to the final product’s

number of input channels and c is set to the number of re-

corded final product’s output channels. On the other hand,

system-internal channels are not part of the final product

but may be necessary to adjudicate a test verdict.

To construct test cases, the morphologic box whose first

column treats the test case’s test type is used. For functional

testing , only conventional input streams and sometimes

the flow of internal communication interfaces is relevant.So, b is set to 0 and d is set to the number of additionally,

probed internal communication channels. In contrast, for

non-functional testing , also flows of system-internal output

streams like:

CPU load,

memory usage,

network I/O, or

event count

are important. Additionally, to have influence on system-

internal parameters (e.g. additional CPU load) is relevant

to target the test goals: performance testing, load testing,stress testing, and reliability testing as already postulated

in Section 4.2. So, b is set to the number of modified sys-

tem-internal parameters (e.g. additional CPU load for stress

testing) and d is set to thenumber of relevant,recorded sys-

tem-internal output streams (e.g. resulting CPU load).

At second, it has to be decided if influences of modifica-

tions of input streams should be taken into account. If influ-

ences of input stream modifications are not targeted, all sets

M con,i = {} with i = 1, . . . , a are empty. Otherwise, the sets

M con,i contain potential modifications of the corresponding

input streams I con,i (see Fig. 5). Since influences of those in-

put stream modifications to the system’s outputs could not

be resolved analytically, randomized algorithms for deter-

mining the modifications may be applied [30].

Moreover, there are these two types of input streams

regarding their variation over time:

input streams which are static within one test run (e.g.

intersection topologies), and

input streams which are not static within one test run

(e.g. time to green/red of traffic light signal timings).

This effects the modification of input streams as stated

so far. For the static type of input streams, a single modifi-

cation of the input stream (e.g. shift) is chosen per test run.

For the non-static type of input streams, modifications of

input streams (e.g. noise, glitches, drift, message loss, or

additional messages) have to be defined as mapping over

a non-modified input stream I = P N , p 2 P , and time t 2 R

such that f : R P ! P with p 0 = f (t , p).

Before modifications of input streams are determined, it

has to be chosen, if the modifications are inside, on, or out of

the specified tolerances. In the first case, all modifications

are inside their ranges of tolerances. In the second case,

at least one modification is chosen exactly on its tolerance

boundaries. In the last case, at least one modification is out

of its specified tolerances.

Next, V2X applications have to deal with many non-

deterministic effects like communication channel distur-

bances, ego-positioning inaccuracies (e.g. limited GPS

accuracy), measurement inaccuracies, or non-determinis-

tic run-time platforms. Those effects might cause varia-

tions of output streams even when reapplying the same

test case. For gathering those output variations, it is sug-

gested to reapply a test case’s input streams several times

in the exactly same fashion. The index set Ind provides the

possibility to distinguish the different test runs.

Additionally, the evaluation of output streams could

either be done by humans reviewing the raw output

streams or by using metrics and assessment strategies. Ta-

bles 1 and 2 list not exhaustive lists of metrics and assess-

ment strategies for the evaluation of the correctness of the

stream’s values and the evaluation of the correctness of the

stream’s timing for single and multiple test runs. Since

streams combine the properties used in the tables, the

evaluation of streams requires a combination of the tables’

metrics and assessment strategies.

5.2.3. Definition of assessment strategies

By using the previous definitions of messages (1),

streams (2), and test cases (3), it is possible to define

assessment strategies formally. To show how this ap-

proach works, the approach is demonstrated with the fol-

lowing two examples:

(1) evaluation of distribution of points in time where

messages appear, and

(2) evaluation of influences of input data tolerances.

For both examples, the RVW application is used to

demonstrate the definition of assessment strategies. In

addition, these examples are picked up in Section 6 toshow their feasibility by a proof of concept.

IVS

System

(OS, HW, ...)

conventional

system internal

value timing

Mcon,1 Tcon,1

ITS Stack

Mcon,a Tcon,a

Icon,1

Icon,a

Ocon,1

Ocon,c

Osi,1

Osi,d

Isi,1

Isi,b

Fig. 5. Input and output of IVSs.


http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-



5.2.3.1. First example. Scenario: A vehicle approaches anintersection with constant speed, whereas, the traffic light

will signalize red light for the vehicle’s driving lane at the

point in time when the vehicle will cross the stop line. The

RVW application (see Section 3.2) is active and should gen-

erate output events because a red light violation will occur.

The events’ content should indicate a red light violation.

Moreover, the IVS’ CPU could be stressed with various

additional CPU load conditions and, thus, the response

time of the RVW application might suffer.

Description: For the first example, the following path

through the morphologic box as depicted in Fig. 4 is used:

test type: non-functional testing ?modification of input

streams: not targeted? gather output variation: yes? evalu-

ation of output stream: metrics and assessment ? output

stream property: correctness of timing . Thus, the variables

are defined as follows: M con,1 = = M con,a = {}, b = 1, c = 1,

and d = 0. So, the test case could be notated as a subset of

the mapping:

TCI # I si;1 Ind ! Ocon;1; ð4Þ

whereas, the set I si,1 contains various additional CPU load

conditions. Furthermore, it is assumed that the RVW appli-

cation’s output stream Ocon,1 is of the type: event based con-

tinuous-time stream over the set {GONG,BRAKE }. So, multiple

test runs are needed and descriptive statistics (see Table 1)have to be applied for the output stream’s evaluation.

Evaluation: In order to not exceed the scope of theexample, only definitions of normalized histograms are gi-

ven exemplarily. Furthermore, it is expected that the RVW

application generates two messages in this example,

whereas, the first one has to have the value GONG and

the second one has to have the value BRAKE to follow

the double-staged warning approach. So, the histogram

H l, x with the k histogram bars H l; xk is defined for the load

condition l and the points in time where the x-th message

with x 2 {1,2} appeared as follows:

H l; xk : ¼ 1

jTCl

I j

tchl;ii 2 TClI jN hl;iicon;1 ¼ 2 ^ u

hl;iicon;1;1n

¼ GONG ^ uhl;iicon;1;2

¼ BRAKE ^ w k 6 t hl;iicon;1; x < w ðk þ 1Þ

o; ð5Þ

with

tchl;ii ¼ hl; ii ! ohl;iicon;1

D ED E 2 TClI ; ð6Þ

TCl¼ x

I ¼ ftchl;ii 2 TCI jl ¼ xg; ð7Þ

ohl;iicon;1 ¼ t

hl;iicon;1;1; u

hl;iicon;1;1

D E; . . . ; t

hl;ii

con;1;N hl;ii

con;1

; uhl;ii

con;1;N hl;ii

con;1

:

ð8Þ

Table 1

Metrics and assessment strategies for timing of streams.

Metrics Assessment strategies

Single run Multiple runs

Stream property: event based appearance

Appeared/not appeared – Ratio

Time stamp of

appearance

– Descriptive statistics

Stream property: sampling based (periodic appearance)

Jitter of sampling time Descriptive statistics for fluctuation of

sampling interval

Descriptive statistics for single-run results (e.g. distribution of mean

values of single runs)

Lack of samples Ratio Descriptive statistics for single-run results (e.g. distribution of ratio

of single runs)

Start/stop time stamp of

sampling

Start and stop time stamp

Duration of sampling

Descriptive statistics for start/stop time stamps

Descriptive statistics for durations

Table 2

Metrics and assessment strategies for messages of streams.

Metrics Assessment strategies

Single run Multiple runs

Stream property: messages in a discrete non-ordered set

Message’s value in subset of correct values Test result Ratio of in set and not in set of correct messages

Jumping message’s values – Set of appeared messages

Count of appearances per message (e.g. stored

in a set like: ðU NÞN )

Stream property: messages in a continuous or discrete totally ordered set

Raw values Descriptive statistics Descriptive statistics

Derivatives of values Descriptive statistics Descriptive statistics

Properties of signals modulated onto the samples (e.g. ramps,

or frequencies) (see [39])

Signal properties (e.g. max.

slope)

Local signal properties

(within window)

Descriptive statistics for signal properties


http://-/?-

http://-/?-



The symbols in (5)–(8) are defined as follows:

H l, x histogram for load condition l and the x-

th appeared message

H l; xk

height of k-th histogram bar of H l, x

N hl;iicon;1

number of messages in conventional

output stream 1 for load condition l and

the i-th test run

l 2 I si,1 additional CPU load

i 2 Ind index of test run

k 2 N [ f0g index of histogram bar

TClI

set of test cases for input streams I and

load condition l

tchl,ii test cases of i-th test run for load

condition l

ohl;iicon;1

conventional output stream 1 for load

condition l and the i-th test run, and

w 2 R>0 width of histogram bar in seconds

The histogram bars H l; xk as defined in (5) have all the

same width. This is not mandatory for all histograms and

thus the histogram bars might also be defined with varying

widths. Please note that Figs. 13 and 14 in Section 6.2.1

show histograms exemplary.

5.2.3.2. Second example. Scenario: A vehicle approaches an

intersection with constant speed, whereas, the traffic light

will signalize red light for the vehicle’s driving lane at the

point in time when the vehicle will cross the stop line. The

RVW application (see Section 3.2) is active and should gen-

erate output events because a red light violation will occur.

The events’ contents should indicate a red light violation.

Moreover, impacts of intersection topology center point

shifts and, thus, shifts of the whole intersection topology,

on the points in time where the RVW implementation de-

tects the red light violation should be analyzed.

Description: For the second example, the following path

through the morphologic box is used as depicted in Fig. 4:

test type: functional testing ? modification of input streams:

static + inside boundaries? gather output variation:

no? evaluation of output stream: metrics and assess-

ment ? output stream property: correctness of values. Thus,

the variables are defined as follows:

M con;1 ¼slatitudenorth ; slatitude

south ½R

slongitudeeast ; slongitude

west ½R

,

M con,2 = = M con,a = {},

b = 0, c = 1, and d = 0

with the following meanings for the variables:

slatitudenorth

max. deviation in northern direction

slatitudesouth

max. deviation in southern direction

slongitudeeast

max. deviation in eastern direction, and

slongitudewest

max. deviation in western direction

respectively for the intersection topology’s center point.

These variables have to be defined for the final test case.

Finally, the test case could be notated as a subset of themapping:

TCI #M con;1 Ind ! Ocon;1; ð9Þ

whereas, the set M con,1 contains the modifications of the

input stream I con,1 which ought to represent the input

stream of intersection topologies. Hence, the intersection

topology is broadcasted periodically, every send-event re-

quires modifications. In addition, it is assumed that the

RVW application’s output stream Ocon,1 is from the same

type as in the first example.For the assessment, test runs for different combinations

of modified input streams are needed and methodologies

from the area of descriptive statistics (see Table 1) have to

be applied. Since the impacts from intersection topology

center point shifts on the points in time where the RVW

implementation detects a red light violation could not be

handled analytically, it was decided to shift the intersec-

tion topology center point inside its tolerance boundaries

stochastically with uniform distribution in longitude and

latitude direction.

Evaluation: In order to not exceed the scope of the

example, only definitions of the minimum ðT xminÞ, maxi-

mum ðT xmaxÞ, and quantile ðT xqquantileÞ time stamp are given:

T xmin :¼ minðseqcon;1; xÞ; ð10Þ

T xmax :¼ maxðseqcon;1; xÞ ð11Þ

according to [40]:

T xqquantile :¼ sortðseqcon;1; xÞ:dq:jseqcon;1; xje; ð12Þ

with

seqcon;1; x :¼ ft con;1; xjN con;1 ¼ 2 ^ ucon;1;1

¼ GONG ^ ucon;1;2 ¼ BRAKE g; ð13Þ

which contains all time stamps t con,1, x where a GONG

( x = 1) or a BRAKE ( x = 2) message appeared. Please notice

that in (12) the sort-function returns the ascending sorted

input sequence, the first dot denotes the access of an ele-

ment of the sequence (seq.n returns the n-th element of

seq), and the second dot denotes a multiplication.

5.3. Usage of real world traces

Since the previous sections deal with test case configu-

rations, description of stream modifications, and assess-

ment strategies for evaluating output streams, it is still

open which streams could be used as input streams. So,this section deals with the question:

How to use real-world traces exhaustively to test inter-

section related V2X applications?

Therefore, a two-layered approach with and without

modifying real-world traces is presented in the next two

subsections.

First, information flows of intersection related V2X

applications (Section 3.1) show two independent informa-

tion flows from IRSs or ICSs, and ECUs to IVSs. Additionally,

the complete information flow shows no feedback loops

and, thus, the content of the flows is not affected by reac-tions or outputs of IVSs. Therefore, it is possible to record


http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-



traces from the information flows. Second, the traces could

be modified as described in Section 5.2 before reapplying

them to IVSs. This limits the amount of required real-world

traces.

Vehicle driving simulators can also generate ECU traces

and, thus, can substitute real-world traces. This may pro-

vide more flexibility by taking traces in required intersec-

tion topologies and traffic situations. Moreover, traces

taken this way are not even affected by disturbances like

real-world traces.

5.3.1. Reapply raw traces time shifted

The simplest way to use information flow traces from

IRSs or ICSs, and ECUs is to leave them unchanged and

reapply them time shifted. Herewith, test cases regarding

the timing between equipped intersections and equipped

vehicles could be stressed and reapplied. For example, it

is possible to stress the following situations by reusing

the same traces for IRS to IVS and ECU traces:

the vehicle crosses the stop line at red light,

the vehicle crosses the stop line at green light,

the vehicle crosses the stop line at the point in time

when the traffic light switches from red/yellow to

green, and so on.

So, the requirements 1, 3, 5, 6, 7, and 9 which are pos-

tulated in Section 4.3 could be fulfilled. Moreover, test runs

could be executed automatized and, thus, the expensive

and time consuming test drives could be minimized. Also,

regression test strategies could be applied even in develop-

ment phases.

To be able to use this approach, two independent sets of

traces are required:

set of intersection IRS traces, and

set of vehicle ECU traces.

The set of intersection IRS traces covers DSRC and cellular

mobile communication and has to include traces covering

all relevant intersection behaviors of all relevant intersec-

tion topologies. The intersection’s behavior targets the sig-

nal state timing as described in Section 3.1. Analogously,

the set of vehicle ECU traces has to include traces covering

all relevant approaching and leaving behaviors of vehicles

on all intersections which are covered by the set of inter-

section traces. Among others this includes modifications

of: speed/acceleration, lane changing, stop & go, and driv-

ing direction. In addition, the GPS stream has to be part

of an ECU trace.

5.3.2. Reapply modified traces time shifted

In this subsection, a refined approach of the previous

subsection’s approach is described to target the remaining

requirements 2 and 4 of Section 4.3. The refinement targets

modifications of raw streams as foreseen in Section 5.2.

This approach uses the same traces as the approach in

the previous subsection. However, those traces combine

several streams and, thus, it is not possible to modify them

directly. To achieve a modification of the streams, the fol-lowing steps have to be applied (see Fig. 6):

(1) split up traces into streams

(2) modify messages per stream

(3) modify timing behavior per stream

(4) combine streams into new traces.

5.4. Test system architecture

A test system for testing intersection related V2X appli-cations needs, on the one hand, the possibility to stimulate

IVSs with test input data and, on the other hand, the pos-

sibility to record the system’s outputs including automatic

verdict building. Fig. 7 depicts a test hull which encloses

IVSs as SUTs and handles test runs. To target the test needs

for intersection related V2X applications, the test hull con-

tains subcomponents for emulating the behavior of IRSs,

ICSs, ECUs, and localization facilities and subcomponents

for emulating the influences of DSRC, cellular mobile com-

munication, vehicular internal communication busses, and

localization methodologies as postulated in Section 4.

In addition, a test system requires a test management

component to initiate and evaluate test runs as depictedin Fig. 8. These points are realized by a test controller sub-

component and a test evaluation subcomponent. The test

controller subcomponent initiates test runs which are

specified in test suit documents, whereupon, the test hull

applies the test input data to the IVSs. Thereupon, the test

evaluation subcomponent takes the recorded IVSs outputs

and evaluates test verdicts. For test cases which require a

replay of the test runs until a test end criterion is fulfilled,

the test controller subcomponent handles the iterations

original

trace

modified

trace

split merge

value timing

value timing

value timing

Mcon,1 Tcon,1

Icon,1

Mcon,2 Tcon,2

Icon,2

Icon,n

Mcon,n Tcon,n

Fig. 6. Trace modification.

<<flow>>

<<flow>> <<flow>>

<<flow>>

TestHull

CeMo

ViBus

DSRC

Rec.

IRS

ICS

ECUs

IVS

<<flow>>

<<flow>>

<<flow>>

Local. GPS<<flow>> <<flow>>

Fig. 7. Test hull.


http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-



and the test end criteria evaluation. Finally, all relevant

information is stored in test report files.

6. Proof of concept

To proof the concept described in the previous section,

the two examples presented in Section 5.2.3 were used.

As IVS, a CarPC which executes the RVW implementation

from the Audi travolution predevelopment project was

used. First, a short description of the Audi travolution sys-

tem architecture, the used test system architecture and the

used input streams is given. At second, the input stream

modifications and the results of the test runs are presented

for both examples. Furthermore, it has to be mentioned

that only DSRC communication and no cellular mobile

communication is used in this proof of concept.

6.1. Environment

6.1.1. Travolution system architecture

As depicted in Fig. 9, the Audi travolution system archi-

tecture consists of the following components:

CarPC The CarPC acts as IVS and executes the

Audi travolution software. Additionally,

the CarPC generates the visualization

which could be seen in the driver’s

instruments display and in the display

located in the center of the instrument

panel.

CarGate The CarGate connects the CarPC to the

relevant vehicle’s CAN-Busses and thus

to the other ECUs. Moreover, the CarGate

includes a GPS receiver which is used for

the positioning of the vehicle.

V2XGate The V2XGate receives the DSRC messages

from the traffic lights and forwards them

to the CarPC. Communication in the

opposite direction is possible but not

used.

UMTS modem The UMTS modem is used to establish

connections to servers which act as ICSs

and are available over internet. Those

servers are used by some applications as

alternatives to the DSRC communication.

As well as for the V2XGate, communica-

tion in the opposite direction is possible

but not used.

HMI The vehicles HMI displays the visualiza-

tions generated by the CarPC. For that

purpose, there is one display which is

part of the driver’s instruments and an

additional display in the center of the

instrument panel.

For this proof of concept, a commercially available per-

sonal computer based on a 2.2 GHz Intel Pentium 4 CPU

with 1.25 GB RAM and Microsoft Windows XP was used

as CarPC.

6.1.2. Test system architecture

Since only information flows originated from ECUs and

IRSs are relevant for the two examples, it is sufficient to

emulate the behavior of the corresponding components.

Hence, traces for testing the Audi travolution system could

be recorded by sniffing the ethernet communication be-

tween CarGate and CarPC, and V2XGate and CarPC. The test

hull used in this proof of concept has only to include a

component to stub a CarGate which includes the emula-

tion of ECUs and localization facilities (see Fig. 7) and a

component to stub a V2XGate which includes the emula-

tion of intersection’s IRSs. Beyond that, the test hull hasto include another component to record the CarPC’s out-

puts. Hence, the Audi travolution CarPC provides an en-

tirely rendered video stream to the vehicle’s HMI display

(see Fig. 9), the RVW application’s output events are addi-

tionally fed out via UDP datagrams and, thus, can be

recorded by the test hull’s recording component.

TestManagement

TestController

TestEvaluation

TestReport

TestSuit

IVS

TestHull

< < f l o w > >

<<flow>>

<<flow>>

<<read>>

< < w r i t e > > <<flow>>

Fig. 8. Test management.

CarPCV2XGate

CarGate

UMTSmodem

ECUs <<flow>>

CAN

HMI

<<flow>>Ethernet

<<flow>>

Ethernet

<<flow>>

<<flow>>

USB

DVI

Fig. 9. Travolution system architecture.

10 20 30 40

time

in sec

25

30

35

40

45

50

in kph vehicle speed

10 20 30 40

time

in sec

-80

-60

-40

-20

20

40

in degree steering wheel angle

Fig. 10. ECU input trace plot.


http://-/?-

http://-/?-



6.2. Examples

This sub section picks up both examples introduced in

Section 5.2.3, whereas, Section 6.2.1 uses the first and

example and Section 6.2.2 uses the second one.

6.2.1. Example 1

As foreseen in Example 1 in Section 5.2.3, the input

streams were applied in exactly the same fashion several

times. In order to not exceed the scope of this proof of con-

cept, only the following two additional CPU load conditions

were used:

no additional CPU load, and

two threads calculating the 35th fibonacci number in

endless loops in parallel.

For both additional CPU load conditions, the input

streams were applied N = 300 times. The resulting histo-

gram plots for the points in time where the BRAKE signal

appeared are depicted in Fig. 13. In the context of this sub-

section, the difference in time between the 0.02-quantile

and the 0.98-quantile is used as definition of the term

t variation.

It can be seen that the Audi travolution system reacts

onto the applied input streams in normal load conditions

within a variation of t variation = 271 ms, whereas, the BRAKE

signal arises as foreseen approximately 2 s before the

vehicle crosses the stopline. In nontypical heavy loadcondi-

tions, the Audi travolution system reacts reliably within a

variationof t variation = 812 ms. Finally,it has to be mentioned

that in all test runs the BRAKE signal appeared and, thus, no

histograms for faulty test results can be generated.

However, this system behavior was not considered as

sufficient. So, after inspecting the involved software com-

ponents, a non-optimal decision in the software design.

After a redesign and modification of this software compo-

nent, the measurements were repeated. As it could easily

be seen in Fig. 14, the Audi travolution RVW implementa-

tion acts now more deterministic. If the Audi travolution

system runs now in normal load conditions, the variation

amounts now to t variation = 174 ms, whereas, in heavy load

conditions the variation amounts now to t variation = 642 ms.

6.2.2. Example 2

As foreseen in Example 2 in Section 5.2.3, modifications

of the intersection’s center point shifted within its bound-

aries were used. Since no exact boundary specifications

had been available, the following three ranges for

slatitudenorth ; slatitude

south ; slongitudeeast , and s longitude

west were used:

5 m,

3 m, and

1 m

for shifting the intersection’s center point in longitude

and latitude direction. So, for each test run, values for the

uniform distributed random variables for shifting the

intersection’s center point ware determined.

Theresulting – not requiredin thisexample– histogramsare depicted in Fig. 15 and the statistical valuations are pos-

tulated in Table 3. The terms zero (and non-zero) in Fig. 15

indicates that the histogram bars in the referred areas are

really zero (not zero) and not just small. The results show,

that specifications of the acceptable inaccuracy of the posi-

tion of intersection center points may have considerable

influence on the behaviorof RVWapplications if theapplica-

tion is not capable to deal with it.For example,T x¼2min :

valuates

to 14.95 s for a variation range of ±3 m, whereas, it valuates

to 21.00 s for a varation range of ±1 m. Thus, RVW imple-

mentations should indeed be tested regarding this issue.

7. Related work

Hence, testing of V2X communication systems is cur-rently a topic of major research interest; many parties

21.0 21.5 22.0 22.5 23.0 23.5 24.0

time in

sec0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Prob. no additional load

21.0 21.5 22.0 22.5 23.0 23.5 24.0

time in

sec0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Prob. two threads calc. fibonacci(35) in endless loops

Fig. 13. Histograms H l=no_add_load, x = 2 and H l=2_fibonacci_35, x=2 for example 1

with not improved software; bar width w = 0.05 s, number of test runs

N = 300 in each case.

21.0 21.5 22.0 22.5 23.0 23.5 24.0

time in

sec0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Prob. no additional load

21.0 21.5 22.0 22.5 23.0 23.5 24.0

time in

sec0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Prob. two threads calc. fibonacci(35) in endless loops

Fig. 14. histograms H l=no_add_load, x=2 and H l=2_fibonacci_35, x=2 for example 1

with improved software; bar width w = 0.05 s, number of test runs

N = 300 in each case.


http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

http://-/?-

A methodology for testing intersection related Vehicle-2-X applications.pdf

Documents

Transcript of A methodology for testing intersection related Vehicle-2-X applications.pdf