Evaluating Road Safety Audit Proceduressites.poli.usp.br/p/hugo.pietrantonio/eRSAStudy.pdf ·...
Transcript of Evaluating Road Safety Audit Proceduressites.poli.usp.br/p/hugo.pietrantonio/eRSAStudy.pdf ·...
EVALUATING ROAD SAFETY AUDIT PROCEDURES: 1
SOME QUESTIONS AND A NEW METHOD OF STUDY 2
Hugo Pietrantonio, D.Sc. (Corresponding Author) 3
Lecturer, Traffic Engineering 4
Department of Transportation Engineering 5
Escola Politécnica – University of São Paulo, Brazil 6
Edifício de Engenharia Civil, Cidade Universitária 7
São Paulo/SP, CEP 05508-900, Brazil 8
Email: [email protected] 9
Phone: +55-11-3091-5492; Fax: +55-11-3091-5716 10
11
Lili L.Bornsztein, M.Sc. 12
Traffic Analyst (Superintendence of Traffic Engineering) 13
Traffic Operations Division 14
Traffic Engineering Company, Prefecture of São Paulo/SP, Brazil 15
Alameda Ministro Rocha Azevedo, 976–ap.72, Cerqueira Cezar 16
São Paulo/SP, CEP 01410-003, Brazil 17
Email: [email protected] 18
Phone: +55-11-3081-9838 19
20
Abstract: The paper discusses current views on the application of road safety audits (RSA) as a tool for 21
the improvement of road safety and proposes a new method of study for detailed evaluation of some 22
features that can influence their effectiveness, which is demonstrated in the practical setting of road safety 23
audits applied to existing roads review or inspection. Starting from a review of published references on 24
RSA in different countries, the identified a set of questions and undertook a case study of the impact of 25
alternative RSA procedures on its final effectiveness. The RSA effectiveness is measured both as 26
agreement to a safety expert assessment and to an accident-based study. Main features analyzed in the 27
case study included the type of observer used for data collection and the type of checklist used as guide to 28
field work. Based on an exploratory study, the performance of RSA application is evaluated using 29
weighted indexes of concordance and disagreement, and the rating of detection or omission for the 30
observations gathered in the accident diagnosis of the safety problems at the intersection studied, 31
supplemented by a statistical analysis of the influence of selected covariates on these scores. The main 32
results of the case study can be summarized as clear support to team work for field observation and 33
support to employing less experienced personnel for field observation. The study also stresses the need of 34
procedures/criteria for priority setting. These conclusions are useful to the selection of alternative RSA 35
procedures in agencies responsible for promoting or enforcing RSA and in professional teams carrying-out 36
RSA tasks. Several features deserving further study are also identified. 37
38
Key words: road safety audits, road safety reviews, statistical analysis of safety measures. 39
40
41
Pietrantonio, Bornsztein 1
1
EVALUATING ROAD SAFETY AUDIT PROCEDURES: 2
SOME QUESTIONS AND A NEW METHOD OF STUDY 3
Hugo Pietrantonio and Lili L. Bornsztein 4
5
1. INTRODUCTION 6
The use of Road Safety Audits (RSA) as a tool for improving traffic safety has received 7
increasing attention in several countries as a result of its development and dissemination from the UK to 8
Australia, New Zealand, to continental Europe, Canada, USA, and also to developing countries such as 9
Brazil and Chile. Examples of recommendations that are widely known and recognized in several of the 10
leading countries can be seen in IHT (1991, 1996, 2008), DoT (1994a, 1994b, 2003), and Austroads 11
(1994, 2002, 2006). Similar recommendations, but with relevant changes, were adopted by late-comers as 12
can also be seen in references INST (1995), Road Directorate (1997), TAC (2001, 2004), CONASET 13
(2003), and FHWA (2006), as examples. 14
There is a favorable position among the general public, politicians and technicians supporting 15
RSA application as a preventive measure that tries to avoid accidents through a (hopefully low cost and 16
effective) revision process and also saving money in future (more expensive) road changes. As usual when 17
reviewing subjects covered by favorable views, the “previous” general agreement on the merit and the 18
purposeful defense of the positions do the bad job of hiding some points that vary among different 19
concepts and practices, features that can change its results and its value. 20
The careful analysis of concepts and practices for RSA application is interesting as it stresses 21
some overlooked questions and clarifies the need of more detailed methods for studying RSAs. Despite 22
the widespread acceptance of the general idea, one can observe that differences in proposals for the RSA 23
technique are not stressed or at least noticed by scholars or practitioners. Also, it is important to 24
disentangle features that are judged to be essential from other features that are a matter of detail and 25
should be tailored to the needs and means of each application. In our view, some of these differences and 26
details can compromise the effectiveness of RSA application and should be carefully studied. 27
In the following sections, the paper tries to carry-out three tasks. First, it discusses the main 28
features that vary among different concepts and practices for RSA (in sections 2 and 3). Then, it presents a 29
method of study that can be used in the comparative effectiveness of alternative procedures for field work 30
and reports the results of its use in the application of RSA procedures at a signalized intersection (in 31
Pietrantonio, Bornsztein 2
section 4 and 5). The final section summarizes the main conclusions that can be delivered by the research 1
on the effectiveness of alternative RSA procedures and on the validity of the method of study. 2
From the outset, one should note that the results in this paper are limited to the RSA application to 3
existing sites (being only partially applicable to road projects) and that the method should be extended and 4
improved before using it for the study of RSA application in the design phase of road projects (usually its 5
main scope). Some authors or institutions distinguish the applications of RSA principles to existing roads 6
and treat it as different technique (called inspection or review, instead of audit, as in EC, 2008, even if 7
considering the monitoring of their first years as in DoT, 2003). However, application to existing roads is 8
acknowledged and described as RSA in several of the original guidelines (e.g. IHT, 1991) and current 9
ones (mainly, Austroads, 1994, 2002, 2006). The vision here is that the techniques, and their problems, are 10
very similar and that the major part of the discussion should apply to both applications. 11
Also, one should note that the problems encountered in the evaluation of RSA procedures are also 12
relevant for other studies (e.g. the evaluation of other surrogate measures of safety, such as traffic 13
conflicts, conflict opportunities and exposure measures). Nevertheless, each technique can have features 14
that can not be evaluated with the methods proposed. This limitation applies even to RSA as some general 15
questions (e.g. on where and when to use RSAs) are outside the realm of this work and its methods (which 16
are more useful in the study of procedural alternatives for RSA work and its tasks). 17
18
2. ALTERNATIVE VIEWS AND CONCEPTS OF ROAD SAFETY AUDITS 19
The very idea of an audit on road safety can be surely traced back to the independent (third party) 20
evaluation of safety by a qualified expert (the way it was originally applied on Surrey County assessments, 21
as reported in Bullpit, 1996). Nevertheless, as the original RSA concept stresses (e.g. IHT, 1991 or 22
Bullpit, 1996), it is not a matter of checking. Instead, RSA must add the view on road safety of a qualified 23
professional. The exclusive focus on traffic safety is another specific point, as a mean for warranting 24
attention to its specific concern (road safety), also a feature universally acknowledged in RSA definitions. 25
Departing from audits in other areas, such as accounting and management, the ex-ante (preventing 26
bad results) feature is stressed in RSA (instead of the ex-post checking of practices). For some authors, 27
RSA when applied to the design stage could be considered more similar to the preventive procedure 28
applied in other areas. However, the justification for taking the application to existing roads as another 29
technique, Road Safety Inspections-RSI, seems to be weak as it can be preventively applied to existing 30
roads as well (the major problem, especially in less developed countries). Otherwise, the word assessment 31
would be better than audit in conveying this peculiar meaning of the RSA concept, similarly to areas such 32
as environmental impact assessment (sharing the same preventive and complementary concerns), but 33
Pietrantonio, Bornsztein 3
distinctively from its general use in accounting and management. Despite agreeing with the latter view, we 1
keep the usual denomination used for the technique, as it is widely adopted nowadays. 2
After this basic agreement, we identified seven features on which the final concept varies: 3
- the essential qualification of experts (especially about accident investigation); 4
- the coverage of RSA application (wideness of the scope for preventive action); 5
- the role of systematic checking (and the importance of checklists); 6
- the requirement for formal reporting (from auditors and from designers); 7
- the focus on human factors and/or all users (from pedestrians with disabilities to truck drivers); 8
- the focus on special road conditions (such as day/night, road works, adverse weather); 9
- the role of multidisciplinary or multirole teams (and their composition). 10
Some of the variations can be seen as inheritance of old questions (traces of the evolution in the 11
concept of RSA) but others are actual and relevant questions meriting careful consideration and study. 12
Among these features, coverage and scope have shown a clear path of evolution. For example, the 13
possibility of applying RSA procedures to existing roads (not only to projects), to maintenance work, to 14
construction planning and execution, to operating procedures, all were finally recognized in a progressive 15
trend, unfolding several fields in which the technique is potentially effective (because they identified areas 16
that bear some burden in road safety and accidents). 17
Another example of evolution is related to the special focus on the needs of all users and on the 18
requirements of all road conditions, just reminding features that are usually present in the causation of 19
road accidents and that supposedly are topics have important shortcomings in current design guidelines (a 20
feature identifying an area of special concern or value for RSA application today). However, these critical 21
concerns can change along time as better guidelines are developed or as new areas of concern appear 22
(maybe due to changes in urban patterns or automation tools, for example). These features can thus be 23
taken as episodic (instead of essential) in defining the concept of RSA. 24
Sometimes it is difficult to distinguish historical questions that evolved during the maturation of 25
the concept and technique (and that were “solved” or “settled” in some sense) from current questions that 26
should be discussed (which leaves some room for personal judgment, of course). 27
For example, in the original sources of the RSA concept, the main requirement for an independent 28
and qualified auditor was expertise on accident investigation. This feature was stressed as a mean to bring 29
the specific knowledge gained in the analysis of traffic accidents to the design process, based on the 30
judgment that design guidelines were insufficient to fully warrant traffic safety (because they stressed 31
other goals or because they use to neglect the importance of some details that matter to safety). 32
Pietrantonio, Bornsztein 4
This view was clearly stated in the historical review by Bullpit (1996, p.211) when the author 1
notes that the basis of RSA are both the principle “prevention is better than cure” and the use of 2
accumulated knowledge in accident investigation and prevention. The same point is highlighted in the first 3
version of the U.K.IHT Guidelines for Road Safety Audits (IHT, 1991, p.9). Nevertheless, the withdrawal 4
of this highlight from the second edition of the U.K.IHT Guidelines must also be noted, where one can 5
find stronger emphasis on the need of improved and updated knowledge for the experts (IHT, 1996, p.20). 6
What about the change? Of course, RSA knowledge can be a source for the development of road 7
design guidelines and the confidence in the accident investigation and prevention expert has to be 8
conditional. Despite recognizing the importance of the specific knowledge gathered in accident 9
investigation, one can expect that its content should be progressively subjected to scientific investigation 10
and integrated in the evolution of design guidelines sooner or later. One must also recognizes that the 11
knowledge of accident investigation experts is subject to idiosyncrasies and changes (and that there are 12
good and bad experts, even among those with experience on accident investigation). 13
Changes in concept can be the result of changes in the specific or historic settings but can also 14
sign changes in points of view concerning the technique or its practice (changes that can perhaps 15
compromise its final efficiency and effectiveness). For example, the pressure for “certified auditors” can 16
also explain the change in the view about qualified experts. 17
For better, the evaluation of current design guidelines and of RSA criteria should be an empirical 18
question as should be the evaluation of actual experts´ competence. The adoption of one or other view on 19
the concept would be a matter of faith unless one can evaluate the practical effectiveness of each option. 20
Another example: the diffusion of RSA procedures in the mid 1990´s brought an over stressed 21
attention to checklists as a tool for applying RSA (and also for communicating its principles and criteria 22
on practical grounds, of course). Some authors mentioned its use as an essential feature of RSA (e.g. 23
Proctor, Belcher, 1993). The original view (IHT, 1991, Bullpit, 1996) took its use as a mean for reducing 24
conflicts between the RSA and design teams and for coding RSA knowledge (to boost its adoption by 25
designers also). Nowadays, several RSA guides prefer to recommend the use of generic prompt-lists (e.g. 26
TAC, 2001, and FHWA, 2006, the more recent sources for Canada and USA), as opposed to checklists, 27
preserving (or restricting) their use by experienced observers. 28
Objectively, the development of lists has been carried-out by every RSA team and there are 29
several examples of real implementations. Today, one can find checklists or prompt-lists with variations in 30
content and organization, in the type of questions and also in their coverage, even in the form and mode of 31
application, but all these features following personal taste rather than application tests. 32
Pietrantonio, Bornsztein 5
Unfortunately, the development of these lists did not track well-defined and certified paths, 1
despite requiring considerable work. Then, one can not expect that their use would surely contribute to 2
deliver better results per se. A critical assessment of their content (against each other or against 3
conventional knowledge) gathers really disappointing results. The analysis carried-out by Bornsztein 4
(2001, chapters 3 and 4), that examined checklists for signalized intersections only, identified several 5
questions that were set on excessively generic terms, failing to encode the available safety knowledge as 6
represented in similar guidance used for conventional road safety analysis (e.g. TRRL, 1991, Gold, 1998). 7
For example, the analysis of road surface on a very detailed Canadian checklist (Hildebrand, 8
Wilson, 1999, A-20 or B-12) asks for checking skid resistance along curves, intersection approaches and 9
steep grades (where we know it is critical), then providing valid hints. Nevertheless, the questions on 10
pavement defects and depression areas are generic. Despite being one of the best checklists available in 11
matter of coverage and detail, it is easy to find similar deficiencies in several subjects regularly considered 12
by road safety analysts (a must for auditors too). 13
Another case: the adequate treatment of pedestrians in traffic signals should avoid usual 14
operational traps (vehicle flows starting/stopping at different stages in the same approach, exposition to 15
permitted left turns or leading protected left turns, etc.) and avoid intended/unintended violations (because 16
of excessive delay to pedestrians, of large gaps in priority vehicular flows having right of way, especially 17
at high speed sites, with large crossing distances or binding sight constraints), all these features are 18
unnoticed in the Canadian checklists quoted (see Hildebrand, Wilson, 1999, A-16, 26 or B-10, 11, 16, 17). 19
In some other areas, even the accumulated knowledge of safety analysts is really vague (e.g. on 20
the effect of signs in accidents or on when they are really critical to their occurrence). Use of checklists 21
can ask for improvement of (current) views, and permit their evaluation, as opposed to prompt-lists. The 22
introduction of another phase for the monitoring of the RSA process (Stage 4 in DoT, 2003) after the first 23
12 and 36 months of project opening, should be welcome for providing a toll for filling this need (even if 24
yet to be fully integrated into standard practice; see IHT, 2008; Belcher et al., 2008). 25
Of course, lists serve more than one function (from a formalizing tool that favors an objective and 26
fair evaluation to an aide-mémoire guide for field work) but its main use is instrumental and should be 27
judged by its effectiveness in contributing to the RSA goal (i.e. the prevention of accidents). Again, their 28
adequacy must be investigated on empiric grounds in the task of identifying accident contributory factors. 29
Some other features can, perhaps, be taken as simple matters of concept and practice. For 30
example, formalization of reports (by the auditor and by the designer) has a clear role and function: 31
warranting explicit consideration to all and every question identified as potentially dangerous and prone to 32
cause accidents. There is no informal RSA (an informal procedure would constitute another practice). This 33
Pietrantonio, Bornsztein 6
feature can be taken as definition, building a practice devoted to warrant that due attention is devoted to a 1
public matter (so the “project owner”, as the sponsor, is also subject to the requirement). 2
Finally, for other questions, the answer can depend on the nature or scale of work (for example the 3
composition of the RSA team and the procedures to be followed on the field tasks). Therefore, concepts 4
and practices tailored to different types of work should be sought. 5
For every advocate or contender of RSA, the precise view on the concept and its essential features 6
should be clearly stated. Our working position is the following one: 7
- the essential qualification of the expert is to have good knowledge about accident causation and 8
correction (not only road design), acquired through experience or at least specific training for the subject; 9
- the preventive action of RSA covers all stages and activities that bear a relevant effect on road safety (i.e. 10
RSA applies to different stages of design and construction, to pre-opening and initial operation, to existing 11
roads and operational procedures, and so on); 12
- systematic checking (and the use of checklists) in RSA procedures is a means (not an essential feature) 13
for achieving a complete evaluation of features, establishing valid and fair criteria, minimizing 14
subjectivity in results and avoiding conflicts with design experts (relevant points, for sure); 15
- formal reporting is a need for warranting real attention to all safety concerns through the documentation 16
of statements and for delimiting responsibilities of designers (the final decision-makers) and auditors (in 17
their advisory role); 18
- the focus on all users and conditions is a technical requirement for RSA, due to their importance in the 19
realm of road safety and the lack of consistent consideration of several needs and limitations (mainly for 20
vulnerable users) in current design guidelines; 21
- the need for multidisciplinary or multirole team, or any other procedural RSA feature, is a practical 22
matter that should justify for itself in each RSA type (multidisciplinary background is a conventional 23
requirement for a traffic safety engineer or expert). 24
Before leaving these conceptual remarks, a final comment should be directed to the role of 25
checklists. Despite over stressed, checklists are the current tools for consolidation of the RSA knowledge 26
base. In our view, their content should receive greater attention than usual and be submitted to systematic 27
and critical evaluation. The lack of studies and methods for justifying the content of proposed checklists as 28
well as the vagueness (more than openness) of most of the questions can undermine their knowledge role. 29
The wide variation in presentation (mostly the level of detail) is also a matter of study (in our study, we 30
related it to the kind of professional performing the extensive field observation). Last, but not least, the 31
need of objective observational hints for identifying a feature as defective or the need of criteria for 32
weighting its relevance to safety is also something to be noted. 33
Pietrantonio, Bornsztein 7
The points made above can be summarized by stressing that most questions should be the subject 1
of careful empirical studies, maintaining a clear concern in warranting or preserving the potential 2
effectiveness and efficiency of the RSA application practice, instead of taking them as a matter of faith or 3
preference only. Our method of study was conceived to contribute to this effort. 4
5
3. ALTERNATIVE PRACTICES FOR ROAD SAFETY AUDITS 6
As regards practice, two conventional sets of questions should be recognized: practice in RSA use 7
(when and where) and practice in the RSA process (who and how). Other more detailed (and disregarded) 8
questions are related to another set: the procedures applied on RSA work. 9
From the RSA guidelines, a clear trade-off can be observed in setting a rigorous procedure or 10
keeping it a simple and straight task that can be applied with small burden on project costs and schedules. 11
This goal stresses the importance of identifying practices that can deliver a reward for their burden. 12
In the first set of questions (on the use of RSA) are the requirements that elevate it to more than a 13
recommended technique for a class of projects, either as a fully mandatory procedure for the overall set 14
(as for trunk road schemes in the UK) or for a sampling rate of the undertaken projects (perhaps 15
depending on their scale or nature, as in Australia and New Zealand, for small scale projects or works). 16
Despite recognizing that the kind of requirement or motivation for undertaking RSAs could be a factor in 17
determining its final effectiveness, these issues are not of main interest here. 18
We concentrated the investigation on the second and third set of questions, viewed as interlinked 19
ones. Among these questions, we identified six main features on which the practice varies: 20
- the degree of formalization in the process and the allocation of roles (for auditors and others); 21
- the composition of the RSA team (if more than the expert is involved in it); 22
- the identification of tasks (and their allocation among team members); 23
- the procedure for field work (observation team, number of visits, periods/tasks of work); 24
- the type of supporting tools used for RSAs (as measuring or registering devices); 25
- the type of checklists used for RSAs, in special (and its relation to qualification of team members). 26
Some of these questions are really unnoticed by current guidelines, despite its apparent 27
importance for the potential effectiveness of RSA application. Some of them may have effects that depend 28
on the particular use of RSA as they can set the context for small or large scale applications. This 29
dependence could be complex and it was a strong motivation for this research. 30
For example, despite being optional instead of mandatory, the most widely known applications of 31
RSA in Brazil are large scale ones. The first application dealt with 1507 signalized intersections in the city 32
of São Paulo, those that were converted to real time control after 1994 (Scaringela et al., 1999). After, 33
Pietrantonio, Bornsztein 8
RSA applications moved to overall networks of highways under concession, also voluntarily applied by 1
some of the new infrastructure operators in the State of São Paulo (following current trends of 2
privatization of road administration based on tolling schemes), covering some hundreds of kilometers, all 3
at once (Martinez, 2000). 4
This practice weakens the requirements for formalization (because the application of RSA were 5
motivated by the contractors) but enforces the interest in efficient processes (combining expert role in 6
judgment matters with clerical roles in field observation) and quality assurance in RSA (including 7
training, checklists, supervision, rechecking and so on). Concerns with the multi-role composition of 8
teams is markedly different from the usual recommendation of multidisciplinary composition by 9
emphasizing the combination of functions (rather than competences in complementary subjects). 10
Furthermore, the adoption of a mandatory requirement for RSA in a country with “soft” legal 11
commitment to tort liability, such as Brazil, means a danger of a “bureaucratic” action (i.e. safe on paper, 12
but not on road). The legal apparatus is not the weak point in Brazil, as codes are very similar to European 13
ones, but the lack of commitment to tort liability comes from a fragile institutional framework that 14
undermines the enforcement of law. Calling for attention to RSA use/guidance should be rooted in greater 15
efficiency and effectiveness than in the conventional institutional framework of a developed country. 16
The importance of the practical issues on RSA application raised above (in any context) seems to 17
be obvious and the main goal of this discussion is to stress the need of informed evaluations for several 18
components of practical RSA procedures. 19
Without being exhaustive, one can suggest that at least the following points should be addressed: 20
- the degree of dependence on a safety expert, as it could be a constraint for application of RSA at large if 21
it is not possible to warrant an efficient work routine, combining clerical personnel (or junior professionals 22
at least) and expert supervision but without compromising effectiveness; 23
- the composition of teams, with evaluation of the comparative performance of personnel with different 24
qualifications in alternative tasks as: 25
. field observation, 26
. specialized evaluations, 27
. rating of risk or danger; 28
- the type of supporting tools used, as guide to work (especially, the type of checklist), studying the 29
importance of tailoring them to each work routine for preserving the efficiency of the RSA process, at 30
least in large scale applications (including RSA preparation, e.g. training tasks); 31
- the degree of integration and enforcement of RSA activities and the commitment to tort liability in 32
agencies or regions, features that should be relevant for the design of RSA process, taking into account: 33
Pietrantonio, Bornsztein 9
. mandatory or recommended RSA application, 1
. level of rigor in calling for attention of RSA use/guidance; 2
- the organization of the RSA work, including field and office activities and the commonly stressed 3
relationship between auditors and designers, their specific requirements and responsibilities, dealing with: 4
. degree of interaction in the RSA team, 5
. procedures for analysis and prioritization, 6
. interaction between RSA team and decision teams. 7
Concluding the theoretical review, our point must be made clearly: the efficiency and 8
effectiveness of RSA must be carefully studied on the field. All effort should be directed to selecting 9
appropriate methods of organization and work that can warrant good results with compatible costs (money 10
and time). Also, recalling what was said at the beginning of this section, rigorous procedures should be 11
kept at a minimum to avoid compromising RSA viability itself. 12
13
4. A NEW METHOD FOR EVALUATING THE EFFECTIVENESS OF ROAD SAFETY 14
AUDITS 15
The previous discussion set forth some questions and the motivation for better evaluation of RSA 16
concepts and their procedures. It remains to deliver a methodological approach that can overcome most of 17
the difficulties one would face in RSA evaluation studies. 18
The study of RSA application has a lot of specific problems coming from its own features (and 19
that should be added to the usual difficulties of traffic safety studies and the analysis traffic accident data). 20
Most data on the effectiveness of RSA application are based on efforts that aimed at evaluating 21
the cost/benefit ratio attained by specific audit trials. For example, based on studies carried-out in Scotland 22
and New Zealand, cost/benefit ratios of 1:15 and 1:20 are usually quoted in several references (Bullpit, 23
1996, pp.413-4, Road Directorate, 1997, p.6-2, CONASET, 2003, p.10). These studies are not clearly 24
presented or discussed in the references but one can grasp that these methods are not prone to the analysis 25
of alternative safety audit procedures in detail. 26
Therefore, a more detailed method of study is needed for evaluating the kind of questions set 27
previously. Nevertheless, difficulties are pervasive. 28
It is usually quoted that, as a preventive action that aims at avoiding traffic accidents, the 29
application of a RSA destroys by itself some of the evidence needed for certifying its value (the accidents 30
that would be caused by the “potential” defects present in the original situation or proposal). Of course, 31
the use of RSA can be regularly evaluated after the implementation of its recommendations, against the 32
alternative scenario of a do-nothing that can be described by the before situation. This procedure will 33
Pietrantonio, Bornsztein 10
evaluate the accepted recommendations from RSA (of RSA in relation to observed accidents if all 1
recommendations were accepted) and will be open to all criticism driven to before and after studies 2
(mainly the presence of selectivity bias and influence of exogenous factors). 3
Either if data on accidents in the before period is unavailable (as when considering a new road), if 4
it can not be used as the base for evaluation (because of differences between before and after scenarios) or 5
if the effects of RSA recommendations were not observed, this procedure can not be applied. This 6
difficulty is especially relevant for evaluating RSA applied to the design stages or applied as preventive 7
action for existing sites (as no before data can make evident the accident risk), at least without confidence 8
on techniques that can be applied for predicting what would be the accident frequency without the 9
interventions recommended by the RSA application (see Macaulay, McInerney, 2002, McInerney, 2003, 10
and Montella, 2005). 11
Of course, one can try to verify the comparative performance of audited projects against control 12
projects but, in a field in which the effect of random and exogenous factors are so important (as traffic 13
safety), the exercise could be subjected to weaknesses that will shed relevant doubts on results (or at least 14
weakens its statistical significance). This kind of with/without method is a much needed approach for 15
evaluating RSA application, through the comparison of the safety performance of audited or non-audited 16
projects, but would be probably insufficient for elucidating all the set of questions previously discussed. 17
The need (and cost) of subsequent corrections is also a clue on the effectiveness of RSA 18
application to projects, yet to be better explored. The responsibility for keeping the road safe should be 19
less burdensome in audited projects (or in the years after auditing roads). Data on these features could be 20
easier to recover but they will leave aside the evaluation of final safety or would have to compensate for 21
different levels of attained safety in projects or works. Ideally both should be studied together (the final 22
level of safety achieved and the need/cost of interventions after project implementation) and will require a 23
long observation time to provide data required for the evaluation. 24
This view suggests a better approach in which the use of RSA is compared to a do-minimum 25
scenario with monitoring of interventions (instead of a do-nothing scenario without interventions). 26
All these approaches (i.e. the monitoring of accidents after changes, the comparison of accidents 27
against control sites and the follow on of corrections after changes) are mandatory (as they can deliver 28
strong evidence of the effects of RSA on the road) and should be improved. But all of them are coarse for 29
the use intended in this study. 30
This paper presents an alternative approach for studying RSA application that is weaker than 31
certifying its final effect based on accident and cost records but that could be potentially richer for 32
evaluating the effects of detailed procedures and tools used on the field. Complementary methods are, 33
Pietrantonio, Bornsztein 11
naturally, desirable. The proposed method aims at scientifically evaluating the promises of RSA 1
applications (that could affirm their potential) against some gold standard of diagnosis. 2
This new method has two tools: an evaluation of alternative RSA results and an evaluation of 3
comparative results to AAD (Accident Analysis Diagnostic). Both tools are based on the availability of 4
some Gold Standard (i.e. an accepted method). These components can be taken almost independently. 5
Both analyses are carried-out in the context of simulated RSA applications and they can not evaluate 6
features that appear on real applications only (as those questions related to commitment or implementation 7
of recommendations). 8
The analysis of alternative RSA results can be used to evaluate the application of RSA in all 9
stages (as it relates RSAs) but the analysis of comparative results to AAD can not evaluate the design of 10
all-new projects or even the design of corrective action without further adaptation (as it partially depends 11
on previous accident records). Each tool is distinguished by the evaluation that is taken as the Gold 12
Standard. 13
The evaluation of alternative RSA results asks for the comparison of each alternative procedure to 14
some standard way of doing a RSA. The evaluation is intermediate in the sense that it compares 15
observations gathered with alternative RSA procedures against the agreed standard and not the quality of 16
the results by themselves. Here, the fundamental hypothesis of evaluation is simple: similar results favor 17
the more cost effective procedure or can permit the selection based on the elicited cost/quality trade-off. 18
The evaluation of comparative results to AAD judges the effectiveness of RSA application by 19
comparing the diagnostic of safety problems based on RSA for real sites against equivalent results based 20
on traditional accident studies. It is also intermediate in the sense that it compares results from the 21
diagnostic of problems (instead of trying to verify the real effect on accident and cost records). But now 22
the fundamental hypothesis is that the validity of the RSA application can be demonstrated by showing 23
that it can reach similar conclusions about the safety problems of the studied sites to those that would be 24
gathered from detailed accident data. So, it will decide corrections that would also be similar to the ones 25
decided by accident studies (usually with reduced costs for the traffic agency and for the road users). At 26
least, this comparison can demonstrate that RSA can reach similar results to accident analysis even if 27
accident data are unavailable (even more if accident data are available). 28
The comparison against a Gold Standard is the conventional method of evaluation for diagnostic 29
methods in health studies (see Zhou et al., 2002). In our case, the Gold Standard of each tool is the 30
diagnosis using the standard way of doing RSA for the evaluation of alternative RSA or the diagnosis 31
using traditional accident analysis in the evaluation comparative to AAD. It is surprising that these 32
Pietrantonio, Bornsztein 12
methods were not previously suggested as a main alternative in the road safety area or, at least, were not 1
similarly conceived. 2
Our study also felt the need to complement these “objective” evaluations with a “subjective” 3
rating (based on technical judgment) about the correctness and importance of further concerns identified 4
by RSA (but not related to the available sample of accident records). Similar approaches were used by 5
other studies. For example, the evaluation of a Road Safety Risk Index by de Leur, Sayed (2002) also felt 6
the need of using an evaluation (partially subjective) of the proneness to accident and its consequences, 7
that was used to weight observations and then called for an objective comparison with accident 8
occurrences (delivering an accident-based measure for validation of the proposed index and procedure). In 9
Australia and New Zealand, this point was developed further and incorporated in a software system (the 10
Road Safety Risk Manager) that can be used in setting priorities to recommendations emanated from 11
RSAs (McInerney, 2003). 12
This effort corresponds to the need for evaluating the consequences of defects. Ideally, measures 13
of the impact on welfare or cost to road users and other roles in society would be required. A large body of 14
literature on the evaluation of diagnostic tests in health studies also deals with the assessment of their 15
economic effectiveness in a wide sense (i.e. considering social costs or welfare, as in Felder, Mayrhofer, 16
2011), by embedding the effects on a framework for assessing/ordering decisions under risk and/or 17
uncertainty. But this further step would require some knowledge about the final effectiveness of RSA 18
recommendations, as in our previous study on the traffic conflict analysis (Pietrantonio, Tourinho, 2006). 19
Our results can be analyzed with simple cross-tabulation methods and subjective evaluation or can 20
be analyzed with conventional statistical modeling methods and inference criteria. Both approaches will 21
be exemplified in the following case study and can assess a wider range of questions on RSA procedures 22
than other methods. Statistical modeling is preferable for evaluating the effect of the procedures 23
controlling for several confounding factors that can affect the results of application work, despite the 24
recognized lack of robustness of techniques based on correlational data (such as regression analysis). 25
Of course, the proposed study approach is not free of criticisms. The case study will demonstrate 26
several shortcomings that can found in the application and the analysis of results as well. The final section 27
will also stress some general points on methodological strengths and weaknesses of the approach. 28
29
5. CASE STUDY OF ALTERNATIVE ROAD SAFETY AUDITS PROCEDURES FOR 30
SIGNALIZED INTERSECTIONS IN THE CITY OF SÃO PAULO/SP, BRAZIL. 31
The city of São Paulo has some 10 million inhabitants and heads a metropolitan region with about 32
18 million inhabitants. It has a very trafficked road network. The congestion level is noticeable for any 33
Pietrantonio, Bornsztein 13
visitor of the city. The freeway network is comparatively small (around 40 km of freeways that almost 1
circulate the main consolidated area of the city) and the overall road network is heavily dependent on the 2
arterial corridors. Signalized intersections with vehicular and pedestrian peak hourly flows reaching the 3
tens of thousands are usual in the city. The sites selected for study are examples of these usual signalized 4
intersections, located in the consolidated part of the city (the fringe area has smaller junctions). 5
The case study selected four observer types and developed three types of checklists for field work. 6
These are the main features of RSA evaluated in this research. All observations were carried-out as 7
simulated RSAs following common required routines that can be summarized as a first visit to the site 8
during the afternoon peak, extended to the subsequent (night) off-peak period, and a second visit at the 9
morning peak, extended to the subsequent inter-peak period, complemented by subsequent visits in special 10
periods as selected by the observer (nevertheless, real routines varied a lot around the requested one). 11
The different observers were classified as Safety Engineering Expert (Exp, with experience in 12
accident investigation and knowledge of RSA application), Senior Traffic Engineer (Des, with experience 13
in road and signal design), a Road Safety Engineer (SAn, with more than 5 years experience in accident 14
analysis) and a Junior Traffic Engineer (Jun, with basic professional training in traffic engineering, 15
covering design and safety). Checklists were developed, after reviewing several sources (IHT, 1991, 1996, 16
DoT, 1994, Austroads, 1994, INST, 1995, , Hildebrand, Wilson, 1999), keeping in mind these types of 17
observers. A Generic Checklist (G) was developed for the Safety Expert (Exp). A Detailed Checklist (D) 18
was developed for the Design Engineer (Des) and for the Safety Analyst (SAn). A Detailed Checklist with 19
Instructions (I) was developed for the Junior Engineer (Jun). 20
The basic applications are named ExpG, DesD, SAnD, JunI, combining both features, an 21
observer type and a checklist type. Some crossed applications were also carried-out for comparison with 22
the same observer (applied with increasing detail) and a Detailed Checklist of the Safety Expert (ExpD) 23
was always gathered after the regular ExpG task, as it is the standard of most comparisons. Table 1 24
presents the general structure of G and D checklists (I checklists are similar in structure to D checklists 25
with added observation instructions). 26
The research team was composed by the RSA supervisor, who carried-out all the research tasks 27
involved in the submission and collection of RSA checklists, and an independent AAD engineer, who 28
collected accident records for both intersections and did all the research tasks involved in the accident 29
analysis diagnostic. Both worked together in evaluating the results. 30
The practical work applied the proposed RSAs at four signalized intersections but, because of the 31
high number of hours required for application of the field work and comparison of observations in 32
checklists, the application of the analysis procedure were tried in two intersections. Furthermore, given the 33
Pietrantonio, Bornsztein 14
lack of confidence on traffic accident records (especially regarding the exact location of accident 1
occurrences on the intersection), full data was considered more adequate for one intersection only and its 2
results are reported here. 3
Note that the large number of hours required for field word and comparison of observations is 4
greatly determined by the large number of questions involved in our checklist (43 to 226 per RSA, as 5
shown in Table 1). From a methodological point of view, the method can be better applied to a smaller 6
number of questions whose status is surely known (a clearly deficient and dangerous feature, a clearly 7
deficient but almost safe feature, a clearly sufficient but critical design, and so on), delivering a more 8
controlled application. Reducing the time required for data acquisition and processing would permit large 9
scale studies on alternative RSA methods. Nevertheless, as the evaluation comparative to AAD depends 10
on accident data, this would be a constraint to any application of its further evaluation. 11
12
Pietrantonio, Bornsztein 15
Table 1. General Features of Generic and Detailed Checklists. 1
Topic Item No.Generic
Questions
No.Detailed
Questions
1. CHOICE OF TYPE AND GENERAL LAYOUT OF INTERSECTION
1.1. INTERSECTION
LOCATION
A. LOCATION IN THE ROAD NETWORK
a. consistency (road and surroundings)
b. functional hierarchy
c. speed approaching the intersection
B. LOCATION IN THE URBAN TISSUE
C. LOCATION OF PEDESTRIAN PATHS
1
1
2
2
1
7
3
11
9
5
1.2. TRAFFIC
CONTROL TYPE
A. SELECTION OF SIGNAL CONTROL 1 5
1.3. INTERSECTION
LAYOUT
A. GENERAL GUIDELINES OF LAYOUT
a. restrictions to movements
b. geometric constraints
c. layout comprehension
d. attention to pedestrians
B. OTHER GEOMETRIC DESIGN ELEMENTS
a. general design
b. physical features of pedestrian crossings
2
1
1
1
3
1
6
5
5
8
12
7
1.4. VISIBILITY IN
THE INTERSECTION
A. VISIBILITY FOR STOPPING IN APPROACHES TO
THE INTERSECTION
B. CONSPICUITY OF INTERSECTION
C. VISIBILITY OF PEDESTRIAN PATHS
1
1
1
6
2
5
2. INTERSECTION
PAVEMENT
A. PAVEMENT FRICTION
B. PAVEMENT DISTRESSES
C. PAVEMENT DRAINAGE
1
1
1
4
8
5
3. INTERSECTION
LIGHTINING
A. VISIBILITY OF THE INTERSECTION AT NIGHT
B. CONTRIBUTION OF LIGHTING TO VISIBILITY OF
SIGNS
1
1
10
9
4. SIGNING A. ROAD MARKINGS
B. VERTICAL SIGNS
C. PEDESTRIAN CROSSINGS
1
1
1
9
9
3
5. TRAFFIC SIGNALS
5.1. VISIBILITY IN
SIGNALS
A. VISIBILITY AT A DISTANCE
a. stopping sight distance
b. conspicuity of signal heads
c. positioning and understanding of signal heads
d. obstructions and interferences
B. COUNFOUNDING AT A DISTANCE
a. vision of signal heads of conflicting movements
b. signal heads of adjacent stop lines sited in the field
of view of approaching drivers
C. VISIBILITY AT THE STOP LINE
a. field of view at the stop line
b. vision of signal heads of conflicting movements
c. intervisibility between conflicting movements
1
1
1
1
1
1
2
1
1
5
7
3
5
3
2
6
3
6
5.2. SIGNAL
TIMING
A. TIMING OF PHASES
B. PHASING PLAN
C. TIMING OF INTERGREENS
D. TIMING FOR PEDESTRIANS
1
2
1
1
9
11
4
9
No.QUESTIONS 43 226
Pietrantonio, Bornsztein 16
A. Data Gathering and Processing 1
The Co-CP intersection, shown in Figure 1, is at the border of the old CBD (well inside the 2
consolidated area of the city), linking some major radial arterials to the old CBD and to a major arterial 3
ring (East-West) road through the Consolação (Co) avenue. Pedestrian flows are heavy and mainly related 4
to local land uses and to a major university campus located after the intersection, in the Maria Antonia 5
(MA) street (the other street, Cesário Mota, CM, is a minor street with predominantly local traffic). 6
Crossing roads are one way from the Caio Prado (CP) road to the Maria Antonia (MA) and Cesário Mota 7
(CM) streets. Left turn movements are forbidden in the two-way road (Co) as usual in the city. The 8
Consolação (Co) radial arterial carries a heavier traffic in the inbound direction all along the day (due to 9
the connection function to the arterial ring road). Nevertheless, the Consolação (Co) road also has heavy 10
flow on the outbound direction especially during the afternoon peak. The Caio Prado (CP) road has one-11
way operation, combining the access function to the arterial ring road with the function of carrying 12
vehicular flows crossing the arterial (including left turn and return flows to the outbound direction). 13
The intersection is operated under the real time control system of the central area (São Paulo has 14
five real time control areas). Traffic signal plans for the intersection are also shown in Figure 1. The Co-15
CP intersection signal operates with three stages just to handle pedestrians in the crossing of the outbound 16
direction (after the intersection) in the Co avenue (stage E1 in Figure 1). The cycle times are around 120s 17
in the peaks (pedestrian delays are high for some movements, mainly during stage E1) and the two-way 18
avenue receives the major share of green. Pedestrian signal heads are used at all painted pedestrian 19
crossings and the number of pedestrians outside these crossings is small. Nevertheless, the number of 20
pedestrian crossings against their red display is significant (mainly pedestrians movements allocated to 21
stage E1 but crossing in other stages, due to their great delay and the availability of gaps in the traffic). 22
Pietrantonio, Bornsztein 17
1
Figure 1. Study Intersections: Consolação x Caio Prado (Co-CP) 2
3
4
Pietrantonio, Bornsztein 18
A representative sample of accident records for pedestrian accidents and vehicular injury 1
accidents was recovered. For PDO accidents, the fraction and number of recovered accident records was 2
smaller and disregarded for the rest of analysis. The AAD engineer carried-out his field observation 3
routines as usual in accident investigations, without having access to data gathered by the RSA observers 4
(the RSA observers had no access to accident data either). Accident records were processed and a list of 5
problems that were related to the recovered sample of accident records was prepared for the intersection. 6
The quality of accident data for the Co-CP intersection for two years (1996, 1997) was better because the 7
accidents could be located precisely in the intersection (as streets change their name before and after the 8
main avenue and pedestrian movements are limited to one side of the main road). This feature is important 9
because precision of the accident analysis diagnostic was felt to be a critical point for further analysis with 10
the proposed method of study (as it is the yardstick for evaluating the final effectiveness of alternative 11
RSA procedures). 12
Our data sample from RSA applications consists of results for each individual question of each 13
subject area. The comparisons were carried-out by the RSA supervisor and submitted to the AAD 14
engineer for discussion only during the research tasks involved in the comparison of results. General 15
criteria of comparison are represented in Figure 2. After RSA observation and data processing, a 16
preliminary comparison of results was subjected to all the field observers. Their comments were used to 17
correct the initial AAD (after a careful discussion between observers and researchers). 18
19
Pietrantonio, Bornsztein 19
Figure 2. Criteria for Comparing Alternative RSAs and for Comparing RSAs and AAD Results. 1
2
For each quotation of a RSA checklist collected in the field observation, a comparison of 3
agreement was done, taking the results from the Detailed Checklist of the Safety Expert (ExpD) as a gold 4
standard. Quotations in common questions were rated for agreement or disagreement. Additions were also 5
rated for value. Figure 2a graphically sketches the criteria used for the comparison of RSAs. Omissions 6
were not rated as it was felt that their importance would appear in the subsequent comparison. One should 7
note that several agreements/disagreements and additions/omissions were partial (i.e. there could be a 8
partial agreement and a simultaneous addition in the comparison for the same item). 9
In the comparison to the accident analysis diagnostic (AAD), the RSA quotations were classified 10
as coincidences (if supported by the AAD results) or precautions (if not) and rated for relevance. Figure 11
2b graphically sketches the criteria used for the comparison with AAD. 12
The rating for value used general criteria studied for prioritizing deficiencies (Navin et al., 1999, 13
TAC, 2001, de Leur, Sayed, 2002, McInerney, 2002), taking into consideration the predicted danger of 14
accident occurrence (considering frequency of risk and proneness to accident), the expected accident 15
severity (in the event of a probable accident) and also the viability of correction (with measures usually 16
applied in São Paulo). A deficiency was rated into high, medium and low degrees for each feature. Any 17
a. comparison between road safety audits (RSAs) quotations
b. comparison of road safety audits (RSAs) and accident analysis diagnostic (AAD) quotations.
Pietrantonio, Bornsztein 20
feature rated as low was taken as a sign of a less relevant quotation or secondary deficiency. Otherwise, 1
the deficiency was taken as a main (relevant) one. 2
Of course, there is some degree of subjectivity in the comparison and rating of several quotations. 3
Most of all, the comparison exercise was shown to be burdensome as there is a need to reach a revised 4
reporting of answers to compensate for errors in the place of quotation (keeping the content of the answer 5
itself as the real point, even when the same observation was made when answering to different questions 6
of the RSA). The complexity of the checklists and of the intersections increased the difficulty in filling the 7
forms. Overall, it was felt that the results were robust to the degree of subjectivity. 8
In the comparison to the AAD, the aggregate analysis was supplemented by a finer analysis, 9
classifying questions by type, based on three subject matters (functional design, geometric design, signs 10
and signals), Other segmentations were considered (for example based on the existence of observational 11
clues in checklists for identification or judgment, among other features) but were not analyzed. 12
13
B. Presentation and Analysis of Results 14
The following results are exploratory and intend to discuss the kind of conclusions that can be 15
drawn from the qualitative analysis of the comparisons or on the statistical analysis of their data. Table 2 16
presents the general results of the comparison between RSAs and to the AAD for the main study in the 17
Co-CP intersection. Table 3 presents the results about the average success of each observer in detecting 18
the features listed in the AAD items (called coincidences) or in a superset of non-AAD items (called 19
precautions), with data on each of the four observers on three subject areas (Functional Design, Geometric 20
Design and Design of Signs and Signals), that gave us a sample of 12 measurements. 21
22
Pietrantonio, Bornsztein 21
1
Table 2. Results on the Comparison of RSAs and the Comparison of RSAs and AAD. 2
3
A qualitative analysis of the results from the comparative performance of alternative RSA 4
procedures (Table 2) can be summarized as: 5
- the degree of concordance achieved was high in all cases; only one average score for agreements was 6
smaller than 9,00 (for the Safety Analyst, mainly for additions of less relevant quotations); 7
- the Safety Expert use of the Generic checklist was comparable to the Detailed one (given the general 8
agreement of observations) but the number of relevant additions was also noticeable; 9
- the number of omissions was high for less qualified personnel (that directly applied the detailed 10
checklist); all of the observers had difficulties in answering questions about lighting and drainage; 11
- the number of disagreements was also high for less qualified personnel (the Safety Analyst disagreed in 12
23,5% of the common quotations with the Safety Expert; nevertheless, the average score for 13
disagreements of 6,60 reveals that some are relevant but most are less important); 14
- the small number of annotations errors and the average score for annotations errors also show a pattern 15
(occurring only for the Junior Professional); 16
Comparison between RSAs ExpDxExpG DesDxExpD SAnDxExpD JunIxExpD
Number of strong agreements 218 189 143 162
Number of weak agreements 1 5 10 2
Average score for agreement 9,91 9,39 8,80 9,40
Number of disagreements 0 25 47 30
Average score for disagreement 0,00 -8,10 -6,60 -8,10
Number of relevant additions 11 10 21 10
Number of secondary additions 6 6 50 38
Average score for additions 4,50 4,90 3,70 3,60
Number of misannotations 0 0 0 4
Average score for misannotations 0,00 0,00 0,00 -10,0
Number of common omissions - 7 7 7
Total number of omissions (7 in ExpD) - 7 24 32
Comparison of RSAs and AAD ExpDxAAD DesDxAAD SAnDxAAD JunIxAAD
Main Coincidences (of 24): 4 3 8 8
Main Precautions: 2 1 7 6
Secondary Coincidences (of 8): 1 0 3 3
Secondary Precautions: 9 9 17 9
Notes: 1- Analysis of quotations of the Road Safety Audits (RSA) and features identified on the Accident Analysis
Diagnostic (AAD); 2- Observers classified as Safety Experts (Exp), Design Engineer (Des), Safety Analyst (SAn) and Junior
Engineer (Jun) and checklists classified as Generic (G), Detailed (D) and Detailed with Instructions (I).
Pietrantonio, Bornsztein 22
- the number of additions also increased for the less qualified personnel but the average score for additions 1
and the proportion of relevant additions decreased (the Senior Designer had the smaller number of 2
additions, probably by knowing that some features are hard to fix by designing countermeasures). 3
A qualitative analysis of the results from the performance obtained by comparison of the RSAs to 4
the AAD and accidents (Table 3), can be summarized as: 5
- the Safety Analyst and also the Junior Professional had the largest number of detections and precautions 6
(they were also the observers with the highest number of quotations); 7
- no single observer was able to reach a high level of detection; nevertheless, combining all the 8
observations, the “team” was able to detect 20 out of 32 deficiencies (with 15 out of 24 main detections). 9
10
Pietrantonio, Bornsztein 23
Table 3. Average Observer Performance by Subject Area as %Hit in Identifying Features of the Global Diagnostic
Observed RSA Expert (Exp) Senior Designer (Des) Safety Analyst (SAn) Junior Engineer (Jun)
No.Cases No.Hits %Cases No.Hits %Cases No.Hits %Cases No.Hits %Cases
Main Accident Functional Design 12 2 16.67% 1 8.33% 7 58.33% 4 33.33%
Factors Geometric Design 4 0 0.00% 1 25.00% 0 0.00% 2 50.00%
Signs and Signals 8 2 25.00% 1 12.50% 1 12.50% 2 25.00%
All Subjects 24 4 16.67% 3 12.50% 8 33.33% 8 33.33%
All Accident Functional Design 18 3 16.67% 1 5.56% 9 50.00% 7 38.89%
Factors Geometric Design 4 0 0.00% 1 25.00% 0 0.00% 2 50.00%
Signs and Signals 10 2 20.00% 1 10.00% 2 20.00% 2 20.00%
All Subjects 32 5 15.63% 3 9.38% 11 34.38% 11 34.38%
Main Preventive Functional Design 9 2 22.22% 1 11.11% 4 44.44% 4 44.44%
Factors Geometric Design 3 0 0.00% 0 0.00% 1 33.33% 1 33.33%
Signs and Signals 2 0 0.00% 0 0.00% 2 100.00% 0 0.00%
All Subjects 14 2 14.29% 1 7.14% 7 50.00% 5 35.71%
All Preventive Functional Design 22 8 36.36% 4 18.18% 12 54.55% 8 36.36%
Factors Geometric Design 7 0 0.00% 1 14.29% 3 42.86% 3 42.86%
Signs and Signals 18 0 0.00% 5 27.78% 9 50.00% 4 22.22%
All Subjects 47 8 17.02% 10 21.28% 24 51.06% 15 31.91%
Both Main Factors All Subjects 38 6 15.79% 4 10.53% 15 39.47% 13 34.21%
All Factors All Subjects 79 13 16.46% 13 16.46% 35 44.30% 26 32.91%
Pietrantonio, Bornsztein 24
For analyzing these results, it was important to keep in mind some peculiarities of each observer. For 1
the study on the Co-CP intersection, all observers had good previous knowledge about the site (as it is a well 2
known intersection in the city) but the Senior Designer and the Safety Analyst had worked previously on the 3
design of the current signal scheme in the past (the Safety Analyst moved from this traffic management area 4
some 5 years ago and the Senior Design remains in charge in this area). The Junior Professional also had some 5
knowledge about the current design and its problems from previous studies. This fact could explain, at least in 6
part, the large number of additions of the Safety Analyst and also the small number of additions of the Senior 7
Designer (who knows and weights most of the constraints to action). Also the performance of the Junior 8
Professional could benefit from his previous knowledge. 9
Some of these features are non-ideal (when considering the proposal for RSAs) and there must be a 10
method to measure and correct their influence. Generally, confounding factors are always present in practical 11
studies and there must be a method for identifying and measuring their effects, as done in our next step. 12
The statistical analysis of the data on the relative performance of each observer was carried-out using 13
logistic regression against a set of explanatory variables. Dependent variables were the relative performance of 14
observers in each of the subject areas (functional design, geometric design and design of signs and signals), 15
measured by the percentage of detections achieved on the features identified by the accident analysis diagnostic 16
(called coincidences) and on the superset of features that were not related to the accident analysis diagnostic but 17
that were judged relevant as preventive actions based on our technical investigation at the site (called 18
precautions). Subject areas were used to allow a more robust aggregate analysis based on the average rating of a 19
group of questions. Analyses could be carried-out using other aggregate groups as types of questions (for 20
example, general or specific questions, questions with or without hints for identification of settings where the 21
deficiency is critical, and so on). Of course, it is possible to carry-out a disaggregate analysis as well. 22
The estimation of logistic regression models is widely discussed in the literature on the analysis of 23
discrete data or choice modeling and is supported by the used of several statistical packages. Our illustrative 24
results were based on the analysis of average detection percentage in each subject area using the nonlinear 25
regression model option (NLSQ) of LIMDEP version 7 (see Greene, 1998, Chapter 19). This option was chosen 26
because the NLSQ procedure permits a better control of model specification and seems to be more robust 27
numerically (also providing a simple tool for evaluating elasticities, as reported ahead). Separate analyses were 28
carried-out for main coincidences, total coincidences, main precautions and total precautions, adding 29
explanatory variables for features of observers: their technical training, their professional experience and by 30
their previous knowledge of the site and their participation in the design of the current project operating at the 31
site for each subject area, represented as scores based on a relative scale, as shown on Table 4. 32
33
Pietrantonio, Bornsztein 25
Table 4. Further Data for Evaluating the Effect of Selected Features on Observer Performance 1
2
Several specifications were tried, searching for a meaning-full and robust set of models. The individual 3
effects of training and experience were hard to distinguish as the variables were naturally correlated. We tried 4
combinations of training and experience scores as we did with site knowledge and current design scores. 5
Combinations by averages and maximum scores were rationalized as complementarity or substitutability of 6
attributes. We also tried generic intercepts, observer specific dummies or subject area specific dummies. The 7
more meaning-full and robust models used the maximum of training and experience scores, individual scores for 8
previous knowledge and current design, also including a set of observer specific dummies in the model. The 9
resulting model coefficients are presented in Table 5. As can be seen, model estimation shows that: 10
- the inclusion of observer features always increase the adjusted explained variance ratio (even if marginally as 11
in the case of main coincidences) but the Akaike information criteria delivers opposite results (an improvement 12
is indicated only for main precautions); 13
- the coefficients of observer specific dummies are statistically significant in the estimation of the simplest 14
model (the dummy-only specification) but their values are not robust to the inclusion of observer attributes in 15
the more complex model; 16
- in general, the coefficients of observer attributes variables are not statistically significant in the estimation of 17
the more complex model but the parameters values are consistent 18
Observer Subject Technical Training (T)
Professional Experience (E)
Knowledge Of Site (K)
Participation In Project (P)
RSA Expert Functional Design 3 3 2 0
Geometric Design 3 3 1 0
Signs and Signals 3 2 2 0
Senior Designer Functional Design 3 2 3 2
Geometric Design 3 2 2 2
Signs and Signals 3 3 3 3
Safety Analyst Functional Design 3 2 1 1
Geometric Design 2 2 1 1
Signs and Signals 3 3 2 1
Junior Engineer Functional Design 1 0 3 0
Geometric Design 1 0 3 0
Signs and Signals 2 0 2 0
Note: Scores were based on the following qualitative scale
Formal Training: 0=no special training, 1=quick courses, 2=specialized training, 3=expert;
Professional Experience: 0=small experience; 1=1-2 years; 2=2-5 years; 3=more than 5 years
Previous Knowledge: 0=no knowledge; 1=site info; 2= accident info; 3=diagnostic info;
Participation in the Project: 0=no participation; 1=shared information; 2=partial tasks; 3=project leader.
Pietrantonio, Bornsztein 26
. all signs are as expected, except for the coefficient of training and experience in the 1
explanation of detection of all precautions; 2
. in all the other cases, trained and experienced observers were more effective and the use of 3
junior professionals is justified based on cost considerations only; 4
. in all the cases, participation in the design of the project operating on the site were correlated 5
with less effective performance (there is thus a commitment effect); 6
. in all the cases, previous knowledge of the site were correlated with more effective 7
performance; 8
- observer specific dummies are different and relevant in value even in the more complex model, remaining the 9
more statistically significant coefficients even in this more complex model (signing the presence of important 10
omitted variables, other that the observer´s attributes considered). 11
Such kind of model can be useful for predicting the performance of some standard observer of each type 12
(for example, a observer without previous knowledge or participation on the design) or measuring the effect of 13
changing some variable (for example, the requirement of training or experience for observers). Table 5 shows 14
elasticities to training and experience (MaxTE). The sensitivity is more relevant for coincidences than 15
precautions (increasing the detection, around 20%, by 7% for main coincidences and 5% for all coincidences). 16
The estimated models are suggestive and would certainly be unsound for use in developing guidelines, giving 17
our limited sample. Nevertheless, it is interesting to note that the results are reasonable even in an illustrative 18
application and to emphasize the usefulness of the method of analysis. 19
However, some conclusions can be drawn from the consistency of these results, mainly the adequacy of 20
using less experienced professionals for field observation (with appropriate checklists, given their positive 21
contribution) and the importance of teamwork in field tasks (as no single observer approached the full set of 22
diagnostic features gathered by the AAD). The importance of expert supervision for evaluating quotations 23
seems also clearly supported (as the reported features included several non-critical problems and even 24
misleading points). The sufficiency of the Safety Expert to the final effectiveness of the RSA is not supported 25
(despite the large experience of the observers of this type that collaborated in the research, the main 26
coincidences to the AAD surpassed the 60% only summing all observers quotations). Nevertheless, the role of 27
the expert supervision in prioritizing quotations and achieving a better final evaluation should be stressed. 28
29
30
Pietrantonio, Bornsztein 27
1
Table 5. Results on the Logistic Regression for Selected %Hit Models (LIMDEP output with NLSQ procedure). 2
3
4
Overall, this research seems to show that the proposed method of study is a valuable alternative. It is 5
important to note that the same approach can also be used to analyzing several finer questions about procedures 6
or tools. For example, our study showed that checklists vary a lot in the way they put questions to the RSA 7
observer and the proposed method can be used to evaluate the comparative success of tort detection in 8
alternative options. Also, our exercise showed that several observations were hard to fill at the sites (especially 9
Accident Factors-Main Coincidences
Model Dummies only dExp dDes dSAn dJun
AdjR2 32.06% Coeff. -1.82455 -1.71298 -1.17412 -0.570545
AIC -0.295 t-ratio -2.52666 -2.56728 -2.45202 -1.52412
Model D&Ob.attributes dExp dDes dSAn dJun MaxTE SKnow ProjP
AdjR2 34.17% Coeff. -3.0216 -2.28376 -1.84587 -1.4823 0.310598 0.183978 -0.38196
AIC 0.173 t-ratio -1.11798 -0.519037 -0.680754 -0.705371 0.45234 0.321319 -0.243038
sensitivity to training or experience at 20% 7.24% at 50% 11.31%
Accident Factors-All Coincidences
Model Dummies only dExp dDes dSAn dJun
AdjR2 39.52% Coeff. -1.97155 -1.85587 -1.18958 -0.562527
AIC -0.511 t-ratio -2.72814 -2.79849 -2.74474 -1.67762
Model D&Ob.attributes dExp dDes dSAn dJun MaxTE SKnow ProjP
AdjR2 44.58% Coeff. -3.29174 -2.66948 -1.8678 -2.10597 0.193672 0.486311 -0.522563
AIC -0.098 t-ratio -1.31521 -5.96E-01 -7.04E-01 -1.08747 0.310301 0.912817 -3.18E-01
sensitivity to training or experience at 20% 4.96% at 50% 7.76%
Preventive Factors-Main Precautions
Model Dummies only dExp dDes dSAn dJun
AdjR2 62.41% Coeff. -2.52573 -3.2581 0.374693 -1.04982
AIC 0.112 t-ratio -1.63671 -1.09787 0.854678 -1.90484
Model D&Ob.attributes dExp dDes dSAn dJun MaxTE SKnow ProjP
AdjR2 97.12% Coeff. -31.8606 -6.55861 3.92506 -43.4787 0.470002 14.1856 -19.7438
AIC -1.959 t-ratio -0.000352 -1.07E-07 1.28E-07 -0.00032 0.16522 0.000313 -6.45E-07
sensitivity to training or experience at 20% 2.64% at 50% 4.13%
Preventive Factors-All Precautions
Model Dummies only dExp dDes dSAn dJun
AdjR2 67.27% Coeff. -1.5384 -1.38119 -0.034636 -0.671579
AIC -1.233 t-ratio -4.14289 -4.10214 -0.160192 -2.78144
Model D&Ob.attributes dExp dDes dSAn dJun MaxTE SKnow ProjP
AdjR2 79.88% Coeff. -2.27385 -3.88083 -1.12265 -2.12014 -0.043931 0.563382 0.455656
AIC -1.220 t-ratio -2.19421 -2.26368 -1.13405 -2.5613 -0.155113 2.24287 0.848456
sensitivity to training or experience at 20% -2.48% at 50% -3.88%
Note: MaxTE=maximum of training and experience scores; SKnow=previous knowledge about the intersection;
ProjP=participation in design projects on the intersection.
Pietrantonio, Bornsztein 28
those related to lighting and drainage) and the effectiveness of engineering measures as supporting data for 1
further analysis can be used as an aid and be evaluated in the same way. These more detailed analyses were not 2
carried-out in this research but seems to be worth of investigation. 3
Note that, comparing to studies in the health area, the confidence on the application of the Gold 4
Standard of diagnosis that is embodied in their terminology (that classify the results of the proposed alternative 5
as True positives, False positives, True negatives, False negatives) we avoided. A similar problem appears on 6
the literature on health studies when recognizing that an action based on the Gold Standard can not be fully 7
effective but this effect is usually attributed to the fail of the remedial action (as therapeutic uncertainty) instead 8
of the accepted diagnosis. There are techniques suggested to correct for imperfect Gold Standards in health 9
studies (see Zhou et al., 2002, for examples) but it is felt that the alternative approach usually applied for 10
evaluating the effect of therapeutic uncertainty is more meaningful also for weighting the diagnostic uncertainty 11
of the Gold Standard and the proposed alternative, similarly. This is our current line of investigation. 12
13
6. CONCLUSIONS. 14
This paper discussed Road Safety Audits (RSA) as a safety improvement technique with the aim at 15
identifying alternative conceptual or practical features that can contribute to the effectiveness of the procedure. 16
A new method of study that can be used to evaluate detailed procedures was proposed. Several features of RSAs 17
were identified and a case study was designed for evaluating three main points: the degree of dependence on a 18
safety expert, the comparative performance of personnel with different qualification for field observation and 19
the adequacy of different types of checklists. Four types of observers and three types of checklists were used for 20
the field study. The new method of study was capable of analyzing these points. 21
After reviewing some methodological questions about the evaluation of RSA applications, the new 22
method proposed two analyses: a comparative analysis of quotations gathered with alternative RSA procedures 23
against a standard (view as an intermediate evaluation task) and a comparative analysis of the RSA results 24
against an accident analysis diagnosis (view as a better evaluation task). Both analyses were revealing in 25
comparing the relative performance of each procedural option. The same procedure can also be applied using 26
any other Gold Standard (e.g. results from the monitoring stage or from an extended RSA process involving 27
supplementary analyses not usually carried-out or further advice from experts not usually participating). 28
The main drawback of the method is the level of effort required for its application. Despite applying the 29
alternative RSA procedures in four intersections, the study was tried in two intersections and finally applied to 30
one of them, due to the large volume of data processing and handling tasks (a feature that can be attributed to 31
the large number of questions of the RSA) and the need of a precise accident data for the comparative diagnostic 32
(needed for the final evaluation task). The comparison of RSAs rated quotations as agreements or disagreements 33
Pietrantonio, Bornsztein 29
with the standard application (a safety expert with a detailed checklist). The comparison to the accident 1
diagnostic rated the quotations as coincidences with the accident analysis diagnosis or as further precautions. 2
Each quotation was also rated as main or secondary based on the predicted danger of accident occurrence 3
(considering frequency of risk and proneness to accident), the expected accident severity (in the event of a 4
probable accident) and also the viability of correction (with measures usually applied in the city of São Paulo). 5
Also, the dependence on accident data is a major problem, at least for developing countries such as Brazil, and 6
the use of alternative standards for comparisons (as a super-RSA, based on all evidences) is worth studying. The 7
time required for data collection and analysis can be reduced by using a smaller set of well defined questions in 8
a simulated RSA (a practice that can potentially deliver also more confident results). 9
The methods used for analyzing the results are widely accessible. Two options were applied: simple 10
cross-tabulation methods and conventional statistical modeling methods using logistic regression. The general 11
evaluation of results based on cross-tabulation and the inference about the influence of covariates based on 12
logistic regression, both delivered consistent results. More advanced methods are also applicable. 13
Despite the need of subjective rating of quotations, it was felt that the method produced useful results on 14
the global performance of each RSA procedure that supports the use of less experienced personnel for field 15
observation (as they can reach a good agreement with expert observations, using detailed checklists) but with 16
recourse to team work (combining professionals with different degrees of qualification and background 17
experience) and to the supervision of a qualified expert (a clear trend for less relevant quotations and even some 18
misleading observations by less experienced professionals was found). Therefore, the proposed method is 19
thought to be a valuable option for further study, being capable of finer analyses (comparing types of questions, 20
for example) and additional investigations (analyzing observation routines or the use of other data or 21
measurements, for example). Additionally, one can note that the same kind of method can be applied to other 22
safety evaluation procedure based on surrogate measures. 23
Last but not least, one should stress the need of complementary methods of study that can be applied to 24
grasp the value of RSA on real applications based on a direct evaluation of its real final effects on the safety 25
improvement and/or reduction of future corrective actions. 26
27
ACKNOWLEDGEMENTS 28
We would like to thank to the Traffic Engineering Company of the City of São Paulo for providing data and support for 29
most of the work, to the professionals who collaborate as observers in the field study, to FAPESP (the Foundation for 30
Research Aid of the State of São Paulo) that funded the project that supplied the accident data, and also especially to 31
Denise Lima Lopes and Veridiana Maria Armond Vasconcelos (who helped us in the data processing tasks) and to 32
Profa.Cristina Borba (who help us in the revision of a previous version of this paper). The remaining errors are ours. 33
34
Pietrantonio, Bornsztein 30
1
REFERENCES 2
Austroads (1994) – Road Safety Audits, Austroads, Australia. 3
Austroads (2002) – Road Safety Audits, Second Edition, Austroads, Australia. 4
Austroads (2006) – Road Safety Audit, Guide to Road Safety, Part 6, Austroads, Australia. 5
Belcher, M.; Proctor, S.; Cook, P. (2008) –Practical Road Safety Auditing, Second Edition, Thomas 6
Telford Publishing, UK. 7
Bornsztein, L.L. (2001) – Road Safety Audits Applied to Signalized Intersections, Master of Science 8
Dissertation, Polytechnic School, University of São Paulo, Brazil (in Portuguese) 9
Bullpit, M. (1996) – “Safety audit - an overview”. In Proceedings of the Institution of Civil 10
Engineers, Transport, v. 117, pp.211-215. 11
CONASET (2003) – Guide for Undertaking a Road Safety Audit, National Comission for Traffic 12
Safety, Chile (in Spanish). 13
de Leur, P; Sayed, T. (2002) – “Development of a Road Safety Index”. In Transportation Research 14
Record 1784, Transportation Research Board, USA, pp.33-42. 15
DoT (1994a) – Road Safety Audits (HD 42/94), Department of Transport, UK. 16
DoT (1994b) – Road Safety Audits (HA 19/94), Department of Transport, UK. 17
DoT (2003) – Road Safety Audits (HD 19/03), Department of Transport, UK. 18
EC (2008) – On Road Infrastructure Safety Management, Directive 2008/96/EC, adopted by The 19
European Parliament/The Council of the European Union. 20
Felder, S.; Mayrhofer, T. (2011) – Medical Decision Making – a Health Economic Primer, Springer. 21
FHWA (2006) – FHWA Road Safety Audit Guidelines, Federal Highway Administration, 22
Publication No. FHWA-SA-06-06, Department of Transport, USA. 23
Gold, P.A. (1998) – Traffic Safety – Using Engineering Measures to Reduce Accidents, Inter-24
American Development Bank. 25
Greene, W.H. (1998) – Limdep Version 7.0 – User’s Manual, Econometric Software Inc.. 26
Hildebrand, E.; Wilson, F. ; Editors (1999) – Road Safety Audit Guidelines, Transportation Group, 27
University of New Brunswick, Canada. 28
IHT (1991) – Guidelines of Safety Audit of Highways, Institution of Highways and Transportation, 29
UK. 30
IHT (1996) – Guidelines of Safety Audit of Highways, Second Edition, Institution of Highways and 31
Transportation, UK. 32
Pietrantonio, Bornsztein 31
IHT (2008) – Road Safety Audits, Institution of Highways and Transportation, UK (Guidelines of 1
Safety Audit of Highways, Third Edition). 2
INST (1995) – Road Safety Audits, Boletim Técnico No.1, National Institute for Traffic Safety, Brazil 3
(in Portuguese). 4
Macaulay, J.; McInerney, R. (2002) – Evaluation of Proposed Actions Emanating from Road Safety 5
Audits, Publication AP-R209/02, Austroads, Australia. 6
Martinez, A (2000) – Road Safety Audit – SP310: km153 to km227; SP225: km91 to km225, 7
CENTROVIAS Highway Systems S/A (in Portuguese). 8
McInerney, R. (2003) – The Road Safety Risk Manager Software Tool: Background Research, 9
Publication AP-R222/03, Austroads, Australia. 10
Montella (2005) - “Safety Review of Existing Roads: Quantitative Safety Assessment Methodology”. In 11
Transportation Research Record 1922, Transportation Research Board, USA, pp.62-72. 12
Navin, F; Ho, T.; Zein, S. (1999) – “Road Safety Audit in Canada Early Success Stories”. In Road 13
Safety Audit Seminar, Institute of Transportation Engineers, USA, pp.101-109. 14
Pietrantonio, H.; Tourinho, L.F.B. (2006) - “A Decision-based Criterion for Selecting Parameters in the 15
Evaluation of Pedestrian Safety Problems with the Traffic Conflict Analysis Technique”. In Transportation 16
Planning and Technology, vol.29, no.3, pp.183-216. 17
Proctor, S.; Belcher, M. (1993) – “The Use of Road Safety Audits in Great Britain”. In Traffic 18
Engineering and Control, v.34, no.2, pp.61-65. 19
Road Directorate (1997) – Manual of Road Safety Audits, Road Directorate, Ministry of Transport, 20
Denmark (available in English). 21
Scaringela, R.S.; Martinez, A.; Rodolpho, A.J. (1996) – “Safety Audit of São Paulo’s City Signaling 22
Projects”. In Eighth International Conference on Road Traffic Monitoring and Control, Conference 23
Publication No.422, Institution of Electrical Engineers, UK, pp.131-135. 24
TAC (2001) – The Canadian Road Safety Audit Guide, Transportation Association of Canada, 25
Canada. 26
TAC (2004) – The Canadian Guide to In-service Road Safety Reviews, Transportation Association 27
of Canada, Canada. 28
TRRL (1991) – Towards Safer Roads in Developing Countries, Transport and Road Research 29
Laboratory, Overseas Development Unit, UK. 30
Zhou, X.-H.; Obuchowski, N.A.; McClish, D.K. (2002) – Statistical Methods in Diagnostic Medicine, 31
Wiley Interscience (Second Edition, 2011, Wiley). 32
33