How to Evaluate How to Evaluate

Driver Education ProgramsDriver Education Programs

Published August 2006Published August 2006

Prepared by:Prepared by:

Northport AssociatesNorthport Associates

Co-funded by:Co-funded by:

BMW of North AmericaBMW of North America

Available online at:Available online at:

www.aaafoundation.org

The ProblemThe Problem

• Traffic crashes remain the #1 killer of teens

• Most research indicates that driver education courses are not effective tools to reduce these deaths and injuries

Purpose of This ResearchPurpose of This Research

To develop comprehensive guidelines on how to evaluate the effectiveness of novice driver training and education programs

Does novice driver training and education make an impact on skills

and safety?

History of Evaluation in Driver Education

• Long history – Jim Nichols NTSB paper

• Recent reviews – little critical analysis

• One recent meta-analysis

• Mainly summative evaluation

• Experimental RCTs – records, surveys

• Quasi-experiments – +/- statistical control

• Ecological studies – population effects

History of Evaluation in Driver Education

Prominent Findings:

• DeKalb

• Reviews

• Mayhew & Simpson

• Australian reviews

• “Systematic” reviews

• Elvik & Vaa meta-analysis

… the systematic assessment of the operation and/or outcomes of

a program or policy, compared to a set of explicit or implicit

standards as a means of contributing to the improvement of the

program or policy.

Weiss, 1998

Management Overview

Program Evaluation Definition

Why Evaluate Driver Education Programs?

1. We have big questions left unanswered:

• Do programs have logical learning objectives?

• Do they achieve their learning objectives?

• Do they have the desired safety impacts?

• Do some programs lead to better results than others?

• Which components of programs contribute to success?

• How can programs be improved to maximize impact

Why Is Systematic Evaluation Important?

2. Credible answers allow us to:

• Identify program strengths & weaknesses

• Measure progress objectively

• Identify ways to improve programs

• Share objective information about what works

• Influence policy & funding

• Build an effective base of knowledge

What Is Important to Know About Evaluation?

Key Concepts:

1. Formative Evaluation

• Intended to improve program logic, content, delivery

• Asks:

• Does the program operate according to plan?

• Is it meeting its teaching objectives?

• How can it be improved?

What Is Important to Know About Evaluation?

Key Concepts:

2. Summative Evaluation

• Intended to prove or “sum-up” results & benefits

• Asks:

• Does the program make students better drivers?

• Does it reduce crashes?

Evaluating Driver Education Evaluation

Has evaluation improved & proved driver education?

• Negative influence on beliefs and policy

• Weak in formative/process areas

• Statistical weaknesses in summative evaluations

• Unsystematic – “hit & run” research

• Lots of room for improvement:

• Positive, cost-effective program impacts

• Continuous improvement

• Knowledge & know-how

• Progressive policy

Step 1FOCUS the EVALUATIO

N

Step 5INTERPRET FINDINGS & TAKE ACTION

Step 4GATHER & ANALYZE

DATA

Step 3IDENTIFY

DATA COLLECTION

PLAN & TOOLS

Step 2SELECT

EVALUATION

METHODS

PROGRAM

Five Steps to Evaluating

Overcoming Barriers to Evaluating?

What if researchers find your program does not work?

• Systematic program evaluation is the key

• Ongoing, stepwise “system” of evaluation

• Small steps & continuous improvement

• There should be little risk of nasty surprises

• Still, plan for unexpected findings

Retooling Driver Education for Progress

• World has changed around driver education – reform is required

• Most jurisdictions lack resources & political will

• Improve Uniformity – there cannot be 60 “best” ways

• Behavior change is tough & programs need to reflect this

• Evidence-based development is badly needed

• R&D & comprehensive evaluation need high priority

The Future: Bridging the Great Divide

• Practitioners & managers “believe in” driver education

• Researchers typically believe “driver education doesn’t work”

“For more effective driver education in the future, key people in both camps must work together toward improving evaluation and using evaluation tools to improve driver education.” Management Overview p.15

Evaluating Driver Education Programs:Comprehensive Guidelines

Management OverviewHow-To Manual

What is next?

Large-Scale Evaluations of Driver Education Programs

Implementation of Guidelines

Guidelines

The AAA Foundation for Traffic Safety is a 501(c)(3) public charity located in Washington, DC that is dedicated to saving lives and reducing injuries.

It is supported by donations from AAA/CAA Clubs, AAA/CAA members, and other organizations associated with AAA/CAA.