An Independent Evaluation of the OK-FIRST Decision Support System

Thomas E. James and Paula O. Long

Institute for Public Affairs / Department of Political Science

University of Oklahoma

Mark Shafer, Oklahoma Climatological Survey

University of Oklahoma

OK-FIRST

• Customized, county-level environmental information

• Target group: emergency managers• Training (computer workshop, data interpretation

workshop)• 3 classes (June 1997, October 1997, March 1998)• Refresher courses with Focus Groups

Independent Evaluator

• Unbiased and objective perspective • A collaborator, NOT a monitor• Combines strengths:

– Institute of Public Affairs: methodological expertise– OK-FIRST staff: substantive context-specific knowledge

• Ensured collection of information useful to the program

• Continuous feedback to improve training methods or data collection techniques

EVALUATION

“The systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.” (Michael Patton 1997)

EVALUATION GOALS

(1) Determine the extent to which OK-FIRST

provided access and training to enhance

participants abilities

(2) Document the use of skills and impacts

(3) Identify lessons learned

EVALUATION DESIGN

• Background Information– self-administered questionnaires

– job experience

– sources of information

– familiarity with computers

EVALUATION DESIGN

• Workshops– pre-test / post-test

– computer workshop: observations of performed tasks coupled with written identification

– data interpretation workshop: identification, definitions, application of concepts (written)

EVALUATION DESIGN

• Self-Assessment– questionnaire administered at the beginning of the data

interpretation workshop, covering computer skills

– questionnaire at refresher workshops on computer skills and use of information

– group discussions focusing on insights and experiences

COMPUTER TRAINING

• Statistically significant increase in skills– 57% average pre-test score; 79% post-test

• Reduced range of post-test scores (those initially at bottom benefited most)

• After receiving training, almost all participants proficient enough to access data

• Despite significant increases in some areas, still room for improvement

DATA INTERPRETATION

• Significant increase in knowledge– 44% average pre-test score; 62% post-test

• Reduced range in post-test scores• Assessing pre-test and post-test knowledge

(clusters) allows targeted training• Feedback on performance from first class allowed

OK-FIRST staff to adjust training, resulting in improved performance of other classes

WEB PAGES

• Follow-up questionnaire to determine utility• 100% satisfied overall

– 98% very satisfied

• 91% very satisfied with content• NIDS (WSR-88D) most frequently accessed

– 61% use daily

• 71% said very easy to navigate

BULLETIN BOARD

• Designed to share information and seek assistance• Not highly used

– 37% did not use at all, rest infrequently

• Of those who used it, majority said posted information was useful

• Integration with web access may make it more useful

MEASURING OUTCOMES

• Long-term Outcomes– broad goals, which program itself cannot accomplish

alone

– e.g., saving lives or property

• Intermediate Outcomes– measurable aspects which facilitate accomplishment of

long-term outcomes

– e.g., application of data, knowledge, and skills

• Primarily measured from focus groups

UTILIZATION

• Overall– “I’m proactive now rather than just reactive”

– “This program has become critical to our organization”

• Access / Application of Data– “OK-FIRST information … has kept disasters from

happening”

– “This allows the emergency manager to check his own specific area”

UTILIZATION

• Severe Weather– most frequent application

– identify threats earlier

– pinpoint storms more accurately

– utilize scarce resources more effectively

• Floods– better management of road closures

– fewer false alarms

UTILIZATION

• Fires– identifying approaching wind shifts

– Fire Danger Model

• Hazardous Materials– trajectories / evacuation decisions

• Public Works (unanticipated outcomes)– assessing need for snow crews

– appropriate conditions for road work

– little league baseball

CONCLUSIONS

• The OK-FIRST team has been successful in meeting the key needs of program participants

• Able to enhance significantly the knowledge and skills of the project participants in a very short period of time

• Integration and coordination of training, access, and ongoing support vital to success

CONCLUSIONS

• “The project was able to change the behavior of local public safety officials and their approach to decision making.”

• Increased participants confidence, abilities, effectiveness, and range of application

• Empowered local officials to make decisions• “The OK-FIRST organization and team should

serve as a model for others.”