An Independent Evaluation of the OK-FIRST Decision Support System Thomas E. James and Paula O. Long...
-
Upload
roy-griffin -
Category
Documents
-
view
213 -
download
0
Transcript of An Independent Evaluation of the OK-FIRST Decision Support System Thomas E. James and Paula O. Long...
An Independent Evaluation of the OK-FIRST Decision Support System
Thomas E. James and Paula O. Long
Institute for Public Affairs / Department of Political Science
University of Oklahoma
Mark Shafer, Oklahoma Climatological Survey
University of Oklahoma
OK-FIRST
• Customized, county-level environmental information
• Target group: emergency managers• Training (computer workshop, data interpretation
workshop)• 3 classes (June 1997, October 1997, March 1998)• Refresher courses with Focus Groups
Independent Evaluator
• Unbiased and objective perspective • A collaborator, NOT a monitor• Combines strengths:
– Institute of Public Affairs: methodological expertise– OK-FIRST staff: substantive context-specific knowledge
• Ensured collection of information useful to the program
• Continuous feedback to improve training methods or data collection techniques
EVALUATION
“The systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.” (Michael Patton 1997)
EVALUATION GOALS
(1) Determine the extent to which OK-FIRST
provided access and training to enhance
participants abilities
(2) Document the use of skills and impacts
(3) Identify lessons learned
EVALUATION DESIGN
• Background Information– self-administered questionnaires
– job experience
– sources of information
– familiarity with computers
EVALUATION DESIGN
• Workshops– pre-test / post-test
– computer workshop: observations of performed tasks coupled with written identification
– data interpretation workshop: identification, definitions, application of concepts (written)
EVALUATION DESIGN
• Self-Assessment– questionnaire administered at the beginning of the data
interpretation workshop, covering computer skills
– questionnaire at refresher workshops on computer skills and use of information
– group discussions focusing on insights and experiences
COMPUTER TRAINING
• Statistically significant increase in skills– 57% average pre-test score; 79% post-test
• Reduced range of post-test scores (those initially at bottom benefited most)
• After receiving training, almost all participants proficient enough to access data
• Despite significant increases in some areas, still room for improvement
DATA INTERPRETATION
• Significant increase in knowledge– 44% average pre-test score; 62% post-test
• Reduced range in post-test scores• Assessing pre-test and post-test knowledge
(clusters) allows targeted training• Feedback on performance from first class allowed
OK-FIRST staff to adjust training, resulting in improved performance of other classes
WEB PAGES
• Follow-up questionnaire to determine utility• 100% satisfied overall
– 98% very satisfied
• 91% very satisfied with content• NIDS (WSR-88D) most frequently accessed
– 61% use daily
• 71% said very easy to navigate
BULLETIN BOARD
• Designed to share information and seek assistance• Not highly used
– 37% did not use at all, rest infrequently
• Of those who used it, majority said posted information was useful
• Integration with web access may make it more useful
MEASURING OUTCOMES
• Long-term Outcomes– broad goals, which program itself cannot accomplish
alone
– e.g., saving lives or property
• Intermediate Outcomes– measurable aspects which facilitate accomplishment of
long-term outcomes
– e.g., application of data, knowledge, and skills
• Primarily measured from focus groups
UTILIZATION
• Overall– “I’m proactive now rather than just reactive”
– “This program has become critical to our organization”
• Access / Application of Data– “OK-FIRST information … has kept disasters from
happening”
– “This allows the emergency manager to check his own specific area”
UTILIZATION
• Severe Weather– most frequent application
– identify threats earlier
– pinpoint storms more accurately
– utilize scarce resources more effectively
• Floods– better management of road closures
– fewer false alarms
UTILIZATION
• Fires– identifying approaching wind shifts
– Fire Danger Model
• Hazardous Materials– trajectories / evacuation decisions
• Public Works (unanticipated outcomes)– assessing need for snow crews
– appropriate conditions for road work
– little league baseball
CONCLUSIONS
• The OK-FIRST team has been successful in meeting the key needs of program participants
• Able to enhance significantly the knowledge and skills of the project participants in a very short period of time
• Integration and coordination of training, access, and ongoing support vital to success
CONCLUSIONS
• “The project was able to change the behavior of local public safety officials and their approach to decision making.”
• Increased participants confidence, abilities, effectiveness, and range of application
• Empowered local officials to make decisions• “The OK-FIRST organization and team should
serve as a model for others.”