&%$#@&%$!!

&%$#@&%$!!Evaluation is NOT a Dirty

WordKathleen Dowell, Ph.D.

EvalSolutions

Epilepsy Foundation: Best Practices InstituteSeptember 29, 2012

Denver, Colorado

Too expensive

Too complicated

Too time consuming

Not a priority

Just don’t know

where to start

Barriers Lack of research/statistics skills Lack of time Lack of resources Other priorities Lack of incentive Fear Don’t see value

What is Evaluation?

The process of determining the merit, worth, or value of a program (Scriven, 1991)

What is Evaluation?

Systematic inquiry that describes and explains, policies’ and programs’ operations, effects, justifications, and social implications (Mark, Henry, & Julnes, 2000)

What is Evaluation?

The systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs (Rossi & Freeman, 1989)

In simpler terms…..

Collection of information to determine the value of a program

eVALUation

Evaluation is NOT…. Auditing Personnel assessment Monitoring (although this can be part

of an evaluation process) Used to end or shut down programs

Evaluation Myth #1

Evaluation is an extraneous activity that generates lots of boring data with useless conclusions

Evaluation Myth #2

Evaluation is about proving the success or failure of a program

Evaluation Myth #3

Evaluation is a unique and complex process that occurs at a certain time in a certain way, and almost always includes the use of outside experts.

How Can Evaluation Help You?

Demonstrate program effectiveness or impacts



Better manage limited resources



Better manage limited resources Document program accomplishments



Better manage limited resources Document program accomplishments Justify current program funding



Better manage limited resources Document program accomplishments Justify current program funding Support need for increased funding



Better manage limited resources Document program accomplishments Justify current program funding Support need for increased funding Satisfy ethical responsibility to clients to

demonstrate positive and negative effects of participation



Better manage limited resources Document program accomplishments Justify current program funding Support need for increased funding Satisfy ethical responsibility to clients to

demonstrate positive and negative effects of participation

Document program development and activities to help ensure successful replication

Ultimately…

To improve program performance which leads to better value for your

resources

No Evaluation Means…. No evidence that your program is working

or how it works


or how it works Lack of justification for new or increased

funding



funding No marketing power for potential clients



funding No marketing power for potential clients Lack of credibility



funding No marketing power for potential clients Lack of credibility Lack of political and/or social support



funding No marketing power for potential clients Lack of credibility Lack of political and/or social support No way to know how to improve

Program Life Cycle

Development

Implementation

Evaluation

Revision Sustainability

Basic Terminology Types of Evaluation

› Outcome (summative)› Process (formative)



Outcomes



Outcomes Indicators



Outcomes Indicators Measures



Outcomes Indicators Measures Benchmarks



Outcomes Indicators Measures Benchmarks Quantitative vs. qualitative

Evaluation Process

Engage stakeholders

Clearly define program

Written evaluation

planCollect

credible/useful data

Analyze data

Share/use results

Engage Stakeholders Those involved in program design,

delivery, and/or funding Those served by the program Users of the evaluation results

Clearly Define Program Resources, activities, outcomes Context in which program operates Logic model

› Explicit connections between “how” and “what”

› Helps with program improvement› Good for sharing program idea with others› Living, breathing model

IF THEN

IF THENI take an

aspirin

IF THENI take an

aspirinMy headache will go away

IF = Inputs & ActivitiesTHEN = Outcomes

Written Evaluation Plan Outcomes Indicators Tools Timelines Person(s) responsible (optional)

Sample Evaluation PlanPROGRAM OUTCOME INDICATOR(S)

DATA COLLECTIO

N TOOL

DATA COLLECTION SCHEDULE

Training participants know how to recognize a seizure

Percent of training participants who correctly identify 10 out of 13 possible symptoms of a seizure

Participant pre, post and follow-up surveys

Pre survey given prior to training; post survey given immediately after training; follow up survey given 30 days after training

Credible Data Collection Tools

Valid and reliable tools› Valid=measures what it is intended to

measure› Reliable=consistent results over time

Qualitative Quantitative Will answer your evaluation questions

and inform decision-making

Collect Credible/Useful Data Quantitative

› Surveys› Tests› Skill assessments

Qualitative› Focus groups› Interviews› Journals› Observations

Analyze Data Many methods Answer evaluation questions Engage stakeholders in interpretations Justify conclusions and

recommendations Get help if needed!

Share/Use Results Reporting format Getting results into the right hands Framing the results Collaborative vs. confrontational

approach Keeping users “in the loop” Debriefs and follow-up

Considerations Purpose

Considerations Purpose Audience

Considerations Purpose Audience Resources

Considerations Purpose Audience Resources Data

Considerations Purpose Audience Resources Data Timeline

Considerations Purpose Audience Resources Data Timeline Planning is key

Considerations Purpose Audience Resources Data Timeline Planning is key Expertise

Contracting out… Staff to perform work


› Available


› Expertise› Available



Credibility



Credibility Technological support




› Collect data




› Collect data› Analyze data




› Collect data› Analyze data

Time frame

Seniors and Seizures Evaluation

Training program for caretakers of seniors with epilepsy/seizures

ADC staff and primary care providers Training provided by affiliates Delivery varies but content is

consistent

Process Meeting with EF staff to learn about the

program Collaboration with affiliate staff to design

logic model Decisions regarding which outcomes to

measure Decisions regarding how to best collect

data Designed data collection tools Pilot testing and revision

Evaluation Questions What impact did the training program

have on knowledge of seizures in seniors?› Pre and post knowledge assessment› Post-training survey

What impact did the training program have on participants’ confidence and comfort in working with seniors ?› Post-training survey

Knowledge

Pre Training Post Training0

102030405060708090

100

54

88

Increased Knowledge After “Seniors and Seizures” Training Program (N=17)

Percentage of test ques-

tions an-swered cor-

rectly

Care of Seniors

Confidence to use material

Quality of care

1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0

8

7.9

Participant Ratings of Training Impact

1=no impact, 10=extensive impact

1=not at all confident, 10=completely confident

Our benchmark is a rating of 7.0 or higher

For More Information….Kathleen Dowell, Ph.D., President

EvalSolutions6408 Whistling Wind Way

Mt. Airy, MD 21771

[email protected]

&%$#@&%$!!

Documents

Transcript of &%$#@&%$!!