EVIDENCE BASED PROGRAMSDr. Carol Albrecht Utah State Extension
Assessment Specialist [email protected]
Evidence Based Programs
Relatively new approach Program decisions are based primarily
on scientific method and scientific evidence
Use best evidence available to make decisions about program
Recognizes that knowledge is provisional and subject to refutation
Evidence Based Programs combines (1) Program Manager’s Expertise, (2) Client Attributes, and (3) Research Findings
Research Findings
Client
Attributes
Program Manage
r’s Expertis
e
EBP
Best Research Evidence Available Can Be Found in Current Literature
Step 1 – Review The Literature “Track down” the evidence- ongoing
process Note Inconsistencies Note research suggesting some
interventions are ineffective
Step 2 – Critique the Literature You Find Examine methods used*
Client Attributes
Take values, concerns, expectations of client into consideration
Involve clients in decision making process
Determine if this program will “fits” your client. Not “one size fits all” situation Consider race, gender, idiosyncratic
characteristics
Program Manager Expertise
Use Common Sense Avoid “fallacies” which include
Overgeneralization - Stereotyping Selective Perception/Observation Ex-post Facto Hypothesizing
Provide alternative explanation after the “study”
Accepting “new” as better Accepting “traditional” as better
How to Complete a Literature Review
Step 1 – Formulate a Question
Step 2 – Search for Evidence
Step 3 – Critically Appraise Relevant Studies
Step 4 – Apply the Literature to Your Question and/or Program
Step 5 – Conduct Your own Research
Step 1 – Formulate a Question What objectives do you want to
accomplish? What intervention is necessary to reach
objectives? Does Intervention A work? Is Intervention A cost effective? Does Intervention A work better than
Intervention B? Does Intervention A work for this type of client? Does intervention A work in this kind of
situation?
Useful Acronym for Formulating Question
Use acronym CIAO Client characteristics Intervention being considered Alternative intervention being considered (if
any) Outcome of intervention (does it work?)
Example
Client CharacteristicsIf children involved in 4-H
Intervention…..personally interact with a mentor for several years…..
Alternative Intervention…..or communicate with a mentor over the internet…..
Outcome…..are they more likely to seek post-high school education?
Step 2 – Search for the Evidence
Helpful “hints” to finding articles Use Google or Google Scholar and type in correct
search terms Combine general terms (e.g., volunteers, evaluation
etc.) With more specific terms (i.e., grades in high school,
crop yield etc.)
Top Down and Bottom Up Searches (Mullen 2006) Top Down – Find Articles/Books that have Compiled the
Literature (i.e., meta-analysis on your topic) Bottom Up – Find a very recently published article and
look at references.
Examples of Resources for “Top Down” Searches
*Campbell collaboration: http://www.campbellcollaboration.org/index.html
**Cochrane Collaboration: http://www.cochrane.org Substance Abuse:http://nrepp.samhsa.gov/htpp:/www.modelprograms.samhsa.gov/index.htm Crisis Intervention and Domestic Violence National Resource: http://www.crisisinterventionnetwork.com Expert Consensus Guidelines series: http://www.psychguides.com National Guidelines Clearinghouse http://www.guidelines.gov National Institute on Drug Abuse: http://www.nida.nih.gov/ Substance Abuse and Mental Health Services Administration:http://www.samhsa.govindex.aspx
*Focuses on Social Welfare, Education and Criminal Justice**Focuses on Health Care Interventions
Step 3 – Critically Appraise Relevant Articles
Emphasis Should be on Methods Used to Collect Empirical Information
How to Complete a Literature Review
Step 1 – Formulate a Question
Step 2 – Search for Evidence
Step 3 – Critically Appraise Relevant Studies
Step 4 – Apply the Literature to Your Question and/or Program
Step 5 – Conduct Your own Research
Step 4 – Apply the Literature to Your Research Question
Which intervention currently has the best empirical support?
Is the situation same/similar to yours? Are the participants in the study the
same or similar to the ones in your program/study?
Step 5 – Conduct Your Own Research
Use Literature Review to help you develop valid and reliable indicators.
Use Literature Review to help you determine your sampling technique and sample size.
Use the Literature Review to help you determine the most appropriate type of data collection.
Use the Literature Review to help you determine ways you can further contribute to the research (i.e., what is missing?)
Evaluate impact of program on your participants
Concerns About Evidence Based Programs
Merely Cost Cutting Have to use the methods “in stock”
Evidence Based May not be Applicable Leads to rigid implementation Your participants may be different
Evidence is in Short Supply – not enough research Real World Obstacles Hinder Implementation of
Programs Time and money are scarce
Devalues Qualitative Research Argument that “Empirical Evidence” is not
obtainable
Suggestions for Appraising Journal Articles in the Literature
Step 1 – Identify Purpose Stated purpose of article
This should appear in first paragraph(s).
Step 2 – Literature Review Is it adequate?
They should have at least 15 references.
Appraising Literature
Step 3 - Identify objectives/outcomes Do they identify and define their
outcomes/objecties It must be clear and succinct. They must be clearly related to the
implementation of program. How do they measure their
concepts/outcomes? Are measurements valid? (face validity) Are measurement reliable (consistent)
Could you repeat study?
Appraising Literature
Step 4 – Appraising Sampling Technique It must be clearly specified?
What is their unit of analysis? Do you know exactly how these units will be
selected? It must be appropriate?
Does the sampling technique “fit” with the data collection technique?
Do they draw valid conclusions based on sampling technique? Note – if they generalize they must have random
sample
Remember the Sampling Techniques
Random Sample – can generalize Simple or Systematic Random Multi-stage Cluster Stratified Random
NonRandom Sample – cannot generalize Theoretical or Purposive Snowball Convenient
Appraising Literature
Step 5 – Research Design/Data Collection Technique It must be clearly specified?
Do you know exactly how they will collect their data?
It must be appropriate Keep in mind the strengths and weaknesses of
the different research designs/data collection techniques
Remember Research Designs or Data Collection Techniques
Data Collection/Research Design Survey or Questionnaire Secondary Experimental
Lab Experiments Quasi – Field Experiment/Trial
Field Research (Qualitative) Participant Observation Content Analysis Intensive Interviews Case Studies
Appraising Literature
Step 6 - Critique Tables/Graphs/Charts Their analysis must be correct?
Did they include descriptive statistics of the sample? Did they include inferential statistics and discuss
level of significance? Are their tables constructed correctly?
They must be interpreted accurately. Does the text accurately describe the data?
They should look professional Do they provide sufficient information?
Do they tell you how they measured the indicators used in the tables?
Appraising Literature
Step 7 – Summary and Conclusions Do they accurately summarize their
research? Do they identify strengths and
weaknesses? Do they acknowledge limitations? Do they make suggestions for future
research?
Easy Steps
1. Identify Purpose 2. Critique Literature Review 3. Identify Success 4. Sample Technique 5. Data Collection Technique/Research
Design 6. Tables and Charts 7. Summary and Conclusions
Contact Information
Dr. Carol Albrecht Assessment Specialist Utah State University 979-777-2421 [email protected]
Top Related