What Works Clearinghouse Susan Sanchez Institute of Education Sciences.
-
Upload
bruce-stevens -
Category
Documents
-
view
219 -
download
0
Transcript of What Works Clearinghouse Susan Sanchez Institute of Education Sciences.
![Page 1: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/1.jpg)
What Works Clearinghouse
Susan Sanchez Institute of Education Sciences
![Page 2: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/2.jpg)
Purpose: To promote education decision making through a web-based dissemination system featuring comprehensive, systematic, high-quality reviews of studies on the effectiveness of educational interventions (programs, products, practices and policies).
![Page 3: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/3.jpg)
The WWC Does Not:
Endorse educational interventions
Conduct field studies of the effects of interventions
Rather, the WWC reports on the effectiveness of educational
interventions as measured by available evidence.
![Page 4: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/4.jpg)
![Page 5: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/5.jpg)
Challenges
Research literature too abundant for individual selection. Where to begin?
Difficult to sift and know what to trust and what to use. Lots of claims of effectiveness out there, but what can you trust?
It has to be transparent if it’s to be believed.
Complex, technical issues can take more than forever and ever to resolve.
![Page 6: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/6.jpg)
Challenges
Opps, we got the answer but now there are new advances in methodology.
Consumers need a fast response for evidence on what works.
Pretty wonky stuff. How do you create user-friendly web-based reports and products that meets your customers’ needs?
![Page 7: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/7.jpg)
WWC Systematic Review Process
A systematic review is a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to extract and analyze data from the studies that are included in the review.
![Page 8: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/8.jpg)
WWC Systematic Review Process
Select the Topic Develop the review protocol
Research questions Search parameters Inclusion/Exclusion criteria
Conduct a literature search & identify the research
Screen and code studies for relevance and methodological quality
Analyze the results of eligible studies Summarize results
![Page 9: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/9.jpg)
Study Review Process
Does Not Pass Screen
Meets WWC Evidence Standards• Meets WWC Evidence Standards•Meets WWC Evidence Standards
with Reservations
Submissions from the Public & Intervention
Developers
Literature Searches
Screening Standards
Pass Screen
Study Reviewed Against the WWC Evidence Standards
Does Not Meet WWC Evidence Screens
Develop ProtocolDevelop Protocol
![Page 10: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/10.jpg)
Three Stages of Review
Screen studies for relevance Select only relevant evidence
Assessing strength of the evidence and sort by rigor Select only credible evidence Sort credible evidence by whether any
reservations
Identify other important study characteristics
![Page 11: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/11.jpg)
Stage 1: Screening for Relevance
All potentially relevant studies identified through the extensive search
Does the study provide evidence that, if credible, would be a valuable addition to the knowledge base regarding the effectiveness of the focal intervention?
![Page 12: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/12.jpg)
Six Essential Screeners
1) Relevant time frame
2) Relevant intervention
3) Relevant sample
4) Relevant outcome
5) Adequate outcome measures
6) Adequate reporting
![Page 13: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/13.jpg)
Stage 2: Assessing Strength of Evidence
The goal is to apply consistent criteria to sort evidence by its credibility into three buckets:
Meets Evidence Standards
Meets Evidence Standards with Reservations
Does Not Meet Evidence Screens
![Page 14: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/14.jpg)
The WWC Evidence Standards (applied to individual studies)
• Meets Evidence Standards
• RCTs without severe design or implementation flaws
• Meets Evidence Standards with Reservations
• RCTs with severe design or implementation flaws
• QEDs with equating and without severe design or implementation flaws
• Does Not Meet Evidence Screens
•Studies not relevant to review
•Studies with fatal design or implementation flaws
![Page 15: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/15.jpg)
Five Factors Govern Judgments
Method for forming intervention and comparison groups
Evidence on baseline equivalence
Sample attrition
Possible contamination of study conditions
Threat of teacher confound
![Page 16: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/16.jpg)
Stage 3: Other Study Characteristics
Variations in people, settings and outcomes represented How generalizable are the findings?
Results reported by subgroups, settings and outcomes What is the breadth of the outcomes
reported?
Statistical results available How complete is the reporting? Do the estimates reflect statistical
controls for baseline characteristics?
![Page 17: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/17.jpg)
![Page 18: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/18.jpg)
Evidence Base of Character Education Programs
Over 70 programs were submitted to or identified by the WWC. 41 school-based programs met WWC definition and were eligible for review. 93 studies on the 41 programs were collected.
13 programs had at least one study meeting evidence standards, either with or without reservations
27 programs had no studies passing evidence screens 1 program is under review
![Page 19: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/19.jpg)
13 Character Education Programs with at Least one Study Meeting Evidence
Standards, either with or without Reservations
1. An Ethics Curriculum for Children 8. Skills for Action
2. Building Decision Skills 9. Skills for Adolescence
3. Caring School Communities 10. Too Good for Drugs
4. Connect with Kids 11. Too Good for Violence
5. Facing History and Ourselves 12. Too Good for Violence and Drugs
6. Lessons in Character7. Positive Action
13. Voices Literature and Character Education Program
![Page 20: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/20.jpg)
Distribution of 92 Studies on 40 Programs
across Evidence Categories7 (8%)
11 (12%)
74 (80%)
Meet evidence standards with reservations
Meet evidence standards
Do not meet evidence screens
![Page 21: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/21.jpg)
Common Reasons for Studies Failing to Meet Evidence
Screens
Not meet relevance screens Lack of a comparison group Severe overall or differential attrition for QEDs Lack of baseline equivalence for QEDs Confound in one-teacher/school-per-condition studies Inadequate statistical reporting for ES computation
![Page 22: What Works Clearinghouse Susan Sanchez Institute of Education Sciences.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649ed95503460f94be7f22/html5/thumbnails/22.jpg)
For More Information on the WWC
• Visit our website at www.whatworks.ed.gov
• Subscribe for WWCUpdate, the WWC’s electronic news alert through our website
• Phone: 1-866-WWC-9799 E-mail: [email protected]