Post on 05-Oct-2020
What Works Clearinghouse:Education
M. C. Bradley
Mathematica Policy Research
Building Blocks of Presentation
• Understanding the What Works Clearinghouse (WWC)• Introduction to the WWC
• Discussion of the backbone of the WWC
• Examining products released by the WWC, including relevant examples
• Identifying whether a study was reviewed, and its WWC rating
• Searching for particular evidence from Intervention Reports
What is the WWC?
Problem and Context
• Substantial amount of research on education interventions• Large variation in quality• No common standards• Synthesizing findings is not
simple• Studies summarized many ways
• How does that affect choices?• Often based on personal
experience and ideology • Infrequently on strong research
WWC Mission
• To be a central and trusted source of scientific evidence for what works in education• Develops and implements standards for reviewing and
synthesizing education research
• Assesses the credibility of research evidence on the effectiveness of interventions (does not directly assess interventions)
• Transparent, objective, and high quality
http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_info_what_061015.pdf
WHAT
The WWC reviews evidence of effectiveness of programs, policies, or practices by using a consistent and transparent set of standards. The WWC doesn’t rank, evaluate, or endorse interventions.
WWC: Influencing Research and Practice
7
Producers of Information
• Universities
• Research organizations
• Developers
Consumers of Information
• State and federal govt.
• Education departments
• Superintendents
• School boards
• Principals and teachers
• Parents and the public
• Funders and consultants
Research Standards Effectiveness Information
Evidence-Based
Decisions
Improved Student
Outcomes
Research
Topics
What Works Clearinghouse
• Develop standards
• Train reviewers
• Identify research
• Review research
• Summarize evidence
• Disseminate findings
The Review: Backbone of a WWC Product
• Review of publication to determine whether it supports causal inferences
• Three sets of design standards • Group design standards
• Pilot single-case design standards
• Pilot regression-discontinuity design standards
• Four ratings• Meets WWC design standards without reservations
• Meets WWC design standards with reservations
• Does not meet WWC design standards
• Not eligible for review
Review Protocols: Set Parameters
http://ies.ed.gov/ncee/wwc/Publications_Reviews.aspx?f=All%20Publication%20and%20Product%20Types,5;#pubsearch
Are the groups formed randomly?
Is sample attrition high or low?
Were groups similar before the intervention began?
Are there confounding factors or concerns with outcomes?
http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_info_rates_061015.pdf
Key questions for a group design review
http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_info_reporting_061015.pdf
Improvement Index
Effectiveness Rating
Extent of Evidence
What does the WWC report?
WWC Products
WWC Website: All Resources and Products
http://whatworks.ed.gov
http://ies.ed.gov/ncee/wwc/
Intervention Reports
An intervention report is a summary of findings of the most reliable research on a given curriculum, program, practice, or policy in education.
http://ies.ed.gov/ncee/wwc/pdf/intervention_reports/wwc_fasttrack_100714.pdfhttp://ies.ed.gov/ncee/wwc/pdf/intervention_reports/wwc_socialskills_020513.pdf
http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_info_making_061015.pdf
1. Define scope of review2. Conduct a comprehensive
search3. Screen and review studies4. Combine findings5. Summarize the review
Steps in Writing an Intervention Report
Searching for an intervention report
http://ies.ed.gov/ncee/wwc/Publications_Reviews.aspx?f=All+Publication+and+Product+Types%2c5%3b
What can you learn from an intervention report?
http://ies.ed.gov/ncee/wwc/interventionreport.aspx?sid=628
Parts of an Intervention Report
• Program description
• Research
• Effectiveness
http://ies.ed.gov/ncee/wwc/pdf/intervention_reports/wwc_careeracademies_092215.pdf
Inside an Intervention Report
• Program Information
• Research Summary
• Effectiveness Summary
• References
• Appendices
http://ies.ed.gov/ncee/wwc/pdf/intervention_reports/wwc_careeracademies_092215.pdf
Single Study Reviews
Assess one study against WWC standards• Not a systematic review on that particular intervention
First stage for some single study reviews is a “Quick Review”• Email blast giving sense of whether study will meet standards,
ideally one week after media attention
What can you learn from a quick review?
http://ies.ed.gov/ncee/wwc/quickreview.aspx?sid=20155
What can you learn from a single study review?
http://ies.ed.gov/ncee/wwc/SingleStudyReview.aspx?sid=10112
http://ies.ed.gov/ncee/wwc/pdf/single_study_reviews/wwc_griffiths_070715.pdf
WWC Practice Guides
A practice guide is a set of recommendations for educators to address challenges in their classrooms and schools. The level of research evidence supporting each recommendation is provided.
• Expert panel to develop recommendations
• Conduct a comprehensive literature search
• Review each study against WWC standards
• Summarize the review
http://ies.ed.gov/ncee/wwc/Publications_Reviews.aspx?f=All%20Publication%20and%20Product%20Types,3;#pubsearch
What is in a practice guide?
http://ies.ed.gov/ncee/wwc/pdf/practice_guides/wwc_algebra_040715.pdf http://ies.ed.gov/ncee/wwc/pdf/practice_guides/early_math_pg_111313.pdf
Recommendations
How to steps
Lots of Examples
What can an example do for you?
What can an example do for you?
If you are a researcher,
http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_gsa_v1.pdf
Has a study been reviewed by the WWC?
http://ies.ed.gov/ncee/wwc/ReviewedStudies.aspx
Find What Works
http://ies.ed.gov/ncee/wwc/findwhatworks.aspx
Contact Information
The What Works Clearinghouse
• http://whatworks.ed.gov
On Facebook at
• www.facebook.com/whatworksclearinghouse
On Twitter at
• www.twitter.com/WhatWorksED
And a widget you can add to your own pages
• http://ies.ed.gov/ncee/wwc/Widget.aspx
M. C. Bradley
• cbradley@mathematica-mpr.com