E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA:...
-
Upload
doris-carroll -
Category
Documents
-
view
213 -
download
0
Transcript of E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA:...
![Page 1: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/1.jpg)
1
EVALUATION TOOLS AS IMPLEMENTATION DRIVERS
An Example from the California SPDG’s
ERIA: Effective Reading Interventions Academy
![Page 2: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/2.jpg)
2
EARLY ERIA: EFFECTIVE READING INTERVENTIONS ACADEMY
ERIA established in 2003-04 before the Era of RTI
upper-elementary and middle schools focus
a diversity of approaches at sites in regional cohorts
recently dev ERIA 2.0 middle and high school focus
Frequently Asked Questions “What is ERIA?”
“How does this relate to RtI2?”
“What is ‘intervention’ and where do we get it?”
How do we Evaluate this?
![Page 3: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/3.jpg)
3
HOW?
![Page 4: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/4.jpg)
4
Evaluation Tools asImplementation
Drivers
HOW?
![Page 5: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/5.jpg)
5
TWO INTEGRATIVE EVALUATION TOOLS SERVE AS IMPLEMENTATION DRIVERS
Program Guide articulates PD model introduces and illustrates contextualizes the training gets away from “you had to be there”
Implementation Rubric operationalizes PD model drives ongoing implementation enables fidelity checks is possible to evaluate
Everyone is on the same page Sustainability (beyond funding, staff turnover) Scale-up (recruit new sites/districts, beyond SPDG) Diversity of approaches enabled
![Page 6: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/6.jpg)
6
EVALUATION DRIVES ERIA’S EVIDENCE-BASED PRACTICES
The Program Guide, a 16-page booklet, explicitly addresses both implementation and intervention practices to guide the design of a site-based program.
The Implementation Rubric is a 10-item instrument which provides a framework for trainers, coaches, site team members, and teachers to evaluate and discuss implementation, fidelity, and next steps.
Some additional tools include: end-of-event training surveys and three-month follow-ups feedback and support from cohort coaches and site team fidelity observations student data
![Page 7: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/7.jpg)
7
ERIA’S EVIDENCE-BASED PRACTICES
The Program Guide articulates a comprehensive set of practices for all stakeholders.
Implementation Practices Intervention Practices
Initial Training
Team-based Site-level Practice and Implementation
Implementation Rubric facilitates self-eval
Ongoing Coaching
Booster Trainings
Implementation Rubric reflection on next steps
The 5 Steps of ERIA
Data-informedDecision-making Screening and
Assessment Progress Monitoring
Tiered Interventions and Learning Supports
Enhanced Literacy Instruction
![Page 8: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/8.jpg)
8
SPDG Evaluators Li Walter and Alan Wood synthesized content expert input and worked to make it readily accessible to a variety of stakeholders.
“5 Steps”• Step 1: Identify• Step 2: Assess• Step 3: Deliver• Step 4: Monitor• Step 5: Improve
EVALUATION TOOL: THE PROGRAM GUIDE
![Page 9: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/9.jpg)
9
Stakeholders from every level were included to provide input both on implementation and intervention practices.
A variety of existing documents and other resources were also synthesized into theProgram Guide.
![Page 10: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/10.jpg)
10
There is a Table of Contents for ease of use.
![Page 11: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/11.jpg)
11
Implementation practices are outlined over a three-year schedule.
![Page 12: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/12.jpg)
12
Implementation practices are described in detail.
![Page 13: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/13.jpg)
13
The Site Team is key to implementing ERIA.
It and its supporting structures are detailed.
![Page 14: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/14.jpg)
14
An expansive list of key roles are described in detail.
![Page 15: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/15.jpg)
15
While the first half of the Program Guide focuses on implementation practices, the second half focuses on intervention practices and the 5 Steps:
Step 1: IdentifyStep 2: AssessStep 3: DeliverStep 4: MonitorStep 5: Improve
![Page 16: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/16.jpg)
16
Step 1: Identify
“Identify struggling readers through universal literacy screening early in the school year using statewide English-language Arts test scores.”
![Page 17: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/17.jpg)
17
Step 2: Assess (1 of 2)
“Assess the decoding, reading fluency, and comprehension skills of struggling readers to guide intervention placement and instruction.”
![Page 18: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/18.jpg)
18
Step 2: Assess (2 of 2)
Presents a variety of assessment tools and strategies, both for basic and advanced implementation.
![Page 19: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/19.jpg)
19
Step 3: Deliver (1 of 2)
“Deliver interventions to assess specific skill needs for success in the core curriculum using evidence-based programs and practices with fidelity.”
![Page 20: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/20.jpg)
20
Step 3: Deliver (2 of 2)
Presents a variety of Intervention topics, programs, and models and how they may be appropriate for implementation, both in basic and advanced implementation.
![Page 21: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/21.jpg)
21
Step 4: Monitor
“Monitor the progress of struggling students to ensure that interventions are helping students improve and to adjust intervention placements accordingly.”
![Page 22: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/22.jpg)
22
Step 5: Improve
“Improve content literacy instructional practices to actively and effectively engage all students in the core curriculum.”
![Page 23: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/23.jpg)
23
Student Outcomes
Past successes, increasing English-Language Arts proficiency inclusive of Students with Disabilities, is detailed.
![Page 24: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/24.jpg)
24
EVALUATION TOOL:IMPLEMENTATION RUBRIC
The 10 items are intervention practices-focused mostly, with site team and fidelity items
The overall tool and process of how the rubric isused drives the implementation practices Self-evaluate and reflect on learning and
implementation. Shared with coaches and trainers to guide activities Evaluates the fidelity of implementation of both the PD
model and the interventions
Former 26-item, 3-point ERIA Checklist lacked the specificity to be meaningful and useful.
![Page 25: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/25.jpg)
25
IMPLEMENTATION RUBRIC, ADAPTED FROM “GOAL ATTAINMENT SCALES” Amy Gaumer Erickson and Monica Ballay presented
“goal attainment scales” on a June 17 SIG Network webinar: http://www.signetwork.org/content_pages/78
Rubric explicitly describes 5 implementation levels for each of 10 items: Levels 1, 2, and 3 reflect the “Not started,” “In
progress,” and “Achieved” implementation levels of former checklist.
Levels 4 and 5 detail concrete steps towards optimal implementation, beyond the basics.
Each implementation level for each item is explicitly described, building more meaning into the tool than our previous checklist format allowed.
![Page 26: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/26.jpg)
26
![Page 27: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/27.jpg)
27
![Page 28: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/28.jpg)
28
![Page 29: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/29.jpg)
29
![Page 30: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/30.jpg)
30
![Page 31: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/31.jpg)
31
IMPLEMENTATION RUBRIC EXCEL FILE:MULTI-YEAR TRACKING AND AUTOMATED REPORTS
The same file is used in all three years of ERIA,reporting both the trend and most-recent entries.
![Page 32: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/32.jpg)
32
EVALUATING PROFESSIONAL DEVELOPMENT:GUSKEY’S FIVE CRITICAL LEVELS
All five levels are addressed through various tools, structures, and practices over ERIA’s three year professional development schedule.
Level 1: Participants’ Reactions End-of-Event Evaluations 3-month Follow-up Surveys
Level 2: Participants’ Learning Enhanced Field-based Training Implementation Rubric Coaching Feedback/Support
![Page 33: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/33.jpg)
33
EVALUATING PROFESSIONAL DEVELOPMENT:GUSKEY’S FIVE CRITICAL LEVELS
Level 3: Organizational Support and Change Team-based Implementation Implementation Rubric Coaching Feedback and Support
Level 4: Participants’ Use of New Knowledge and Skills Fidelity Observations Integrated with Coaching Implementation Rubric’s advanced implementation
levels
Level 5: Student Learning Outcomes Student Data guides Coaching and Booster Training
content to address gaps and capitalize on successes
![Page 34: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/34.jpg)
34
Evaluation Tools asImplementation
Drivers
![Page 35: E VALUATION T OOLS A S I MPLEMENTATION D RIVERS An Example from the California SPDG’s ERIA: Effective Reading Interventions Academy 1.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649cf15503460f949c0d86/html5/thumbnails/35.jpg)
35
ERIA on the Web:http://calstat.org/effectivereading.html
Li Walter: [email protected]
Alan Wood: [email protected] (707) 287-0054