Using the PMRN and ECI and EI Indices to Make Effective
Decisions
Elizabeth Crawford, MS, CCC-SLPDirector of Interventions
The Florida Center for Reading Researchwww.fcrr.org
2
Agenda
General Uses of Data PMRN 101 Highlights of DIBELS Use Data to:
manage school level intervention resources monitor progress of specific groups of students run an effective class/student level data
meeting interpret ECI and EI tables
Concluding Thoughts and Resources
3
General Uses of Data
4
Barriers to Using Data to Inform Instruction
Grunwald Associates (out of Maryland) surveyed 353 small, medium and large school districts asking them “What are the barriers to you using data-driven decision making more effectively?”
The next slides list the results
5
Grunwald Associates Page 0
Barriers to More Effective Data-Driven Decision MakingLack of training and interoperability are the main barriers to more effective data-driven decisions. A lack of understanding in what to do with the data is a barrier that’s particular to large and medium districts.
Barriers to Data-Driven Decision MakingTo what degree does your district consider the following to be barriers to
a more effective data-driven decision making process? (N=353)(Percent Responding 8, 9, or 10 on 10-point scale) Large (A)
School District Size
Significantly higher
Medium (B) Small (C)
50%
42%
39%
36%
35%
31%
24%
24%
22%
Lack of training
Interoperability - systems that are unable toshare/exchange data
Lack of understanding in what to do with thedata
Absence of clear priorities on what datashould be collected
Failure to collect data in a uniform manner
Outdated technology / legacy systems
Low quality data - inaccurate or incomplete
Timing of data collection
User interface is too complicated tounderstand reports
47%
44%
41%
36%
38%
34%
24%
23%
19%
55%
42%
44%
37%
35%
30%
25%
25%
24%
42%
36%
17%
28%
28%
22%
20%
23%
21%
C
C
6
Barriers to more effective data-driven
decision making Lack of training 50%
Interoperability – systems that can’t share data 42%
Lack of understanding about what to do with the data 39%
Absence of clear priorities 36%
Failure to collect data uniformly 35%
Outdated technology 31%
Incomplete or incorrect data 24%
Timing of data collection 24%
User interface is too complicated to understand reports
22%
7
PMRN 101
8
What is the PMRN?(Progress Monitoring and Reporting Network)
A Web-based data management system that provides:
a convenient place for entering and organizing the results of student assessments
a secure, centralized, easily accessible location for the storage of student information
a tool for timely and helpful reports so that educators can effectively analyze data, plan instruction, and communicate student progress
9
The PMRN is located at https://pmrn.fcrr.org.
To Sign In to the PMRN, you will need the User Name and Password that were E-mailed to you, as well as your Identifier which was received in a letter from the FCRR.
Signing In to the PMRN
Please note the “S” in https. The “S” indicates a secure network and is required for your web browser to find the PMRN.
10
11
Active Header Menu
Availability of information is limited by the Users access level. The level of access determines if the User can select all or specific grades, teachers, students, and/or tests.
12
Types of Reports and how data is reported Status - Table Progress – Box and Whiskers and Pie chart (except at the
student level which only has Box and Whiskers) Class Progress Tracking – Bar graph Historical - Box and Whiskers Risk Level - Table Recommended Level of instruction – Table and/or Pie
chart Student Recommended Instructional Level - Table Grade Summary – Box and Whiskers Comparison - Box and Whiskers Demographics – Table and Bar graph Cumulative - Table Ongoing Progress Monitoring – Line graph Year End Outcome – Pie chart and Table
13
Box & Whiskers Graph
The Box & Whiskers Graph is a type of distribution report. It is used to report how an individual or group compares to a larger group. The Box & Whiskers used in the PMRN is a modified version of the those that report quartiles.
14
Highlights of DIBELS
15
GRADE DIBELS MEASURE READING COMPONENT
ASSESSEDKindergarten Initial Sounds, Phoneme
Segmentation
Letter Naming, Nonsense Words
Phonemic Awareness
Phonics
First Phoneme Segmentation
Letter Naming, Nonsense Words
Oral Reading
Phonemic Awareness
Phonics
Reading Fluency
Second Nonsense Words
Oral Reading
Phonics
Reading Fluency
Third Oral Reading Reading Fluency
16
Risk Level Key
17
18
SY0607 Florida DIBELS Recommended Level of Instruction Decision Rules
http://www.fcrr.org/assessment/pdf/ril_decisionrules_0607.pdf
19
Use Data to…
Manage school level intervention resources
Monitor progress of specific groups of students
Run an effective class/student level data meeting
Interpret ECI and EI tables
20
Strong Leadership is critical to success
The great leaders are like the best conductors - they reach beyond the notes to reach the magic in the players. - Blaine Lee
21
Use Data to… manage school level intervention resources
22
School Level Reports
“The Chart” Highlight:
School Status (“Home Base”) School Demographics
23
“The Chart”
Use the chart as a “quick start” guide to the PMRN for the School Level Reports
Refer to separate handout The PMRN User’s Guide is the
complete manual and should be read and referenced for specific questions and information
24
25
26
Using the School Status Report – “Home Base”
Make a list of school wide resources Who needs extra support? (High Risk,
Moderate Risk, Low Risk) Which Grades? Which Teachers?
Which skills need to be emphasized in professional development?
27
28
School Status Report – “Home Base”
29
School Status Report – “Home Base”
30
School Status Report – “Home Base”
31
Navigating from the School Status Report – “Home Base”
School Grade Summary – Click on the words ‘grade summary’ under the grade want to see
Class Status – From the extended view, click on the teacher’s name
School Progress – From Grade Summary, click on ‘show progress report’ at bottom
Class Progress – From the School Progress, use active header menu to select teacher/class of interest
School Comparison – From School Progress, click on ‘show school comparison’ at bottom
** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get
32
School Status Report – “Home Base”
33
School Grade Summary Report
34
Navigating from the School Status Report – “Home Base”
School Grade Summary – Click on the words ‘grade summary’ under the grade want to see
Class Status – From the extended view, click on the teacher’s name
School Progress – From Grade Summary, click on ‘show progress report’ at bottom
Class Progress – From the School Progress, use active header menu to select teacher/class of interest
School Comparison – From School Progress, click on ‘show school comparison’ at bottom
** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get
35
School Status Report – “Home Base”
36
Class Status Report
37
Navigating from the School Status Report – “Home Base”
School Grade Summary – Click on the words ‘grade summary’ under the grade want to see
Class Status – From the extended view, click on the teacher’s name
School Progress – From Grade Summary, click on ‘show progress report’ at bottom
Class Progress – From the School Progress, use active header menu to select teacher/class of interest
School Comparison – From School Progress, click on ‘show school comparison’ at bottom
** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get
38
School Grade Summary Report
39
School Progress Report
40
Navigating from the School Status Report – “Home Base”
School Grade Summary – Click on the words ‘grade summary’ under the grade want to see
Class Status – From the extended view, click on the teacher’s name
School Progress – From Grade Summary, click on ‘show progress report’ at bottom
Class Progress – From the School Progress, use active header menu to select teacher/class of interest
School Comparison – From School Progress, click on ‘show school comparison’ at bottom
** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get
41
School Progress Report
42
Class Progress Report
43
Navigating from the School Status Report – “Home Base”
School Grade Summary – Click on the words ‘grade summary’ under the grade want to see
Class Status – From the extended view, click on the teacher’s name
School Progress – From Grade Summary, click on ‘show progress report’ at bottom
Class Progress – From the School Progress, use active header menu to select teacher/class of interest
School Comparison – From School Progress, click on ‘show school comparison’ at bottom
** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get
44
School Progress Report
45
School Comparison Group
46
Navigating from the School Status Report – “Home Base”
School Grade Summary – Click on the words ‘grade summary’ under the grade want to see
Class Status – From the extended view, click on the teacher’s name
School Progress – From Grade Summary, click on ‘show progress report’ at bottom
Class Progress – From the School Progress, use active header menu to select teacher/class of interest
School Comparison – From School Progress, click on ‘show school comparison’ at bottom
** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get
47
School Status Report – “Home Base”
48
School Progress Report
49
Use Data to… monitor progress of specific groups of students
50
51
School Demographic Reports
Demographic data is available, by assessment period, at the school and district levels for individual measures and the Recommended Level of Instruction.
The data file allows all of the data to be downloaded so that Users can complete their own analyses.
52
53
54
School Demographic Reports
You can also display a bar graph for a specific demographic group from this report. The next slides shows a bar graph for kindergarten students at the third assessment by Ethnicity.
The total number of students in each category is in ( ) next to the group name.
For example Hispanic (17)
55
56
57
Use Data to…run an effective class/student level data meeting
58
Does the district provide the tools and training to interpret and query data?
Have data teams developed a process for identifying, recommending and implementing intervention based on data?
Do teachers have access to data in an easy-to-use format soon after assessment?
Using Data Checklist
59
Data Utilization and Analysis
Key Characteristics of Data Meetings: Held on a regular basis (bi-weekly or
monthly) Attendees were able to make school
level decisions and personnel changes (usually the principal)
Systems/worksheets used for structure Data Compiled from multiple sources Follow-up
60
Class/Student Level Data Meeting Some principals start with discussing:
Any students who made great gains (celebrations)
Surprises in results (or discrepancies between impressions and actual data – if so, might look at student answer sheet to get more specific info)
Any students of great concern (and any circumstances that would be of importance – for example, the student has been tardy for the past 20 days missing all whole group instruction)
If any ESE, social work or other services are involved
Remind staff to look at scores not just colors and to keep ‘next target’ at bottom of column in mind when looking at scores
61
Class Level Reports Two main class level reports can be used to
run the meeting (with worksheets) Class Status Reports
To determine groups and areas of focus You might also want to have the status report from
the previous assessment period to compare scores
Class Recommended Level of Instruction To determine effectiveness of instruction and
target students
62
Class Status Report The Class Status report from the PMRN will
help you answer 3 main questions:
1. Who needs extra support?
2. How should groups be formed?
3. Which skills need to be emphasized?
Refer to “Class Status Report” Worksheet Handout
63
64
Class Status Report
• 1st Grade Class• Assessment
Period 2• Full year
students only• You can get here
from ‘School Status’ by clicking on teacher’s name when the grade is extended
65
Who Needs Extra Support?
• High Risk:5• Moderate Risk: 7• Low Risk: 4
• Look at the “Next Target” to
help guide grouping
66
How will small groups be formed?
• Group 1:
• Group 2:
• Group 3:
• Group 4:
67
How will small groups be formed?
• Group 1: Students 2,4
• Group 2: Students 1,3,5,6,7
• Group 3: Students 8,9,11,12
• Group 4: Students 13,14,15,16,10 (10 to get peer modeling to improve fluency)
68
Which skills need to be emphasized?
• Group 1:
• Group 2:
• Group 3:
• Group 4:
69
Which skills need to be emphasized?
• Group 1: PA and Phonics
• Group 2: Phonics and Fluency
• Group 3: Phonics and Fluency
• Group 4: Fluency and Comprehension
70
Calculate the Class Level ECI, EI-I, and EI-S
How do you do it? AND…what does it all mean??
Refer to “DIBELS Data Analysis Conference” Worksheet Handout
71
72
Class Recommended Level of Instruction Report
73
Class Recommended Level of Instruction Report
This report provides a summary of the students’ overall progress. It can be used to get an overall sense of instructional/risk levels in the class and to calculate the Effectiveness of Core Instruction (ECI) index and the three Effectiveness of Intervention (EI) indices.
74
Class Recommended Level of Instruction Report
This is the same type of report with “all” students included. Students 5, 6, 7, and 19 would not be used to calculate the ECI and EI indices because they were not in the school for all three assessment periods. This is why we select the full year only function for calculation.
75
Class Recommended Level of Instruction Report
You can use the pie charts to help you fill out the top part of the DIBELS worksheet
Initial Level = Low risk
Strategic Level = Moderate Risk
Intensive Level = High Risk
Assessment #3
% Initial = 44
% Strategic = 50
% Intensive = 6
76
Class Recommended Level of Instruction Report
To calculate ECI – Effectiveness of Core Instruction for the year, use only those students that were at Initial Instruction during the 1st assessment and remained in the class throughout the year (that is why we select full year only).
Divide the number of the students that finished at Initial Instruction for the 3rd assessment (5) by the number of students that were at Initial Instruction for the 1st assessment (9). The higher the percentage the better.
77
Class Recommended Level of Instruction Report
Effectiveness of Core Instruction
# stayed initial 5--------------------- = -------# started initial 9
5
--- = 56% ECI = 56% 9
Target Students:
Students 3, 6, 8, 9 (Initial to Strategic)
78
Class Recommended Level of Instruction Report
To calculate EI – Effectiveness of Intervention for the year, use only those students that were in the class for the first assessment and the last assessment (full year only).
Take the number of students who finished the year at ‘initial’ or ‘strategic’ (if they started at intensive) level by the students who started the year at some level of risk (intensive or strategic)
The higher the percentage the better.
79
Class Recommended Level of Instruction Report
Effectiveness of Intervention
# moved to initial or strategic from Intensive 2--------------------- = -------# started strategic & intensive 7
2
--- = 29% EI = 29% 7
Target Students:
Students 4, 7, and 10 (strategic to intensive during assessment 2)
80
Class Recommended Level of Instruction Report
Effectiveness of Intervention-Strategic
# moved to initial 1--------------------- = -------# began strategic 5
1
--- = 20% EI-S = 20% 5
Target Students: Students 4, 7, and 10 (strategic to intensive during assessment 2)
81
Class Recommended Level of Instruction Report
Effectiveness of Intervention - Intensive
# moved to strategic or initial 1--------------------- = -------# began intensive 2
1
--- = 50% EI-I = 50% 2
Target Students:
Student 5
82
What do all of these indices mean for my school and teachers? This particular teacher in 1st grade is struggling both with
maintaining children at grade level and reaching her struggling readers:
ECI = 56% EI = 29% EI-S = 20% EI-I = 50%
You may want to first determine if other first grade teachers have similar indices and if so, you want to examine your Core Reading Program and especially your Intervention Programs
You can look at the School Recommended Level of Instruction Report to see how different grade levels are doing by comparing the pie charts from one assessment period to the next
The goal is to see the green increase and the red decrease – to make the pie chart look more like an olive!!
83
School Level Indices
Breaking down the tables and text you receive in your ECI and EI letters
84
ECI Definition Effectiveness of Core Instruction
(ECI) — this index shows the percentage of students who began the year reading “at grade level” and are continuing to meet grade level expectations at the mid-year or end of year assessment. This index tells you whether the core instruction is sufficiently powerful to produce a year’s growth in reading skill for a year’s worth of instruction for all students who begin the year at grade level.
85
ECI – “Technical”
Percentage of students who started on grade level and remained on grade level
Number of children used to calculate the index score
Percentile Rank compared to other RF schools
86
ECI – “School”
91% of the 45 kindergarteners who started the year on grade level remained on grade level.91 x 45 = 41 children41 out of 45 children remained on grade level
.97 x 75 = 73 children73 out of 75 children remained on grade level, 2 children did not remain on grade level
In first grade, my school is doing equal to or better than 96% of the RF schools on keeping our 1st grade students on grade level.
87
EI Definition Effectiveness of Interventions (El)* —this
index shows the percentage of your students who began the year at some level of risk for reading difficulties (reading below grade level) but who have grown rapidly enough to advance to a lower level of risk by the mid year or end of year assessment (i.e. they move from intensive to strategic, or from strategic to grade level). This index, and the two explained below, provide information about how effectively intervention programs are working to accelerate the development of students who are lagging behind in reading growth.
88
EI – “Technical”
Percentage of children who progressed to a lower level of risk
Percentile rank when compared to the other Reading First schools in Florida
The number of children used to calculate the index
89
EI – “School”
T81% of the 42 kindergartners who started the year at some level of risk, have made gains and are now at a lower level of risk. .81 x 42 = 34 students
In first grade, my school is doing equal to or better than 44% of the RF schools in moving children to strategic or grade level.
.23 x 30 = 7 students7 out of 30 3rd graders who started at some level of risk have moved to a lower level of risk. 23 students are still at some risk.
90
EI-I Definition
Effectiveness of Interventions for Intensive Students (EI-I) — this index shows the percentage of students that moved from the “intensive” risk category to the “strategic” risk level or to grade level.
91
EI – I “Technical”
Percentage of students who started the year at “intensive” and made enough progress to be “strategic” or “grade level”
Percentile rank when index score is compared to other Reading First schools in Florida
Number of students used to calculate the index.
92
EI – I “School”
93% of our 14 kindergarten students who started the year at “intensive” made progress to a lower risk level..93 x 14 = 13 students made growth, 1 is still “intensive”
In second grade, we are doing an equal or better job than 47% of the RF schools at moving our children out of the “intensive” category.
In all grades, K-3, of the 46 students who started at “intensive” 50% (or 23) of them made growth to “strategic” or “grade level.”
93
EI-S Definition
Effectiveness of Interventions for Strategic Students (EI-S) — this index shows the percentage of students that moved from the “strategic” risk level to grade level.
94
EI – S “Technical”
Percentage of students who started the year at “strategic” moved to “grade level”
Percentile Rank of index when compared to all other RF schools in Florida
Number of students used to calculate the index
95
EI – S “School”
.56 x 27 kindergarten students who started the year at “strategic” moved to “grade level” = 15
27-15 = 12 are still “strategic”
In 1st grade, when compared to the other 583 RF schools in FL, We are doing equal to or better than 48% of them at moving first grade “strategic” students to “grade level”
In 3rd grade, 62% of the 16 children (10) who started at “strategic” are now at “grade level”
Worksheet Slides
97
ECI – “Worksheet”
___% of the ___ kindergarteners who started the year on grade level remained on grade level.___ x ___ = ___ children___ out of ___ children remained on grade level
.___ x ___ = ___ children___ out of ___ children remained on grade level, ___ children did not remain on grade level
In first grade, my school is doing equal to or better than ___% of the RF schools on keeping our 1st grade students on grade level.
98
EI – “Worksheet”
T___% of the ___ kindergartners who started the year at some level of risk, have made gains and are now at a lower level of risk. .___ x ___ = ___students
In first grade, my school is doing equal to or better than ___% of the RF schools in moving children to strategic or grade level.
.___ x ___ = ___ students___ out of ___ 3rd graders who started at some level of risk have moved to a lower level of risk. ___ students are still at some risk.
99
EI – I “Worksheet”
___% of our ___ kindergarten students who started the year at “intensive” made progress to a lower risk level..___ x ___ = ___ students made growth, ___ is/are still “intensive”
In second grade, we are doing an equal or better job than ___% of the RF schools at moving our children out of the “__________” category.
In all grades, K-3, of the ___ students who started at “intensive” ___% (or __) of them made growth to “strategic” or “grade level.”
100
EI – S “Worksheet”
.___ x ___ students who started the year at “strategic” moved to “grade level” = ___
___-___ = ___ are still “strategic”
In 1st grade, when compared to the other 583 RF schools in FL, We are doing equal to or ____ than ____% of them at moving first grade “strategic” students to “grade level”
In ___ grade, ___% of the ___ children (___) who started at “strategic” are now at “grade level”
101
ECI – “Worksheet”
___% of the ___ kindergarteners who started the year on grade level remained on grade level.___ x ___ = ___ children___ out of ___ children remained on grade level
.___ x ___ = ___ children___ out of ___ children remained on grade level, ___ children did not remain on grade level
In first grade, my school is doing equal to or better than ___% of the RF schools on keeping our 1st grade students on grade level.
102
EI – “Worksheet”
T___% of the ___ kindergartners who started the year at some level of risk, have made gains and are now at a lower level of risk. .___ x ___ = ___students
In first grade, my school is doing equal to or better than ___% of the RF schools in moving children to strategic or grade level.
.___ x ___ = ___ students___ out of ___ 3rd graders who started at some level of risk have moved to a lower level of risk. ___ students are still at some risk.
103
EI – I “Worksheet”
___% of our ___ kindergarten students who started the year at “intensive” made progress to a lower risk level..___ x ___ = ___ students made growth, ___ is/are still “intensive”
In second grade, we are doing an equal or better job than ___% of the RF schools at moving our children out of the “__________” category.
In all grades, K-3, of the ___ students who started at “intensive” ___% (or __) of them made growth to “strategic” or “grade level.”
104
EI – S “Worksheet”
.___ x ___ kindergarten students who started the year at “strategic” moved to “grade level” = ___
___-___ = ___ are still “strategic”
In 1st grade, when compared to the other 583 RF schools in FL, We are doing equal to or ____ than ____% of them at moving first grade “strategic” students to “grade level”
In ___ grade, ___% of the ___ children (___) who started at “strategic” are now at “grade level”
105
Key ‘follow-up’ sections on worksheet
Items Discussed Roles and Responsibilities
Teacher will… Reading Coach will… Principal will…
106
Follow Up is Key
107
Concluding Thoughts and Resources
108
Concluding Thoughts Use PMRN Reports to make school,
classroom, and student level decisions: Monitor progress Determine which students need more
intensive instruction Offer instructional strategies to the teacher Determine types of professional
development by grade level or individual teacher
By asking some of the following questions…
109
Is the core reading program implemented with fidelity? Effectiveness?
Are the interventions being implemented with fidelity? Effectiveness?
Is there a need for more professional development for a particular area of reading?
Is there a need for more intensive coaching? Are materials allocated to maximize learning? Is personnel use maximized to focus on reading
instruction? Is the instructional day scheduled such that reading is
THE priority?
Important Considerations
110
Florida Center for Reading Research 227 North Bronough Street,
Suite 7250 Tallahassee, FL 32301(850) 644-9352 - Phone (850) 644-9085 (Fax)www.fcrr.org
PMRN Help Desk 850-644-0931 (Voice)1-866-471-5019 (Toll Free)850-645-1700 (Fax)[email protected]
User’s guidehttp://www.fcrr.org/pmrn/userguides.htm
111
PROCESS TO PLAN FOR FOLLOW-UP
FOCUS ROLES
School-based Non-school Based
Principal Reading Coach
Teachers District RFPD Coordinator
Grade-level Leader Classroom Teacher
Instructional Content
Explicit Delivery
Assessment
Professional Development
A) Resources 1.Human resources 2.Materials
B) Time 1. Master schedule 2. Planning period
C) Services 1.Workshops 2.Follow-up
Top Related