Seven purposes presentation
-
Upload
john-cronin -
Category
Education
-
view
108 -
download
0
Transcript of Seven purposes presentation
Multiple measures done right: The purpose-driven assessment system
John Cronin, Ph.D.VP – Education ResearchNorthwest Evaluation Association
The pursuit of compliance is exhausting because it is always a moving target. Governors move on, the party in power gets replaced, a new president is elected, and all want to put their own stamp on education.
It is saner and less exhausting to define your own course and align compliance requirements to that.
The problem of compliance-based assessment systems
Seven standards that define the purpose driven assessment
system
The purposes of all assessments are defined and the assessments are valid and useful for their purposes.
DEFINE YOUR PURPOSES
Northwest Evaluation Association and Gallup - 2016
https://www.nwea.org/content/uploads/2016/05/Make_Assessment_Work_for_All_Students_2016.pdf
Make Assessment Work for All Students: Multiple Measures Matter
TEACHERS, PRINCIPALS AND SUPERINTENDENTS say students spend
too much time taking assessments.
Percent of students who say the results from state accountability tests are useful
29%Gallup (2016, May). Make Assessment Work for All Students: Multiple Assessments Matter.
https://www.nwea.org/content/uploads/2016/05/Make_Assessment_Work_for_All_Students_2016.pdf
Teacher and Administrator perceptions on the purposes of assessments
State Poli-cymakers
Parents Students Teachers Administrators0%
10%
20%
30%
40%
50%
60%
70%
80%
% State accountability assessments % Classroom tests and quizzes
Principals: Do you believe the purpose of data from (state accountability assessments, classroom assessments and quizzes) in your district is well understood by each of the following:
Students find many assessment types useful when the assessment is tied to a specific purpose
Classroom tests and quizzes developed by teachers to evaluate their students’ learning process
Interim assessments that are given to students two to three times over the year so teachers can see where students are growing and where they need to learn more
Performance tasks that require students to apply things they have learned to a real-world problem or situation
Formative assessments
78%
76%
75%74%
One action to take
First, document and prioritize your purposes for assessment then…inventory the assessments in use and evaluate their alignment with your purposes.
It’s a big job but worth the time!
Teachers are educated in the proper administration and application of the assessments used in their classrooms.
EDUCATE TEACHERS
What types of extra help, if any, does your teacher or school give you if you do not do well on a…
CL AS S ROOM T EST STATE STA N DAR D I Z E D TEST
Before and/or after school support 43% 24%
One to one support 32% 19%
Access to added resources 29% 22%
Small group support 24% 17%
No extra support provided 20% 38%
Teachers need to …
Understand the purpose
Follow the procedures
Properly interpret and act
One action to take
Develop internal structures and policies to provide training for all teachers (and administrators) in your assessments and promote proper use. This training should help educators understand what the results can and can’t tell them, and how best to use it to make a difference for students.
Assessment results are aligned to the needs of their audiences.
ALIGN RESULTS
Assessment audiences
• School Board• Students• Teachers• Parents• Principals, school
administrators, and teacher leaders
• District Administrators• Community members• State and federal officials
Percentage of parents who say teachers rarely or never discuss their child’s
assessment results with them
61%Gallup (2016, May). Make Assessment Work for All Students: Multiple Assessments Matter.
https://www.nwea.org/content/uploads/2016/05/Make_Assessment_Work_for_All_Students_2016.pdf
Questions parents want answered from assessment• Core question - Do you know
and care for my child?
• What are my child’s strengths and weaknesses?
• Is my child on track for the next grade level?
• Is my child on track for college?
• Is my child showing improvement?
• Should I be concerned?
What kind of data do parents want?
Monitoring general progress
Knowing when to be concerned
Determining readiness for next grade
Knowing if I need to seek extra help
Monitoring standards
Communicating with teacher/admin
Measuring high quality teaching
Helping my child with homework
Adjust content to student needs
Providing activities for home
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
95%
95%
93%
92%
91%
90%
88%
84%
79%
77%
Source – Northwest Evaluation Association (2012) – NWEA Assessment Perceptions Study. Survey conducted by Grunwald Associates LLC
Questions teachers want answered from assessment• What does each student
know and not know?
• What does this student need to learn next?
• What resources will help this student?
• How can I group these students for instruction?
One action to take
Train teachers in the skills needed to have a positive and productive parent conversation. Many teachers find these conversations intimidating. When educators are well prepared they gain confidence and develop higher levels of trust and support from most parents.
Redundant, mis-aligned, or unused assessments are eliminated.
STREAMLINE MEASURES
Your current assessment tools
• What purposes do each assessment meet well?• Comes from assessment design and capabilities –
not your current use• Is there alignment between the assessment and the
information needed?
• How are you using the assessments?• Reduce use to within the design and capability• Use assessments more fully to meet other purposes
One action to take
Identify the places where you have multiple assessments in-use validly performing the same purpose. Determine whether you can leverage other assessments to meet these needs.
Assessment results are delivered in a timely and useful manner.
DELIVER RESULTS
Percent of students who say they do not receive their state accountability test results.
37%
For parents, assessment results begin losing their relevance within…
1month
after the assessments are administered
Source – Northwest Evaluation Association (2012) – NWEA Assessment Perceptions Study. Survey conducted by Grunwald Associates LLC https://www.nwea.org/content/uploads/2014/07/NWEAGRUNWALD_Assessment_Perceptions_b.pdf
Timely results are related to use
If needed for instruction, results need to be faster than school improvement conversations
Reports designed to fulfill purpose
• School administrators want reports that quickly identify students who need help.
• Teachers want reports that help them make instructional decisions.
• Parents want to know whether there student is on track and to be warned when trouble is ahead.
One action to take
Review your assessment calendar, with a focus on when data is being delivered to your audiences. Adjust your testing calendar or practices to ensure that data gets to educators when they most need it. Make sure assessment results are timed to fit your schedule for professional days to encourage dialogue.
The metrics and incentives used encourage a focus on all learners.
SUPPORT ALL LEARNERS
Grade Students Athletes % Fine Arts %
9 3276 1487 50% 2256 76%
10 3190 1215 41% 1656 56%
11 2967 932 31% 1233 42%
12 2795 703 24% 1028 35%
Percent of students participating in athletics and fine arts programs
What gets measured and attended to really does matter
165169
176183
187190
194197
200203
206209
212215
218221
224227
230233
236239
242245
248251
254258
0
5
10
15
20
25
30
35
40Mathematics
No ChangeDownUp
Fall RIT
Num
ber o
f Stu
dent
s
Proficiency College Readiness
One district’s change in 5th grade mathematics performance relative to the KY proficiency cut scores
Changing from Proficiency to Growth means all kids matter
166173
178185
188191
195198
201204
207210
213216
219222
225228
231234
237240
243246
249252
255262
0
5
10
15
20
25
30
Mathematics
Below projected growthMet or above projected growth
Student’s score in fall
Num
ber o
f Stu
dent
s
Number of 5th grade students meeting projected mathematics growth in the same district
One action to take
Ask the question – “What 3 to 5 metrics drive educational decision making in our school or district? Now ask – “What behavior do those metrics incentivize and are all students encompassed in those metrics?
Make changes as needed to be sure your metrics encourage focus on all students.
The assessment program contributes to a climate of transparency and objectivity with a long-term focus
INVITE COLLABORATION
“Everyone is entitled to their opinion, but not to their own facts.” Daniel Patrick Moynihan
What is accountability?Accountability is a dialogue between the stakeholders and the leaders of their schools. Part of that dialogue is understanding and aligning the goals and objectives of the parents, the schools, and the larger community. Another part of that dialogue is discussing how your schools are doing in reaching those goals.
The most important part of the dialogue is what you’re doing to improve performance based on this information, which is leading.
The four principles that guide your communications to your audiences.
• Clearly identify who your audience is and address them as your partner.
• Understand what that audience wants to know about your schools and their performance.
• Be transparent about how you are performing on these objectives.
• Communicate your strategy to improve performance, and corrective action when strategy fails.
Florida District
Highly Effective
Effective Needs Improvement
Developing Unsatisfactory VA Score Florida Ranking
1 44.4% 55.6% 0.0% 0.0% 0.0% 0.39 1092 25.0% 75.0% 0.0% 0.0% 0.0% 0.37 1213 90.9% 9.1% 0.0% 0.0% 0.0% -0.14 28024 60.7% 39.3% 0.0% 0.0% 0.0% -0.14 27975 81.2% 18.8% 0.0% 0.0% 0.0% -0.16 28316 37.3% 54.2% 1.7% 0.0% 6.8% 0.12 8807 81.3% 18.8% 0.0% 0.0% 0.0% 0.22 4028 41.7% 55.6% 1.4% 1.4% 0.0% -0.34 32749 52.2% 47.8% 0.0% 0.0% 0.0% 0.16 66410 27.0% 66.2% 1.4% 0.0% 5.4% 0 176411 7.1% 72.6% 9.5% 10.7% 0.0% -0.08 2445
Teacher Evaluation Ratings in Eleven Florida Schools 2013
Source: Aviv, R (2014, July 21). Wrong Answer. The New Yorker. Retrieved on June 16, 2016 from http://www.newyorker.com/magazine/2014/07/21/wrong-answer
“After more than two thousand interviews, the investigators concluded that forty-four schools had cheated and that a “culture of fear, intimidation and retaliation has infested the district, allowing cheating—at all levels—to go unchecked for years.” They wrote that data had been “used as an abusive and cruel weapon to embarrass and punish.” Several teachers had been told that they had a choice: either make targets or be placed on a Performance Development Plan, which was often a precursor to termination. At one elementary school, during a faculty meeting, a principal forced a teacher whose students had tested poorly to crawl under the table.”
Differences in fall-spring test durations
An illustration of gaming
15%
25%
61%
Mathematics
Spring < Fall Spring = Fall Spring > FallSpring < Fall Spring = Fall Spring > Fall
0.0
1.0
2.0
3.0
4.0
5.0
6.0
Mathematics
Grow
th In
dex
Differences in growth index score based on fall-spring test
durations
One action to take
Review the tone of a few of your recent communications to stakeholders about student learning. As you read these does it feel like your touting, or are you truly educating your partners? Try writing an op-ed about your educational program that would surprise readers because of its honesty and frankness.
A purposeful assessment program assessment
system ensures you have direction, that your
direction stays true, and that you and your students
eventually reach their intended destination
Thank you for joining us!
http://info.nwea.org/MMDR-Guide-LP.html?utm_medium=press-release&utm_source=rss&utm_campaign=multiple-measures&utm_content=text-link
Multiple Measures Done Right – can be downloaded from