Multiple measures done right: The purpose-driven assessment system

John Cronin, Ph.D.VP – Education ResearchNorthwest Evaluation Association

The pursuit of compliance is exhausting because it is always a moving target. Governors move on, the party in power gets replaced, a new president is elected, and all want to put their own stamp on education.

It is saner and less exhausting to define your own course and align compliance requirements to that.

The problem of compliance-based assessment systems

Seven standards that define the purpose driven assessment

system

The purposes of all assessments are defined and the assessments are valid and useful for their purposes.

DEFINE YOUR PURPOSES

Northwest Evaluation Association and Gallup - 2016

https://www.nwea.org/content/uploads/2016/05/Make_Assessment_Work_for_All_Students_2016.pdf

Make Assessment Work for All Students: Multiple Measures Matter

TEACHERS, PRINCIPALS AND SUPERINTENDENTS say students spend

too much time taking assessments.

Percent of students who say the results from state accountability tests are useful

29%Gallup (2016, May). Make Assessment Work for All Students: Multiple Assessments Matter.

https://www.nwea.org/content/uploads/2016/05/Make_Assessment_Work_for_All_Students_2016.pdf

Teacher and Administrator perceptions on the purposes of assessments

State Poli-cymakers

Parents Students Teachers Administrators0%

10%

20%

30%

40%

50%

60%

70%

80%

% State accountability assessments % Classroom tests and quizzes

Principals: Do you believe the purpose of data from (state accountability assessments, classroom assessments and quizzes) in your district is well understood by each of the following:

Students find many assessment types useful when the assessment is tied to a specific purpose

Classroom tests and quizzes developed by teachers to evaluate their students’ learning process

Interim assessments that are given to students two to three times over the year so teachers can see where students are growing and where they need to learn more

Performance tasks that require students to apply things they have learned to a real-world problem or situation

Formative assessments

78%

76%

75%74%

One action to take

First, document and prioritize your purposes for assessment then…inventory the assessments in use and evaluate their alignment with your purposes.

It’s a big job but worth the time!

Teachers are educated in the proper administration and application of the assessments used in their classrooms.

EDUCATE TEACHERS

What types of extra help, if any, does your teacher or school give you if you do not do well on a…

CL AS S ROOM T EST STATE STA N DAR D I Z E D TEST

Before and/or after school support 43% 24%

One to one support 32% 19%

Access to added resources 29% 22%

Small group support 24% 17%

No extra support provided 20% 38%

Teachers need to …

Understand the purpose

Follow the procedures

Properly interpret and act

One action to take

Develop internal structures and policies to provide training for all teachers (and administrators) in your assessments and promote proper use. This training should help educators understand what the results can and can’t tell them, and how best to use it to make a difference for students.

Assessment results are aligned to the needs of their audiences.

ALIGN RESULTS

Assessment audiences

• School Board• Students• Teachers• Parents• Principals, school

administrators, and teacher leaders

• District Administrators• Community members• State and federal officials

Percentage of parents who say teachers rarely or never discuss their child’s

assessment results with them

61%Gallup (2016, May). Make Assessment Work for All Students: Multiple Assessments Matter.

https://www.nwea.org/content/uploads/2016/05/Make_Assessment_Work_for_All_Students_2016.pdf

Questions parents want answered from assessment• Core question - Do you know

and care for my child?

• What are my child’s strengths and weaknesses?

• Is my child on track for the next grade level?

• Is my child on track for college?

• Is my child showing improvement?

• Should I be concerned?

What kind of data do parents want?

Monitoring general progress

Knowing when to be concerned

Determining readiness for next grade

Knowing if I need to seek extra help

Monitoring standards

Communicating with teacher/admin

Measuring high quality teaching

Helping my child with homework

Adjust content to student needs

Providing activities for home

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

95%

95%

93%

92%

91%

90%

88%

84%

79%

77%

Source – Northwest Evaluation Association (2012) – NWEA Assessment Perceptions Study. Survey conducted by Grunwald Associates LLC

Questions teachers want answered from assessment• What does each student

know and not know?

• What does this student need to learn next?

• What resources will help this student?

• How can I group these students for instruction?

One action to take

Train teachers in the skills needed to have a positive and productive parent conversation. Many teachers find these conversations intimidating. When educators are well prepared they gain confidence and develop higher levels of trust and support from most parents.

Redundant, mis-aligned, or unused assessments are eliminated.

STREAMLINE MEASURES

Your current assessment tools

• What purposes do each assessment meet well?• Comes from assessment design and capabilities –

not your current use• Is there alignment between the assessment and the

information needed?

• How are you using the assessments?• Reduce use to within the design and capability• Use assessments more fully to meet other purposes

One action to take

Identify the places where you have multiple assessments in-use validly performing the same purpose. Determine whether you can leverage other assessments to meet these needs.

Assessment results are delivered in a timely and useful manner.

DELIVER RESULTS

Percent of students who say they do not receive their state accountability test results.

37%

For parents, assessment results begin losing their relevance within…

1month

after the assessments are administered

Source – Northwest Evaluation Association (2012) – NWEA Assessment Perceptions Study. Survey conducted by Grunwald Associates LLC https://www.nwea.org/content/uploads/2014/07/NWEAGRUNWALD_Assessment_Perceptions_b.pdf

Timely results are related to use

If needed for instruction, results need to be faster than school improvement conversations

Reports designed to fulfill purpose

• School administrators want reports that quickly identify students who need help.

• Teachers want reports that help them make instructional decisions.

• Parents want to know whether there student is on track and to be warned when trouble is ahead.

One action to take

Review your assessment calendar, with a focus on when data is being delivered to your audiences. Adjust your testing calendar or practices to ensure that data gets to educators when they most need it. Make sure assessment results are timed to fit your schedule for professional days to encourage dialogue.

The metrics and incentives used encourage a focus on all learners.

SUPPORT ALL LEARNERS

Grade Students Athletes % Fine Arts %

9 3276 1487 50% 2256 76%

10 3190 1215 41% 1656 56%

11 2967 932 31% 1233 42%

12 2795 703 24% 1028 35%

Percent of students participating in athletics and fine arts programs

What gets measured and attended to really does matter

165169

176183

187190

194197

200203

206209

212215

218221

224227

230233

236239

242245

248251

254258

0

5

10

15

20

25

30

35

40Mathematics

No ChangeDownUp

Fall RIT

Num

ber o

f Stu

dent

s

Proficiency College Readiness

One district’s change in 5th grade mathematics performance relative to the KY proficiency cut scores

Changing from Proficiency to Growth means all kids matter

166173

178185

188191

195198

201204

207210

213216

219222

225228

231234

237240

243246

249252

255262

0

5

10

15

20

25

30

Mathematics

Below projected growthMet or above projected growth

Student’s score in fall

Num

ber o

f Stu

dent

s

Number of 5th grade students meeting projected mathematics growth in the same district

One action to take

Ask the question – “What 3 to 5 metrics drive educational decision making in our school or district? Now ask – “What behavior do those metrics incentivize and are all students encompassed in those metrics?

Make changes as needed to be sure your metrics encourage focus on all students.

The assessment program contributes to a climate of transparency and objectivity with a long-term focus

INVITE COLLABORATION

“Everyone is entitled to their opinion, but not to their own facts.” Daniel Patrick Moynihan

What is accountability?Accountability is a dialogue between the stakeholders and the leaders of their schools. Part of that dialogue is understanding and aligning the goals and objectives of the parents, the schools, and the larger community. Another part of that dialogue is discussing how your schools are doing in reaching those goals.

The most important part of the dialogue is what you’re doing to improve performance based on this information, which is leading.

The four principles that guide your communications to your audiences.

• Clearly identify who your audience is and address them as your partner.

• Understand what that audience wants to know about your schools and their performance.

• Be transparent about how you are performing on these objectives.

• Communicate your strategy to improve performance, and corrective action when strategy fails.

Florida District

Highly Effective

Effective Needs Improvement

Developing Unsatisfactory VA Score Florida Ranking

1 44.4% 55.6% 0.0% 0.0% 0.0% 0.39 1092 25.0% 75.0% 0.0% 0.0% 0.0% 0.37 1213 90.9% 9.1% 0.0% 0.0% 0.0% -0.14 28024 60.7% 39.3% 0.0% 0.0% 0.0% -0.14 27975 81.2% 18.8% 0.0% 0.0% 0.0% -0.16 28316 37.3% 54.2% 1.7% 0.0% 6.8% 0.12 8807 81.3% 18.8% 0.0% 0.0% 0.0% 0.22 4028 41.7% 55.6% 1.4% 1.4% 0.0% -0.34 32749 52.2% 47.8% 0.0% 0.0% 0.0% 0.16 66410 27.0% 66.2% 1.4% 0.0% 5.4% 0 176411 7.1% 72.6% 9.5% 10.7% 0.0% -0.08 2445

Teacher Evaluation Ratings in Eleven Florida Schools 2013

Source: Aviv, R (2014, July 21). Wrong Answer. The New Yorker. Retrieved on June 16, 2016 from http://www.newyorker.com/magazine/2014/07/21/wrong-answer

“After more than two thousand interviews, the investigators concluded that forty-four schools had cheated and that a “culture of fear, intimidation and retaliation has infested the district, allowing cheating—at all levels—to go unchecked for years.” They wrote that data had been “used as an abusive and cruel weapon to embarrass and punish.” Several teachers had been told that they had a choice: either make targets or be placed on a Performance Development Plan, which was often a precursor to termination. At one elementary school, during a faculty meeting, a principal forced a teacher whose students had tested poorly to crawl under the table.”

Differences in fall-spring test durations

An illustration of gaming

15%

25%

61%

Mathematics

Spring < Fall Spring = Fall Spring > FallSpring < Fall Spring = Fall Spring > Fall

0.0

1.0

2.0

3.0

4.0

5.0

6.0

Mathematics

Grow

th In

dex

Differences in growth index score based on fall-spring test

durations

One action to take

Review the tone of a few of your recent communications to stakeholders about student learning. As you read these does it feel like your touting, or are you truly educating your partners? Try writing an op-ed about your educational program that would surprise readers because of its honesty and frankness.

A purposeful assessment program assessment

system ensures you have direction, that your

direction stays true, and that you and your students

eventually reach their intended destination

Thank you for joining us!

http://info.nwea.org/MMDR-Guide-LP.html?utm_medium=press-release&utm_source=rss&utm_campaign=multiple-measures&utm_content=text-link

Multiple Measures Done Right – can be downloaded from