Based on the work of Vicki Bernhardt

ITF• January 27, 2009

Replace hunches with facts concerning changes Facilitate clear understanding of gaps between where

school is and where it needs to be, and identify root causes-not treat symptoms

Provide information to eliminate ineffective practices Ensure efficient use of $$$ School if school goals and objectives are being

accomplished Ascertain if school staffs are implementing their visions Generate answers and effectively educate the community Predict and prevent failures Predict and ensure successes Improve instruction Provide students with feedback on their performance.

Gain understanding of what quality is and how close we are to achieving it

Make sure students are not “falling through the cracks

Get the “root causes” of the problems

Guide curriculum development and revision

Meet state and federal requirements Promote accountability

In contrast to business, work culture in education usually doesn’t focus on data

Few are trained and few see it as part of their jobs Schools don’t have data warehousing solutions that

make it easier Teachers are generally trained to be subject

oriented, not data oriented There’s a perception that data is gathered for other

people not mine The legislature keeps on changing the rules! (e.g.

NHEIAP, NECAP)

Data analysis should beSystematic, Systemic

All of the the processes and procedures that contribute to learning-the whole and the interrelationships of the parts of the whole to each other

Continuous Measuring and evaluating processes on an

ongoing basis to identify and implement improvement

“Upstream process improvement not downstream damage control” Teams and Tools (1991)

One measure gives useful information, but…

Comprehensive, multiple measures give much richer info.

Demographics (enrollment, drop out rate, gender, ethnicity)

Perceptions (values/beliefs, attitudes)

Student Learning (standardized tests, authentic assessments, teacher observations)

School Processes (how we teach children to read, our Math scope and sequence through the years)

Each of the four measures, by themselves, give valuable information. But the deeper we dig and the more levels we utilize, the more effective the resulting information.

How many students are enrolled at MSS this year? (Demographics)

How satisfied are parents, students, and/or staff with the learning environment at HMS? (Perceptions)

How did students at the Hopkinton High School score on the SATs this year? (Student Learning)

What new courses are being offered by the high school this year? (School Processes)

How has enrollment changed throughout the HSD over time? (Demographics)

How have student perceptions of the A.P. program changed over time? (Perceptions)

Are there differences in boy’s 6th grade math scores in the NHEIAP over time? (Student Learning)

What programs have been consistent at MSS over the last five years? (School Processes)

Looking at more than one type of data within each of the circles..

What percentage of students in the HSD are fluent speakers of languages other than English and are there equal numbers of males and females? (Demographics)

Are staff, students, and parent perceptions of the culture within HMS in agreement? (Perceptions)

Are student’s scores on the 6th grade NHEIAP consistent with K-6 assigned grades and performance assessment rubrics? (Student Learning)

What are the major instructional strategies used by the Math and Science departments at the high school? (School Processes)

Similar to Level 3…just does the analysis over time…

For example: How has the enrollment of non-English speaking children changed in the last three years? (Demographic)

Do students who have perfect attendance perform better on the NHEIAP than students who miss more than five days per month? (Demographics by Student Learning)

Is there a gender difference in students’ perceptions of The Senior Project? (Perceptions by Demographics)

Do students engaged in classrooms utilizing differentiated instruction perform better on the NHEIAP than in those classrooms that do not employ this strategy? (Student Learning by School Processes)

Same as Level 6, just over time, for example…

How have students of different ethnicities scored on the SAT over the past three years? (Demographics by Student Learning)

Is there a difference in students’ reports of what they like most about MSS by whether or not they participate in extracurricular activities? Do these students have higher grades than those who do not participate in extracurricular activities? (Perceptions by Student Learning by School Processes)

What program at the high school is making the biggest difference with respect to student achievement for at-risk students this year and is one group of students responding better to the processes? (School Processes by Student Learning by Demographics)

Same as Level 7, but over time, for example…

What programs throughout the district do all types of students like the most every year? (Demographics by Perceptions by School Processes)

This is the ultimate analysis. It allows us to answer questions such as:

Are there differences in NWEA scores for 8th grade girls and boys who report they like school, by the type of program in which they are enrolled? (Demographics by Perceptions by School Processes by Student Learning)

Same as Level 9, but over time, for example…

Based on whom we have as students, how they prefer to learn, and what programs they are in, are all students learning at the same rate? (Student Learning by Demographics by Perceptions by School Processes)

Demographics, Perceptions, Student Learning, and School

Processes

CommunityLocation, history, population trends…

School DistrictDescription and history, number of

schools… School

Grants and awards received, class sizes, facilities

Students Over Time, and by Grade LevelAttendance, gender, race…

Staff Over TimeYears of experience, ethnicity,

certifications… Parents

Educational levels, involvement with child’s learning...

The separation of data into subgroups. Why is it important?

Disaggregation shows if there are differences in subgroups-there should be few differences.

Helps us find subgroups that are not responding to our techniques.

A generally held view. A belief stronger than impression and

less strong than positive knowledge.

Teachers Parents Students Administrators Community members

Give parents a chance to experience the approach, e.g. parent night, open house.

Cognitive dissonance; the discomfort one feels when two thoughts, opinions, or ideas are inconsistent.

Questionnaires Interviews Focus Groups

(All three were done as part of the SPEDMIP report)

They must be: Valid Reliable Understandable Quick to complete

Interview (face to face) Telephone Interview (person to person) Mailed Paper Online (recommended, e.g.

SurveyMonkey.com

Standardized testsNorm-referenced, criterion referenced

Authentic assessments Teacher-made test Teacher-assigned grades Performance assessments Standards based assessments Diagnostic testing

Compares test performance of a school, group, or individual with the performance of a norming group (e.g. CAT, Iowa)

Compares individuals performance to a set of standards and not to the performance of other test takers. (e.g. NHEIAP, NECAP)

Tests given to students to know the nature of a student’s difficulty but not necessary the cause of that difficulty.(e.g. Sped testing)

Measures skill and knowledge directly, e.g. if you want students to learn to write, assess it on a writing assignmentAdvantages: measures process, can match

state standards, is a learning tool in and of itself, provides reflection for students...

1. Identify desired results

2. Determine acceptable evidence

3. Plan learning experiences

& instruction

(from UbD PPT)

The evidence should be credible & helpful. Implications: the assessments should –

Be grounded in real-world applications, supplemented as needed by more traditional school evidence

Provide useful feedback to the learner, be transparent, and minimize secrecy

Be valid, reliable - aligned with the desired results of Stage 1 (and fair)

(from UbD PPT)

We need patterns that overcome inherent measurement errorSound assessment (particularly of State

Standards) requires multiple evidence over time - a photo album vs. a single snapshot

(from UbD PPT)

Varied types, over time:authentic tasks and projectsacademic exam questions, prompts, and

problemsquizzes and test items informal checks for understanding student self-assessments(from UbD PPT)

ProgramsReading Recovery, Write Traits, UBD

PracticesHBA, Mini-Mentors, Student Councils

Instructional strategiesDifferentiated Instruction, Cooperative

Learning

What teachers and administrators do to achieve the vision of the school What do we want students to be able to know and do? How are we enabling students to learn in terms of

Instructional strategies, learning strategies, instructional time, student-teacher ratio, philosophies of classroom management

How will we know if any given approach works? What will we do with students who don’t learn this way? How will the parts of the curriculum relate? What learning strategies do successful learners use?

Solid training in UbD Conducting a School Portfolio Setting up a systematic system for data

collection and analysis Selecting a data warehouse Training the staff in general