The DATA WISE Process

and Data-Driven Dialogue

Presented by: Lori DeForestldeforest@cnyric.org (315)433-2247

“Using data effectively does not mean getting good at crunching numbers. It means getting good at working together to gain insights from student-assessment results and to use the insights to improve instruction.”

- Kathryn Boudett, Elizabeth City, & Richard Murnane, “When 19 Heads Are Better Than One,” Education Week, December 7, 2005.

The DATA WISE Process

and Data-Driven Dialogue

Organize for Collaborative Work

Build Data Teams Establish team structure to allow

for data discussions Establish norms Utilize protocols Complete a Data Inventory

Data Teams Raise important questions about

student learning and achievement Assist in organizational aspects of

data Dialogue about multiple data

sources Examine and interpret data Investigate ways to improve

teaching and learning

Curriculum CouncilsInitial Data Teams

October and November

- Setting Norms Protocol

- Compass Points Protocol

Setting Norms ProtocolWhat norms do we need? Brainstorm Discuss Synthesize Build consensus

We need to build emotional safety to reach cognitive complexity.

B. Wellman and L. Lipton

Curriculum CouncilsInitial Data Teams

October and November - Setting Norms Protocol - Compass Points ProtocolDecember and January

- Data Inventory- Data Analysis Tools

Purpose of Data Inventory

Summarizes all the types of data that are available and helps to determine what other data is needed

Builds assessment literacy Assists in the planning of using data

effectively Begins conversation about educational

questions

Data Inventory

Data Source

Dates of

Collection

Students

Assessed

PurposeCurrent

Data Use

More Effective Use

ELA State Asmt

January (results in summer)

Grades 3, 4, 5, 6, 7, 8

State Accountability purposes and to evaluate program and students’ …

Program evaluation and intervention placement

Data Team analyzes data to inform instruction

Other Student Level-Information Race/Ethnicity

Data Wish List

English Proficiency Attendance Race/Ethnicity

Data Inventory

Data Source

Dates of

Collection

Students

Assessed

PurposeCurrent

Data Use

More Effective Use

Running Record

Ongoing

Grades K - 2

Evaluate a students’ reading skill and level

To inform planning of guided rdg. lessons and inform new teacher of rdg level

District Writing Folder

3 times per year per grade

Grades K-12

Document a students’ writing progress over time

To archive student work

Teacher review regularly to inform instruction

The DATA WISE Process

and Data-Driven Dialogue

Assessment Literacy

Data Analysis Tools COGNOS Report Net COGNOS Power Play Cubes Data Mentor nySTART NYS State Report Card Databases and ELA and Math Media Databases Student Management System

Demonstrate tools at Curriculum Councils, Grade Level and Department Meetings

Offer training opportunities

The DATA WISE Process

and Data-Driven Dialogue

Data Overview

Determine audience

Decide on educational questions

Create graphic displays of standardized test results

Engage in conversations around initial data set

2006 English Language Arts Performance

Data Sources: CNYRIC COGNOS PowerPlay Cubes, NYSED School Report Card and ELA Assessment Databases

2006 English/Language Arts Assessment Performance

0%

10%

20%

30%

40%

50%

60%

70%

80%

Grade Levels

% o

f st

ud

ents

at

or

abo

ve l

evel

3

Tully ES or JSHS 74.68% 70.73% 70.87% 69.39% 70.65% 56.60%

Sim.Schools 74.82% 74.58% 74.33% 67.87% 65.13% 57.49%

BOCES 73.14% 75.62% 74.37% 69.29% 67.52% 58.02%

State 69.00% 68.60% 67.10% 60.40% 56.40% 49.30%

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

2006 Mathematics Performance

Data Sources: CNYRIC COGNOS PowerPlay Cubes, NYSED School Report Card and Math Assessment Databases

2006 NYS Mathematics Assessment Performance

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Grade Levels

% o

f st

ud

ents

at

or

abo

ve l

evel

3

Tully ES or JSHS 77.50% 82.72% 67.96% 57.00% 52.69% 41.51%

Sim.Schools 86.59% 84.30% 74.81% 67.30% 66.64% 66.06%

BOCES 84.80% 84.65% 75.60% 69.81% 70.54% 67.21%

State 80.50% 77.90% 68.40% 60.40% 55.60% 53.90%

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

The DATA WISE Process

and Data-Driven Dialogue

“Without an investigation of the data, schools risk misdiagnosing the problem.”

Data Wise, 2005

Data Analysis Protocol

Activate and Engage Set norms Articulate predictions and assumptions

Explore and Discover Begin with a single data source First describe what you see Ask questions Identify additional data needs

The DATA WISE Process

and Data-Driven Dialogue

Data Analysis Writers Club

Informing Planning for Writers Club Use COGNOS PowerPlay to identify needs of struggling students.

Item Difficulty(% of points earned out of total possible points)

2006 Grade 3 English Language Arts Assessment 

SCHOOL DISTRICT  BOCES  Region 

MC 01 Level_3 L3_Low 89.7% 90.0% 92.4% 92.6%

Level_3 93.8% 95.5% 95.2% 95.1%

Level_2 L2_High 85.7% 74.1% 86.4% 86.7%

L2_Med 100.0% 85.7% 81.9% 80.9%

L2_Low 100.0% 100.0% 72.5% 72.5%

Level_2 91.7% 79.5% 82.5% 82.2%

Level_1 L1_High 80.0% 55.6% 54.0% 56.6%

L1_Med /0 /0 23.5% 20.5%

L1_Low /0 /0 20.0% 14.3%

Level_1 80.0% 55.6% 51.6% 53.6%

All Performance Levels 94.1% 92.7% 90.1% 89.6%

MC 02 Level_3 L3_Low 100.0% 100.0% 97.6% 97.5%

Level_3 100.0% 99.5% 98.3% 98.4%

Level_2 L2_High 100.0% 85.2% 95.3% 95.6%

L2_Med 100.0% 71.4% 93.2% 93.9%

L2_Low 100.0% 100.0% 91.0% 91.7%

Level_2 100.0% 84.6% 93.9% 94.4%

Level_1 L1_High 80.0% 88.9% 72.1% 74.6%

L1_Med /0 /0 23.5% 17.9%

L1_Low /0 /0 20.0% 14.3%

Level_1 80.0% 88.9% 68.3% 70.1%

All Performance Levels 99.2% 97.4% 95.6% 95.6%

A distracter analysis may help you understand children’s incorrect thought processes.

Distracter Analysis Item Count

as values

2006 Grade 3 English Language Arts Assessment 

ELEMENTARY SCHOOL 

Choice 1 

Choice 2 

Choice 3 

Choice 4 

Blank 

MC 01 Level_3 L3_Low 1 26 2 0 0

Level_3 2 76 3 0 0

Level_2 L2_High 0 6 1 0 0

L2_Med 0 4 0 0 0

L2_Low 0 1 0 0 0

Level_2 0 11 1 0 0

Level_1 L1_High 1 4 0 0 0

L1_Med 0 0 0 0 0

L1_Low 0 0 0 0 0

Level_1 1 4 0 0 0

All Performance Levels 3 112 4 0 0

MC 02 Level_3 L3_Low 0 0 29 0 0

Level_3 0 0 81 0 0

Level_2 L2_High 0 0 7 0 0

L2_Med 0 0 4 0 0

L2_Low 0 0 1 0 0

Level_2 0 0 12 0 0

Level_1 L1_High 1 0 4 0 0

L1_Med 0 0 0 0 0

L1_Low 0 0 0 0 0

Level_1 1 0 4 0 0

All Performance Levels 1 0 118 0 0

Data Analysis Writers Club Physical Education program Intervention Analysis

Intervention Analysis2006 Cohort ELA 8 Performance of students

who performed at Level 1 or 2 on ELA 4 and remained in district (n=43)

Tracking cohort performance may give you some information about program or student growth.

Data Analysis Writers Club Physical Education program Intervention Analysis English Language Arts Analyses

English Language Arts

Educational question: Is student performance declining in reading comprehension but increasing in listening comprehension?

Utilize COGNOS PowerPlay item analysis and Scoring Key to classify questions by subtest

Compile data using formulas and functions Study trends over time Use comparative data to inform analysis

2006 ELA Grade 5Multiple Choice: Reading and Listening Comprehension

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

1 2 3 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25

Reading Comprehension ListeningComprehension

2006 ELA Grade 5 Multiple Choice: Reading and Listening Comprehension

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

50.0%

55.0%

60.0%

65.0%

70.0%

75.0%

80.0%

85.0%

90.0%

95.0%

100.0%

1 2 3 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25

Reading Comprehension ListeningComprehension

The graph above illustrates the importance of using comparative data to inform analysis.

English Language Arts Analyses

COGNOS Report Net now houses analysis reports which can provide you information about student performance.

English Language Arts Analyses

LastName

FirstName

School

Teacher

Reading Comp

(MC-28)

Listening

Writing

(5)

Reading

Writing (5)

Writing Mechani

cs (3) LevelTotal Score

Sample Abby EH Eladata 18 3 3 1 Level 3 652

Sample Bob EH Eladata 26 3 3 2 Level 3 695

SampleCatherin

e EH Eladata 28 3 3 2 Level 3 711

Sample David EH Eladata 27 3 4 2 Level 3 711

Sample Erin EH Eladata 28 3 4 2 Level 4 721

Sample Frank EH Eladata 11 2 1 1 Level 1 606

Sample Gabby EH Eladata 22 3 3 2 Level 3 673

Sample Harold EH Eladata 27 3 3 2 Level 3 703

Sample Iris EH Eladata 15 3 2 1 Level 2 632

Sample Jacob EH Eladata 24 4 4 3 Level 3 711

Sample Karen EH Eladata 20 2 2 2 Level 3 652

Sample Luke EH Eladata 28 2 3 2 Level 3 703

Sampling Principle“Because a test is not a direct

measure of a student’s degree of mastery of an entire domain, any conclusion you reach about proficiency in that domain is based on an inference from proficiency on the smaller sample. …Even a test that provides good support for one inference may provide weak support for another.”

Data Wise, 2005

Data Analysis Writers Club Physical Education program Intervention Analysis English Language Arts Analyses Math Analyses

Data Overview of NYS Assessment Performance

Considerations when reviewing summary data Different cohort groups Different samples of items each year Different test blueprints

Importance of comparison data sets

Grade 4 NYS Mathematics Performance % of students scoring at or above Level 3

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

Year

Fayetteville-Manlius 90.88% 92.82% 87.50% 86.21% 91.21% 95.04% 95.04% 93.03%

OCM BOCES 81.65% 81.17% 85.65% 79.50% 87.23% 87.36% 89.64% 84.65%

Similar Schools 91.50% 90.47% 92.04% 90.33% 94.42% 95.03% 96.12% 92.85%

1999 2000 2001 2002 2003 2004 2005* 2006*

Go to www.emsc.nysed.gov/osa and the Report Card link to access data for similar schools.

Note: Dual enrollments taken into account for total annual percentage

Enrollment in College Level Math Courses

Utilize your own student management system to analyze additional data as well.