ESMSJ ISSN: 2247 2479 ISSSN L: 2247 2479 Vol IV, Issue 2 2014
The DATA WISE Process and Data- Driven Dialogue Presented by: Lori DeForest [email protected]...
-
Upload
isabel-randall -
Category
Documents
-
view
220 -
download
0
Transcript of The DATA WISE Process and Data- Driven Dialogue Presented by: Lori DeForest [email protected]...
“Using data effectively does not mean getting good at crunching numbers. It means getting good at working together to gain insights from student-assessment results and to use the insights to improve instruction.”
- Kathryn Boudett, Elizabeth City, & Richard Murnane, “When 19 Heads Are Better Than One,” Education Week, December 7, 2005.
The DATA WISE Process
and Data-Driven Dialogue
Organize for Collaborative Work
Build Data Teams Establish team structure to allow
for data discussions Establish norms Utilize protocols Complete a Data Inventory
Data Teams Raise important questions about
student learning and achievement Assist in organizational aspects of
data Dialogue about multiple data
sources Examine and interpret data Investigate ways to improve
teaching and learning
Curriculum CouncilsInitial Data Teams
October and November
- Setting Norms Protocol
- Compass Points Protocol
Setting Norms ProtocolWhat norms do we need? Brainstorm Discuss Synthesize Build consensus
We need to build emotional safety to reach cognitive complexity.
B. Wellman and L. Lipton
Curriculum CouncilsInitial Data Teams
October and November - Setting Norms Protocol - Compass Points ProtocolDecember and January
- Data Inventory- Data Analysis Tools
Purpose of Data Inventory
Summarizes all the types of data that are available and helps to determine what other data is needed
Builds assessment literacy Assists in the planning of using data
effectively Begins conversation about educational
questions
Data Inventory
Data Source
Dates of
Collection
Students
Assessed
PurposeCurrent
Data Use
More Effective Use
ELA State Asmt
January (results in summer)
Grades 3, 4, 5, 6, 7, 8
State Accountability purposes and to evaluate program and students’ …
Program evaluation and intervention placement
Data Team analyzes data to inform instruction
Other Student Level-Information Race/Ethnicity
Data Wish List
English Proficiency Attendance Race/Ethnicity
Data Inventory
Data Source
Dates of
Collection
Students
Assessed
PurposeCurrent
Data Use
More Effective Use
Running Record
Ongoing
Grades K - 2
Evaluate a students’ reading skill and level
To inform planning of guided rdg. lessons and inform new teacher of rdg level
District Writing Folder
3 times per year per grade
Grades K-12
Document a students’ writing progress over time
To archive student work
Teacher review regularly to inform instruction
The DATA WISE Process
and Data-Driven Dialogue
Assessment Literacy
Data Analysis Tools COGNOS Report Net COGNOS Power Play Cubes Data Mentor nySTART NYS State Report Card Databases and ELA and Math Media Databases Student Management System
Demonstrate tools at Curriculum Councils, Grade Level and Department Meetings
Offer training opportunities
The DATA WISE Process
and Data-Driven Dialogue
Data Overview
Determine audience
Decide on educational questions
Create graphic displays of standardized test results
Engage in conversations around initial data set
2006 English Language Arts Performance
Data Sources: CNYRIC COGNOS PowerPlay Cubes, NYSED School Report Card and ELA Assessment Databases
2006 English/Language Arts Assessment Performance
0%
10%
20%
30%
40%
50%
60%
70%
80%
Grade Levels
% o
f st
ud
ents
at
or
abo
ve l
evel
3
Tully ES or JSHS 74.68% 70.73% 70.87% 69.39% 70.65% 56.60%
Sim.Schools 74.82% 74.58% 74.33% 67.87% 65.13% 57.49%
BOCES 73.14% 75.62% 74.37% 69.29% 67.52% 58.02%
State 69.00% 68.60% 67.10% 60.40% 56.40% 49.30%
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
2006 Mathematics Performance
Data Sources: CNYRIC COGNOS PowerPlay Cubes, NYSED School Report Card and Math Assessment Databases
2006 NYS Mathematics Assessment Performance
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Grade Levels
% o
f st
ud
ents
at
or
abo
ve l
evel
3
Tully ES or JSHS 77.50% 82.72% 67.96% 57.00% 52.69% 41.51%
Sim.Schools 86.59% 84.30% 74.81% 67.30% 66.64% 66.06%
BOCES 84.80% 84.65% 75.60% 69.81% 70.54% 67.21%
State 80.50% 77.90% 68.40% 60.40% 55.60% 53.90%
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
The DATA WISE Process
and Data-Driven Dialogue
“Without an investigation of the data, schools risk misdiagnosing the problem.”
Data Wise, 2005
Data Analysis Protocol
Activate and Engage Set norms Articulate predictions and assumptions
Explore and Discover Begin with a single data source First describe what you see Ask questions Identify additional data needs
The DATA WISE Process
and Data-Driven Dialogue
Data Analysis Writers Club
Informing Planning for Writers Club Use COGNOS PowerPlay to identify needs of struggling students.
Item Difficulty(% of points earned out of total possible points)
2006 Grade 3 English Language Arts Assessment
SCHOOL DISTRICT BOCES Region
MC 01 Level_3 L3_Low 89.7% 90.0% 92.4% 92.6%
Level_3 93.8% 95.5% 95.2% 95.1%
Level_2 L2_High 85.7% 74.1% 86.4% 86.7%
L2_Med 100.0% 85.7% 81.9% 80.9%
L2_Low 100.0% 100.0% 72.5% 72.5%
Level_2 91.7% 79.5% 82.5% 82.2%
Level_1 L1_High 80.0% 55.6% 54.0% 56.6%
L1_Med /0 /0 23.5% 20.5%
L1_Low /0 /0 20.0% 14.3%
Level_1 80.0% 55.6% 51.6% 53.6%
All Performance Levels 94.1% 92.7% 90.1% 89.6%
MC 02 Level_3 L3_Low 100.0% 100.0% 97.6% 97.5%
Level_3 100.0% 99.5% 98.3% 98.4%
Level_2 L2_High 100.0% 85.2% 95.3% 95.6%
L2_Med 100.0% 71.4% 93.2% 93.9%
L2_Low 100.0% 100.0% 91.0% 91.7%
Level_2 100.0% 84.6% 93.9% 94.4%
Level_1 L1_High 80.0% 88.9% 72.1% 74.6%
L1_Med /0 /0 23.5% 17.9%
L1_Low /0 /0 20.0% 14.3%
Level_1 80.0% 88.9% 68.3% 70.1%
All Performance Levels 99.2% 97.4% 95.6% 95.6%
A distracter analysis may help you understand children’s incorrect thought processes.
Distracter Analysis Item Count
as values
2006 Grade 3 English Language Arts Assessment
ELEMENTARY SCHOOL
Choice 1
Choice 2
Choice 3
Choice 4
Blank
MC 01 Level_3 L3_Low 1 26 2 0 0
Level_3 2 76 3 0 0
Level_2 L2_High 0 6 1 0 0
L2_Med 0 4 0 0 0
L2_Low 0 1 0 0 0
Level_2 0 11 1 0 0
Level_1 L1_High 1 4 0 0 0
L1_Med 0 0 0 0 0
L1_Low 0 0 0 0 0
Level_1 1 4 0 0 0
All Performance Levels 3 112 4 0 0
MC 02 Level_3 L3_Low 0 0 29 0 0
Level_3 0 0 81 0 0
Level_2 L2_High 0 0 7 0 0
L2_Med 0 0 4 0 0
L2_Low 0 0 1 0 0
Level_2 0 0 12 0 0
Level_1 L1_High 1 0 4 0 0
L1_Med 0 0 0 0 0
L1_Low 0 0 0 0 0
Level_1 1 0 4 0 0
All Performance Levels 1 0 118 0 0
Data Analysis Writers Club Physical Education program Intervention Analysis
Intervention Analysis2006 Cohort ELA 8 Performance of students
who performed at Level 1 or 2 on ELA 4 and remained in district (n=43)
Tracking cohort performance may give you some information about program or student growth.
Data Analysis Writers Club Physical Education program Intervention Analysis English Language Arts Analyses
English Language Arts
Educational question: Is student performance declining in reading comprehension but increasing in listening comprehension?
Utilize COGNOS PowerPlay item analysis and Scoring Key to classify questions by subtest
Compile data using formulas and functions Study trends over time Use comparative data to inform analysis
2006 ELA Grade 5Multiple Choice: Reading and Listening Comprehension
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
1 2 3 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25
Reading Comprehension ListeningComprehension
2006 ELA Grade 5 Multiple Choice: Reading and Listening Comprehension
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
55.0%
60.0%
65.0%
70.0%
75.0%
80.0%
85.0%
90.0%
95.0%
100.0%
1 2 3 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25
Reading Comprehension ListeningComprehension
The graph above illustrates the importance of using comparative data to inform analysis.
English Language Arts Analyses
COGNOS Report Net now houses analysis reports which can provide you information about student performance.
English Language Arts Analyses
LastName
FirstName
School
Teacher
Reading Comp
(MC-28)
Listening
Writing
(5)
Reading
Writing (5)
Writing Mechani
cs (3) LevelTotal Score
Sample Abby EH Eladata 18 3 3 1 Level 3 652
Sample Bob EH Eladata 26 3 3 2 Level 3 695
SampleCatherin
e EH Eladata 28 3 3 2 Level 3 711
Sample David EH Eladata 27 3 4 2 Level 3 711
Sample Erin EH Eladata 28 3 4 2 Level 4 721
Sample Frank EH Eladata 11 2 1 1 Level 1 606
Sample Gabby EH Eladata 22 3 3 2 Level 3 673
Sample Harold EH Eladata 27 3 3 2 Level 3 703
Sample Iris EH Eladata 15 3 2 1 Level 2 632
Sample Jacob EH Eladata 24 4 4 3 Level 3 711
Sample Karen EH Eladata 20 2 2 2 Level 3 652
Sample Luke EH Eladata 28 2 3 2 Level 3 703
Sampling Principle“Because a test is not a direct
measure of a student’s degree of mastery of an entire domain, any conclusion you reach about proficiency in that domain is based on an inference from proficiency on the smaller sample. …Even a test that provides good support for one inference may provide weak support for another.”
Data Wise, 2005
Data Analysis Writers Club Physical Education program Intervention Analysis English Language Arts Analyses Math Analyses
Data Overview of NYS Assessment Performance
Considerations when reviewing summary data Different cohort groups Different samples of items each year Different test blueprints
Importance of comparison data sets
Grade 4 NYS Mathematics Performance % of students scoring at or above Level 3
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
Year
Fayetteville-Manlius 90.88% 92.82% 87.50% 86.21% 91.21% 95.04% 95.04% 93.03%
OCM BOCES 81.65% 81.17% 85.65% 79.50% 87.23% 87.36% 89.64% 84.65%
Similar Schools 91.50% 90.47% 92.04% 90.33% 94.42% 95.03% 96.12% 92.85%
1999 2000 2001 2002 2003 2004 2005* 2006*
Go to www.emsc.nysed.gov/osa and the Report Card link to access data for similar schools.
Note: Dual enrollments taken into account for total annual percentage
Enrollment in College Level Math Courses
Utilize your own student management system to analyze additional data as well.