Quality data, sound judgements,powerful learning.

Sara Glover – Executive Director, Portfolio CoordinationStephen Fraser – Data, Outcomes and EvaluationOffice for Children and Portfolio Coordination

2010 NMR ConferenceMantra Erskine Beach Resort, Lorne,May 31st – June 1st 2010

The essence of accountability is looking forward, using data to inform judgments about current performance and formulate plans for reasonable actions.

(Earl and Katz, 2006)

Outline of the institute

• Working with data• Building a data story

• Break

• Building a data story contd.

• Leading change – a process to construct change theories• Engaging stakeholders• Monitoring impact

A framework for thinking and action

Engagement

AnalysisPlanning

and action

Monitoring progress

Knowledge sharing and

capacity building

School Level Report

Attitudes to SchoolStaff Opinion Parent Opinion

VCE Data Service (online by VCAA)

NAPLAN Data Service (online by VCAA)

School Performance SummaryPercentile/SFO Report

• Build data literacy across your school – data managers, data teams

• You are not your data.

• Data is half the story; context is the other half.

• Look for the main stories. Don’t get bogged down in the detail and miss the main stories.

• Don’t react to any one particular data set, for any one particular year. Always look at a range of data sets over time. Look for a consistent story.

• Beware of small numbers (e.g. in small schools). Use the raw data (number of students) or aggregate statistics (e.g. over several years).

Working with data

• Absolute scores (usually shown as vertical blue columns, e.g., mean, % “C”) are useful for;– knowing how you compare to some standard (e.g. the VELS),– monitoring over time (i.e. are the scores increasing, declining or showing

no clear pattern) and– specifying targets.

• Percentiles (usually shown as horizontal orange bars) are useful for;– comparing your own performance to other schools, and – identifying strengths and weaknesses.

Interpretation Advice (… continued)

Always remember how the data are constructed

1

1.5

2

2.5

3

3.5

4

4.5

52

00

7

20

08

20

09

20

07

20

08

20

09

20

07

20

08

20

09

20

07

20

08

20

09

20

07

20

08

20

09

20

07

20

08

20

09

20

07

20

08

20

09

20

07

20

08

20

09

Yr 5 Yr 6 Yr 7 Yr 8 Yr 9 Yr 10 Yr 11 Yr 12

Teacher Effectiveness

1=

stro

nlg

y d

isa

gre

e &

5=

stro

ng

ly a

gre

e

0th percentile

100th percentile

10th percentile

90th percentile

50th percentile

Targeting low-SES students vs Targeting low performing students Source: Masters (2009) using PISA data

A comparison of NAPLAN scores for low SES students and all students

0%

2%

4%

6%

3

127

181

219

250

277

302

324

346

367

387

408

429

452

477

506

544

608

NAPLAN scaled score

pe

r ce

nt

All

Low SES*

* "low SES" defined as occupation of parent is … machine operator, hospitality staff, assistant, labourer or related worker, or not in paid work in last 12 months

Leading change: using data to engage people in the change process

• Build a data story• Develop a focus• Dig into the detail• Examine practice – what are we doing now – how do we know if it’s

working?• Develop a change theory – what would happen if we did things differently?• Engage stakeholders• Do the work• Monitor impact

Building a data story along a developmental continuum

Engagement

AnalysisPlanning

and action

Monitoring progress

Kindergarten Participation in our community

Early Childhood Community Profile

Kindergarten Participation rate for 4 year olds across the LGA: 91%

• Less than the metropolitan average• Less than the rural average

Children’s development across our community (AEDI)

Developmental Vulnerability• Physical Health and

Wellbeing: 9%• Social Competence: 10%• Emotional Maturity: 8%• Language and Cognitive:

13%• Communication and General

Knowledge: 24%

Early Years Reading

English Online Interview• High proportions of

students in the lower bands

Assessment of Reading Prep – Year 2

• - declining Prep outcomes (improving towards Year 2)

Reading Prep to Year 6 - VELS

• Mismatch with EOI and with AoR• Greater distribution of outcomes towards Year 6 – more at the top,

more at the tail

Reading Prep to Year 6 - NAPLAN

• Improvement over the last two years• Greater distribution of outcomes towards Year 6 – more at the top, more

at the tail (consistent with VELS)

Reading Year 7 to 10 - VELS

• Decline over three year period

• Greater distribution of outcomes towards Year 10 – more at the top, more at the tail (consistent with Prep to Year 6)

Numeracy Year 7 to 10 - VELS

• Mixed over three year period

• Improvement at Year 10

Reading Year 7 to 10 - NAPLAN

• Improvement over last two years• Greater proportions of students in the lowest two bands by Year 10

(somewhat consistent with VELS)

Year 7 Reading Year 9 Reading

Numeracy Year 7 to 10 - NAPLAN

• Improvement over last two years• Stronger than Literacy• Lower proportions of students in the lowest two bands by Year 10

(somewhat consistent with VELS)

Year 7 Numeracy Year 9 Numeracy

Post-Compulsory Outcomes - VCE

• Decline in English• Increase in Maths (very strong)

• Overall (allstudy) steady improvement

Post-Compulsory Outcomes - VET

• High participation• High completion

Post-Compulsory Outcomes - VCAL

• Low participation• High completion

Transitions and Pathways

• Excellent and improving retention across the board

7-10

10-11

11-12

7-12

Transitions and Pathways

• Overall – excellent proportions of students continuing with their education or training

• Declining University enrolments.

Engagement and Wellbeing

Strengths• Motivated students• Positive teacher-learner

relationships

Opportunities• Stimulating learning – high

demand content – lacking• Safety an issue link with

distress

• Attendance excellent.

Climate

Strengths• High Expectations• High motivation and

goal alignment

Opportunities• Collegiality and decision

making

Building a data story

• We have lower kinder participation rates• We have vulnerability in our community – language/general

knowledge/communications• Literacy a challenge from the outset• We are seeing a decline in literacy outcomes at most transition points• Strong mathematics performance• We have very strong retention and pathways through to further education

and training, with declining University enrolment.

• We have engaged and motivated students – is the level of task challenge lacking?

• We have motivated teachers with high expectations• Student and staff alike have a lack of say

Out

com

esE

xper

ienc

es

Dig into the detail

• What can we learn about mathematics instruction in the Secondary years – about the students undertaking further maths in VCE

• NAPLAN Data Service – Item Analysis Report (SREAMS SPA).

Developing the plan…examining practice

Engagement

AnalysisPlanning

and action

Monitoring progress

Examine practice – what are we doing now

• What are our relationships with kindergartens like?• What are our transition arrangements/plans like? Are teachers getting

the right information about literacy at the start of each year?• What level of involvement do teachers and students have in decisions

that affect them?

Developing a change theory

• Deciding what to do.

• The most important, but often overlooked aspect of these theories is that they should be falsifiable – that is, they can be proven false.

If we do …x , then we expect to see …y.

Monitoring progress…communicating successes and challenges against what we set out to do

Engagement

AnalysisPlanning

and action

Monitoring progress

Monitoring Impact

• If the data doesn’t really capture what you’re trying to achieve in your school, target on the behaviours that will.

Behaviour Impact Data (if any)

If …

then …

If …

then …

If …

then …

If … x

then … y

Engaging stakeholders – building ownership

• Thinking about the traction achieved by engaging stakeholders vs going it alone

• A question of how to harness the enthusiasm/effort/ownership of stakeholders at all levels

• And what information/communication tools can be used to do so

Engaging stakeholders

Key Messages Communication Tool

School Leaders

Classroom teachers

Parents

Students