ASSESSMENT FOR BETTER LEARNING USING NAPLAN DATA Presented by Philip Holmes-Smith School Research...

ASSESSMENT FOR BETTER LEARNING ASSESSMENT FOR BETTER LEARNING USINGUSING

NAPLAN DATANAPLAN DATA

Presented byPresented by

Philip Holmes-SmithPhilip Holmes-SmithSchool Research Evaluation and Measurement ServicesSchool Research Evaluation and Measurement Services

OverviewOverview

1. The National Scale1. The National Scale

National Scale Scores National Scale Scores ororEstimated VELS Equivalent Scores?Estimated VELS Equivalent Scores?

You can choose to look You can choose to look at at National Scale ScoresNational Scale Scores or or Estimated VELS Estimated VELS Equivalent scoresEquivalent scores

What’s the Difference?What’s the Difference?

Understanding the National ScaleUnderstanding the National Scale• The National Scale is an arbitrary scale – at this stage it is not

related to points along a developmental curriculum. But, it is highly likely that it will be mapped onto the National Curriculum at some time in the future.



• The National Scale was fixed in 2008 as follows:– Range: 0-1000– Mean: 500– Standard Deviation: 100 (i.e. 68% students between 400-600)

+1 STD

2008 National Mean

-1 STD

900

800

700

600

500

400

300

200

100

National Averages (2008)National Averages (2008)

Only 20.1% of Australian school communities are

more advantaged

My School Website (2009)My School Website (2009)

YearLevel

Reading Writing SpellingGrammar &Punctuation

Numeracy

3 400.4 414.2 399.3 402.9 396.7

5 484.3 486.4 483.6 496.0 475.7

7 536.6 533.7 538.6 529.0 544.9

9 578.0 569.3 577.0 569.2 582.2

School’s Average

National Average

Average for school’s with similar

ICSEA




• The Scale has been divided into ten Bands which are used for reporting to parents. – Band 1 covers all scores equal to or less than 270.– Bands 2 – 9 increment by 52 score points each Band– Band 10 covers all scores above 686.

Parent ReportsParent Reports

Year 3Year 3

Year 5Year 5

Year 7Year 7

Year 9Year 9




• The Scale has been divided into ten Bands which are used for reporting to parents. – Band 1 covers all scores equal to or less than 270.– Bands 2 – 9 increment by 52 score points each Band– Band 10 covers all scores above 686.

• At this stage Bands have no explicit curriculum meaning but results show that for Victorian students in 2009:– A typical Yr3 level of performance is at the bottom of Band 5– A typical Yr5 level of performance is almost halfway into Band 6– A typical Yr7 level of performance is a third into Band 7– A typical Yr9 level of performance is at the bottom of Band 8

Yr 3

Yr 5

Yr 7Yr 9

Victorian State Averages - 2009Victorian State Averages - 2009(By Year Level and Dimension)(By Year Level and Dimension)

Comparing National Scale Scores to Comparing National Scale Scores to Estimated VELS Equivalent scoresEstimated VELS Equivalent scores

• The Victorian means for Year 3 and Year 9 Reading and Numeracy on the The Victorian means for Year 3 and Year 9 Reading and Numeracy on the National scale are compared to the estimated VELS equivalent scores below:National scale are compared to the estimated VELS equivalent scores below:

• Compared to our expected curriculum outcomes for Year 3 students (2.175), Compared to our expected curriculum outcomes for Year 3 students (2.175), the State Reading mean is about the State Reading mean is about 1½ term ahead 1½ term ahead of where we expect a of where we expect a typical Year 3 student to be. However, the State Numeracy mean is about typical Year 3 student to be. However, the State Numeracy mean is about 2½ terms below 2½ terms below where we expect a typical Year 3 student to be.where we expect a typical Year 3 student to be.

• Compared to our expected curriculum outcomes for Year 9 students (5.175), Compared to our expected curriculum outcomes for Year 9 students (5.175), the State Reading mean is about the State Reading mean is about one (1) term ahead one (1) term ahead of where we expect a of where we expect a typical Year 9 student to be. However, the State Numeracy mean is about typical Year 9 student to be. However, the State Numeracy mean is about 2½ terms below 2½ terms below where we expect a typical Year 9 student to be.where we expect a typical Year 9 student to be.

Year 3 Victoria Means Year 9 Victoria Means

DimensionNational

ScaleVELS

Equivalent National

ScaleVELS

Equivalent

Reading 430.6430.6 2.372.37 587.9587.9 5.285.28Numeracy 411.0411.0 1.831.83 596.3596.3 4.884.88

Cautionary Note #1Cautionary Note #1

• Equal scores amongst different dimensions Equal scores amongst different dimensions (on the National Scale) do not equate to (on the National Scale) do not equate to equal levels of performance in terms of equal levels of performance in terms of expected VELS levels.expected VELS levels.

• For example a National Yr9 Reading score of For example a National Yr9 Reading score of 587.9 is equivalent to a VELS score of 5.28 587.9 is equivalent to a VELS score of 5.28 but a higher National Yr9 Numeracy score but a higher National Yr9 Numeracy score of 596.3 is equivalent to a lower VELS score of 596.3 is equivalent to a lower VELS score of 4.88.of 4.88.

Cautionary Note #2Cautionary Note #2

• Some “Estimated VELS Equivalent Scores” Some “Estimated VELS Equivalent Scores” CAN CAN NOTNOT be read as VELS scores. be read as VELS scores.

• Specifically, it is doubtful that the “Estimated Specifically, it is doubtful that the “Estimated VELS Equivalent Scores” for VELS Equivalent Scores” for WritingWriting or or SpellingSpelling are truly VELS scores*.are truly VELS scores*.

• On the other hand, “Estimated VELS Equivalent On the other hand, “Estimated VELS Equivalent Scores” for Scores” for ReadingReading and and NumeracyNumeracy appear to be appear to be trustworthy*.trustworthy*.

*As evidence, see following AIM/NAPLAN results*As evidence, see following AIM/NAPLAN results

The National Minimum StandardThe National Minimum Standard(against Victoria’s State Averages)(against Victoria’s State Averages)

Year 3Year 3

Year 5Year 5

Year 7Year 7

Year 9Year 9

Yr 5

Yr 3

Yr 5

Yr 3

Yr 9

Even students at lower levels of “above Minimum Standard” may be “at risk”

Understanding Minimum StandardsUnderstanding Minimum Standards

• Yr 3, 5, 7 & 9 “Minimum Standards” are all Yr 3, 5, 7 & 9 “Minimum Standards” are all very low very low (> 2 years below (> 2 years below expected level.). In fact they are so low I think they offer no expected level.). In fact they are so low I think they offer no assistance in identifying “at risk” students so consider the following assistance in identifying “at risk” students so consider the following alternative.alternative.

• Use the “Holmes-Smith” Minimum Standard instead – anyone 0.5 of Use the “Holmes-Smith” Minimum Standard instead – anyone 0.5 of a VELS level a VELS level belowbelow expected is expected is in need of in need of focused intervention or focused intervention or additional supportadditional support.. For example:For example:– Yr5: Expected VELS score ~ 3.175Yr5: Expected VELS score ~ 3.175– Yr5: Holmes-Smith “low achiever” ~ 2.675 or lowerYr5: Holmes-Smith “low achiever” ~ 2.675 or lower

• Also anyone 0.5 of a VELS level Also anyone 0.5 of a VELS level aboveabove expected is expected is in need of further in need of further extensionextension (to avoid boredom). For example: (to avoid boredom). For example:– Yr7: Expected VELS score ~ 4.175Yr7: Expected VELS score ~ 4.175– Yr7: Holmes-Smith “high achiever” ~ 4.675 or higherYr7: Holmes-Smith “high achiever” ~ 4.675 or higher

The School Summary ReportThe School Summary Report

Choose “School Summary Report”

Click on “Preview Report” to view results

Choose to look at All Students or Girls vs. Boys orLBOTE Students orATSI Students

Choose to look at National Scale Scores or Estimated VELS scores

Interpreting “box and whisker” graphsInterpreting “box and whisker” graphs

• There are 30 students in this Year Level. Therefore:– 50% (or 15 students) are above the median.– 50% (or 15 students are below the median.

– 50% (or 15 students) are inside the “box”. Half of these (7-8 students) are above the median and half (7-8 students) are below the median.

– 10% (or 3 students) are at or below the 10th percentile “whisker”.

– 10% (or 3 students are at or above the 90th percentile “whisker”.

– 15% (or 4-5 students) are spread between the 25th down to the 10th percentile.

– 15% (or 4-5 students) are spread between the 75th up to the 90th percentile.

15 s

tude

nts

abov

e m

edia

n15

stu

dent

s be

low

med

ian

7-8 spread above the median.

15 students inside the box.

7-8 bunched below the median.

3 students on or below the 10th

percentile. (Note, all we know is at least one student scored on the 10th percentile.)

3 students on or above the 90th percentile. (Note, all we know is at least one student scored on the 90th percentile.)

4-5 students between the 75th up to the 90th percentile.

4-5 students between the 25th down to the 10th percentile.

Interpreting the School Summary ReportInterpreting the School Summary Report1. Is the school’s median

above, at or below the State median?

School’s median is about half a band below the State median



2. Is one or more dimension very different from the other dimensions?


Writing and Spelling are lower than Reading and Grammar & Punctuation



2. Is one or more dimension very different from the other dimensions?

3. How does the school’s spread compare to the State spread?


School has far fewer high performers than the State

School has far more low performers than the State

School’s median is at the State’s 25th percentile

Writing and Spelling are lower than Reading and Grammar & Punctuation

Interpreting the Interpreting the “Standard Error of the Mean” - “Standard Error of the Mean” - se(mean)se(mean)

Your reported school mean simply reflects the performance of the students who were present on the day and how they felt on that day. What if some of the smartest students had been away? What if there had been a bad accident in the schoolyard just prior to the test and the students’ minds weren’t 100% on task? Differences in results due to such events are referred to as measurement errormeasurement error.

State State MeanMean

School School MeanMean

Standard Standard error of the error of the

MeanMean



Statistically, we can allow for such errors in measurement errors in measurement by building a “confidence interval” around the reported school mean using the “Standard error of the mean” – se(mean).

We can be 95% confident that the true school mean for Reading is no lower than 488.1 – 1.96 * 7.6 (473.2)(473.2) and no higher than 488.1 + 1.96 * 7.6 (503.0)(503.0). That is, the true school meantrue school mean is somewhere between 473.2 – 503.0473.2 – 503.0.




MeanMean



Statistically, we can allow for such errors in measurement errors in measurement by building a “confidence interval” around the reported school mean using the “Standard error of the mean” – se(mean).

We can be 95% confident that the true school mean for Reading is no lower than 488.1 – 1.96 * 7.6 (473.2)(473.2) and no higher than 488.1 + 1.96 * 7.6 (503.0)(503.0). That is, the true school meantrue school mean is somewhere between 473.2 – 503.0473.2 – 503.0.

Now, because the State mean for reading (506.5) (506.5) is above the highest estimate of the school reading mean we conclude that the school mean is significantly belowsignificantly below the State mean.




MeanMean

The Five Year Trend ReportThe Five Year Trend Report

Choose “Five YearTrend Report”

Choose one dimension

The Five Year Trend ReportThe Five Year Trend Report

Using the Five Year Trend ReportUsing the Five Year Trend Report• Use the Five Year Trend Report to determine whether:– this year’s result was a one-offone-off or whether it is consistent

with previous results,– the trend over time is showing an improvingimproving trend, a steadysteady

trend or a decliningdeclining trend relative to the State.

• Remember, however, each year’s data comes from a different cohort of students different cohort of students and in some years, students are just much better or much worse than the typical cohort of students.

• Remember also that for small cohorts (< 10 students)small cohorts (< 10 students), a few extra high performing students can significantly increase your school mean. Likewise, a few extra low performing students can significantly decrease your school mean.

Analysing extracted data in SPAAnalysing extracted data in SPA

A-E Grades:A-E Grades:Year 3Year 3

A-E Grades:A-E Grades:Year 7Year 7

Year 3 – Year 5 GrowthYear 3 – Year 5 Growth

The Item Analysis ReportThe Item Analysis Report

Roughly equalequal numbers selecting each of the wrongwrong answers = guessing guessing

Nearly allall students who got this wrong gave the same wrong same wrong answer = common misconceptioncommon misconception

Year 3 Reading (Q21)Year 3 Reading (Q21)Year 5 Reading (Q9)Year 5 Reading (Q9)

Correct = ??

Most common incorrect = ??

Year 3 Reading (Q21)Year 3 Reading (Q21)Year 5 Reading (Q9)Year 5 Reading (Q9)

Correct = BB (Yr3 – 36%; Yr5 – 56%)

Most common incorrect = CC (Yr3 – 49%; Yr5 – 34%)

WHY?WHY?

Year 3 NumeracyYear 3 Numeracy

Correct = ??



Correct = BB (60%)

Most common incorrect = AA (28%)

WHY?WHY?


Correct = ??



Correct = BB (55%)

Most common incorrect = AA (26%)

WHY?WHY?

Zone of Proximal Development

The Student Response ReportThe Student Response Report(Reading or Numeracy – Difficulty Order)(Reading or Numeracy – Difficulty Order)

Choose “Student Response Report”

Choose “Reading or Numeracy – Difficulty Order”

The Student Response ReportThe Student Response Report

Data sorted by Item Difficulty and Student AbilityData sorted by Item Difficulty and Student Ability

Increasing Level of DifficultyIncreasing Level of Difficulty

Incr

easi

ng L

evel

Abi

lity

Incr

easi

ng L

evel

Abi

lity

Mostly correct Mostly incorrect

Zone of Proximal Development (Vygotsky)Zone of Proximal Development (Vygotsky)

The known: What students can already do independently.

Increasing Level of DifficultyIncreasing Level of Difficulty

Incr

easi

ng L

evel

Abi

lity

Incr

easi

ng L

evel

Abi

lity

Zone of Proxim

al Development:

What students a

re capable of le

arning with th

e assista

nce

of explici

t instr

uction from th

e teacher (s

caffolding).

The unknown: What students are incapable of learning before prior concepts are taught.

Summarising

strengths and weaknesses

Year 3 Reading – The KnownYear 3 Reading – The Known(What students can already do independently)(What students can already do independently)

Year 3 Reading – The UnknownYear 3 Reading – The Unknown((What students are incapable of learning before prior concepts are What students are incapable of learning before prior concepts are taughttaught))

Year 3 Reading – The Zone of Proximal DevelopmentYear 3 Reading – The Zone of Proximal Development((What students are capable of learning with the assistance of explicit What students are capable of learning with the assistance of explicit instruction from the teacher [scaffolding]instruction from the teacher [scaffolding]))

The Writing Criteria ReportThe Writing Criteria Report

Choose “Writing Criteria Report”

Click on “Preview Report” to view results

The Writing TaskThe Writing Task

The Writing Marking RubricThe Writing Marking Rubric

Interpreting the Writing Criteria ReportInterpreting the Writing Criteria Report

In this school, about 20% of students received a score of “1” but the majority (nearly 60%) received a score of “2”.

To improve, the teacher needs to move the 1s onto 2s, the 2s onto 3s, etc.

Assessment Assessment asas Learning Learning

Students writing like a “1” could be shown examples of how they are currently writing (Dungaun, The casel, BMX, etc.) and shown examples of what is now

expected of them to improve to a “2” (Living dead, Woodern box, etc.)

Another assessment Another assessment asas learning example learning exampleUsing the “Paragraphing” rubric and accompanying sample scripts, a student could be shown that their writing demonstrates no paragraphing.

More importantly, the rubric shows what is expected next and gives examples that students could read to get an idea of what writing in paragraphs looks like.

The Student Response ReportThe Student Response Report(Writing Test – by criteria)(Writing Test – by criteria)

Choose “Student Response Report”

Choose “Writing Test – by criteria”

The Student Response Report (Writing)The Student Response Report (Writing)

ASSESSMENT FOR BETTER LEARNING USING NAPLAN DATA Presented by Philip Holmes-Smith School Research...

Documents

Transcript of ASSESSMENT FOR BETTER LEARNING USING NAPLAN DATA Presented by Philip Holmes-Smith School Research...