LANGUAGES

The links between the:Standards

TextsQuestions

and

Schedules

2

The CurriculumMoE

The CurriculumMoE

Achievement standards

MoE

Achievement standards

MoE

The Grade scoreNZQA

The Grade scoreNZQA

The hierarchy

Assessment Task

NZQA

Assessment Task

NZQA

A happy student

3

The Listening standards

Level 1 (1.1)

Demonstrate understanding of a variety of spoken LANGUAGE texts on areas of most immediate relevance

Level 2 (2.2)

Demonstrate understanding of a variety of spoken LANGUAGE texts on familiar matters

Level 3 (3.1)

Demonstrate understanding of a variety of extended spoken LANGUAGE texts

4

The reading standards

Level 1 (1.4)

Demonstrate understanding of a variety of LANGUAGE texts on areas of most immediate relevance

Level (2 .4)

Demonstrate understanding of a variety of written and/or visual LANGUAGE texts on familiar matters

Level 3 (3.4)

Demonstrate understanding of a variety of extended written and/or visual LANGUAGE texts

5

The standards, a colourful summary

The colours indicate

Grade: Achievement Merit Excellence

NCEA level: Level 1 Level 2 Level 3

The listening standard

Demonstrate clear / thorough understanding of a variety of extended spoken Tongan texts on areas of most immediate relevance / on familiar matters

The reading standard

Demonstrate clear / thorough understanding of a variety of extended written and/or visual Tongan texts on areas of most immediate relevance / on familiar matters

6

The definitions that matter

For Merit

Clear understanding means the relevant information ideas and/or opinions from the texts are selected and unambiguously communicated.

For Excellence

Thorough understanding means relevant information, ideas and/or opinions from the texts are expanded on with supporting detail to show understanding of the implied meanings or conclusions within the text.

A BAD TEXT and A BAD QUESTION

Example of a bad text - Liz made this up, it’s not taken from a real exam

 Below is some information for the exchange student who is coming in a month to stay with your family and go to your school.

Our family consists of five people. My mother, my father, myself, my older brother called William who is at university studying to be an accountant and my younger brother called Thomas who goes to the same school as I do. He wants to be a musician when he grows up so he can be rich and famous. I have lots of aunts and uncles and cousins, and many of them live in the Auckland like I do. We often meet up for birthdays and other family celebrations.

When you come to our you will be able to choose some interesting subjects to study such as Drama, Media studies, Technology, German, French, Japanese, Māori and Music. Other classes are compulsory, for example Maths, Science, English and PE. Our school day starts at 8/45 with a Form class meeting, and finishes at 3/15. We have an hour for lunch, and can buy food at the canteen but we are not allowed to go to the shops.

We have a lot of sports teams at my school. Mostly we practice at lunchtime and after school, and play against other schools on Saturday. In summer I like to play tennis, and in winter I like to play hockey. My father likes to come and watch me play sport, and afterwards we often go to a café for a delicious lunch. If you don’t like playing sport we have all sorts of other clubs such as drama, music and debating. I hope you join one of our after school clubs, you will enjoy the activities and the company of the other students.

In the weekends and holidays there are lots of places you could visit with me and my family. We could go to the beaches or the hot springs, there are museums and art galleries and concerts, and many interesting small towns to visit, we could even stay on a farm and help with the animals.

I hope very much you enjoy your stay in New Zealand, I am looking forward to your visit.

Kim

 A bad question to match

Describe Kim’s family

Explain what the exchange student can do during the school day• during class time • outside of class time

Describe a typical weekend for Kim. Use information from the text to support your answer.

Consider now:

1. What’s the problem with the text?

2. Can it be improved without too much hassle?

3. What’s the problem with the question?

4. Can it be improved without too much hassle?

5. What’s more important / difficult to fix, the text or the questions?

And, going back a couple of steps:

6. What’s the function of an exam question?

7. What things characterise a good exam question?8

9

The Assessment ScheduleThe schedule is formatted the same way for both standards at all three levels.

It is intended to enable holistic judgement for each question, and avoid focussing on counting up fragmented pieces of evidence / lexical items.

It is a guideline to how to make judgements, not a black and white list of ‘right’ and ‘wrong’ answers.

Judgements between grades (that is N and A, A and M, M and E) must be qualitative

Judgements within grades may be qualitative or quantitative

Although final cut-scores may vary from exam to exam (depending on relative difficulty) adherence to the intent of the standard supersedes all other considerations. That is, final judgements must be made against the standard.

10

Grade scores

• Each question receives a grade first, then a score – a grade score!• There are nine possible grade scores:

• Between grades – steps up are always qualitative.• Within grades – steps up are quantitative and/or qualitative • The scores are totalled once all the questions have been marked• The total score is written on the front of the answer booklet

Top-down Marking

All standards are marked using top-down marking which conforms to the fundamental principle on which standards based assessment sits.

Top down marking is fairer and more valid for the candidates because:

• It recognises all evidence (including N1 or N2) and aggregates it across the paper.

• The absence of lower level evidence does not ‘knock’ a candidate out of a higher level of achievement.

The steps in top down marking

1. Markers look for Excellence evidence. If there is sufficient to indicate the candidate is performing at E level, they: – award E– decide if it is ‘a good E’ (E8) or ‘just an E’ (E7)– write the appropriate grade score in the box provided – move on to the next question.

2. If Excellence evidence is not there, they look for Merit evidence

3. If Merit evidence is not there, they look for Achievement evidence

4. If Achievement evidence is not there, they decide whether to award NØ, N1 or N2

5. Finally they total the scores and write the total on the front of the answer booklet.

13

Quick analysis of a test

On one page, draw up a tally table like the one below.

Go through the class results, one student and a time.

Enter (as a tally) each student’s results for the four questions.

You can now see which questions:

• are discriminating well• seem to be too ‘hard’ or too ‘easy’• have a very strange spread.

14

Cut scores

The languages exams all have four questions, therefore the highest possible total score is 32.

While there is some variation possible in cut-scores, there are limits. For example:

8 must always be N, because it could be gained from N2, N2, N2 and N2. Clearly that’s not an A.

Equally, 12 must be an A because it could be gained from A3, A3, A3 and A3 which cannot logically or validly be N or M.

Task

Work out the highest and lowest possible scores that for the valid award of A and M and E in a four question exam.

15

Cut score ranges – an almost complete illustration

Why must Merit start at 17 and not 16?

Achievement could start at which four scores?

One of the above ‘starting/minimum’ scores is extremely unlikely.

• What score is that?

• Why is it unlikely?

Which 4 scores could M and E start at?

The nuts and bolts of adapting

* Texts * Questions* Schedules

so they are valid for the new standards

The hierarchy

What order do you think these should be put in?

Curriculum

Cut score decision

Grade scores

Questions

Schedule

Standard

Student

Text

Top down marking

Total score

17

This order

Student

Curriculum

Standard

Text

Questions

Schedule

Top down marking

Grade scores

Total score

Cut score decision

18

The text

The text is the most important thing, it must contain:

• straight forward information (Achievement)• material that can used to justify opinions / draw conclusions /

infer / link / analyse (Merit and Excellence)

Texts and questions can link in several ways:• one text, one question• one text, more than one question • several linked texts, one question

If the text is good, the questions ‘fall out’ of it easily.

The question – some guidelines

The question should elicit sufficient information for the marker to be able to validly judge the level of understanding the student is demonstrating; don’t ask more just because you can.

As you are writing / critiquing a question, bear in mind two students:• the ‘just achieved’, literal, conscientious one• the excellence one who thinks outside the box

If your question can be successfully attempted by both those candidates, it will discriminate successfully at all levels.

Scaffold if possible, and try to avoid very broad ‘drag net’ questions; candidates at all levels tend to respond to them by writing everything.

A BAD TEXT and A BAD QUESTION

Example of a bad text - Liz made this up, it’s not taken from a real exam

 Below is some information for the exchange student who is coming in a month to stay with your family and go to your school.

Our family consists of five people. My mother, my father, myself, my older brother called William who is at university studying to be an accountant and my younger brother called Thomas who goes to the same school as I do. He wants to be a musician when he grows up so he can be rich and famous. I have lots of aunts and uncles and cousins, and many of them live in the Auckland like I do. We often meet up for birthdays and other family celebrations.

When you come to our you will be able to choose some interesting subjects to study such as Drama, Media studies, Technology, German, French, Japanese, Māori and Music. Other classes are compulsory, for example Maths, Science, English and PE. Our school day starts at 8/45 with a Form class meeting, and finishes at 3/15. We have an hour for lunch, and can buy food at the canteen but we are not allowed to go to the shops.

We have a lot of sports teams at my school. Mostly we practice at lunchtime and after school, and play against other schools on Saturday. In summer I like to play tennis, and in winter I like to play hockey. My father likes to come and watch me play sport, and afterwards we often go to a café for a delicious lunch. If you don’t like playing sport we have all sorts of other clubs such as drama, music and debating. I hope you join one of our after school clubs, you will enjoy the activities and the company of the other students.

In the weekends and holidays there are lots of places you could visit with me and my family. We could go to the beaches or the hot springs, there are museums and art galleries and concerts, and many interesting small towns to visit, we could even stay on a farm and help with the animals.

I hope very much you enjoy your stay in New Zealand, I am looking forward to your visit.

Kim

 A bad question to match

Describe Kim’s family

Explain what the exchange student can do during the school day• during class time • outside of class time

Describe a typical weekend for Kim. Use information from the text to support your answer.

Some questions apropos the previous

1. What’s the problem with the text?

2. Can it be improved without too much hassle?

(Pause and consider)

3. What’s the problem with the question?

4. Can it be improved without too much hassle?

(Pause and consider)

5. What’s more important / easier to fix, the text or the questions?

And, going back a couple of steps:

6. What’s the function of an exam question?

7. What things characterise a good exam question?

The Assessment ScheduleThe schedule is formatted the same way for both standards, all languages, at all three levels.

It is intended to enable holistic judgement for each question, and avoid focussing on counting up fragmented pieces of evidence / lexical items.

It is a guideline to how to make judgements, not a black and white list of ‘right’ and ‘wrong’ answers.

Judgements between grades (that is N and A, A and M, M and E) must be qualitative

Judgements within grades may be qualitative or quantitative

Although final cut-scores may vary from exam to exam (depending on relative difficulty) adherence to the intent of the standard supersedes all other considerations. That is, final judgements must be made against the standard.

23

Adapting a schedule

Schedules for the old standards have:• bolds• underlines• question judgements that specify how many ‘pieces of

evidence’ are required

Schedules for new standards have:• each level of achievement, as defined in the standards,

at the top (this can me omitted in an adapted schedule)• nine ‘grade score descriptors’• a specific evidence box• brief examples for each of the nine levels

24

Quick analysis of a test

On one page, draw up a tally table like the one below.

Go through the class results, one student and a time.

Enter (as a tally) each student’s results for the four questions.

You can now see which questions:

• are discriminating well• seem to be too ‘hard’ or too ‘easy’• have a very strange spread.

Score

Question 0 1 2 3 4 5 6 7 8

One ll lll l llll llll ll lll l lll ll

Two llll llll ll lll l l lll ll llll lll

Three llll llll l lll lll l ll ll l

Four l l lll ll llll llll llll ll lll llll ll

25