Asking the Right Questions

Assessing Language Skills

2008 Presentation to ATESL Central Local Sheri Rhodes,

Mount Royal College

What is the first thing you think of when you hear the word

“evaluation”?“assessment”?

“test”?

Example of a Speaking Test

• One item• Single-word repetition• Testing a discrete point • Stimulus is the prompt, “Say now Shibboleth.”• Scoring criteria – right / wrong• Consequences – potentially dire!!

Purpose of Assessment

Why?

Formative vs. Summative

• How will we use the test?

Reasons for Testing

• Placement Tests• Diagnostic Tests• Progress or Achievement Tests• Proficiency Tests• Portfolio Assessments

What makes a good test?

• Validity• Reliability

Test Design

Discrete

•A verb tense•A word

Integrative

•Use a variety of language and skills for a task

Indirect

•Knowledge•Study

Direct •Activation•Integrative

Test Design

Question Types - Indirect

•Multiple Choice

Question Types - Indirect

•Gap-fill and Cloze

•www.lextutor.ca

Question Types - Indirect

•Transformation•Sentence re-ordering•Edit•Match

Question Types - Direct

• May use items that look like indirect items • Others are specific tasks to the skills• Replicate real-life interaction

– Mirror an authentic purpose

Question Types - Direct

• Remember– Items should look like the kind of tasks students

have been practicing in their lessons.– Create a “level playing field”.– Marking direct test items will be more

difficult than indirect items.

Constructed-response assessments

• Fill-in assessment

Performance-based assessments

• Simulates real-life language use• Offers open-ended tasks• Allows for creative and divergent responses• Often employs strategies• Requires students to synthesize information• Promotes recursive learning • Evaluates performance based on

well-defined critera

Testing Reading and Listening

• Choose the best summary• Put pictures in order• Complete a phone message form (listening)• Fill out a summary form (reading)

Testing Reading and ListeningGlobal understanding

Testing Reading

• Many skills at different levels of reading from word recognition, reading speed, skimming techniques, etc.

• Primary evaluation is looking at reading comprehension at every level.

Testing Listening

• Use a recorded audio track to ensure test relability

• Ensure the recording is clearly and easily audible

Testing Writing

• Produce written texts – Leaflets based on information in a text– Compositions (narrative or discursive essays)– Transactional letters (reply to advertisement)

Testing Speaking

• Seen as the most challenging of exams to prepare, administer and score.

Testing Speaking

• Interview students• Pair tasks

– Compare two pictures– Solve a problem– Discuss a topic– Role-play a situation

Grammar and Vocabulary

• Knowledge versus Ability• Isolated (discrete points) or integrated

Marking

• Backwash– Positive– Negative

Marking Writing

A + ?

Are you ever tempted to do this?

Marking Writing

• Holistic Scoring– Rubrics assigning a single score

• Analytic Scoring– 5 aspects of writing weighted and combined

(Jacob)– 7 scales (TEEP by Weir)– 3 scales (Michigan Writing Assessment)

Other Applications of Rubrics and Scales

• Speaking• Performance of Tasks• Portfolio evaluation

1. List the things you want to test.2. Decide how important each item is.

• Give them more time or space on the test

• Weight the marks to reflect the importance

3. Consider the test situation.4. Write the test. 5. Consult and share.6. Test the test. (the trial)7. Reflect and refine.

Steps in Test Design

Another list (Brown and Hudson)• Require students to perform, create, produce, or do something• Use real-world contexts or simulations• Allow students to be assessed on what they normally do in class every day• Use tasks that present meaningful instructional activities• Focus on processes as well as products• Tap into higher-level thinking and problem-solving skills• Provide information about both the strength and weaknesses of students• Are multi-culturally sensitive when properly administered• Ensure that people, not machines, do the scoring using human judgment• Encourage open disclosure of standards and rating criteria• Call upon teachers to perform new instructional and assessment roles