Automated Assessment of 4-Skills

in English (and Other Languages)

for Placement and Vocational Purposes

Ryan Downey

Pearson

Katrin Windsor

Pearson

CINOP workshop

24 May 2012

The problem• Large volume of students requiring English skills

– HBO (from hbo-raad.nl; estimated 2011)

• 102.525 intake first year students

• 423.776 total enrollment

– MBO (from mboraad.nl; estimated 2009)

• 58.000 graduating students

• 230.000 total enrollment• 230.000 total enrollment

• Assessing language skills:

– time-consuming

– resource intensive

– managing subjectivity / training raters

2

A solution

• Solution: Automated test delivery and scoring– objective, standardized, time-efficient testing with immediate results

– used now for “passive” language skills such as listening and reading

• Ideal solution: Automated assessment of all 4 major language skills: – Reading, Writing, Listening, Speaking

• Technology allows innovative task design– Integrative tasks = more data in less time

– Tasks relevant to target language use domain• Listen to a story and retell it

• Read and summarize an article for a colleague

• Read and respond to an email

• Testing efficiency returns time to the teacher

3

What is meant by “completely” automated?

Testing Type Responses

collected by…

Scoring

Direct Human Human judges

4

- Speech recognition technology

- Natural language processing

- Latent Semantic Analysis

Semi-directComputer

(recording device)Human judges

Automated Computer Computer

What it is /How it works

Student Student Responses are

Scoring system scores the test Student

accesses the Versant Testing

System by computer

Student responds to

the test questions or

prompts

Responses are sent to the automated

scoring system

scores the test and returns

scores to teachers via

web reporting tool

50-110 minutes for 4-skills tests 1-2 minutes for most tests

Outline

1. Introduction

2. The Versant English Placement Test

3. The Versant Professional Test3. The Versant Professional Test

4. Automated language assessment technology

5. Discussion

6

Versant English Placement Test

• Sample Demo

7

Scoring structurePart Group Name # of Items

PresentedScores Produced

A Passage Reading 2 Pronunciation, Fluency

B Repeats 16 Pronunciation, Fluency, Grammar

C Sentence Builds 10 Pronunciation, Fluency, Grammar

D Conversations 12 Listening Comprehension

8

E Typing 1 N/A, but…

F Sentence Completion 20 Reading Comprehension

G Dictation 16 Listening Comprehension

H Passage Reconstruction 3 Reading Comprehension, Writing

I Summary & Opinion 1 Reading, Writing (Opinion, Content)

Automated Scoring Technology:Speaking

Content Manner

Vocabulary Sentence Mastery Pronunciation Fluency

w1 w2 w3 w4 w5 w6 75-90 Words/Min

p p pppp p p p p p pp ppp pp p p p p p 5.8 Phones/Sec

waveform

Automated Scoring Technology:Speaking

10

Phoneme & Word Alignment

spectrum

segmentation

words

AccuracyFluencyPronunciation

Automated Scoring Technology:Speaking

11

Performance Comparison

5.502 seconds

Learner

3.026 seconds

Native speaker

Automated Scoring Technology:Writing

• “Latent Semantic Analysis” knows that…

“Surgery is often performed by a team of doctors.”

“On many occasions, several physicians are involved in

an operation.”an operation.”

= mean basically the same thing, even though they share no words.

• Enables evaluating the content of what is written rather than just matching keywords

12

Automated Scoring Technology:Writing

• Content

– How well did the writer address the topic?

• Voice & Tone

– Word choice – like that in high scoring essays or low

Latent Semantic Analysis and other scoring technologies compare each written

response to a large (n = hundreds) set of reference responses with human-

assigned scores

– Word choice – like that in high scoring essays or low

scoring essays?

• Organization

– Word and sentence flow – do the words and

sentences naturally follow each other?

– Coherence – does each sentence logically follow the

next? Does each sentence contribute to the essay as

a whole?

• Grammatical range & accuracy

– Grammar, word usage, punctuation, spelling, …

Speaking Listening Reading Writing

Overall

RepeatsRead Alouds

Sentence Builds

Conversations DictationSentence CompletionTyping PassageReconstruction

Summary and Opinion

15

Outline

1. Introduction

2. The Versant English Placement Test

3. The Versant Professional Test3. The Versant Professional Test

4. Automated language assessment technology

5. Discussion

16

You have successfully entered your TIN and are now ready to take Versant Professional English Test.

MENU

You are now ready to take the Versant Professional English Test.

The Versant Professional English Test has 13 parts. The test takes approximately 100 minutes. Click ‘Start’ to start the test.

Task Items Presented Scores Produced

Passage Reading 2

Repeats 16

Questions 20

Sentence Builds 10

Story Retells 3 SpeakingStory Retells 3 Speaking

Response Selection 16 Listening Comprehension

Conversations 12

Passage Comprehension 9 Listening Comprehension

Sentence Completion 20

Dictation 16

Passage Reconstruction 4

Email Writing 2 Grammar, Vocabulary, Voice &

Tone, Organization, Reading

Comprehension

MENU

MENU

MENU

Versant Professional

5-page score report (each module)

• Scores & subscores

• Explanation of subskills and

capabilities

• Detailed can-do statements

(linked to CEFR)

• Recommendations for • Recommendations for

improvement

• Relationship to other

tests/scales (CEFR, TOEFL, etc.)

21

Outline

1. Introduction

2. The Versant English Placement Test

3. The Versant Professional Test3. The Versant Professional Test

4. Automated language assessment technology

5. Discussion

22

Technology

• Patented, researched, and validated over last 15 years

• Publications describing technology and validation process are publicly

available (most on website)

• Technology certified, recognized, or approved by external (objective) • Technology certified, recognized, or approved by external (objective)

agencies

– 2012 SIIA CODiE Award finalist

– 2011 Tech and Learning Award of Excellence

– 2009 ICC Certificate of Quality Assurance

– 2009 CRM Excellence Award From "Customer Interaction Solutions“

23

Tests

ENGLISH

SPANISH

DUTCH

C2

C1

B2

CEFR

Testing

System

24

ARABIC

DUTCH

CHINESE

B1

A2

A1

FRENCH

Test validation process is rigorous and includes aligning to CEFR

Technology in use•Teacher/TA certification

•ELL placement

•Oral reading fluency

•Student assessment

•Study abroad

•Recruiting selection

•Training placement

•Leadership programs

•Promotion

CorporateSchools and Universities

•ESL, EFL language skills

•Oral reading fluency

•Assessment and scoring service for publishers

•Training

•Employment screening

•Language certification

•Immigrants screening

Private Language

Schools and Publishers

Government

25

Technology in useSpoken Tests Sample Users

SpanishUS Government: Department of Homeland Security and US Dept of

Defense; UC, Davis

Dutch Dutch Government: civic integration and naturalization exams

Arabic US Defense Language Institute: Arabic training program

English

Singapore Ministry of Education, Texas school districts, Stanford,

UConn, U Washington, Rhode Island Dept. of Education, Navitas,

Laureate, Hong Kong University of Science & Technology, FIFA

AT&T, Dell, IBM, P&G, Accenture, CitiBank, LG, Convergys, Amazon,

Deloitte, 3M, Bell Deloitte, 3M, Bell

Academic EnglishStudents for college and university entrance; recognized by >1,300

institutions

Junior EnglishGovernments of Singapore, S Korea, Chile

Yoons English School

Aviation EnglishFAA, Boeing, Emirates Airlines, Belgian Government, Indian

Government, Air Asia

French Canadian Government: Department of Education, Bell Canada

(Chinese) Peking University and Consortium

Outline

1. Introduction

2. The Versant English Placement Test

3. The Versant Professional Test3. The Versant Professional Test

4. Automated language assessment technology

5. Discussion

27

DiscussionTest Duration Skills required Target educational level

Placement Test 50 mins Basic English skills Entrance for higher

professional education

(HBO)

Professional Test Speaking module = 25 mins

Writing module = 45 mins

Combined = 110 mins

Basic English skills,

with work-related

content

Final exam, secondary

vocational education

(MBO)

28

CEFR Score

A1 20-29

A2 30-43

B1 44-53

B2 54-66

C1 67-76

C2 77-80

CEFR Score

A1 26-35

A2 36-46

B1 47-57

B2 58-68

C1 69-78

C2 79-80

Wri

tin

g

Sp

ea

kin

g

Discussion

Benefits Caveats

Accurate, consistent, objective School administrators and ICT prepare for

initial set-up

Time-efficient and cost-effective May be most appropriate for candidates

A2/B1/B2

Scalable and practical (any time, any place)

Automated solution for 4-skills testing

29

Scores returned immediately, available for

batch download by teacher

Administrators/teachers can view scores

online

Can be delivered on any supported

computer with Internet connection

(including WiFi)

Bedankt, en… Take a Test!

• Check the Pearson table to take your own test

– Sampler

– Demo

• Contact Thomas.dezeeuw@pearson.com for more • Contact Thomas.dezeeuw@pearson.com for more

information

• Check website @ www.kt.pearsonassessments.com

30