The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP...

21
An ArgumentBased Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute of Management, Economics and Strategic Research (KIMEP)

Transcript of The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP...

Page 1: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

An Argument‐Based Validation for the KIMEP English Entrance Test  (KEET)

Marina Gerassimenko,University

of

Leicester

Kazakhstan Institute of Management,  Economics and Strategic Research (KIMEP)

Page 2: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Outline

BackgroundValidity as an argument

Toulmin’s structure of an argumentKane (1992, 2006) and Chapelle et al. 

(2008)The KEET’s validity argumentSummary of the KEET’s validity argumentLessons  learned

Page 3: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Background

KIMEP: Kazakhstan Institute of Management, Economics and  Strategic Research, Almaty, Kazakhstan, Central Asia

An English‐medium university

Degrees in economics, management, finance

87% undergraduate students; 13% graduate; 3.6% international

3000‐3500 applicants annually

Page 4: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

The KEET: paper‐based 100 items

multiple‐choice three sections – Grammar, Vocabulary in Use, Reading 

Comprehension parallel forms

Page 5: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

A validity argument

“A validity argument should present and  integrate evidence and rationales from which 

validity conclusion can be drawn pertaining to  particular score‐based inferences and uses of 

a test.”

Chapelle (1999:263) 

Page 6: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

An Argument Structure (Toulmin,  1958, 2003) :

WARRANT 

Backing 1

Backing 2

Assumption 

1

Assumption 

2 Assumption 

3

Backing 3

Grounds 1

Claim (Grounds 2)

Rebuttals

Page 7: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Kane (1992, 2006):

Two Stages of  Arguments

• An interpretative argument

specifies the  proposed interpretations and uses of test  results by laying out the network of inferences 

and assumptions leading from the observed  performances to the conclusions and decisions 

based on the performances

• The validity argument

provides an evaluation  of the interpretative argument 

Page 8: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Chapelle et al. (2008)Test use

Construct

Target 

score

Expected

score

Observation

Observed 

score

Target 

domain

UTILIZATION (6)

EXPLANATION (4)

EXTRAPOLATION

(5)

GENERALIZATION (3)

EVALUATION (2)

DOMAIN  DESCRIPTION (1)

Page 9: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Step 1: The Domain Description  Inference

WARRANT

1: The KEET is a reliable and valid test for 

admission purposes; the test reveals important 

English language skills and abilities

B 1.1 Tasks are 

selected as 

representative 

of General 

English domain

B 1.2 Important 

skills were 

selected based 

on format and  

administration 

constraints

A 1.1 Important tasks 

are identified

A 1.2 Performance on the 

test reveals abilities and 

skills relevant at  English‐

medium university

A 1.3 Scores are not 

affected by the test format 

(multiple‐choice)

B 1.3 Scoring is 

objective

Target 

Domain

Observations

Page 10: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Step 2: The evaluation InferenceWARRANT 2: The KEET’s observed scores 

reflect targeted language abilities

A 2.1 Scoring 

rubrics are 

appropriate 

as evidenceA 2.2 Task 

administration 

conditions are 

appropriate evidence 

for targeted abilitiesA 2.3 The KEET has 

appropriate 

psychometric 

properties

Observations

Observed 

score

Page 11: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

B2.1

Rubrics:  grammar, use 

of English,  reading 

comprehension  provide 

evidence for  targeted 

abilities

B 2.2  Task  administration 

conditions are  standardized 

by Admission  Office 

protocol

B 2.3 Acceptable  for standardized  tests 

coefficients  Alfa: 

form A .935;  form B .922

Both forms have  distribution 

close to normal

Page 12: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Backing 3.3: The KEET psychometric data for forms A and B:

Scale Form A Form B

N of items 100 100

Sample  size 232 241

Mean 58.375 54.095

Variance 279.467 232.584

Median 58.000 55.000

Std. Dev. 16.717 15.251

SEM 4.235 4.273

Mean p 0.584 0.541

Alfa 0.935 0.922

ITEMAN software Research Questions: How significant is the difference?Why do parallel forms have different means, variances 

and SD?Do parallel forms have items of similar difficulty level?

Page 13: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

How parallel is parallel?

Research methods:  (1) parallel test theory as part of Classical Test Theory(Kline, 2005):

•Modes•Means•Medians•Variances(2) t‐test: how significant is the difference?(3) Cohen d‐test: what is the effect‐size?

Findings:(2) T = 2.9108, df (degrees of freedom) is 471, 2‐tail confidence at99.62% the difference in means

is significant;

(3) Cohen’s d = 0.27 Cohen (1988) – not significant; Wolf (1986) –

practically significant

Page 14: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Step 3: The Generalization Inference

WARRANT 3: Observed scores are indicative 

of expected scores over parallel versions of 

tasks and test forms

B 3.1 Secondary 

school EFL 

programme

B 3.2 The KEET’s 

specifications:Three fixed rubrics;100 free items

A 3.1 Test domain is 

sampled from 

general English 

proficiency area: 

Problematic 

Grammar Structures, 

vocabulary, reading

A 3.2 Test 

specifications  as 

test template 

Observed score

Expected score

Page 15: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

R 3.2Test Specifications : fixed and 

free elements

Backing for assumption 3.2Test Specifications

METHODOLOGY:

could free elements 

in test specifications 

cause test SD, 

means vary?

RESEARCH 

QUESTION:

p‐values compared  

across sections of 

grammar, vocabulary in  

context and reading 

comprehension and 

across two parallel forms

Page 16: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

FINDINGS:

Grammar Form  A Form B

Mean p‐value .66 .59

P‐value range .11 ‐

.91 .18 ‐

.97

Vocabulary Form A Form B

Mean p‐value .49 .57

P‐value range .09 ‐

.87 .17 ‐

.89

Reading 

comprehensionForm A Form B

Mean p‐values .52 .34

P‐values range .38 ‐

.76 . 12 ‐

.60

Page 17: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

The KEET example

Form A

14. This time next week, we.....the chemistry exam.A.

had finished

B.

have been finishingC.

will have finished

D.

will have been finishing

Prop.  Disc.            Point

Correct        Index             Biser.0.61             0.31                0.24

Form B

14. Don’t worry – I ..... writing my  report by Friday next week. I’ll  give it to you by then.

A.

finishB.

will be finished

C.

am finishingD.

will have finished

Prop.            Disc.          PointCorrect        Index         Biser.0.72               0.56          0.50

Page 18: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Reading ComprehensionForm A Form B

Length 600 700

Topics General interest Specific interest

Discourse features

Narrative Narrative 

imbedded with 

descriptions

Vocabulary General Specific

Question types Asking for explicit 

informationAsking for 

inferences

Page 19: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Summary of the claims about the KEET

1st claim: the test purpose

2nd claim: the test construct

3d claim: the test specifications

4th claim: the item analysis

5th claim: the use of the test scores

Page 20: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Conclusion: Lessons Learned

• The KEET’s construct• The test specifications• Analysis of parallel forms

• Item analysis

Page 21: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

Thank you for your attention!