The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP...
Transcript of The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP...
An Argument‐Based Validation for the KIMEP English Entrance Test (KEET)
Marina Gerassimenko,University
of
Leicester
Kazakhstan Institute of Management, Economics and Strategic Research (KIMEP)
Outline
BackgroundValidity as an argument
Toulmin’s structure of an argumentKane (1992, 2006) and Chapelle et al.
(2008)The KEET’s validity argumentSummary of the KEET’s validity argumentLessons learned
Background
KIMEP: Kazakhstan Institute of Management, Economics and Strategic Research, Almaty, Kazakhstan, Central Asia
An English‐medium university
Degrees in economics, management, finance
87% undergraduate students; 13% graduate; 3.6% international
3000‐3500 applicants annually
The KEET: paper‐based 100 items
multiple‐choice three sections – Grammar, Vocabulary in Use, Reading
Comprehension parallel forms
A validity argument
“A validity argument should present and integrate evidence and rationales from which
validity conclusion can be drawn pertaining to particular score‐based inferences and uses of
a test.”
Chapelle (1999:263)
An Argument Structure (Toulmin, 1958, 2003) :
WARRANT
Backing 1
Backing 2
Assumption
1
Assumption
2 Assumption
3
Backing 3
Grounds 1
Claim (Grounds 2)
Rebuttals
Kane (1992, 2006):
Two Stages of Arguments
• An interpretative argument
specifies the proposed interpretations and uses of test results by laying out the network of inferences
and assumptions leading from the observed performances to the conclusions and decisions
based on the performances
• The validity argument
provides an evaluation of the interpretative argument
Chapelle et al. (2008)Test use
Construct
Target
score
Expected
score
Observation
Observed
score
Target
domain
UTILIZATION (6)
EXPLANATION (4)
EXTRAPOLATION
(5)
GENERALIZATION (3)
EVALUATION (2)
DOMAIN DESCRIPTION (1)
Step 1: The Domain Description Inference
WARRANT
1: The KEET is a reliable and valid test for
admission purposes; the test reveals important
English language skills and abilities
B 1.1 Tasks are
selected as
representative
of General
English domain
B 1.2 Important
skills were
selected based
on format and
administration
constraints
A 1.1 Important tasks
are identified
A 1.2 Performance on the
test reveals abilities and
skills relevant at English‐
medium university
A 1.3 Scores are not
affected by the test format
(multiple‐choice)
B 1.3 Scoring is
objective
Target
Domain
Observations
Step 2: The evaluation InferenceWARRANT 2: The KEET’s observed scores
reflect targeted language abilities
A 2.1 Scoring
rubrics are
appropriate
as evidenceA 2.2 Task
administration
conditions are
appropriate evidence
for targeted abilitiesA 2.3 The KEET has
appropriate
psychometric
properties
Observations
Observed
score
B2.1
Rubrics: grammar, use
of English, reading
comprehension provide
evidence for targeted
abilities
B 2.2 Task administration
conditions are standardized
by Admission Office
protocol
B 2.3 Acceptable for standardized tests
coefficients Alfa:
form A .935; form B .922
Both forms have distribution
close to normal
Backing 3.3: The KEET psychometric data for forms A and B:
Scale Form A Form B
N of items 100 100
Sample size 232 241
Mean 58.375 54.095
Variance 279.467 232.584
Median 58.000 55.000
Std. Dev. 16.717 15.251
SEM 4.235 4.273
Mean p 0.584 0.541
Alfa 0.935 0.922
ITEMAN software Research Questions: How significant is the difference?Why do parallel forms have different means, variances
and SD?Do parallel forms have items of similar difficulty level?
How parallel is parallel?
Research methods: (1) parallel test theory as part of Classical Test Theory(Kline, 2005):
•Modes•Means•Medians•Variances(2) t‐test: how significant is the difference?(3) Cohen d‐test: what is the effect‐size?
Findings:(2) T = 2.9108, df (degrees of freedom) is 471, 2‐tail confidence at99.62% the difference in means
is significant;
(3) Cohen’s d = 0.27 Cohen (1988) – not significant; Wolf (1986) –
practically significant
Step 3: The Generalization Inference
WARRANT 3: Observed scores are indicative
of expected scores over parallel versions of
tasks and test forms
B 3.1 Secondary
school EFL
programme
B 3.2 The KEET’s
specifications:Three fixed rubrics;100 free items
A 3.1 Test domain is
sampled from
general English
proficiency area:
Problematic
Grammar Structures,
vocabulary, reading
A 3.2 Test
specifications as
test template
Observed score
Expected score
R 3.2Test Specifications : fixed and
free elements
Backing for assumption 3.2Test Specifications
METHODOLOGY:
could free elements
in test specifications
cause test SD,
means vary?
RESEARCH
QUESTION:
p‐values compared
across sections of
grammar, vocabulary in
context and reading
comprehension and
across two parallel forms
FINDINGS:
Grammar Form A Form B
Mean p‐value .66 .59
P‐value range .11 ‐
.91 .18 ‐
.97
Vocabulary Form A Form B
Mean p‐value .49 .57
P‐value range .09 ‐
.87 .17 ‐
.89
Reading
comprehensionForm A Form B
Mean p‐values .52 .34
P‐values range .38 ‐
.76 . 12 ‐
.60
The KEET example
Form A
14. This time next week, we.....the chemistry exam.A.
had finished
B.
have been finishingC.
will have finished
D.
will have been finishing
Prop. Disc. Point
Correct Index Biser.0.61 0.31 0.24
Form B
14. Don’t worry – I ..... writing my report by Friday next week. I’ll give it to you by then.
A.
finishB.
will be finished
C.
am finishingD.
will have finished
Prop. Disc. PointCorrect Index Biser.0.72 0.56 0.50
Reading ComprehensionForm A Form B
Length 600 700
Topics General interest Specific interest
Discourse features
Narrative Narrative
imbedded with
descriptions
Vocabulary General Specific
Question types Asking for explicit
informationAsking for
inferences
Summary of the claims about the KEET
1st claim: the test purpose
2nd claim: the test construct
3d claim: the test specifications
4th claim: the item analysis
5th claim: the use of the test scores
Conclusion: Lessons Learned
• The KEET’s construct• The test specifications• Analysis of parallel forms
• Item analysis
Thank you for your attention!