The KIMEP English Entrance Test (KEET) Validity Argumentâ€Based Validation for the KIMEP...

download The KIMEP English Entrance Test (KEET) Validity Argumentâ€Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute

of 21

  • date post

    27-Apr-2018
  • Category

    Documents

  • view

    218
  • download

    5

Embed Size (px)

Transcript of The KIMEP English Entrance Test (KEET) Validity Argumentâ€Based Validation for the KIMEP...

  • AnArgumentBasedValidationfor theKIMEPEnglishEntranceTest (KEET)

    MarinaGerassimenko,University

    of

    Leicester

    KazakhstanInstituteofManagement, EconomicsandStrategicResearch(KIMEP)

  • Outline

    BackgroundValidityasanargument

    ToulminsstructureofanargumentKane(1992,2006)andChapelleetal.

    (2008)TheKEETsvalidityargumentSummaryoftheKEETsvalidityargumentLessonslearned

  • Background

    KIMEP:KazakhstanInstituteofManagement,Economicsand StrategicResearch,Almaty,Kazakhstan,CentralAsia

    AnEnglishmediumuniversity

    Degreesineconomics,management,finance

    87%undergraduatestudents;13%graduate;3.6%international

    30003500applicantsannually

  • TheKEET: paperbased 100items

    multiplechoice threesections Grammar,VocabularyinUse,Reading

    Comprehension parallelforms

  • Avalidityargument

    Avalidityargumentshouldpresentand integrateevidenceandrationalesfromwhich

    validityconclusioncanbedrawnpertainingto particularscorebasedinferencesandusesof

    atest.Chapelle(1999:263)

  • AnArgumentStructure(Toulmin, 1958,2003):

    WARRANT

    Backing1

    Backing2

    Assumption

    1

    Assumption

    2 Assumption 3

    Backing3

    Grounds1

    Claim(Grounds2)

    Rebuttals

  • Kane(1992,2006):

    TwoStagesof Arguments

    Aninterpretativeargument

    specifiesthe proposedinterpretationsandusesoftest resultsbylayingoutthenetworkofinferences

    andassumptionsleadingfromtheobserved performancestotheconclusionsanddecisions

    basedontheperformances Thevalidityargument

    providesanevaluation

    oftheinterpretativeargument

  • Chapelleetal.(2008)Testuse

    Construct

    Target

    score

    Expected

    score

    Observation

    Observed

    score

    Target

    domain

    UTILIZATION(6)

    EXPLANATION(4)

    EXTRAPOLATION

    (5)

    GENERALIZATION(3)

    EVALUATION(2)

    DOMAIN DESCRIPTION(1)

  • Step1:TheDomainDescription Inference

    WARRANT

    1:TheKEETisareliableandvalidtestfor

    admissionpurposes;thetestrevealsimportant Englishlanguageskillsandabilities

    B1.1Tasksare

    selectedas

    representative

    ofGeneral

    Englishdomain

    B1.2Important

    skillswere

    selectedbased

    onformatand

    administration

    constraints

    A1.1Importanttasks areidentified

    A1.2Performanceonthe

    testrevealsabilitiesand

    skillsrelevantatEnglish

    mediumuniversity

    A1.3Scoresarenot

    affectedbythetestformat

    (multiplechoice)

    B1.3Scoringis

    objective

    Target

    Domain

    Observations

  • Step2:TheevaluationInferenceWARRANT2:TheKEETsobservedscores

    reflecttargetedlanguageabilities

    A2.1Scoring

    rubricsare appropriate asevidence

    A2.2Task

    administration conditionsare appropriateevidence

    fortargetedabilities A2.3TheKEEThas appropriate

    psychometric properties

    Observations

    Observed

    score

  • B2.1

    Rubrics: grammar,use

    ofEnglish, reading

    comprehensionprovide

    evidencefor targeted

    abilities

    B2.2Task administration

    conditionsare standardized

    byAdmission Office

    protocol

    B2.3Acceptable forstandardized tests

    coefficients Alfa:

    formA.935; formB.922

    Bothformshave distribution

    closetonormal

  • Backing3.3:TheKEETpsychometricdataforformsAandB:

    Scale FormA FormB

    Nofitems 100 100

    Samplesize 232 241

    Mean 58.375 54.095

    Variance 279.467 232.584

    Median 58.000 55.000

    Std.Dev. 16.717 15.251

    SEM 4.235 4.273

    Meanp 0.584 0.541

    Alfa 0.935 0.922

    ITEMANsoftware ResearchQuestions:Howsignificantisthedifference?Whydoparallelformshavedifferentmeans,variances

    andSD?Doparallelformshaveitemsofsimilardifficultylevel?

  • Howparallelisparallel?

    Researchmethods:(1)paralleltesttheoryaspartofClassicalTestTheory(Kline,2005):

    ModesMeansMediansVariances(2)ttest:howsignificantisthedifference?(3)Cohendtest:whatistheeffectsize?

    Findings:(2)T=2.9108,df(degreesoffreedom)is471,2tailconfidenceat99.62%thedifferenceinmeans

    issignificant;

    (3)Cohensd=0.27Cohen(1988) notsignificant;Wolf(1986)

    practicallysignificant

  • Step3:TheGeneralizationInference

    WARRANT3:Observedscoresareindicative

    ofexpectedscoresoverparallelversionsof tasksandtestforms

    B3.1Secondary

    schoolEFL

    programme

    B3.2TheKEETs

    specifications:Threefixedrubrics;100freeitems

    A3.1Testdomainis

    sampledfrom

    generalEnglish

    proficiencyarea:

    Problematic

    GrammarStructures,

    vocabulary,reading

    A3.2Test

    specificationsas testtemplate

    Observedscore

    Expectedscore

  • R3.2TestSpecifications:fixedand

    freeelements

    Backingforassumption3.2TestSpecifications

    METHODOLOGY:

    couldfreeelements

    intestspecifications causetestSD,

    meansvary?

    RESEARCH

    QUESTION:

    pvaluescompared

    acrosssectionsof grammar,vocabularyin

    contextandreading comprehensionand acrosstwoparallelforms

  • FINDINGS:

    Grammar FormA FormB

    Meanpvalue .66 .59

    Pvaluerange .11

    .91 .18

    .97

    Vocabulary FormA FormB

    Meanpvalue .49 .57

    Pvaluerange .09

    .87 .17

    .89

    Reading

    comprehensionFormA FormB

    Meanpvalues .52 .34

    Pvaluesrange .38

    .76 .12

    .60

  • TheKEETexample

    FormA

    14.Thistimenextweek,we.....thechemistryexam.A.

    hadfinished

    B.

    havebeenfinishingC.

    willhavefinished

    D.

    willhavebeenfinishing

    Prop. Disc.Point

    CorrectIndexBiser.0.610.310.24

    FormB

    14.Dontworry I.....writingmy reportbyFridaynextweek.Ill giveittoyoubythen.

    A.

    finishB.

    willbefinished

    C.

    amfinishingD.

    willhavefinished

    Prop.Disc.PointCorrectIndexBiser.0.720.560.50

  • ReadingComprehensionFormA FormB

    Length 600 700

    Topics Generalinterest Specificinterest

    Discoursefeatures

    Narrative Narrative

    imbeddedwith descriptions

    Vocabulary General Specific

    Questiontypes Askingforexplicit

    informationAskingfor

    inferences

  • SummaryoftheclaimsabouttheKEET

    1stclaim:thetestpurpose

    2ndclaim:thetestconstruct

    3dclaim:thetestspecifications

    4thclaim:theitemanalysis

    5thclaim:theuseofthetestscores

  • Conclusion:LessonsLearned

    TheKEETsconstruct Thetestspecifications Analysisofparallelforms Itemanalysis

  • Thankyouforyourattention!

    An Argument-Based Validation forthe KIMEP English Entrance Test (KEET)OutlineBackgroundKIMEP: Kazakhstan Institute of Management, Economics and Strategic Research, Almaty, Kazakhstan, Central AsiaAn English-medium universityDegrees in economics, management, finance87% undergraduate students; 13% graduate; 3.6% international3000-3500 applicants annuallyThe KEET:paper-based100 itemsmultiple-choicethree sections Grammar, Vocabulary in Use, Reading Comprehensionparallel formsA validity argumentAn Argument Structure (Toulmin, 1958, 2003) :Kane (1992, 2006): Two Stages of ArgumentsChapelle et al. (2008)Step 1: The Domain Description InferenceStep 2: The evaluation InferenceBildnummer 11Bildnummer 12How parallel is parallel?Step 3: The Generalization InferenceBildnummer 15FINDINGS:The KEET exampleReading ComprehensionSummary of the claims about the KEETConclusion: Lessons LearnedBildnummer 21