‘Tes%ngatest’–Evalua%ngourAssessmentTools

EddyWhite,Ph.D.

AssessmentCoordinator

CenterforEnglishasaSecondLanguage

UniversityofArizona

1.  Mybackground

2.  Classroombasedassessment

3.  Tests‐purposes/func%ons

4.  The‘cardinalcriteria’forevalua%ngatest

5.  Conclusions

Targets

2

1.  Mybackground

2.  Classroombasedassessment

3.  Tests‐purposes/func%ons

4.  The‘cardinalcriteria’forevalua%ngatest

5.  Conclusions

Targets

3

(1994‐2009)

Classroom-based Assessment

•  AssessmentofLearning

•  AssessmentforLearning

1.  Mybackground

2.  Classroombasedassessment

3.  Tests‐purposes/func%ons

4.  The‘cardinalcriteria’forevalua%ngatest

5.  Conclusions

Targets

8

Thegoalofassessmentisto...

9

The goal of assessment has to be, above all, to support the improvement of

learning and teaching.

(Fredrickson&Collins,1989)

10

definiGon:ClassroomAssessment

Assessment

Planning

CollecGng

Analyzing

ReporGng

ESLAssessment‐Purposes•  idenGfystrengthsandweaknessesofindividualstudents,

•  adjustinstrucGontobuildonstudents’strengthsandalleviateweaknesses,

•  monitortheeffecGvenessofinstrucGon,•  providefeedbacktostudents(sponsors,parents,etc.),and

•  makedecisionsabouttheadvancementofstudentstothenextleveloftheprogram.

12(Source:ESLSeniorHighGuidetoImplementaGon,2002)

Consider •  Researchsuggeststhatteachersspendfromone‐quartertoone‐thirdoftheirprofessionalGmeonassessment‐relatedacGviGes.

•  Almostalldosowithoutthebenefitofhavinglearnedtheprinciplesofsoundassessment.

(S%ggins,2007)

Teacherslearnhowtoteachwithoutlearningmuchabouthowtoassess.(Heritage,2007)

14

Assessmentliteracy

•  thekindsofassessmentknow‐howandunderstandingthatteachersneedtoassesstheirstudentseffecGvely

•  AssessmentliterateeducatorsshouldhaveknowledgeandskillsrelatedtothebasicprinciplesofqualityassessmentpracGces

(SERVECenter,UniversityofNorthCarolina,2004)

AssessmentLiteracy

Know‐howandunderstandingteachersneedtoassessstudentseffec%velyand

maximizelearning

•  Wemaynotlikeit,butstudentscananddoignoreourteaching;

•  howeveriftheywanttogetaqualificaGon,theyhavetoparGcipateintheassessmentprocesseswedesignandimplement.

(Brown,S.2004.Assessmentforlearning.LearningandTeachinginHigherEduca0on,1,81‐89)

Importanceofclassroomassessment

43/W

Whoaretheassessment‘deciders’atyour

insGtuGon?

Classroom-Based Assessment: Challenges, Choices, and Consequences

AssessmentFrameworks

Assessmentframework

•  ‐theseriesofassessmenttools(exams,tasks,projects,etc.)thatarescoredandusedtoarriveatasumma%vegradeforacourse

•  ‐itshouldbeskills‐basedandknowledge‐based(i.e.SsdemonstratewhattheyknowaboutandcandowithEnglish)

•  basedonlearningoutcomes

(Rowntree,1987)

• Thespiritandstyleofstudentassessmentdefinesthedefactocurriculum.

defacto=exisGnginfact,actual,whetherintendedornot

1.  Mybackground

2.  Classroombasedassessment

3.  Tests‐purposes/func%ons

4.  The‘cardinalcriteria’forevalua%ngatest

5.  Conclusions

Targets

25

QuizGme!

26

AssessinganEnglisharGclesquiz

Context

• ConversaGonclass(listening&speaking)

• high‐beginnerlevel27

Whatisafundamentalproblemwiththisquiz?

28

Answer

29

1.  Mybackground

2.  Classroombasedassessment

3.  Tests‐purposes/func%ons

4.  The‘cardinalcriteria’forevalua%ngatest

5.  Conclusions

Targets

30

Whatisatest?

31

Atest...•  isamethodofmeasuringaperson’sability,knowledge,orperformanceinagivendomain.

•  isaninstrument–asetoftechniques,procedures,oritems–thatrequiresperformanceonthepartofthetest‐taker.

32

Tests–measuringfunc%on

33

Atestmustmeasure•  Sometestsmeasuregeneralability,whileothersfocusonveryspecificcompetenciesorobjecGves.

•  Examples

•  AmulG‐skillproficiencytestmeasuresgeneralability;

•  aquizonrecognizingcorrectuseofdefinitearGclesmeasuresveryspecificknowledge.

34

•  Atestmeasuresperformance,...

• but,theresultsimplythetest‐takersability,orcompetence.

35

•  Performance‐basedtestssamplethetest‐takersactualuseoflanguage,

•  butfromthosesamplesthetestadministratorinfersgeneralcompetence.

36

•  Awell‐constructedtestisaninstrumentthatprovidesanaccuratemeasureofatest‐taker’sabilitywithinaparGculardomain.

•  Construc%ngagoodtestisacomplextask.

37

Yourassessmentprac%ces?

38

Thinkaboutwhatis

happeninginyourcontextandyour

assessmentpracGces

•  Inventories•  Checklists•  PeerRaGng•  SelfRaGng•  Journals•  Porkolios•  ObservaGons•  Discussions•  Interviews

•  True–FalseItem•  MulGpleChoice•  CompleGon•  ShortAnswer•  Essay•  PracGcalExam•  Papers/Reports•  Projects•  QuesGonnaires•  PresentaGons

YourassessmentpracGces?

Howdoyouassessyourstudents?

Foryou,whichofthefourskillsaremore/lesschallengingtotest?

41

1.  Mybackground

2.  Classroombasedassessment

3.  Tests‐purposes/func%ons

4.  The‘cardinalcriteria’forevalua%ngatest

5.  Conclusions

Targets

42

QuizGme!

43

2010

•  Exploringhowprinciplesoflanguageassessmentcanandshouldbeappliedtoformaltests.

•  Theseprinciplesapplytoassessmentofallkinds.

•  Howtousetheseprinciplestodesignagoodtest.

45

• Whatarethe‘fivecardinalcriteria’thatcanbeusedtodesignandevaluatealltypesofassessment?

46

Q.HowdoyouknowifatestiseffecGve,appropriate,useful,or,indown‐to‐earth

terms,a“good”test?

47

Fivekeyassessmentprinciples?

• Discuss• 3minutes

• Hint(fivenouns)48

Fivekeyassessmentprinciples

• PracGcality• Reliability• Validity• AuthenGcity• Washback

49

50

KeyAssessmentPrinciples

•  ThesequesGonsprovideanexcellentcriteriontoevaluatethetestswedesignanduse.

52

53

1.PracGcality

54

• IstheprocedurerelaGvelyeasytoadminister?

Prac%calityconsidera%ons

•  thelogisGcalandadministraGveissuesinvolvedinmaking,givingandscoringanassessmentinstrument

•  theamountofGmeittakestoconstructandadminister

•  theeaseofscoring•  easeofinterpreGng/reporGngtheresults

55

AneffecGvetestisprac%cal.

Thismeansthatit:•  isnotexcessivelyexpensive•  stayswithinappropriateGmeconstraints

•  isrelaGvelyeasytoadminister,and

•  hasascoring/evaluaGonprocedurethatisspecificandGmeefficient

56

ThevalueandqualityofatestsomeGmeshingeonsuchni`y‐gri`yprac%cal

considera%ons.

57

•  InclassroombasedtesGng,_________isalmostalwaysacrucialpracGcalfactorforbusyteachers.

58

59

2.Reliability

60

• Is all work being consistently marked to the

same standard?

• Areliabletestisconsistentanddependable.

•  Ifyougivethesametesttothesamestudentormatchedstudentsontwodifferentoccasions,thetestshouldyieldsimilarresults.

61

Whatfactors

contributetothe

unreliabilityofatest?

62

TestUnreliability‐contribuGngfactors

• Studentrelatedreliability• Raterreliability(inter,intra)• Testadministra%onreliability

• Testreliability63

Q.Whatisonekeywaytoincreasereliability?

A.Userubrics

64

• Rubricsarescoringguidelines.•  Theyprovideawaytomakejudgmentsfairandsoundwhenassessingperformance.

•  Auniformsetofpreciselydefinedcriteriaorguidelinesaresetforthtojudgestudentwork.

65

66

3.Validity

‐ mostcomplexcriteria

‐ mostimportantprinciple

• Does the assessment

measure what we

really want to

measure?

Validity‐definiGon

•  ‘Theextendtowhichinferencesmadefromassessmentresults

areappropriate,meaningful,andusefulintermsofthepurposeof

theassessment.’

(Gronlund,1998,p.226)68

•  Avalidtestofreadingability...

•  actuallymeasuresreadingability–

•  notmathskills

•  orpreviousknowledgeinasubject

•  norwriGngskills•  norsomeothervariableofquesGonablerelevance

69

Howisthevalidityofatestestablished?

1. Contentvalidity

2. Facevalidity

70

Contentvalidity•  Ifatestrequiresthetest‐takertoperformthebehaviorthatisbeingmeasured...

•  itcanclaimcontent‐relatedevidenceofvalidity(contentvalidity)

•  e.g.Atestofaperson’sabilitytospeakanL2requiresthestudenttoactuallyspeakwithinsomesortofauthenGccontext.

•  AtestwithpaperandpencilmulGplechoicequesGonsrequiringgrammaGcaljudgmentsdoesnotachievecontentvalidity.

71

AnotherwayofunderstandingcontentvalidityistoconsiderthedifferencebetweendirectandindirecttesGng.

•  directtes%ng–involvesthetest‐takerinactuallyperformingthetargettask

•  indirecttes%ng‐studentsnotperformingthetaskitself,butarelatedtask.

•  e.g.tes%ngoralproduc%onofsyllablestress

72

Toachievecontentvalidityin

classroomassessment,trytotestperformance

directly.

73

74

Howisthevalidityofatestestablished?

1. Contentvalidity

2. Facevalidity

75

Facevalidity

•  Theextenttowhichstudentsviewtheassessmentas:

1.  fair2.  relevant3.  usefulforimprovinglearning

•  Facevalidityreferstothedegreetowhichatestlooksright,andappearstomeasuretheknowledgeorabiliGesitclaimstomeasure.

76

Highfacevalidity:thetest...•  iswell‐constructed,expectedformatwithfamiliartasks

•  isclearlydoablewithinallouedGme•  hasitemsthatareclearanduncomplicated•  direcGonsthatarecrystalclear•  hastasksrelatedtocoursework(contentvalidity)

•  hasadifficultylevelthatpresentsareasonablechallenge

77

• Mostsignificantcardinalprincipleofassessmentevalua%on.

•  Ifvalidityisnotestablished,allotherconsideraGonsmayberendereduseless.

78

79

4.AuthenGcity

80

• Arestudentsaskedtoperformreal‐worldtasks?

Testtaskauthen%city

• tasksrepresent,orcloselyapproximate,real‐worldtasks

• thetaskislikelytobeenactedinthe“realworld”

• notcontrivedorarGficial81

AuthenGcitychecklist•  Isthelanguageinthetestasnaturalaspossible?

•  Aretopicsascontextualizedaspossibleratherthanisolated?

•  AretopicsandsituaGonsinteresGngenjoyable,and/orhumorous?

•  IssomethemaGcorganizaGonprovided,suchasthroughastorylineorepisode?

•  Dotasksrepresent,orcloselyapproximate,real‐worldtasks?

82

83

5.Washback

84

• Does the assessment have positive effects on learning

and teaching?

Washback=theeffectoftesGngonteaching

andlearning

85

‐ posi%vewashback‐ nega%vewashback

Washback•  Classroomassessment:theaffectsofanassessmentonteachingandlearningpriortotheassessmentitself(preparaGon)

•  Anotherformofwashback=theinformaGonthat‘washesback’tostudentsintheformofusefuldiagnosesofstrengthsandweaknesses.

•  Formaltestsprovidenowashbackifstudentsreceiveasimpleleuergradeorsingleoverallnumericalscore.

86

Atestthatprovidesbeneficialwashback...

•  posiGvelyinfluenceswhatandhowteachersteach

•  posiGvelyinfluenceswhatandhowstudentslearn

•  offerslearnersachancetoadequatelyprepare•  giveslearnersfeedbackthatenhancestheirlanguagedevelopment

•  providescondiGonsforpeakperformancebythelearner

87

Teachers’challenge

• tocreateclassroomteststhatserveaslearningtoolsthroughwhichwashbackis

achieved

88

89

1.  Mybackground

2.  Classroombasedassessment

3.  Tests‐purposes/func%ons

4.  The‘cardinalcriteria’forevalua%ngatest

5.  Conclusions

Targets

Q.HowdoyouknowifatestiseffecGve,appropriate,useful,or,indown‐to‐earthterms,a“good”

test?

91

Answer.A‘good’test:•  canbegivenwithinappropriateadministraGveconstraints,

•  isdependable,•  accuratelymeasureswhatyouwantittomeasure,

•  thelanguageinthetestisrepresentaGveofreal‐worldlanguageuse,and

•  thetestprovidesinformaGonthatisusefulforthelearner.

92

•  TheseprincipleswillhelpyoumakeaccuratejudgmentsabouttheEnglishcompetenceofyourstudents.

•  TheyprovideusefulguidelinesforevaluaGngexisGngtests,anddesigningourown.

93

AssessmentLiteracy

Know‐howandunderstandingteachersneedtoassessstudentseffec%velyand

maximizelearning

•  Thereisnogewngawayfromthefactthatmostofthethingsthatgowrongwithassessmentareourfault,

•  theresultofpoorassessmentdesign‐andnotthefaultofourstudents.

(Raceetal.,2005)

•  Improvingstudentlearningimpliesimprovingtheassessmentsystem.

•  Teachersoxenassumethatitistheirteachingthatdirectsstudentlearning.

•  InpracGce,assessmentdirectsstudentlearning,becauseitistheassessmentsystemthatdefineswhatisworthlearning.

(Havnes,2004,p.1)

•  ThereissubstanGalevidencethatassessment,ratherthanteaching,hasthemajorinfluenceonstudents’learning.

•  ItdirectsauenGontowhatisimportant,actsasanincenGveforstudy,andhasapowerfuleffectonstudent’sapproachestotheirwork.

(Boud&Falchikov,2007)

RethinkingAssessmentinHigherEduca0on

“We owe it to ourselves and our students to devote at least as much energy to ensuring that our assessment practices are worthwhile as we do to ensuring that we teach well”.

Dr.DavidBoud,UniversityofTechnology,Sydney,Australia

98

ThankyouforyourGmeandparGcipaGon