Measurement in the context of formative classroom ......• Wilson, M. (1989). Saltus: A...

Measurementinthecontextofformativeclassroomassessment

MarkWilsonUCBerkeley

PresentedattheUniversityofCalifornia,BerkeleyMarch7,2017

Abstract

Thistalkwillsurveythelast15yearsofworkthatwehavebeendoingattheBEARCenteronthe BEARAssessmentSystem(BAS). Itwillbeginbynotingtheinitialmotivationsfordevelopingthisapproachtomeasurement/assessment,focusingonthequestionofwhatarethemeasurementdemandsinthecontextofformativeclassroomassessments. ThiswillbefollowedbyabriefdescriptionoftheBAS,accompaniedbydiscussionofhowitreflectsaresponsetothisquestion.Followingthatitwillalsoexplorethefollowing.(a) Whatare themostimportantdevelopmentsintheBASsince2000?(b)WhataresomeimportantexamplesofBASassessments?(c)HowshouldBASinterfacewithstatetesting?(d)Whatarethechallengesandopportunities?

Outline• Whyshouldwethinkofclassroomassessmentas“measurement”?

• WhatshouldwewantfromanAssessmentSystem?• HowistheBEARAssessmentSystem(BAS)aresponsetothequestion?

• Whatare themostimportantdevelopmentsintheBASsince2000?

• WhataresomeimportantexamplesofBASassessments?

• HowshouldBASinterfacewithstatetesting?• Whatarethechallengesandopportunities?

Abitofmyownbackground

• Q1.Howcaneducationalmeasurementhelpclassroomassessment?

• Worriedonthisin1970s,andsince– Sawtheneedtogiveteachersbetterassessmentinformationintheclassroom

– Sawtheneedfortheassessmenttotrackthe“logicbehindthecurriculum”

– B/cthatwaytheteacherscanunderstandwhattodowiththatinformation.

Abitofmyownbackground

• Q2.Wheredoesthescale (aka,thecontinuum,theconstruct,thelearningprogression)intestscomefrom?

• Workedonthisin1980sandearly90s– Discoveringandinterpretingeffectsonitemdifficultyfor“wild” items

– Lookingforaconsistentstoryaboutwhyitemsareorderedthewaytheyare

• Published10papersonthesetopicsduringthatperiod…

• Wilson,M.,&Bock,R.D.(1985).Spellability:alinearly-orderedcontentdomain.AmericanEducationalResearchJournal,22(2),297-307.

• Wilson,M.(1989).Empiricalexaminationofalearninghierarchyusinganitemresponsetheorymodel.JournalofExperimentalEducation,57(4),357-371.

• Wilson,M.(1989).Saltus:Apsychometricmodelofdiscontinuityincognitivedevelopment.PsychologicalBulletin,105(2),276-289.

• Wilson,M.(1990).MeasuringavanHiele geometrysequence:Areanalysis.JournalforResearchinMathematicsEducation,21(3),230-237.

• Wilson,M.(1990).Investigationofstructuredproblemsolvingitems.InG.Kulms (Ed.),Assessinghigherorderthinkinginmathematics.Washington,DC:AmericanAssociationfortheAdvancementofScience.

• Wilson,M.(1992).Theorderedpartitionmodel:Anextensionofthepartialcreditmodel.AppliedPsychologicalMeasurement.16(3),309-325.

• Masters,G.N.,Adams,R.J.,&Wilson,M.(1990).Chartingofstudentprogress.InT.Husen &T.N.Postlethwaite (Eds.),InternationalEncyclopediaofEducation:ResearchandStudies.SupplementaryVolume2(pp.628-634). Oxford:Pergamon Press.Reprintedin:T.Husen &T.N.Postlethwaite (Eds.),(1994).InternationalEncyclopediaofEducation(2nd.Ed.) (pp.5783-91). Oxford:Pergamon Press.

• Wilson,M.(1990).Measurementofdevelopmentallevels.InT.Husen &T.N.Postlethwaite (Eds.),InternationalEncyclopediaofEducation:ResearchandStudies.SupplementaryVolume2. Oxford:Pergamon Press.

• Wilson,M.(1992).Measurementmodelsfornewformsofassessmentinmathematicseducation.InJ.F.Izard&M.Stephens(Eds.)Reshapingassessmentpractices:Assessmentinthemathematicalsciencesunderchallenge.Hawthorn,Australia:ACER.

• Wilson,M.(1992).Measuringlevelsofmathematicalunderstanding.InT.Romberg(Ed.),Mathematicsassessmentandevaluation:Imperativesformathematicseducators.NewYork:SUNYPress.

From:Wilson,2010.

Moreofmyownbackground

• Concludedthattherewaslittleenlightenmentaboutlearningprogressionsinresultsfromtheseanalyses– Theitemssetsweretoodiverse,hencetoosparseinwhattheyconveyedaboutstudentlearning,

– Theitemsdifferedinidiosyncraticways.• Gaveupdoingthat

– neededtofindanotherway...

ThingsIlearned…• MaximNo.1.Assessmentcontentspecialists(“itemwriters”)donotknowwhytheybuilditemsthewaytheydo.Corrollary:Theycannotpredictthedifficultyoftheiritems.

• MaximNo.2.“Wild”itemsconfounddeepercognitive/structuralist/diagnosticunderpinningswithsurfacefeatures(suchasitemwording,etc.).

ThingsIlearned…

• ButGoodcurriculumdeveloperscreatetheircurriculausingadevelopmentalwayofthinkingaboutlearning,

• HenceNeedtoengagewithcurriculumdevelopers,notitemwriters

• ButCurriculumdevelopersdonotknowhowtowriteitems.

Moreofmyownbackground

• Changedmyentireresearchagenda– Becameanitem-designer,notjustasecondarydataanalyst

– Becamea“sortof” curriculumdesigner—alearningprogressiondesigner

– Becameadeveloperofnewtypesofitemresponsemodels(“explanatory”)

Learnedthatthereisafurtherquestion...

• Q3.Howcanwecreate“summativetests”ifwedonotknowwhatclassroomtestsaretesting?

• thatis...• Q3.Howcanclassroomassessmenthelpeducationalmeasurement?





• HowshouldBASinterfacewithstatetesting?• Whatarethenextsteps?

WhatshouldwewantfromanAssessmentSystem?

• Assessmentandaccountabilityinformationavailableandusefultoimportantaudiences:

Audiences• Classroom

– Teachers,Students,Parents• schoolsite

– otherteachersandspecialists,administrators• schooldistrict

– administrators• Stateandpublic

– SchoolBoard,legislators,StateDept.ofEducation,Public

Audiences,andwhattheyoughttoget

Audiences What they need/want What they get from What the State now they ought to get

classroom results useful for (a) individual student feedback SBAC, etc. (b) planning for individual progress SBAC, etc (c) planning for class progress SBAC, etc schools results useful for: (a) school, school district, State SBAC, etc school districts planning (b) school, school district, State SBAC, etc State and public accountability

WhattheyoughttogetFirst, some background. 1. Different sorts of assessment Assessment to assist learning only in classrooms Summative assessment of individuals some in classrooms, some from State Assessment to evaluate programs mainly State (or school districts, federal, etc.) 2. What should be the relationships among these different sorts of assessment? “...teachers’ goals for learning should be consistent with those of large-scale assessments and vice-versa... one challenge is to make stronger connections between the two so they work together to support a common set of learning goals.” (Knowing What Students Know, National Research Council, 2001, p. 41)

3. Assessment in education is a major industry State spends millions on it, testing companies make millions out of it. Assessment is one of the major components of every classroom. Not “State” assessments....teachers’ assessments. The instructional cycle....

Assessment Instruction

Curriculum

Assessmentis...

An activity that goes on at almost every minute of every school day. In every classroom, in every school in this State. An activity that is crucial to the success of every piece of instruction that every teacher carries out. This assessment enterprise dwarfs the assessment system of the State,

in time it takes, in effect on the students, in cost

in how it affects the practices of teachers, and hence in how it effects student learning What will teachers do when the have to comply with State tests that are inconsistent with what they do in their instruction?

Thus,therealquestionis...NOT

• howtoconstructasystemofStateassessmentsthatwillcarryouttheprogramevaluationtasksoftheState?

BUT• howtoconstructasystemthat

(a)helpsintheclassroomby(i)definingclassroomlearninggoals,(ii)isaresourceforclassroomassessmentand(iii)focusesinstruction,and

(b)helpschannelusefulinformationto(i)summativeindividualassessmentand(ii)programevaluationtasksoftheState

Howdowedothat?• (a)Needalearningframeworkthatallowscommunicationamongthedifferentaudiencesforassessmentinformation

• (b)Needmethodsofgatheringdatathatareacceptableandusefultoallaudiences

• (c)Needwaystoevaluate(score)responsesthatisconsistentwiththeframeworkin(a)

• (d)Needatechniqueofinterpretingdatathatallowsmeaningfulreportingtomultipleaudiences

Howdowedothat?• (a)Needalearningframeworkthatallowscommunicationamongthedifferentaudiencesforassessmentinformation

• (b)Needmethodsofgatheringdatathatareacceptableandusefultoallaudiences

• (c)Needwaystoevaluate(score)responsesthatisconsistentwiththeframeworkin(a)

• (d)Needatechniqueofinterpretingdatathatallowsmeaningfulreportingtomultipleaudiences

• AKA...4“BuildingBlocks”

TheBEARAssessmentSystem(BAS)

• HerbThier,LawrenceHallofScience– ScienceEducationforPublicUnderstanding(SEPUP)– Issues,EvidenceandYou(IEY)Curriculum

• Unhappythat:– TeacherslovetheIEYcurriculum,butnogainsonstandardizedtests

– Teachersdon’treallyknowwhethertheirstudentshavelearnedthe“stuff”(Despite“unittests”)

– Studentsdon’tknowwhattheyhavelearned– Lookingforthe“assessablemoment”

Image of a Curriculum/Learning Progression

Relating measurement dimensions to a curriculum?

SEPUP’sIEYCurriculum(circa1990)

• UnderstandingConcepts(UC)--understandingscientificconcepts(suchaspropertiesandinteractionsofmaterials,energy,orthresholds)inordertoapplytherelevantscientificconceptstothesolutionofproblems.

• DesigningandConductingInvestigations(DCI)--designingascientificexperiment,carryingthroughacompletescientificinvestigation,performinglaboratoryprocedurestocollectdata,recordingandorganizingdata,andanalyzingandinterpretingresultsofanexperiment.

• EvidenceandTradeoffs(ET)--identifyingobjectivescientificevidenceaswellasevaluatingtheadvantagesanddisadvantagesofdifferentpossiblesolutionstoaproblembasedontheavailableevidence.

• CommunicatingScientificInformation(CM)--organizingandpresentingresultsinawaythatisfreeoftechnicalerrorsandeffectivelycommunicateswiththechosenaudience.

From:Wilson&Sloane,2000

Category ScoreUsing Evidence:

Response uses objective reason(s) based onrelevant evidence to support choice.

Beyond correct4

Response accomplishes Level 3 AND goesbeyond in some significant way, such asquestioning or justifying the source, validity,and/or quantity of evidence.

Correct 3 Response provides major objective reasonsAND supports each with relevant & accurateevidence.

Partially complete (ii)--some objective reasons

and some evidence2

Response provides some objective reasonsAND some supporting evidence, BUT atleast one reason is missing and/or part of theevidence is incomplete.

Partially complete (i)--subjective and/orinaccurate reasons

1 Response provides only subjective reasons(opinions) for choice and/or uses inaccurateor irrelevant evidence from the activity.

Missing or irrelevant 0 No response; illegible response; responseoffers no reasons AND no evidence tosupport choice made.

No opportunity X Student had no opportunity to respond.

TheIEYEvidenceandTradeoffsconstructmap.

Direction of increasing sophistication in using evidence.

Students Responses to Items

No relevant response

Subjective or inaccurate evidence

Something important missing

All major points included

Goes beyond complete and correct

Direction of decreasing sophistication in using evidence.

Relating measurement dimensions to a curriculum?

Itemsdesign

YouareapublichealthofficialwhoworksintheWaterDepartment.Yoursupervisorhasaskedyoutorespondtothepublic'sconcernaboutwaterchlorinationatthenextCityCouncilmeeting.Prepareawrittenresponseexplainingtheissuesraisedinthenewspaperarticles.Besuretodiscusstheadvantagesanddisadvantagesofchlorinatingdrinkingwaterinyourresponse,andthenexplainyourrecommendationaboutwhetherthewatershouldbechlorinated.

“As an edjucated employee of the Grizzelyville water company, I am well aware of the controversy surrounding the topic of the chlorination of our drinking water. I have read the two articals regarding the pro’s and cons of chlorinated water. I have made an informed decision based on the evidence presented the articals entitled “The Peru Story” and “700 Extra People May bet Cancer in the US.” It is my recommendation that our towns water be chlorin treated. The risks of infecting our citizens with a bacterial diseease such as cholera would be inevitable if we drink nontreated water. Our town should learn from the country of Peru. The artical “The Peru Story” reads thousands of inocent people die of cholera epidemic. In just months 3,500 people were killed and more infected with the diease. On the other hand if we do in fact chlorine treat our drinking water a risk is posed. An increase in bladder and rectal cancer is directly related to drinking chlorinated water. Specifically 700 more people in the US may get cancer. However, the cholera risk far outweighs the cancer risk for 2 very important reasons. Many more people will be effected by cholera where as the chance of one of our citizens getting cancer due to the water would be very minimal. Also cholera is a spreading diease where as cancer is not. If our town was infected with cholera we could pass it on to millions of others. And so, after careful consideration it is my opion that the citizens of Grizzelyville drink chlorine treated water.”

Category ScoreUsing Evidence:

Response uses objective reason(s) based onrelevant evidence to support choice.

Beyond correct4

Response accomplishes Level 3 AND goesbeyond in some significant way, such asquestioning or justifying the source, validity,and/or quantity of evidence.

Correct 3 Response provides major objective reasonsAND supports each with relevant & accurateevidence.

Partially complete (ii)--some objective reasons

and some evidence2

Response provides some objective reasonsAND some supporting evidence, BUT atleast one reason is missing and/or part of theevidence is incomplete.

Partially complete (i)--subjective and/orinaccurate reasons

1 Response provides only subjective reasons(opinions) for choice and/or uses inaccurateor irrelevant evidence from the activity.

Missing or irrelevant 0 No response; illegible response; responseoffers no reasons AND no evidence tosupport choice made.

No opportunity X Student had no opportunity to respond.

TheIEYEvidenceandTradeoffsoutcomespace.

Astudent’sprofileontheIEYconstructs.

Astudent’slongitudinalprofileontheDCIconstruct.

GeneralStrategythe4“BuildingBlocks”...

• (a)Needaframeworkthatallowsteacherstointerpretdevelopmentintheconstruct– "Constructmap”

• (b)Needmethodsofgatheringdatathatareacceptableandusefultoallaudiences– "Itemsdesign”

• (c)Needawaytovaluewhatweseeinstudentwork– "Outcomespace”

• (d)Needatechniqueofinterpretingdatathatallowsmeaningfulreportingtomultipleaudiences– "Measurementmodel"

TheBearAssessmentSystem(2000AD)


Reliability&Validity


Construct Item Responses

Measurement Model

Outcome Space

Causality

Inference

AreflectivemodelFrom:Wilson,2005



• Whatare themostimportantdevelopmentsinBASsince2000?



Whatare themostimportantdevelopmentsinBASsince2000?

• QualitativeInnerTriangle• FeedbackLoops,etc.• OMC• Banding• DimensionalComparability• StandardSetting• LearningProgressions

1.QualitativeInnerTriangle

QualitativeInnerTriangle

2.FeedbackLoops,etc.

From:Brown&Wilson,2011.

3.OrderedMultipleChoice:OMCWhichisthebestexplanationforwhyitgetsdarkatnight?A.TheMoonblockstheSunatnight.[Level1response]B.TheEarthrotatesonitsaxisonceaday.[Level4response]C.TheSunmovesaroundtheEarthonceaday.[Level2response]D.TheEarthmovesaroundtheSunonceaday.[Level3response]

E.TheSunandMoonswitchplacestocreatenight.[Level2response]

item will change as student proficiency (θ) changes. Figure 6 gives an example ofan ICC for a hypothetical item analyzed with the Rasch Model. Note that only twocurves are plotted; one curve (X = 1) represents the correct answer on a multi-ple-choice item, and the other curve (X = 0), which is simply the inverse of the first,represents any incorrect response (i.e., a student who chooses any of the possibleMC distractors).

A problem with the Rasch Model as reflected by Figure 6 is that informationabout specific item responses is lost. One alternative approach, which takes this in-formation into account, is known as the Ordered Partition Model (OPM; Wilson,1992). Importantly, the OPM assumes that we have an a priori ordering of item re-sponse options, a point we will return to. The OPM is related to the Rasch Model,and is more formally described by Wilson (1992) as an extension of the PartialCredit Model (Masters, 1982). The advantage of using the OPM is that it becomespossible to plot a curve for each response possibility in an OMC item. We call thisan item options characteristic curve (IOCC). For an OMC item with five responseoptions, the IOCC plots five different curves, as in Figure 7, whereas the ICC plotsonly two (as in Figure 6).

We briefly describe the OPM more formally. Let Xni be a random variable thatrepresents the response of student n, with proficiency θn, to item i. Each item has Ki

possible response categories (k = 1, … , Ki) which are scored into Mi + 1 possiblescore levels (m = 0, … , Mi). The probability of observing a response in category kcan be written as

DIAGNOSTIC ASSESSMENT 51

FIGURE 7 OPM-based plot of IOCCs for Grade 5, ESS item 1.

From:Briggs,Alonzo,Schwab&Wilson,2006Seealso:Wilson&Adams,1992

4.Banding

• Findinglevelsincomplexassessments...

Whatwehopedtosee:

48

Whatwegot

49

NewECCWrightMaps

50

Strand-A Strand-B Strand-C

Modelling:CoS WrightMap

51

RevisedCoS Map| former&labels NL CoS1 CoS2 CoS3 CoS4| new&labels NL CoS1A CoS1B CoS2* CoS3*|| ErinQ2

4 | ClassQ2*| Range||||

3 ||

X || Range HeightQ2

X | Caff2,WaterX | ErinQ1X | HeightQ2MC

2 XX | ClassQ1 TempQ2MCXX | HeightQ2 HeightQ2MC Swim3

XXX | HeightQ2 Water,9TempQ2 TempQ2MCXXXX | FreeThrowXXXXX | TempQ2 Ball2

XXXXXX | Range Batt2,&Ball3XXXXX | Caff2 Kayla1

1 XXXXXX |XXXXXX | Swim1

XXXXXXXXX | GardC1,&Batt1XXXXXXXXX | FreeThrow ErinQ1,&Kayla1 Swim2

XXXXXXXXXX | Batt2XXXXXXXX |XXXXXXXX | Swim1 Tree2MC

0 XXXXXXXX | ErinQ2XXXXXXXX | Swim2,0Caff2 BallMean GardB1XXXXXXXX | Swim3XXXXXXX | ErinQ2 Max4MCXXXXXXX | ErinQ1 Tree1MCXXXXXX | ClassQ2

XXX | BallMode

;1 XXX |FreeThrow,0

Range,BallMean ClassQ1XXXX | Kayla1 GardMean,&BallMed

XX | BallMode Batt1XX | ClassQ2XX | BallMedX | GardMean,Ball3X |

;2 X | GardC1X |X | Water,Ball2

| GardMedX |

| GardMed;3 | GardB1

| Batt1,Batt2 GardMode|| Gardmode|||

5.Dimensionalcomparability

• StandardassumptioninIRT:– Meanofpersondistributionis0.0

or– Meanofitemdifficultiesis0.0

• Inmultidimensionalmodels,makesameassumptions(asinFactorAnalysis).

• Consequences?

MultidimensionalWrightMap

54

55

Delta'Dimensional'Alignment'(DDA)'•  Transforms'the'item'loca/ons'and'step'parameters'from'the'7:dimensional'analysis,'using'the'results'from'the'unidimensional'analysis'

•  Then,'re:run'7:dimensional'analysis'using'these'transformed'item'es/mates'and'step'parameters'as'anchored'values.'' 16 €

τ ij( transformed) = τ ij(multi)σ d (uni)

σ d (multi)

$

% &

'

( )

€

δ i(transformed ) = δ i(multi)σ d (uni)

σ d (multi)

$

% &

'

( ) + µd (uni)

€

δ ij = δ i +τ ij

MultidimensionalWrightMap

56

6.Standard-Setting

• Settingcut-pointsbetweenreportinglevels...

6.Construct-MappingSoftware

• Dynamicdisplayofitemmap• Displays,foranychosenproficiencylevel

– probabilityofpassingallmultiple-choiceitems– probabilityofattainingeverylevelonopen-endeditems

– expectedtotalscoreonmultiple-choicesection;expectedscoreoneachopen-endeditem

• Allowschoiceofweightsforitemtypes

ItemsGSE Multiple Choice WR 1 WR 2Scale Prob Prob Prob750740 2.5 .00730720710 1.5 .00700690 2.4 .00680670660 37 .31650640630 15 .35 1.4 .00620610 28 39 .39600590 27 .42580 38 .43 2.3 .07570 19 .45560550 43 .49540 34 45 48 .50 1.3 .47530 17 18 20 50 .52520 40 .53510 4 31 .55500 11 33 44 .57490 32 47 .58480 5 9 12 46 .59470 3 7 16 .61 2.2 .92460 6 10 29 .63 1.2 .52450 36 .65440 22 35 .66430 8 14 23 26 .67420 13 25 .69410 24 42 .70400 41 .71390 1 49 .73380 21 30 .74370360350340330 2 .79320310

Persons GSE ItemsFR Percentile Histogram Scale MC WR 1 WR 2

0 100.0 740 1.40 100.0 7350 100.0 730

42 100.0 < 7250 99.9 720 2.40 99.9 715 60 99.9 7100 99.9 7050 99.9 7000 99.9 6950 99.9 690

41 99.9 < 68563 99.7 < 68099 99.5 < 6750 99.2 670

83 99.2 < 665 1.323 98.9 < 66019 98.9 < 655

192 98.8 X 65066 98.2 < 645 2.389 98.0 < 640

256 97.7 XX 635182 96.9 X 630 5242 96.3 XX 625224 95.5 XX 620184 94.8 X 615203 94.2 XX 610266 93.5 XX 605276 92.7 XX 600308 91.8 XXX 595327 90.8 XXX 590413 89.7 XXXX 585 1.2433 88.4 XXXX 580301 87.0 XXX 575581 86.1 XXXXX 570 8475 84.2 XXXX 565598 82.7 XXXXX 560 4461 80.7 XXXX 555661 79.2 XXXXXX 550596 77.1 XXXXX 545491 75.2 XXXX 540 21655 73.6 XXXXXX 535 2.2916 71.5 XXXXXXXXX 530 52459 68.6 XXXX 525 38744 67.1 XXXXXXX 520 43 46857 64.7 XXXXXXXX 515 35996 61.9 XXXXXXXXX 510 12 22 24 41 47899 58.7 XXXXXXXX 505 37661 55.8 XXXXXX 500 10 20 29 36 45 50 1.1853 53.7 XXXXXXXX 495 26 30 40 42 3

1035 51.0 XXXXXXXXXX 490 5 14 25 48675 47.6 XXXXXX 485 23 33 34 44837 45.5 XXXXXXXX 480 39975 42.8 XXXXXXXXX 475 28967 39.6 XXXXXXXXX 470 51987 36.5 XXXXXXXXX 465 2.1

1316 33.3 XXXXXXXXXXXXX 460 27746 29.1 XXXXXXX 455936 26.7 XXXXXXXXX 450 7 11 49894 23.7 XXXXXXXX 445 15 16 19

1032 20.8 XXXXXXXXXX 440755 17.5 XXXXXXX 435 6 9 13491 15.1 XXXX 430 17 31 32 2656 13.5 XXXXXX 425701 11.4 XXXXXXX 420646 9.1 XXXXXX 415 4491 7.0 XXXX 410 3

Final Standards Map

Persons GSE ItemsFR Percentile Histogram Scale MC WR 1 WR 2

0 100.0 740 1.40 100.0 7350 100.0 730

42 100.0 < 7250 99.9 720 2.40 99.9 715 60 99.9 7100 99.9 7050 99.9 7000 99.9 6950 99.9 690

41 99.9 < 68563 99.7 < 68099 99.5 < 6750 99.2 670

83 99.2 < 665 1.323 98.9 < 66019 98.9 < 655

192 98.8 X 65066 98.2 < 645 2.389 98.0 < 640

256 97.7 XX 635182 96.9 X 630 5242 96.3 XX 625224 95.5 XX 620184 94.8 X 615203 94.2 XX 610266 93.5 XX 605276 92.7 XX 600308 91.8 XXX 595327 90.8 XXX 590413 89.7 XXXX 585 1.2433 88.4 XXXX 580301 87.0 XXX 575581 86.1 XXXXX 570 8475 84.2 XXXX 565598 82.7 XXXXX 560 4461 80.7 XXXX 555661 79.2 XXXXXX 550596 77.1 XXXXX 545491 75.2 XXXX 540 21655 73.6 XXXXXX 535 2.2916 71.5 XXXXXXXXX 530 52459 68.6 XXXX 525 38744 67.1 XXXXXXX 520 43 46857 64.7 XXXXXXXX 515 35996 61.9 XXXXXXXXX 510 12 22 24 41 47899 58.7 XXXXXXXX 505 37661 55.8 XXXXXX 500 10 20 29 36 45 50 1.1853 53.7 XXXXXXXX 495 26 30 40 42 3

1035 51.0 XXXXXXXXXX 490 5 14 25 48675 47.6 XXXXXX 485 23 33 34 44837 45.5 XXXXXXXX 480 39975 42.8 XXXXXXXXX 475 28967 39.6 XXXXXXXXX 470 51987 36.5 XXXXXXXXX 465 2.1

1316 33.3 XXXXXXXXXXXXX 460 27746 29.1 XXXXXXX 455936 26.7 XXXXXXXXX 450 7 11 49894 23.7 XXXXXXXX 445 15 16 19

1032 20.8 XXXXXXXXXX 440755 17.5 XXXXXXX 435 6 9 13491 15.1 XXXX 430 17 31 32 2656 13.5 XXXXXX 425701 11.4 XXXXXXX 420646 9.1 XXXXXX 415 4491 7.0 XXXX 410 3

Exampleofresulting"mappingmatrix"

Multiple choice Written response problem scoresnumber correct 0 1 2 3 4

0 1 1 1 1 11 1 1 1 1 22 1 1 1 1 23 1 1 1 1 24 1 1 1 2 25 1 1 1 2 26 1 1 2 2 37 1 1 2 2 38 1 1 2 2 39 1 2 2 3 3

10 1 2 2 3 311 2 2 2 3 412 2 2 3 3 413 2 2 3 4 414 2 2 3 4 415 2 2 3 4 416 2 3 3 4 417 2 3 3 4 518 2 3 3 4 519 3 3 4 4 520 3 3 4 5 521 3 3 4 5 622 3 4 4 5 623 3 4 5 5 624 3 4 5 6 625 3 4 5 6 626 4 4 5 6 6

7.LearningProgressions

• Learningprogressionsaredescriptionsofthesuccessivelymoresophisticatedwaysofthinkingaboutanimportantdomainofknowledgeandpracticethatcanfollowoneanotheraschildrenlearnaboutandinvestigateatopicoverabroadspanoftime.Theyarecruciallydependentoninstructionalpracticesiftheyaretooccur.(Corcoran,Mosher,&Rogat,2009,p.37)

Image of a Learning Progression

From:Wilson,2009

Image of a Construct Map

One Possible Relationship:the levels of the learning progression are levels of several

construct maps

Another possible relationship:the levels are staggered

Possibilityofqualitativechange

69

An extreme case

A “within-levels” case

Level What the Student Knows

A

AAAAAAAaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

BBBBBBB

Bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

C

CCCCCCCCccccccccccccccccccccccccccccccCccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc

D E

DDDDDdddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd

EEEEEeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

FFFFFFFFFF

Ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff


A


BBBBBBB


C


D E



FFFFFFFFFF



A


BBBBBBB


C


D E



FFFFFFFFFF



A


BBBBBBB


C


D E



FFFFFFFFFF



A


BBBBBBB


C


D E



FFFFFFFFFF



A


BBBBBBB


C


D E



FFFFFFFFFF



A


BBBBBBB


C


D E



FFFFFFFFFF


A relationship with complex dependencies:

A more complicated relationship

WhataresomeimportantexamplesofBASassessments?

• SEPUP(withHerbThier,UCBerkeley)• ADM(withRichLehrer&LeonaSchauble,VanderbiltUniversity)

• DRDP(withPeterMangione,WestEd)• ScientificReasoningProject(KarenDraney,UCBerkeley,withBrianReiser ,NorthwesternUniversity)

• CarbonCycleProject(KarenDraney,UCBerkeley,withAndyAndersen,MichiganStateUniversity)

• LPS– AtomicMolecularModel(withPaulBlack,KingsCollege)– Argumentation(withJonathanOsborne,StanfordUniversity

WhataresomeimportantexamplesofBASassessments?

Out-of-normalrangeapplications

• Fidelitymeasurement– withRichLehrerand

• Linkedconstructs/SCMs– withRichLehrer,– withKathleenMetz(UCBerkeley)

StructuredConstructModels(SCMs)

• Thelearningprogressioninvolvescross-dimensionallinks:

77

!

Level!4!

Level!3!

Level!2!

Level!3!

Level!2!

Level!1!

Level!4!

Level!1!

Requirement!Variable:!R!

Target!Variable:!T!

Relationshipsbetweendimensions

• Inordertogo:– fromlevel2 oftargetvariableT– tolevel3 oftargetvariableT

• Youneedtobeat– levelt-1ofvariableT

and– Level2 ofvariableR

78

!

Level!4!

Level!3!

Level!2!

Level!3!

Level!2!

Level!1!

Level!4!

Level!1!

Requirement!Variable:!R!

Target!Variable:!T!

NCME2016Symposium• Wilson,M.(2016,April).Introductiontotheconceptofastructured

constructsmodel(SCM). PaperpresentedattheNCMEannualmeeting,Washington,DC.

• Irribarra,D.T.,&Diakow,R.(2016,April).Modelingstructuredconstructsasnon-symmetricrelationsbetweenordinallatentvariables.PaperpresentedattheNCMEannualmeeting,Washington,DC.– Latentclassmodeling

• Choi,I-H.,&Wilson,M.(2016,April).Astructuredconstructsmodelforcontinuouslatenttraitswithdiscontinuityparameters.PaperpresentedattheNCMEannualmeeting,Washington,DC.– Continuousmodelingwithfixedcut-points

• Shin,H-J.&Wilson,M.(2016,April).Astructuredconstructsmodelbasedonchange-pointanalysis.PaperpresentedattheNCMEannualmeeting,Washington,DC.– Continuousmodelingwithestimatedcut-points

ThefullADMview…

80

• See:Wilson,M.,&Santelices,M.V.(inpress).Weaknessesofthetraditionalviewofstandardsettingandasuggestedalternative.In,Blomeke,S.&Gustaffson,J-E.(Eds.),StandardSetting:InternationalStateofResearchandPracticesintheNordicCountries.Dordrecht,Germany:Springer



• Whatare themostimportantdevelopmentsin• BASsince2000?• WhataresomeimportantexamplesofBASassessments?


BASChallengesandOportunities:ForMeasurement

• “Standard”challenges– Definingvariableswell– Creatingitemsinadesign-specificway– Developingsoundcoding/scoringsystems– Applyinguni- andmultidimensionalmodelstointerrogatethedataappropriately

• Should-be“standard”challenges– Makingreportsusefulforteachers– Helpingteachersdesign/adaptassessments

BASChallengesandOpportunities:ForMeasurement

• “Non-Standard”challenges– Incorporatingthemetricinmultidimensionalmodels

– Representing/modelling “links”acrossdimensions– Developingnewmodelsthatwell-representthelatentcontinuumand/orlatentclassesinLPs

– Representing/modelinglongitudinalchangeacrossaLP

BASChallengesandOpportunities:FromMeasurement

• Tothedevelopmentofcurricula(C&I,policymakers,etc.)– Readiness(andexpectancy)tobeprovenwrongbyempiricalobservations

– ThinkingofLearningProgressions/Curriculaashypotheses• Tofidelitymeasurement

– --ditto--– Providesawaytotranscend“theobvious”

• Toprofessionaldevelopment– (followingonfromtheprevious)providesanewwaytolinkteacherpracticestolarge-scaleassessmentandaccountability

References• Briggs,D.,Alonzo,A.,Schwab,C.,&Wilson,M.(2006).Diagnosticassessmentwith

orderedmultiple-choiceitems.EducationalAssessment,11(1),33-63.• Brown,N.J.S.,&Wilson,M.(2011).Modelofcognition:Themissingcornerstone

ofassessment.EducationalPsychologyReview,23(2),221-234.• Corcoran,T.,Mosher,F.A.,&Rogat,A.(2009,May).Learningprogressionsin

science:Anevidence- basedapproachtoreform(CPREResearchReport#RR-63).Philadelphia,PA:ConsortiumforPolicyResearchinEducation.

• Wilson,M.(1992)Theorderedpartitionmodel:Anextensionofthepartialcreditmodel.AppliedPsychologicalMeasurement,16,309–325.

• Wilson,M.(2009).Measuringprogressions:Assessmentstructuresunderlyingalearningprogression.JournalforResearchinScienceTeaching,46(6),716-730.

• Wilson,M.(2010,March).DiscussantNotes.Discussant atasymposiumon“CognitionandValidInferencesaboutStudentAchievement” attheannualmeetingoftheAmericanEducationalResearchAssociation,Denver,Colorado.

• Wilson,M.&Sloane,K.(2000).Fromprinciplestopractice:Anembeddedassessmentsystem.AppliedMeasurementinEducation,13(2),181-208.

Measurement in the context of formative classroom ......• Wilson, M. (1989). Saltus: A...

Documents

Transcript of Measurement in the context of formative classroom ......• Wilson, M. (1989). Saltus: A...