A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages...
Transcript of A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages...
![Page 1: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/1.jpg)
AProgrammaticApproachtoAssessmentADEA/ADEEMeeting:ShapingtheFutureofDentalEducation8-9May,London,UK
CeesvanderVleutenMaastrichtUniversityTheNetherlandswww.maastrichtuniversity.nl/shewww.ceesvandervleuten.comWith gratitude to the supportofLiftUPP!
![Page 2: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/2.jpg)
![Page 3: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/3.jpg)
Overview
• Frompracticetoresearch• Fromresearchtotheory• Fromtheorytopractice• Conclusions
![Page 4: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/4.jpg)
TheToolbox
• MCQ,MEQ,OEQ,SIMP,Write-ins,Key Feature,Progress test,PMP,SCT,Viva,Longcase,Shortcase,OSCE,OSPE,DOCEE,SP-based test,Videoassessment,MSF,Mini-CEX,DOPS,assessmentcenter,self-assessment,peerassessment,incognitoSPs,portfolio………….
![Page 5: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/5.jpg)
Knows
Shows how
Knows how
Does
Knows Fact-oriented assessment:MCQ, write-ins, oral…..
Knows howScenario or case-based assessment:MCQ, write-ins, oral…..
Shows howPerformance assessment in vitro:Assessment centers, OSCE…..
DoesPerformance assessment in vivo:In situ performance assessment, 360◌,۫ Peer assesment
Thewayweclimbed......
![Page 6: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/6.jpg)
Validity
Characteristicsofinstruments
Reliability
Educationalimpact
Acceptability
Cost
Validity
Reliability
Educationalimpact
![Page 7: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/7.jpg)
Validity:whatareweassessing?
• Curriculahavechangedfromaninputorientationtoanoutputorientation
• Wewentfromhaphazardlearningtointegratedlearningobjectives,toendobjectives,andnowto(generic)competencies
• Wewentfromteacherorientedprogramstolearningoriented,self-directedprograms
![Page 8: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/8.jpg)
Competency-frameworks
CanMeds§ Medical expert§ Communicator§ Collaborator§ Manager§ Health advocate§ Scholar§ Professional
ACGMEn Medical knowledgen Patient caren Practice-based learning
& improvementn Interpersonal and
communication skillsn Professionalismn Systems-based practice
GMCn Good clinical caren Relationships with
patients and familiesn Working with
colleaguesn Managing the
workplacen Social responsibility
and accountabilityn Professionalism
![Page 9: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/9.jpg)
Knows
Shows how
Knows how
Does
Knows
Knows how
Shows how
Does
Validity:whatareweassessing?
Standardizedassessment(fairlyestablished)
Unstandardizedassessment(emerging)
![Page 10: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/10.jpg)
Messagesfromvalidityresearch
• Thereisnomagicbullet;weneedamixtureofmethodstocoverthecompetencypyramid
• WeneedBOTHstandardizedandnon-standardizedassessmentmethods
• Forstandardizedassessmentqualitycontrolaroundtestdevelopmentandadministrationisvital
• Forunstandardizedassessmenttheusers(thepeople)arevital.
![Page 11: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/11.jpg)
Methodreliabilityasafunctionoftestingtime
TestingTimeinHours
1
2
4
8
MCQ1
0.62
0.77
0.87
0.93
Case-BasedShortEssay2
0.68
0.81
0.89
0.94
PMP1
0.36
0.53
0.69
0.82
OralExam3
0.50
0.67
0.80
0.89
LongCase4
0.60
0.75
0.86
0.92
OSCE5
0.54
0.70
0.82
0.90
PracticeVideoAssess-ment7
0.62
0.77
0.87
0.93
1Norcinietal.,19852Stalenhoef-Hallingetal.,19903Swanson,1987
4Wassetal.,20015VanderVleuten,19886Norcinietal.,1999
In-cognitoSPs8
0.61
0.76
0.86
0.93
MiniCEX6
0.73
0.84
0.92
0.96
7Ram etal.,19998Gorter,2002
![Page 12: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/12.jpg)
Reliabilityasafunctionofsamplesize(Moonen etal.,2013)
0.65
0.7
0.75
0.8
0.85
0.9
4 5 6 7 8 9 10 11 12G=0.80 KPB
Mini-CEX
![Page 13: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/13.jpg)
0.65
0.7
0.75
0.8
0.85
0.9
4 5 6 7 8 9 10 11 12
G=0.80 KPB OSATSMini-CEX OSATS
Reliabilityasafunctionofsamplesize(Moonen etal.,2013)
![Page 14: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/14.jpg)
0.65
0.7
0.75
0.8
0.85
0.9
4 5 6 7 8 9 10 11 12
Mini-CEX OSATS MSF
Reliabilityasafunctionofsamplesize(Moonen etal.,2013)
![Page 15: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/15.jpg)
Effectofaggregationacrossmethods(Moonen etal.,2013)
Method
Mini-CEXOSATSMSF
Sampleneeded
whenusedasstand-alone
899
Sampleneeded
whenusedasacomposite
562
![Page 16: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/16.jpg)
ResultatenBetrouwbaarheidPerinstrumentAllejaren:
8KPB,9OSATS,9MSFEerstejaar:
6KBP,6OSATS,6MSF
GezamenlijkAllejaren:
7KPB,8OSATS,1MSFof5KPB,6OSATS,2MSF
Eerstejaar:5KBP,6OSATS,1MSF
![Page 17: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/17.jpg)
Messagesfromreliabilityresearch
• Acceptablereliabilityisonlyachievedwithlargesamplesoftestelements(contexts,cases)andassessors
• Nomethodisinherentlybetterthananyother(thatincludesthenewones!)
• ObjectivityisNOTequaltoreliability• Manysubjectivejudgmentsareprettyreproducible/reliable.
![Page 18: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/18.jpg)
Educationalimpact:Howdoesassessmentdrivelearning?
• Relationshipiscomplex(cf.Cilliers,2011,2012)• Butimpactisoftenverynegative
– Poorlearningstyles– Gradeculture(gradehunting,competitiveness)– Gradeinflation(e.g.intheworkplace)
• AlotofREDUCTIONISM!– Littlefeedback(gradeispoorestformoffeedbackonecanget)– Non-alignmentwithcurriculargoals– Non-meaningfulaggregationofassessmentinformation– Fewlongitudinalelements– Tick-boxexercises(OSCEs,logbooks,work-basedassessment).
![Page 19: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/19.jpg)
![Page 20: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/20.jpg)
• Alllearnersconstructknowledgefromaninnerscaffoldingoftheirindividualandsocialexperiences,emotions,will,aptitudes,beliefs,values,self-awareness,purpose,andmore...ifyouarelearning…..,whatyouunderstandisdeterminedbyhowyouunderstandthings,whoyouare,andwhatyoualreadyknow.
PeterSenge,DirectoroftheCenter forOrganizationalLearningatMIT (ascitedinvanRyn etal.,2014)
![Page 21: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/21.jpg)
Messageslearningimpactresearch
• Noassessmentwithout(meaningful)feedback
• Narrativefeedbackhasalotmoreimpactoncomplexskillsthanscores
• Provisionoffeedbackisnotenough(feedbackisadialogue)
• Longitudinalassessmentisneeded.
![Page 22: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/22.jpg)
Overview
• Frompracticetoresearch
• Fromresearchtotheory• Fromtheorytopractice• Conclusions
![Page 23: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/23.jpg)
Limitationsofthesingle-methodapproach
• Nosinglemethodcandoitall• Eachindividualmethodhas(significant)limitations
• Eachsinglemethodisaconsiderablecompromise onreliability,validity,educationalimpact
![Page 24: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/24.jpg)
Implications
• Validity:amultitudeofmethodsneeded• Reliability: alotof(combined)informationisneeded
• Learningimpact: assessmentshouldprovide(longitudinal)meaningfulinformationforlearning
Programmaticassessment
![Page 25: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/25.jpg)
Programmaticassessment
• Acurriculumisagoodmetaphor;inaprogramofassessment:– Elementsareplanned,arranged,coordinated– Issystematicallyevaluatedandreformed
• Buthow?(theliteratureprovidesextremelylittlesupport!)
![Page 26: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/26.jpg)
Programmaticassessment
• Dijkstraetal2012:73genericguidelines
• To be done:– Further validation– Afeasible (self-assessment)instrument
• ASPIREassessmentcriteria
![Page 27: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/27.jpg)
Buildingblocksforprogrammaticassessment1
• Everyassessmentisbutonedatapoint(Δ)• Everydatapointisoptimizedforlearning
– Informationrich(quantitative,qualitative)– Meaningful– Variationinformat
• Summativeversusformativeisreplacedbyacontinuumofstakes(stakes)
• Ndatapointsareproportionallyrelatedtothestakesofthedecisiontobetaken.
![Page 28: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/28.jpg)
Continuumofstakes,numberofdatapointandtheirfunction
Nostake
Very highstake
OneDatapoint:
• Focusedoninformation
• Feedbackoriented
• Notdecisionoriented
Intermediateprogressdecisions:
• Moredatapointsneeded
• Focusondiagnosis,remediation,prediction
Finaldecisionsonpromotionorselection:
• Manydatapointsneeded• Focusedona(non-
surprising)heavydecision
![Page 29: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/29.jpg)
Assessmentinformationaspixels
![Page 30: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/30.jpg)
Classicalapproachtoaggregation
Method1toassessskillA Σ
Method2toassessskillB Σ
Σ
Σ
Method3toassessskillC
Method4toassessskillC
![Page 31: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/31.jpg)
Moremeaningfulaggregation
Method1
Σ
Method2
Σ
Method3
Σ
Method4
Σ
SkillA
SkillBB
SkillC
SkillD
![Page 32: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/32.jpg)
![Page 33: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/33.jpg)
![Page 34: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/34.jpg)
![Page 35: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/35.jpg)
Overview
• Frompracticetoresearch• Fromresearchtotheory
• Fromtheorytopractice• Conclusions
![Page 36: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/36.jpg)
Fromtheorybacktopractice
• Existingbestpractices:– VeterinaryeducationUtrecht– ClevelandLearnerClinic,Cleveland,Ohio
– DutchspecialtytraininginGeneralPractice
– McMasterModularAssessmentPrograminEmergencyMedicine
– GraduateentryprogramMaastricht
![Page 37: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/37.jpg)
Fromtheorybacktopractice
• Existingbestpractices:– VeterinaryeducationUtrecht– ClevelandLearnerClinic,Cleveland,Ohio
– DutchspecialtytraininginGeneralPractice
– McMasterModularAssessmentPrograminEmergencyMedicine
– GraduateentryprogramMaastricht
![Page 38: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/38.jpg)
Fromtheorybacktopractice
• Existingbestpractices:– VeterinaryeducationUtrecht– ClevelandLearnerClinic,Cleveland,Ohio
– DutchspecialtytraininginGeneralPractice
– McMasterModularAssessmentPrograminEmergencyMedicine
– GraduateentryprogramMaastricht
![Page 39: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/39.jpg)
Physician-clinical investigator program
• 4 year graduate entry program• Competency-based (Canmeds) with emphasis on
research• PBL program
– Year 1: classic PBL– Year 2: real patient PBL– Year 3: clerkship rotations– Year 4: participation in research and health care
• High expectations of students: in terms of motivation, promotion of excellence, self-directedness
![Page 40: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/40.jpg)
The assessment program• Assessment in Modules: assignments, presentations, end-examination,
etc.• Longitudinal assessment: assignments, reviews, projects, progress
tests, evaluation of professional behavior, etc.• All assessment is informative and low stake formative• The portfolio is central instrument
Module-overstijgendetoetsingvanprofessioneelgedrag
Module2 Module3 Module4Module1
PT 1 PT2 PT 3 PT 4
Longitudinal Module exceeding assessment of knowledge, skills and professional behavior
portfolioCounselormeeting
Counselormeeting
Counselormeeting
Counselormeeting
Module exceeding assessment of knowledge in Progress Test
![Page 41: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/41.jpg)
Longitudinal total test scores across 12 measurement moments and predicted future performance
![Page 42: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/42.jpg)
Maastricht Electronic portfolio(ePass)
Comparisonbetween the score of the student and the average score of his/her peers.
![Page 43: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/43.jpg)
Every blue dot corresponds to an assessment form included in the portfolio.
Maastricht Electronic portfolio (ePass)
![Page 44: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/44.jpg)
![Page 45: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/45.jpg)
Coaching by counselors• Coaching is essential for successful use of reflective
learning skills • Counselor gives advice/comments (whether asked or not)• He/she counsels if choices have to be made• He/she guards and discusses study progress and
development of competencies
![Page 46: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/46.jpg)
Decision-making by committee• Committee of counselors and externals• Decision is based on portfolio information & counselor
recommendation, competency standards• Deliberation is proportional to clarity of information• Decisions are justified when needed; remediation
recommendation may be provided
![Page 47: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/47.jpg)
Strategytoestablishtrustworthiness Criteria
PotentialAssessmentStrategy(sample)
Credibility Prolongedengagement Trainingofexaminers
Triangulation Tailoredvolumeofexpertjudgmentbasedoncertaintyofinformation
Peerexamination Benchmarkingexaminers
Memberchecking Incorporate learnerview
Structuralcoherence Scrutinyofcommitteeinconsistencies
Transferability Timesampling Judgment basedonbroadsampleofdatapoints
Thickdescription Justifydecisions
Dependability Stepwisereplication Usemultipleassessorswhohavecredibility
Confirmability Audit Givelearnersthepossibilitytoappealtotheassessmentdecision
![Page 48: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/48.jpg)
Progress test embedded in programmatic assesssment – use of information and feedback
to selfdirect learning
![Page 49: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/49.jpg)
Overview
• Frompracticetoresearch• Fromresearchtotheory• Fromtheorytopractice
• Conclusions
![Page 50: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/50.jpg)
Conclusions1:Thewayforward• Wehavetostopthinkingintermsofindividualassessmentmethods
• Asystematicandprogrammaticapproachisneeded,longitudinallyoriented
• Everymethodofassessmentmaybefunctional(oldandnew;standardizedandunstandardized)
• Professionaljudgmentisimperative(similartoclinicalpractice)
• Subjectivityisdealtwiththroughsamplingandproceduralbiasreductionmethods(notwithstandardizationorobjectification).
![Page 51: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/51.jpg)
Conclusions 2:Thewayforward
• Theprogrammaticapproachtoassessmentoptimizes:– Thelearningfunction(throughinformationrichness)
– Thepass/faildecisionfunction(throughthecombinationofrichinformation)
![Page 52: A Programmatic Approach to Assessmentshapingdentaleducation.org/meetings/london2017/... · Messages from validity research • There is no magic bullet; we need a mixture of methods](https://reader033.fdocuments.us/reader033/viewer/2022050423/5f927d81b0790f75c31a5c9c/html5/thumbnails/52.jpg)
Furtherreading:www.ceesvandervleuten.com