0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’
Transcript of 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’
!"#$%&'()*)"'+#%&,)-()+.'
/%+0)"'12345'67)+$8%+9'
():8%;'.<)'=8>)"'.)+.'
Phil SmithConsultant Neurologist and Sub-Dean for Assessments
Cardiff
Learning outcomes
To appreciate the characteristics of high quality SBA questions (statistical and question construction)
To discuss ways to construct better ‘difficult-to-write’ SBAs (sociology, professionalism, ethics)
Learning outcomes
To appreciate the characteristics of high quality SBA questions (statistical and question construction)
To discuss ways to construct better ‘difficult-to-write’ SBAs (sociology, professionalism, ethics)
Discriminationscore on
this question
score on this test upper
quartile
lower
quartile
Questions must discriminate in favour of the best candidates
Distractor efficiencyThree-item questions performed equally well to four-item questions
Tarrant M, Ware J. A comparison of the psychometric properties of three-
and four-option multiple-choice questions in nursing assessments. Nurse
Educ Today 2010; 30: 539-43.
Reducing from 5 items to 4 or 3. Pass rates and reliability remained
constant. Pass rates increased by 1% (for 4-item) and 3% (for 3-item)
questions
Kilgour JM, Tayyaba S. An investigation into the optimal number of
distractors in single-best answer exams. Adv Health Sci Educ Theory Pract.
2016;21:571-85.
It’s fine to have questions that work with only 2 or 3 distractors
Learning outcomes
To appreciate the characteristics of high quality SBA questions (statistical and question construction)
To discuss ways to construct better ‘difficult-to-write’ SBAs (sociology, professionalism, ethics)
Taken as read
• Has a scenario: tests application of knowledge
• Question is positive and expresses uncertainty
• Passes the cover tests: traditional and science
Question construction
Science cover test!"#$%&'()%*+,"-(."/012"3'.')(+04',"/'(5.'44"2(4"("60*74&"1(5'."8)*-"204"9:(,)0;'74"-:4;+'<"
/20;2"04"'=(-0.',":40.3"'+';1)*."-0;)*4;*7&>"?2'"7(12*+*3041")'7*)14"(."(6.*)-(+"8*)-(@*."*8"
12'"A*+30"(77()(1:4>"""
B2(1"04"12'"-(0.")*+'"*8"12'"A*+30"(77()(1:4C"
!>";*.1)*+4"(-0.*"(;0,"6)'(5,*/."
D>";*.1)*+4"+070,"6)'(5,*/."
E>";*.1)*+4"7)*1'0."7)*;'440.3"
F>"0.0@(1'4"7)*1'0."(44'-6+&"
G>"0.0@(1'4"7)*1'0."6)'(5,*/."
!.4/')"E<";*.1)*+4"7)*1'0."7)*;'440.3"
!!!!!!""######$$$$$$$%%%&&&&&&'''''''(((((())))%%%%*******++++++,,,,,,"------((((((......""//////000001111112222222""""333333'''''......''''''''))))))((((((
///////22222220000;;;;;22222222"000000444444""''''''========((((((------0000000......'''''''',,,,,,"""::::::44444400000......33333333""""'''''''''++++'''''';;;;;;;11111))))
111111222222''''''"AAAAAA******+++++33333300000""""((((((777777777777(((((())))))(((((((111111:::::::444444>>>>>"""""
((((+++++0000044444'''''',,,,,,,"""///////''''''((((((555555.......'''''444444444444""22222222((((((44444"""((((((""6666660000******777777444444&&&&&&&""11111111((((((
))))))******......."------000000;;;;;;;;))))))******444444;;;;;;;;******777777&&&&&&>>>""????????222222''''''""777777((((((1111112222222******+++++++******3333333330000044444411111
55555''''''......"8888888)))))*******-------""22222200000000444444"999999::::::((((((,,,,,,))))))000000;;;;;;''''''7777744444444""""----::::::4444;;;;;;;;++++++'''''
111111""))))))'''''''7777777****))))))))111111444444"((((((........""((((((666666......********))))))------((((((++++++""888888*******))))))---------((((((@@@@@@@@@*******
''''<<<<<"
****......""******888888""
!"#$%&'()%*+,"-(."/012"3'.')(+04',"/'(5.'44"2(4"(."(6.*)-(+01&"*8"12'"A*+30"(77()(1:4<"
0,'.@H',"0."("-:4;+'"60*74&>"
B2(1";'++:+()"8:.;@*."04"-*41"+05'+&"1*"6'"(6.*)-(+C"
!>"(-0.*"(;0,"6)'(5,*/."
D>"+070,"6)'(5,*/."
E>"7)*1'0."(44'-6+&"
F>"7)*1'0."6)'(5,*/."
G>"7)*1'0."7)*;'440.3"
!.4/')"G<"7)*1'0."7)*;'440.3
Science cover test
• Minimising construct-irrelevance
consistent precise additional words relate to an option
• Testwise traps
convergence strategyquestion interdependence
Writing styleImproving reliability
Testwise candidates
A. CT scan of abdomen and pelvis
B. CT scan of thorax, abdomen and pelvis
C. gallium scan of whole body
D. positron-emission tomography scan of abdomen
E. ultrasound scan of testes and pelvis
What is the best next investigation?
• Authenticity
realistic scenario
presenting complaint in first line data not summary
• Best options all plausible
in same domain includes best answer
• Best practiceevidence-based practice good practice
non-discriminatory language
Question constructionImproving validity
Learning outcomes
To appreciate the characteristics of high quality SBA questions (statistical and question construction)
To discuss ways to construct better ‘difficult-to-write’ SBAs (sociology, professionalism, ethics)
Is it worth it??0I."J!<"K0.."AL<"L;M(;2+(."NE>"GO(+:(@.3"7)*8'440*.(+04-"
0."-',0;(+":.,')3)(,:(1'4":40.3"4'+';1',")'47*.4'"
9:'4@*.4P"H.,0.34"8)*-"(."01'-")'47*.4'"-*,'++0.3"41:,&>"
!"#$"%&'()*$+&,()-./"$QRRSRRPT#>
“Professionalism SBAs show poor psychometrics compared to anatomy questions, relatively poor psychometric properties, and lack associations with other
constructs associated with undergraduate professional behaviour correlation to conscientiousness index.”
Advises not to include as SBAs
Need more care to write