Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.
-
Upload
earl-hillen -
Category
Documents
-
view
217 -
download
0
Transcript of Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.
![Page 1: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/1.jpg)
Blueprint analysis and feedback based on the Progress Test
Lee CoombesJeff Stratford
![Page 2: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/2.jpg)
Progress Test
• 125 MCQ items, 4 times a year• Don’t Know option (negative marking)• 1 formative test, 19 summative• All students take the same test• Set at the level of a newly qualified doctor
![Page 3: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/3.jpg)
Progress Test
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0PT
01PT
02PT
03PT
04PT
05PT
06PT
07PT
08PT
09PT
10PT
11PT
12PT
13PT
14PT
15PT
16PT
17PT
18PT
19PT
20PT
21PT
22PT
23PT
24PT
25PT
26PT
27PT
28PT
29PT
30PT
31PT
32PT
33PT
34PT
35PT
36PT
37PT
38PT
39PT
40PT
41PT
42PT
43PT
44PT
45
Perc
enta
ge
Student Progress02/03 Cohort03/04 Cohort04/05 Cohort05/06 Cohort06/07 Cohort07/08 Cohort08/09 Cohort09/10 Cohort10/11 Cohort11/12 Cohort12/13 Cohort
![Page 4: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/4.jpg)
Student Feedback
![Page 5: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/5.jpg)
Blueprinting
• Based on the GMC PLAB blueprint• 17 domains, 17 Skills• Classified by main question theme• Some areas have more questions than others• Aim has been to provide detailed feedback on
each area for students• Both single test and aggregated yearly average
![Page 6: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/6.jpg)
Student Feedback
• ATLAS
![Page 7: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/7.jpg)
Questions
• How can we use the information?– Provides valuable feedback
• Differences in gender/ethnic groups?– We’ve previously seen gender and ethnicity
differences in the data– Are these general, or specific to certain areas?
![Page 8: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/8.jpg)
Methods
• Tests covering 7 year period – 2006/07 to 2012/13
• Examining the 20 tests students take• Different tests are grouped together• No adjustment for test difficulty• # items varies in each test/skill/domain
![Page 9: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/9.jpg)
PCMD Progress Test
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Applied Medical Knowledge ScoresScoreCorrectIncorrectDon't Know
![Page 10: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/10.jpg)
Gender
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Applied Medical Knowledge ScoresFemale ScoreFemale CorrectFemale IncorrectFemale Don't KnowMale ScoreMale CorrectMale IncorrectMale Don't Know
![Page 11: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/11.jpg)
Gender
Domain: Neurological Skill: Pathology
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
NeurologicalScore (F)Correct (F)Incorrect (F)Don't Know (F)Score (M)Correct (M)Incorrect (M)Don't Know (M)
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
SBM PathologyScore (F)Correct (F)Incorrect (F)Don't Know (F)Score (M)Correct (M)Incorrect (M)Don't Know (M)
![Page 12: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/12.jpg)
Gender
Domain: Women's Health Skill: Diagnosis
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Womens HealthScore (F)Correct (F)Incorrect (F)Don't Know (F)Score (M)Correct (M)Incorrect (M)Don't Know (M)
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
DiagnosisScore (F)Correct (F)Incorrect (F)Don't Know (F)Score (M)Correct (M)Incorrect (M)Don't Know (M)
![Page 13: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/13.jpg)
Ethnicity
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Applied Medical Knowledge ScoresBME ScoreBME CorrectBME IncorrectBME Don't KnowWhite ScoreWhite CorrectWhite IncorrectWhite Don't Know
![Page 14: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/14.jpg)
Ethnicity
Domain: Neurological Domain: Digestive
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
NeurologicalScore (BME)Correct (BME)Incorrect (BME)Don't Know (BME)Score (W)Correct (W)Incorrect (W)Don't Know (W)
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
DigestiveScore (BME)Correct (BME)Incorrect (BME)Don't Know (BME)Score (W)Correct (W)Incorrect (W)Don't Know (W)
![Page 15: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/15.jpg)
Ethnicity
Skill: Management Plan Skill: Diagnosis
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Management PlanScore (BME)Correct (BME)Incorrect (BME)Don't Know (BME)Score (W)Correct (W)Incorrect (W)Don't Know (W)
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
DiagnosisScore (BME)Correct (BME)Incorrect (BME)Don't Know (BME)Score (W)Correct (W)Incorrect (W)Don't Know (W)
![Page 16: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/16.jpg)
Conclusions
• Gender differences are skill/domain specific• Ethnic difference appears more general• Caused by the test, or is the test accurately
measuring pre-existing differences?– Evidence of similar differences elsewhere
• What role do other demographics play?– Home\EU\Other– Entry Type (Graduate\Adult)
![Page 17: Blueprint analysis and feedback based on the Progress Test Lee Coombes Jeff Stratford.](https://reader035.fdocuments.us/reader035/viewer/2022062417/551738aa550346fe558b6042/html5/thumbnails/17.jpg)
Thank You
Lee CoombesJeff Stratford