School Performance School Performance Variation on COMLEX Level Variation on COMLEX Level
11
Linjun Shen, Ph.D., MPHLinjun Shen, Ph.D., MPH
National Board of Osteopathic Medical National Board of Osteopathic Medical
ExaminersExaminers
PurposesPurposes
Provide more information to school Provide more information to school administrators and educatorsadministrators and educators
Help school administrators and Help school administrators and educators to interpret and analyze educators to interpret and analyze COMLEX dataCOMLEX data
Demonstrate analytical approaches Demonstrate analytical approaches individual schools can adopt in individual schools can adopt in digesting their own COMLEX datadigesting their own COMLEX data
LimitationsLimitations
Analysis is limited to Level 1 June Analysis is limited to Level 1 June 2002 administration2002 administration
Variables available for this analysis Variables available for this analysis is limited, so as the resultsis limited, so as the results
A model of the medical A model of the medical school-licensing examination school-licensing examination
relationshiprelationship
Practice
LicensingExams
Pre-MedMed Sch
Scope of This AnalysisScope of This Analysis
Interest of the analysis: school mean Interest of the analysis: school mean scores and school passing rates on scores and school passing rates on Level 1.Level 1.
Potentially influential factors: gender Potentially influential factors: gender of students, sector of school, school of students, sector of school, school mean MCAT.mean MCAT.
Analytical approach: mainly descriptive Analytical approach: mainly descriptive and graphical with a brief statistical and graphical with a brief statistical modeling.modeling.
Summary of June 2002 Summary of June 2002 Level 1Level 1
N Mean/SD Passing
2696 500/79 90%
June 2002 Level 1June 2002 Level 1
StandardScore
Min. Passing 400(-1.26sd)
Mean 500
Lowest 225(-3.48sd)
Highest 862(4.58sd)
Level 1 School MeanLevel 1 School Mean
440
450
460
470
480
490
500
510
520
530
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Highest and Lowest on Highest and Lowest on Level 1Level 1
RANK
19.001.00
TO
TA
L800
750
700
650
600
550
500
450
400
350
300
250
200
School Mean Score School Mean Score DistributionDistribution
LEVEL1
530.0
525.0
520.0
515.0
510.0
505.0
500.0
495.0
490.0
485.0
480.0
475.0
6
5
4
3
2
1
0
Std. Dev = 16.54
Mean = 500.3
N = 19.00
School Failing Rate School Failing Rate DistributionDistribution
FAILING PERCENTAGE
16.014.012.010.08.06.04.0
5
4
3
2
1
0
Std. Dev = 4.65
Mean = 9.5
N = 19.00
Failing Rate by Ranking of Failing Rate by Ranking of MeansMeans
Rank by Mean
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
FA
ILIN
G P
ER
CE
NT
AG
E18
16
14
12
10
8
6
4
2
0
Passing Rate by Ranking of Passing Rate by Ranking of MeansMeans
Rank by Mean Score
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
PA
SS
ING
RA
TE
98
96
94
92
90
88
86
84
82
80
School Mean PerformanceSchool Mean Performance
Two ways to rank schools: by mean Two ways to rank schools: by mean score and passing ratescore and passing rate
Numerical ranking is not as Numerical ranking is not as important as qualitative groupingimportant as qualitative grouping
Variation among school mean scores Variation among school mean scores is substantial (53 point between the is substantial (53 point between the highest and lowest, 40 point between highest and lowest, 40 point between above and below average groups)above and below average groups)
Significant Gender Significant Gender Differences on June 2002 Differences on June 2002
Level 1Level 1
NMean/SD Passing
Male 1534 509/82 91.3%
Female 1162 486/73 88.6%
Gender Difference in Gender Difference in Mean Scores by SchoolMean Scores by School
400
420
440
460
480
500
520
540
560
1 3 5 7 9 11 13 15 17 19
MaleFemale
Gender Difference in Gender Difference in Failing Percentage by Failing Percentage by
SchoolSchool
0
5
10
15
20
25
1 3 5 7 9 11 13 15 17 19
MaleFemale
Gender Difference in Gender Difference in Osteopathic Medical Osteopathic Medical
SchoolsSchools
There are substantial gender There are substantial gender differences in terms of individual differences in terms of individual student Level 1 performancestudent Level 1 performance
For a few schools, the differences in For a few schools, the differences in both mean scores and passing rates are both mean scores and passing rates are alarmingalarming
Gender differences can also be an Gender differences can also be an opportunity for schools to improve if opportunity for schools to improve if efforts are made to narrow the gap efforts are made to narrow the gap
Sector Differences on June Sector Differences on June 2002 Level 12002 Level 1
Mean/SD Passing
Sector A 498/79 89.9%
Sector B 505/79 91.2%
Gender and Sector Gender and Sector InteractionInteraction
on Mean Scoreon Mean Score
516
492
508
484
460
470
480
490
500
510
520
Male Female
Sector ASector B
Gender and Sector Gender and Sector InteractionInteraction
on Failing Rateon Failing Rate
9.38.3
12
8.8
0
2
4
6
8
10
12
14
Male Female
Sector ASector B
Effects of Sector of Effects of Sector of Osteopathic Medical Osteopathic Medical
SchoolsSchools
There is no evidence that sector There is no evidence that sector significantly affects Level 1 performancesignificantly affects Level 1 performance
According to a study 10 years ago, sector According to a study 10 years ago, sector was a powerful factor in determining was a powerful factor in determining osteopathic Board exam performanceosteopathic Board exam performance
It is a very positive improvement of It is a very positive improvement of osteopathic medical educationosteopathic medical education
More formal study is neededMore formal study is needed
School Mean MCAT School Mean MCAT DistributionDistribution
MCAT
9.008.758.508.258.007.757.50
6
5
4
3
2
1
0
Std. Dev = .42
Mean = 8.32
N = 19.00
School Mean MCAT of Class School Mean MCAT of Class 20002000
Mean 8.32
Lowest 7.40
Highest 9.00
Sector A 8.36
Sector B 8.29
Mean MCAT by RankMean MCAT by Rank
Rank
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
Me
an
MC
AT
9.5
9.0
8.5
8.0
7.5
7.0
Mean MCAT and Mean Mean MCAT and Mean Level 1Level 1
(r = .30, not significant)(r = .30, not significant)
Mean MCAT
9.59.08.58.07.57.0
Me
an
LE
VE
L1
530
520
510
500
490
480
470
Mean MCAT and Mean Mean MCAT and Mean Passing Rate (r = .27)Passing Rate (r = .27)
Mean MCAT
9.59.08.58.07.57.0
Pa
ssin
g R
ate
98
96
94
92
90
88
86
84
82
MCAT and Level 1 MCAT and Level 1 PerformancePerformance
The implication of correlation of The implication of correlation of school means of MCAT and a single school means of MCAT and a single Level 1 administration is highly Level 1 administration is highly limitedlimited
Individual student MCAT and Level Individual student MCAT and Level 1 score relationship can be very 1 score relationship can be very different from school mean MCAT different from school mean MCAT and school mean Level 1and school mean Level 1
Conventional RegressionConventional Regression
GenderSectorscoreLevel *23*85081
Results of Conventional Results of Conventional RegressionRegression
The model is significant. Both gender The model is significant. Both gender and sector are significant predictor.and sector are significant predictor.
This regression model with two This regression model with two variables only explained 3% of the variables only explained 3% of the student. performance variation. Most student. performance variation. Most of the variation remains unexplained.of the variation remains unexplained.
More information is needed to explain More information is needed to explain student performance.student performance.
Hierarchical Linear Hierarchical Linear RegressionRegression
11211101
00201000
10score1Level
MCATSector
MCATSector
Gender
Models Level School
Model LevelStudent
Results of HLM AnalysisResults of HLM Analysis Difference of students Level 1 Difference of students Level 1
performance across school is significantperformance across school is significant Gender is a predictor for Level 1 scores Gender is a predictor for Level 1 scores
but the difference of gender is not but the difference of gender is not significant across schoolssignificant across schools
Neither Sector and school mean MCAT Neither Sector and school mean MCAT predict student Level 1 performancepredict student Level 1 performance
More data will help: student MCAT, race, More data will help: student MCAT, race, curriculum type, etc.curriculum type, etc.
Thank You!Thank You!
Top Related