PREDICTORS OF ACHIEVEMENT ON USMLE STEP I · PDF filepredictors of achievement on usmle step i...
Transcript of PREDICTORS OF ACHIEVEMENT ON USMLE STEP I · PDF filepredictors of achievement on usmle step i...
PREDICTORS OF ACHIEVEMENT ON USMLE STEP I FOR STUDENTS AT UNIVERSITY OF PUERTO RICO
SCHOOL OF MEDICINE: IMPLICATIONS FOR ACADEMIC LEADERSHIP
Irma L. Rivera, EdD1 Claudia X. Alvarez, PhD2
Ricardo R. González-Méndez, PhD3
1Evaluation and Research on Medical Education Office, School of Medicine, Medical Sciences Campus, University of Puerto Rico, San Juan, PR 2Faculty of Education, University of Puerto Rico, Río Piedras Campus 3Department of Radiological Sciences, School of Medicine, Medical Sciences Campus, University of Puerto Rico, San Juan, PR
Disclosure
This study is part of the dissertation work of Irma L. Rivera, as part of the requirements towards the degree of Education Doctor at the University of Puerto Rico, Río Piedras Campus.
The authors do not have any financial interests and
do not have any conflicts of interest to declare.
Purpose
This study has two major objectives: Identify key performance variables predictive for passing
USMLE Step I for students at University of Puerto Rico School of Medicine.
*Characteristics before being admitted to Medical School *Performance of students during first and second year of Medical School
Understand how useful this knowledge is for the educational
leadership of the UPRSOM in order to develop early intervention strategies.
Background
93 92 90 91 90 92 93 93 94 93 93 91
83 90
68
87 83
78
89 85 88
77 77 76
50 55 60 65 70 75 80 85 90 95
100
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Perc
ent
Percent Passing U.S & Canada
Percent Passing P.R
Year Prepared by: Evaluation and Research on Medical Education Office
Percent Passing for USMLE Step I First Time Takers P.R. vs. USA-Canada, 1999-2010
University of Puerto Rico School of Medicine has been fully accredited by the LCME since 1954 Puerto Rico’s population are US citizens or legal resident aliens In national studies we are considered as minorities Being a US school we are not considered as a Caribbean School of Medicine
Background
National Studies identify the Medical College Admission Test (MCAT) as a good predictor of performance on USMLE Step I (Julian, 2000).
MCAT Biological Sciences score combined with General GPA were good predictors for performance on USMLE Step I (Collin, 2005).
MCAT Biological Sciences score combined with grades obtained during the first two years at medical school were significant predictors for performance on USMLE Step I (Coumarbatch, 2010).
Background
The practice of medicine is a shared social contract between the medical profession and the public. Educational leaders have to be aware about the constant state of change on medical education (McMahon, 2010).
Leaders keep things simple and make information accessible. This
helps individuals understand what’s going on, hence they are better able to achieve success (Rich, 2008).
Top performing leaders seem to be self-effacing team builders who focus on strategic and incremental changes that will gradually transform their organizations (Naylor, 2006)
Methods
IRB approval: (having advisors and involvement from two campuses at UPR,
approval was obtained from both IRB’s) University of Puerto Rico, Río Piedras Campus (00000944) IRB Medical Sciences Campus (A9170111).
Mixed Method Explanatory Sequential Design (Creswell, 2007)
Phase I
QUAN
Phase II
qual Interpretation based on QUAN qual results
Methods Phase I (Quantitative)
Graduates from years 2001 to 2011 (N=1,113) Frequencies, Chi-Square, Linear and Logistic Regression Logistic Regression, Backward Likelihood Ratio (Kleinbaum, 2002)
( )( ) ( )( ) ∑+=
−
=i
ii XXPXPXP βα
1lnlogit
Phase II (Qualitative) Dean, Associate Deans, Committees Coordinators (N=7)
Focus Group (1 Hour) 8 key questions on Educational Leadership
( ) ieXOR iβ=
Results Statistically Significant Variables
Variable Chi
Square df p
GPA sciences 157.67 131 0.056
GPA general 136.63 106 0.024
Age 753.32 666 0.010
Gender 4.36 1 0.037
MCAT total 50.60 21 0.0001
MCAT Physical Sciences 39.83 9 0.0001
MCAT Biological Sciences 36.00 9 0.0001
High School 17.07 5 0.004
1st Year Courses Chi
Square df p
Anatomy 72.11 2 0.0001
Embriology 64.96 2 0.0001
Histology 62.97 2 0.0001
Biochemistry 99.80 2 0.0001
Physiology 101.19 2 0.0001
Neurosciences 105.69 2 0.0001
Human Development 44.64 2 0.014
2nd Year Courses Chi Square df p
Pathology 159.83 2 0.0001
Infectious Diseases 132.47 3 0.0001
Pharmacology 152.36 2 0.0001
Fundamentals of Clinical Diagnosis 56.53 2 0.0001
Psicopathology 30.61 2 0.0001
Mechanisms of Diseases 109.77 2 0.0001
Public Health 48.26 2 0.0001
Results Logit Model (Admissions)
Predicting Variable Estimate of Parameter
Standard Error
Wald df p OR
GPA sciences 1.176 1.176 15.843 1 .000 3.241
MCAT Physical Sciences 0.337 0.085 15.863 1 .000 1.400
MCAT Biological Sciences 0.264 0.078 11.435 1 .001 1.302
Constant -6.743 1.293 27.212 1 .000
Variables entered: GPA sciences, GPA general, MCAT Physical Sciences, MCAT Biological Sciences. R² = 6% Power = 99%
Results Logit Model (Combined)
Predicting Variable Estimate of Parameter
Standard Error
Wald df p OR
MCAT Biological Sciences .220 0.086 6.498 1 0.011 1.24
Physiology .792 0.336 5.552 1 0.018 2.207
Neurosciences .430 0.197 4.787 1 0.029 1.538
Biochemistry .500 0.218 5.274 1 0.022 1.648
Pathology .962 0.230 17.545 1 0.000 2.616
Pharmacology .900 0.239 14.179 1 0.000 2.458
Constant -9.164 1.052 75.877 1 0.000
Variables entered: GPA sciences, MCAT Physical Sciences, MCAT Biological Sciences, Physiology, Neurosciences, Pathology, Pharmacology and Biochemistry. R² = 19% Power = 100%
Results
Educational Leadership: Consensus of the Focus Group
The University of Puerto Rico School of Medicine has a reliable study that will be helpful to make appropriate decisions regarding admissions and students at risk of academic performance problems.
The school needs to review: the admissions formula, the course
content and the assessment methodology.
The school should continue to strengthen faculty development.
The leadership of the Second Year made a decision to develop an early intervention program to address the students at risk of failing USMLE Step 1 based on the factors identified in this study.
New Intervention Strategies
Postpone the taking of the USMLE Step 1 Counseling by Psychologist, Counselors and Faculty (individualized study plan) Faculty development to improve questions in tests. Improve the integration of basic and clinical sciences Open Forum in which third year students talk about their experiences in taking the exam NBME practice test recommended (score >200 desired) Early access during 2nd year to USMLE World question bank.
93 92 90 91 90 92 93 93 94 93 93 91 94 96
83 90
68
87 83
78
89 85
88
77 77 76
92 98
50
60
70
80
90
100
110
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Perc
ent
Percent Passing U.S & Canada
Percent Passing P.R
Year Prepared by: Evaluation and Research on Medical Education Office
Percent Passing for USMLE Step I First Takers P.R. vs USA-Canada, 1999-2011
Conclusions
The PREDICTIVE variables identified by our logistic model at ADMISSION for passing the USMLE Step I were: GPA sciences, MCAT Biological Sciences, MCAT
Physical Sciences
The PREDICTIVE variables identified by our logistic model for passing the USMLE Step I were: MCAT Biological Sciences Grades on courses of Physiology, Neurosciences,
Biochemistry, Pathology and Pharmacology
Conclusions
Leaders reflected about their leadership. They concluded that transformational leadership style
will be more effective in the development of strategies that will promote the incremental changes needed for continuous improvement at the UPRSOM.
The leadership of the Second Year designed an early intervention program that has substantially improved the passing rate of first time takers of the USMLE Step 1 over the last two years.
Leaders think and talk about the solutions. Followers think an talk about the problems. —Brian Tracy
Thank you!
References
Collin, V.T. (2005). Predictive validity for the medical college admission test using multiple regression and latent variable path analysis (LVPA). (Disertación doctoral). Universidad de Calgary, Alberta, Canadá.
Coumarbatch, J., Robinson, L., Thomas, R., Bridge, P.D. (2010). Strategies for identifying students at risk for usmle step 1 failure. Family Medicine, 42, 105-110.
Creswell, J. W. (2009). Research design: qualitative, and mixed methods approaches. California: SAGE Publications. Julian, E.R. (2000). The predictive validity of the medical college admission test. Association of American Medical
Colleges, 3, 1-3. Kleinbaum, D.G., Klein, M. (2002). Logistic regression a self-learning text (second edition). New York: Springer-Verlag. McMahon, G.T., Tallia, A.F. (2010). Anticipating the challenges of reforming the united states medical licensing
examination. Academic Medicine, 85, 453-456. Naylor, D. (2006). Leadership in academic medicine: reflections from administrative exile. Clinical Medicine, 6, 488-492. Rich, E.C., Magrane, D., Kirch, D.G. (2008). Qualities of the medical school dean: insights from the literature.
Academic Medicine, 83, 483-487.
Graduated Class
53 54 10749.5% 50.5% 100.0%
53 53 10650.0% 50.0% 100.0%
50 48 9851.0% 49.0% 100.0%
57 58 11549.6% 50.4% 100.0%
37 63 10037.0% 63.0% 100.0%
35 54 8939.3% 60.7% 100.0%
54 61 11547.0% 53.0% 100.0%
48 52 10048.0% 52.0% 100.0%
50 42 9254.3% 45.7% 100.0%
47 46 9350.5% 49.5% 100.0%
52 46 9853.1% 46.9% 100.0%
536 577 111348.2% 51.8% 100.0%
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
Total
M FGender
Total
Average size graduating classes =102 (SD=8.8)
Focus Group Key Questions
Strategic Leadership 1-As with the results of phase 1 of this study a plan could be designed to achieve the mission and vision of
University of Puerto Rico School of Medicine? Didactic Leadership 2-Which educational practices can be integrated into the courses to strengthen the performance of medical
students on USMLE Step I and facilitate students will approve it on the first attempt? 3-How technology (computer programs, simulators, electronic data banks and so on) could be integrated
into the curriculum to strengthen the teaching and learning process, that has an impact for a better medical students performance on USMLE Step I and promoting students will approve it on the first attempt?
4- How faculty could be developed professionally so can be strengthening the teaching and learning process and thus the performance of medical students on USMLE Step I?
Administrative Leadership 5- With the results obtained in phase 1 of the study. Additional fiscal, human resources and technology are
required to strengthen the performance of medical students on USMLE Step I and facilitate students will approve it on the first attempt?
6- How you will use the results from phase 1 of the study to prepare action plans directed on how you as a leader can effectively manage fiscal and human sources that have an impact on students’ success?
Community Leadership 7-How important are the results of this study so you as a leader can respond to changes and challenges that
have today medical education and train doctors that respond to society needs? Implications for leadership 8-You as a leader in the school of medicine, How important is to knowing the predictive variables for
medical students performance on USMLE Step 1, and how helps you to exercise educational leadership at University of Puerto Rico School of Medicine to facilitate that students can pass USMLE Step 1 on the first attempt?