Reply to Letter

1
PREDICTORS OF PERFORMANCE OF STUDENTS FROM THE CANADIAN MEMORIAL CHIROPRACTIC COLLEGE ON THE LICENSURE EXAMINATIONS OF THE CANADIAN CHIROPRACTIC EXAMINING BOARD To the Editor: Lawson and Till 1 conclude bthe admissions interview is not a predictor of successQ on various outcome measures yet fail to offer any data in their article to inform the reader or allow such a conclusion. They give a purpose of their study as being the determination of whether bthe Canadian Memorial Chiro- practic College [CMCC] structured admissions interview and other outcomes measures predict successQ on the Canadian external licensure examination. The reader may thus reasonably expect to see data of the admissions interview in addition to the reported data such as Grade Point Average. In the absence of either summary data or a table identifying the elements within the structured interview, the authors can only offer an invalid conclusion. The absence of reported data should but does not prevent the authors making the unsubstantiated statement in their discussion that bnone of the noncognitive characteristics used in the admissions process at CMCC were identified as predictors of success on the CCEB examinations.Q Apart from stating that bappropriate noncognitive qualities (are) requiredQ and admitting to a black of reliable and valid processes...to identify the noncognitive qualities of ...applicants,Q they fail to provide any description of the noncognitive components of the CMCC admissions process. The reader is thus forced to presume that these include a structured interview with no knowledge of the content or structure of that interview. Quite simply, the claimed lack of any predictive value of the admission interview could be due to either poor quality content of the interview or poor processes to accord ranking or other outcome measures. In the absence of the data, the reader is unable to make their own interpretation or indeed replicate the processes, both being strong indicators of weak research methodology and immature reporting. Phillip Ebrall, PhD Associate Professor, Chiropractic Education Department Head, Division of Chiropractic School of Health Sciences RMIT University, Victoria 3083 Australia 0161-4754/$32.00 Copyright D 2007 by National University of Health Sciences doi:10.1016/j.jmpt.2006.12.012 REFERENCE 1. Lawson DM, Till H. Predictors of performance of students from the Canadian Memorial Chiropractic College on the licensure examinations of the Canadian Chiropractic Examining Board. J Manipulative Physiol Ther 2006;29:566 - 9. REPLY TO LETTER In Reply: Doctor Till and I would like to thank the reader for bringing this matter to our attention. The reader is correct in stating that the data on the admissions interview should have been included with the other data. Because of the low correlation of the admissions interview to outcomes data from both the Canadian Memorial Chiro- practic College (CMCC) and Canadian Chiropractic Examining Board (CCEB) and its poor prediction as based on the regression analysis, the values for the admissions interview were left out of the article. The correlations between the admissions interview and CMCC grade point averages for years 1, 2, 3, and 4 were 0.19, 0.18, 0.16, and 0.09. The correlations between the admissions interview and CCEB examinations of Basic Science, Applied Science, Clinical Decision Making, and the Clinical Skills Examination were 0.14, 0.18, 0.08, and 0.01. The reader also requests ba table identifying the elements within the structured interview.Q At CMCC, the individual marks allocated to different elements of the interview are not used individually but are combined to calculate a final score for the interview; and that score is used in the selection process. Only this final score could therefore be used in the regression analyses of the study reported in this article. Just as the authors did not explain in depth how CMCC calculates their grade point average scores for each year or how the CCEB determines the scores on their individual papers, we also felt that it was not necessary to explain the different elements of the admissions interview and exactly how the individual checklist items or structured questions are used to calculate the final interview score. A further study could be planned to investigate these aspects in more depth. Comments on the admissions instrument with regard to its structure and makeup, the suitability of the rating scale used, the reliability of the instrument, or the validity of its scores—other than its low predictive value—was outside the scope of this study. Our conclusions were not meant for generalization to other interview processes. Simply, we conclude that the admissions interview as administered to this one cohort of applicants was not predictive. No such claim is made against any other admissions interview. Douglas Lawson DC, MSc DOI of original article 10.1016/j.jmpt.2006.12.012 0161-4754/$32.00 Copyright D 2007 by National University of Health Sciences doi:10.1016/j.jmpt.2007.01.001 160

Transcript of Reply to Letter

Page 1: Reply to Letter

PREDICTORS OF PERFORMANCE OF STUDENTS FROM THE CANADIANMEMORIAL CHIROPRACTIC COLLEGE ON THE LICENSURE EXAMINATIONS OFTHE CANADIAN CHIROPRACTIC EXAMINING BOARDTo the Editor:

Lawson and Till1 conclude bthe admissions interview is

not a predictor of successQ on various outcome measures yet

fail to offer any data in their article to inform the reader or

allow such a conclusion.

They give a purpose of their study as being the

determination of whether bthe Canadian Memorial Chiro-

practic College [CMCC] structured admissions interview

and other outcomes measures predict successQ on the

Canadian external licensure examination.

The reader may thus reasonably expect to see data of the

admissions interview in addition to the reported data such as

Grade Point Average. In the absence of either summary data

or a table identifying the elements within the structured

interview, the authors can only offer an invalid conclusion.

The absence of reported data should but does not prevent

the authors making the unsubstantiated statement in their

discussion that bnone of the noncognitive characteristics

used in the admissions process at CMCC were identified

as predictors of success on the CCEB examinations.QApart from stating that bappropriate noncognitive qualities

(are) requiredQ and admitting to a black of reliable and

valid processes. . .to identify the noncognitive qualities of

. . .applicants,Q they fail to provide any description of the

noncognitive components of the CMCC admissions process.

The reader is thus forced to presume that these include a

structured interview with no knowledge of the content or

structure of that interview.

Quite simply, the claimed lack of any predictive value of

the admission interview could be due to either poor quality

content of the interview or poor processes to accord ranking

or other outcome measures. In the absence of the data, the

reader is unable to make their own interpretation or indeed

replicate the processes, both being strong indicators of weak

research methodology and immature reporting.

Phillip Ebrall, PhD

Associate Professor, Chiropractic Education

Department Head, Division of Chiropractic

School of Health Sciences

RMIT University, Victoria 3083

Australia

0161-4754/$32.00 Copyright D 2007 by National University of Health Sciences

doi:10.1016/j.jmpt.2006.12.012

REFERENCE

1. Lawson DM, Till H. Predictors of performance of students fromthe Canadian Memorial Chiropractic College on the licensure

160

examinations of the Canadian Chiropractic Examining Board.J Manipulative Physiol Ther 2006;29:566-9.

REPLY TO LETTERIn Reply:

Doctor Till and I would like to thank the reader for

bringing this matter to our attention. The reader is correct

in stating that the data on the admissions interview

should have been included with the other data. Because

of the low correlation of the admissions interview to

outcomes data from both the Canadian Memorial Chiro-

practic College (CMCC) and Canadian Chiropractic

Examining Board (CCEB) and its poor prediction as

based on the regression analysis, the values for the

admissions interview were left out of the article. The

correlations between the admissions interview and CMCC

grade point averages for years 1, 2, 3, and 4 were �0.19,�0.18, �0.16, and �0.09. The correlations between the

admissions interview and CCEB examinations of Basic

Science, Applied Science, Clinical Decision Making, and

the Clinical Skills Examination were �0.14, �0.18,�0.08, and �0.01.

The reader also requests ba table identifying the

elements within the structured interview.Q At CMCC, the

individual marks allocated to different elements of the

interview are not used individually but are combined to

calculate a final score for the interview; and that score is

used in the selection process. Only this final score could

therefore be used in the regression analyses of the study

reported in this article. Just as the authors did not

explain in depth how CMCC calculates their grade point

average scores for each year or how the CCEB

determines the scores on their individual papers, we

also felt that it was not necessary to explain the different

elements of the admissions interview and exactly how

the individual checklist items or structured questions are

used to calculate the final interview score. A further

study could be planned to investigate these aspects in

more depth. Comments on the admissions instrument

with regard to its structure and makeup, the suitability of

the rating scale used, the reliability of the instrument, or

the validity of its scores—other than its low predictive

value—was outside the scope of this study.

Our conclusions were not meant for generalization to

other interview processes. Simply, we conclude that the

admissions interview as administered to this one cohort of

applicants was not predictive. No such claim is made

against any other admissions interview.

Douglas Lawson DC, MSc

DOI of original article 10.1016/j.jmpt.2006.12.012

0161-4754/$32.00 Copyright D 2007 by National University of Health Sciencesdoi:10.1016/j.jmpt.2007.01.001