DOCUMENT RESUME

ED 067 003 HE 003 276

AUTHOR Olson, Carl J.; And OthersTITLE The Orthopaedic Training Study, Phase II 1968-1972.

Final Report, VOlume I.INSTITUTION Illinois Univ., Urbana. Medical Center.SPONS AGENCY Public Health Service (DHEW), Washington, D.C. Bureau

of Health Professions Education and ManpowerTraining.

PUB DATE [72]NOTE 120p.

EDRS PRICE MF-$0.65 HC-$6.58DESCRIPTORS *Higher Education; *Laboratories; *Medical Education;

*Medical Students; Skill Development; *TrainingLaboratories

ABSTRACTPhase two of the Orthopaedic Training Study was

designed to examine time, sequence, and content requirements ofexisting crthopaedic programs. Specifically, the proposal wasdesigned to achieve the following objectives: (1) to provide a modelof individualized graduate education in medicine in which thedemonstration of individual competence marks the end of formaltraining; (2) to document the nature and variation of orthopaedictraining in the U.S.; (3) to devise and test methods for increasingthe efficiency and effectiveness of orthopaedic training; (4) todetermine the relationships between input training and outputvariables; (5) to develop mechanisms that will facilitate continuinginstitutional self-study of training programs; and (6) to develop apool of educational specialists in orthopaedics who can providecontinuing leadership in the field. This document presents a reportof the results of the study with remarks about implications forfurther study and development of programs. See also HE 003 277 and HE003 275. (HS)

THE ORTHOPAEDIC TRAINING STUDY

Phase II 1968 - 1972

FINAL REPORTVOLUME I

?/11- Doa /.

1

ORTHOPAEDIC TRAINING STUDY STAFF

Professional StaffThomas J. nigh M.Ed.James L. Coole, M.S.Penny Edgert, M.A.Brian Huncke, M.D.James Monahan, M.A.

Fellowship StaffGary Barham, M.D.Richard Dimond, M.D.Henry Hood, M.D.Joseph Kopta, M.D.Henry LaRocca, M.D.Edward Mc Carthy. M.D.Clyde L. Nash, M.D.H. Bates Noble, M.D.

2

TABLE OF CONTENTS

VOLUME I PAGE

I. INTRODUCTION AND SPECIFIC AIMS

A. BACKGROUND 1

B. METHODS 4

II. NATURE AND VARIATION OF

A. PROGRAMS 19

B. RESIDENTS AND ATTENDINGS 41

III. ANALYTIC STUDIES 67

IV. PREDICTION STUDY 98

VOLUME II

SUPPLEMENT

ILIST OF TABLES

SEC.NO.

TABLENO. TITLE PAGE

II - 1 Programs Reporting Subspecialists(Percentages) 23

2 Resident Patient. Care Responsibilityby Year in Training (Percentages) 37

3 Program Characteristics by Type ofProgram 38-39-40

4 Satisfaction with Attendings asTeachers 44

5 Resident Teaching Responsibilities 45

6 Dissatisfaction with HospitalProcedures 46

7 Resident Attitudes (In agreement) 47-48

8 Resident Attitudes (Disagreement) 49

9 Mean Hours per Week Residents Spentin Selected Activities 52

10 Weekly Hours per Category 53

11 Attending's role at the operation 61

12 Attending Attitude Survey (Agreement) 64-65

13 Attending Attitude Survey (Disagreed) 66

III 1 Population-Study Sample Comparison 72

2 Correlations of Chiefs' Ratings ofResidents at the First, Second andThird Years of Training 75

4

LIST OF TABLES(Continued)

SEC.Ncit

TABLENO. TITLE PAGE

III 3 Mean Chiefs' Ratings for each Yearin Training '16

4 Selected Correlations between Chiefs'Ratings and OITE Scores for eachYear in Training 81

5 Significant Correlations betweenResident Attitude Statements andOTTE Score 86

6 Average Percentile Rank of Ar(BCs Com-pared to Residents at the same Yearof Training on the Multiple ChoiceSection of the OITE 90

7 Average Chief's Evaluation of ARBCsand All other Residents in thesame Year in Training 92

8 Correlations between OITE and OCEScores 97

TV 1 Items Included in the PredictionStudy 100

2 Interaction of Resident and ProgramCharacteristics in Predicting Mul-tiple Choice Scores 102

3 Program Characteristics Related toMultiple Choice Score Prediction 103

4 Characteristics of Residents andPrograms Related to OITE MultipleChoice Scores (University Programs) 104

I

I

LIST or TATMES(Continued)

SEC. TABLENO. NO. TITLE 1311GF

IV 5 Charactericties of Residents andPrograms Related to OITE MultipleChoice Scor's (Independent Programs) 105

6 Interaction of Resident and ProgramChe'.racterisvics in Predicting PMPScores

7 Charecteristics of Residents andPrograms Related to OITE PMr? Scores(University Programs)

8 Characteristics of Residents andPrograms Related to OITE PMPScores (Independent Programs)

3.07

108

309

INTRODUCTION

Phase II of the Orthopaedic Training Study representsa landmark in the fullest sense of the word. It is thehope of the Study Staff that the information and experiencegained in this, the first intensive study of medical speci-alty training, will serve as a model for others. However,the landmark indicates the beginning rather than the end ofthe trail. Even the beginning could not have been achievedwithout the cooperation of many representatives of the or-thopaedic community and the efforts of the research staff.Most notable among the former were the members of the Advi-sory Committee, Drs. George T. Aitken, Paul Curtiss, CharlesHerndon, Walter Hoyt, Jr., Paul Lipscomb and Fred C. Rey-nolds. It is to these men the Staff looked for guidance,they were never found wanting.

The Final Report is divided into two volumes, thefirst briefly contains the following information:

I. Introduction and specific aims

A. Background

B. Methodology

II. Nature and Variation of --

A. Programs

B. Residents and attendings

III. Analytic Study

IV. Prediction Study

The second volume, or Supplement, contains copies ofmost of the Study instruments, working papers, and reportdocuments which will be helpful to those wishing to utilizethe methodology developed.

The report contains only a small portion of the mate-rials developed for the Study and the data generated throughdata collection efforts. Complete information is availablefrom the Center for Educational Development provided therequest is approved by the Advisory committee.

I - INTRODUCTION AND SPECIFIC AIMS

FOREWORD

There has been a steady trend toward increased speciali-zation in medicine during the last 20 years. Parallel with,and indeed a part of this development has been increased em-phasis on formal board certific!ation, both in the traditionalspecialities and the growing number of sub-specialitics. Evi-dence of this can be seen in two areas: First, a generalagreement that both services at the community level have im-proved, both in quality and variety. Second, a growing con-ccrn that the potential practitioner is being required toinvest greater periods of time in training, thus reducing hisavailability to deliver these health services. Concurrentwith this concern is a question about the rigidity of timerequirements which appear to deny an opportunity for individ-ual differences and the structuring of time requirements onsome basis other than empiric.

A general review of Board speciality requirements suggeststhat, while attempts are being made to apply our general know-ledge to differences in learning speed and patterns to trainingprograms, a systematic effort to apply this knowledge had notbeen attempted. Finally, an empiric justification for the re-structuring of both training programs and board requirementsto allow for these differences could not he found in theresearch literature extant.

It was discussions of this sort that led to an agreementbetween the American Board of Orthopaedic Surgery (ABOS) andthe Center for Educational Development which culminated in aresearch project, "Efficient Use of: Medical Manpower," fundedby the Bureau of State Services. This Study was to focus onthe definition and measurement of professional competence inorthopaedics, with the aim of increasing the validity and thereliability of such appraisals so that the Board could acceptmeasures of competence as the primary criterion for certifica-tion. After reviewing the research completed and the resultantevaluation procedures developed from that research, the Board,in a meeting on June 30, 1967, agreed that it was time to beginexperimental modification of sequence, time, and content re-quirements on a controlled Study sample.

9

-2-

At this same time the Skeletal Systems Committee ofthe National Research Council - National Academy of Sciencebecame concerned over the failure of orthopaedic trainingto produce academically oriented orthopao.'sts. It wasappLrent that these programs produced highly skilled ortho-paedists, many of whom became outstanding program directorsbut who had little familiarity with the vast body of know-ledge in education and with application to instructionalpractices in orthipaedics.

On April 21., 1967, the NRC-NAS Skeletal SyL.temq Com-mittee agreed to lend support to the orthopaedic communityfor the development and implementation of a study designedto examine. time, sequence and content requirements of ex-isting orthopaedic programs. Specifically, the proposal,an extention of Phase I*, was designed to achieve the follow-ing broad objectives:

1. To provide a model of individualized graduateeducation in medicine in which the demonstrationof individual competence, rather than the fulfill-ment of rigid tame and content requirements, markthe end point of formal training.

2.. To document the nature and variation of ortho-paedic training in the United States.

3. To devise and test methods for increasing theefficiency and effectiveness of orthopaedictraining.

4. To determine the relationships between inputtraining and output variables.

5. To develop mechanisms that will facilitate con-tinuivig institutional self-study of trainingprograms.

*This report of Phase I is available from the Centerfor Educational Development.

.t̂ 0

-3-

6. To develop a pool of educational specialistsin orthopaedics who can provide continuingleadership in the field.

Study Organization

In order to achieve these objectives, a rather uniqueconsortium was established with three principal elements:a proiesslonal education staff, an advisory committee anda cadre of trained orthopaedic educators. Each of thesegroups provided a unique contribution to the Study. First,the professional education staff was able to supply anextensive knowledge of the science of education and researchmethodology to clarify questions and propose mechanisms forobtaining their answers. In addition, they provided theoperating personnel for the Study, implementing plans, andproviding coordination among the other groups involved inthe Study. This staff varieC from one to four educationprofessionals, the number dependant upon the particularportion of the Study being unrIertahen at the time. Second,an Advisory Committee* perfofmed several unique services inaddition to serving as represenitives of the orthopaediccommunity and voicing their coneJrns: They provided broadrange policies under which the Study operated, served as mon-itors of progress and direction, and finally, provided liaisonwith members of the orthopaedic community. The third groupconsisted of consultants. Under provisions of the Grantselected residents could substitute a portion of their resi-dency requirement by serving as consultants to the Ortho-paedic Training Study. This arrangement permitted them togain expertise in the field of education while providingvalualla subject matter input to the educational staff ofthe Study. Two men availed themselves of one-year fellowshipsand earned Masters' degrees in Medical Education. Two spentsix months at the Center for Educational Development, whileone spent approximately four months at the Center, and twoothers 30 days each.

*The Advisory Committee consisted of two representa-tives from the Board, the Academy, and the MSC-NCR who wererespectively Drs. Walter A. Hoyt, Jr. and Paul R. Lipscomb;Drs. George T. Aitken and Charles H. Herndon; and Drs. Fred

Reynolds and Paul H. Curtiss, Jr.

-4-

Methods of Procedure

The accomplishment of these goals within the proposedfoir -year study period required that several. phases of theStudy be conducted simultaneously, but for clarity of pre-sentation, the elements of the Study will be described in-dependently.

1. Introduction of Flexibility. Early data from theOrthopaedic In-Training Examination (OITE) had shown thewide range of achievement among residents in each year oftraining programs and the substantial overlap of achievementamong individuals in all four years. The evidence suggestedthat some residents might qualify for certification in asignificantly shorter time than was presently required evenwithout modification of iniuctional content or methodology.In order to provide an early opportunity to demonstrate thevalidity of this hypothesis, 16 training programs, consistingof 280 residents* were designated as an intensive studygroup. The criteria for selection was developed jointly byan Advisory committee representing the three participatingagencies.

The cooperation of these program directors was enlistedto peimit their residents, with the authorization of theAmerica.. Board nt Orthopaedic Surgery, to depart from thepresent time and ccntent distribution requirements, withoutjeopardizing their eligibility for Board certification.*The participating residents were authorized to present anapplication for examination to the Board Committee on eligi-bility when, in the opinion of their program director, theywere prepared to take the Board examination. Candidates whosuccessfully completed the Board examination could proceedimmediately to fulfill the practice requirement (presentlylne year) and, at the conclusion of that period, apply forfinal certification.

* See Supplement.

-5-

In order to maximize the opportunity for increasingthe speed and efficiency of learning, program directorsin the intensive study group were provided with assist-ance in introducing a process of continuous performancemonitoring and giving feedha,A to the resident in orderthat both staff and resident might be aware of the individ-u(1 progress toward achievement of the critical componentsof competence in orthopaedics.

The instruments to be used in this assessment includedthe annual II,-Training Examinationgiven by the AmericanAcademy of Orthopaedic Surgery (developed by the Center,with the assistance of professional staff members from theBoard) , and a variety of check lists, rating scales, andtests of complex cognitive and interpretive skills devel-oped in the original study.

A second consideration in flexibility was the pro-posal for program modification that would.have the effectof introducint greater variety within a program in orderto respond to differences in resident needs and interests.

Resident in some programs were encouraged to sub-stitute a 6 to 12 month fellowship in re:,,earch in medicaleducation for more conventional rotaticns. Residents whotook their Board exam earlier than us-al opted from amongseveral alternatives.*

2. Intensive Study of Training Experiences. Thecritical components of competence in orthopaedic surgerydefined the educational goals toward which training pro-grams should be directed.** The purpose of this phaseof the Study identified the extent to which 1) trainingprograms provide opportunities for residents to gain thedefined knowledge, skills and attitudes, 2) the curricularorganization, instructional materials and methods conformto generally accepted principles of learning and 3) system-

*For a study of these early board candidates, see part 2,section 2.

**Critical Components of Competency are discussed in theFinal Report of Phase I.

-6-

atic evaluation is conducted and utilized for continuousprogram assessment.

A stratified study sample (the intensive study groupand the survey group*) of the 187 approved residencyprograms was selected to represent variations in academicaffiliation, hopsital size, nature of population served,and geographic location. Each program was analyzed todocument the following.

A. Program organization - including schedule of resi-dent rotation, the personnel who supervise training, thefacilities and resources to support the training.

B. Program objectives - the mechanism created forestablishment, review, and communication of the objectivesto staff and residents.

C. Program operation - activities and responsibil-ities of a resident sample, the nature or instructionalprocedures, of both a formal and informal nature, thenature of feedback to residents of their individualstrengths and weaknesses as training progresses.

D. Program evaluation - the mechanisms employed toaccumulate data about resident progress, program effec-tiveness and the utilization of these data in continuingprogram review.

E. Program perceptions - identification of simi-larities among residents and staff in the perception ofpurposes, procedures and effectiveness.

The variables (See Table IV - 1, p. 100) identified forobjective 2 required diverse data, some of which alreadyexisted. Data included information on a resident's sex,marital status, academic background, internship, knowledgeof science, ability at patient interviewing, etc.

*The survey sample consisted of 35 resident trainingprograms which received only the survey instruments.See supplement for a list of their programs.

14

-7-

Generally, the material which had to be gathered includedinformation concerning background characteristics andtraining experiences with specific orthopaedic procedures.Type, location, and size of the program setting were areasof availzble institutional information while checklists,interviews, and questionnaires generated details of theprogram's mission or role in the community, associationwith other programs, room arrangement, support services(brace shops, physical therapy, special laboratories), etc.In addition to the general and specific characteristics ofthe program setting, other training variables for whichinformation had to be gathered were the nature of residentresponsibility for patient care, patterns of rotation, andprofessional qualifications of attendings. Knowledge,interpretive skills, and problem solving skills of theresidents constituted the output variables and were measuredby the Academy's In-Training Examination.

The instruments* constructed to gather the above infor-mation were the following:

The Program Description Questionnaire was designed toelicit information for classifying currently approved pro-grams according to affiliation, sources of financial support,administrative organization, primary commitment, orientation,physical facilities, clinical material, staff resources,size, program objectives and organization, education activ-ities, resident responsibilities and the like. This documentwas completed by 80 percent of the directors of all approvedresidency programs in orthopaedics in the United States.**

The Institutional Description Form was designed toelicit detailed information from all institutions affili-ated with each program regarding the type of institution,primary mission, sources of financial support, administra-tive structure, edv'cational resources, characteristics ofthe patient population served, interrelations among sub-specialty services, patient care facilities and supporting

*See Supplement for samples.

** There was a 70 percent return on a re- distribution`of this questionnaire in 1971.

"5

0-8-

services and decision-making process in screening patientsfor admission and directing their total health care. Theform was completed by 78 percent of the directors of eachinstitution affiliated with each program of the Studysamples.

The Resident Evaluation Form and The Candidate Eval-uation Form were designed to obtain preceptors ratings ofresidents and of candidates for certification, with respectto each of a number of performance factors related to theseveral components of competence identified as criticalperformance requirements. The Candidate Evaluation Formwas completed by at least two persons who had known eachresident over an extended period of time. The ResidentEvaluation Form was completed by the chief.

The Resident Evaluation of Operative Procedures wasdesigned to identify the nature of the resident's experi-ence and the character of his responsibility with regardto various decision points in the operative management ofpatients and was returned by 90 percent of the residentsin the Study.

The Resident Procedures Form was designed to determinethe resident's experience with technical procedures and hisdegree of confidence in ability to perform them. The formenabled identification of the setting in which he learnedto perform a procedure and the nature of the supervisionunder which he learned it. The form listed 58 sampletreatment and orthopaedic procedures and was completed by78 percent of residents in the Study sample.

The Resident Time Log was designed to discover vari-ations in the work pattern in different orthopaedic residencyprograms. It was completed by 66 percent of residents inthe Study samples at specified calendar periods selectedto sample resident activities in each program.

The Resident Attitude Survey was designed to identify.both within and between program variations in the attitudesof residents toward patients, colleagues, instructionalstaff, physical and clinical facilities, and the professionand the training program. It was completed by 88 percent of.

-9-

residents in the Study groups.

The Resident Background Survey was designed to identifydifferences among residents in terms of their personal 'ack-ground, education experience and attitudes. It was com-pleted by 98 percent of residents in the Study ;:ample.

The Attending Attitude Survey was designed to elicitattending staff attitudes toward program organization,teaching styles, resident behavior, interaction with resi-dents and with staff, etc. The survey was completed by 35percent of attendings in the Study.

In addition to specifically designed instruments,data collection for this study employed other survey andobservational techniques. In the first round of sitevisits, survey data was verified in an interview* withprogram chiefs and residents in training by requesting de-scriptive information on the educational resources actuallyused in training, the nature and amount of instructionalactivity in which residents actually participate; the extentand character of resident responsibility for patient care,teaching, and research; the amount and character of feedbackon performance regularly provided to each resident fromsenior residents, attending staff, and training chief.Observational data was obtained from site visits by the pro-ject staff. In the course of these two-day visits, a ran-dom sample of instructional experiences (rounds, conferences,seminars, operative and emergency room teaching) were de-scribed through carefully structured checklists and ratingscales designed to document the quality of instructionalexchange between trainees and their mentors, the educationalgoals (cognitive,. psychomotor, effective) that the exchangeis most likely to serve, and the pedagogic quality of theencounter (in terms of facilitation of learning, not thebiomedical content).

3. Increasing Program Efficiency and Effectiveness.Through the intensive review described above, identificationof areas in which new organization of training systems or

See Supplement for site visit procedures.

17

-10-

utilization of alternative instructional modes wouldincrease either a program's effectiveness of efficiencywas made. Such changes included the following:

Staff Development Programs. A brief seminar ongraduate education designed for attending staff and someof the intensive study programs was conducted during thesecond round of site visits. Almost uniformly the attend-ings from these programs voiced a desire for additionalmaterials in the general field of educational science.Many attendings commented on the fact that these sessionswere the first time in which they had dealt with educationalprinciples without regard for the particular subject matterand clinical considerations in 'the hospital setting. Ini-tially, two films illustrating the application (or violation)of fundamental principles of learning in characteristic edu-cational settings (lectures, conferences, rounds, operatingroom, and individual instruction) of residency training wereprepared.* Their effectiveness resulted in the preparationof six additional films which were used on the subsequentsite visits. These materials are available in the supplement.

Instructional Innovation**. One of the components ofprogram efficiency is the appropriate utilization of ateaching staff. This utilization should appropriate highlyskilled staff members to the task for which they are trainedand avoid use of staff to accomplishing routine tasks forwhich another way is available. This rationale lay behindthe type of instructional innovation promoted among theintensive study programs. Some of the instruction in plasterapplication, traction hanging, interpretation of x-rays, andthe appropriate use and handling of orthopaedic surgicalinstruments is possible by organizing the Study. This typeof study substitutes for the instructor a set of materials

*See Supplement for the folio of staff development

materials.

**See also p. 13 for discussion of the task forces

related to this area.

guiding the resident through the more elementary aspectsof the subjects and allows the resident to procede throughthe set at his own pace and whenever he has the time. Theinstructor's time is thus saved from repetitious instruc-tion for that instruction which only he can provide.

1. Radiology Teaching Package. The Radiology Teach-ing Package was an outgrowth of the material devel-oped at the University of California at San Fran-cisco by Doctor Steven Ross, a radiologist.Dr. Ross developed teaching materials in responseto a number of needs that he saw in the field ofratiology teaching. The first was that the resi-dents needed to be presented with radiologicalmaterials in the order of increasing difficultyso that in the learning process he was not encoun-tered by the wide variety of trauma that enteredthe clinical situation but was able to progros];from the more easily recognized cases to the moredifficult ones. Secondly, that the resident couldstudy these materials on an independent and self-paced basis. As a result of the interest of theOrthopaedic Training Study in his work Dr. Rosshas begun development, with the assistance of theOrthopaedic Department of the University ofCalifornia, San Francisco, of several sets oforthopaedic resident radiology teaching packages.The Orthopaedic Training Study itself, to deter-mine the effectiveness of such a teaching method,has developed a four series set of materials whichare designed to familiarize chiefs, attendings andresidents with the concept, and provide a modelfor whidh they can develop their own teachingmaterials. The radiographic series we have devel-oped, illustrating the various types of pathology,are given anatomic location with attached atten-uating histories and obscured correct interpreta-tions and bibliography viewed by the trainee onlyafter he has committed himself to an interpreta-tion of the films. Overlays are available for preand post-testing and help to coordinate the de-scription of fine and obscure detail. .

-12-

2. Plaster Skills. It was observed in.many of theprograms that the basic principles underlyingthe use of plaster were never systematicallypresented to the resident. It was expected thathe would obtain this knowledge from one or anotherof the attending staff involved but no attendingwas individually responsible for teaching it.Plaster skills, being one of the most basic oforthopaedic psychomotor skills was selected firstin our development of psychomotor innovations.The initial phase of the plaster lab was developedby one of the six month.Fellows in OrthopaedicEducatior, who had just completed his residencytraining program. Under his direction an intensiveinvestigation of the literature and rationale thathad been developed for psychomotor training, whichcame mostly from industry, was investigated and apilot unit was prepared. This unit was then cir-culated to the intensive study programs duringsite visits and comments and suggestions aboutboth the methodology involved and its potentialvalue to residency training programs was solicited.One of the curious phenomenon which resulted fromour site visits was that there was a divergence ofopinion between residents and attend:;_ngs aboutplaster skills instruction. Residents saw a realneed for the plaster laboratory early in theirtraining, but attendings felt that the instructionwas given to all residents at some time. Theattendings, while they understood the rationalepresented, did not feel that the laboratory wasa necessary adjunct to their programs. Upon dis-cussion of the residents and attendings it becameobvious that much of the instruction that theattendings took for granted as having been taughthad been missed in many of the residents' training.Therefore, such a planned sequential program forevery resident, seemed to the residents to be mostdesirable. After considerable discussion and lay-ing of the evidence, the attendings arrived at thesame conclusion. This discussion and consensusoccurred in about 80 percent of the programs inthe second round of site visits. Subsequently,

20

-13-

the total package of plaster skills,was developedand expanded to a series of eight lessons whichwas then field tested in three of the intensivestudy programs. Based on their comments to date,the materials have been revised and reviewed bythe chiefs of the programs in the intensive studygroup for further development.

3. Traction Laboratory. The second innovation tobe developed was the traction laboratory. Thisinnovation was developed by two residents, oneof whom wrote the instructor's manual, while asecond provided the ancillary (slides, tapes, etc.)material which were the. actual instructional toolsfor the laboratory. Again, this was tried out intwo of the intensive study programs. Commentsand suggestions were then used to further refinethe laboratory.

4. Psychomotor Skills - Surgical. Due to the perceivedneed for further development of the basic psycho-motor skills needed by beginning orthopaedic resi-dents, a task force was assembled to study thearea of surgical technique and instrumentation whichshould be included in a basic surgical skillslaboratory. After defining the instruments whichwould be considered, each of the participantsselected an area to be developed both in the formof a sound slide presentation and a video tape ofthe dynamic aspects of instrument usage, ratherthan the static one, which could be demonstratedwell on sound slides. It was decided to focus thelaboratory around the use of particular instrumentsrather than around surgical procedures, both as amechanism for the organization of individual studyunits independent of one another and for the simpli-fication in the presentation of the large amount ofinformation, considering the large number of indi-vidual instruments which could be used in anyprocedure. The focus on instruments themselvesavoided a duplication of instrument usage in thevarious operative procedures. The instrumentswhich were selected for development in the first

-14-

meeting were as follows: scalpel, needle holdersand needles (suturing), hand saws, rieriostealelevators, hand drills, drill bits, depth gauges,screw drivers, hemostats, rongeurs, wires, pincutters, bending iron, wires and surgical pins,reamers, bone halters, rasps, osteotmee, curettesand gouges.

4. Relationship of Input Training and Output Variables.The Study produced, in 1967, a data bank with the guaranteeof anonymity for any individual resident, that allows aninvestigation access to a continuous' record of resident pro-gress as it is assessed through annual In-Training Exami-nation (OITE). Additionally, the data bank containsstandardized perceptor ratings of resident achievement ofknowledge, problem solving, and interpretive skills. Resultsof the Board Certifying Examination, which probes complexcognitive prodesses, professional behavior, and selectedtechnical skills, are also part of the orthopaedic databank.

The model chosen for the study design was borrowedfrom the economics model of input-process-output. Theelements of each are examined individually; for example, thecharacterization of the input is considered first, with theresident's background, attitude, fund of knowledge, etc.,with which he comes to the field of orthopaedics, then themethod of which this individual is treated or processedthrough a program for example, the settings the hospitalsand various kinds of service rotations, etc. These arethen related individually to the output measures. Finally,a variety of statistical techniques are useful in studyingthe interaction between various elements of the input andprocessing or treatment.

`,4

r.I

:.

se Li

rt

.f

!Cisf"stI:

!

I1,

a

t,7?^ / F

ll'

;.

stool,1

I

4,-.I

fi-

C!r.

t

r;"*.

r

i1 ("1,1 !.

-

L'.; g

-16-

The analysis of the data was done at two levels.First, each of the many responses given by program chiefs,hospital administrators, attending staff, and residents,to the questionnaires were tabulated then summarized inthe reports in Section II. Second, from the questionnaireresults, hypotheses were generated at Advisory CommitteeMeetings, intensive study program chiefs' meetings, andpersonal observations of the Study Staff during site visits.These hypotheses stated expected relationships of charac-teristics of orthopaedic training program's facilities andorganization, and characteristics and attitudes of residentsand attendings to a criterion measure of residentperformance. The criterion selected bpon which to judgethe success of a resident's education was the CITE totalmultiple choice score and the total PMP score. The hypoth-eses tested in this report are by no means exhaustive ofthe areas in which investigation is needed nor are they,in many instances, refined enough for the results to beapplicable to any particular program. Further evaluationsof program efficiency and effectiveness need to be carriedout in individual training programs to assure applicabilityof results. Hoever, the results of the hypothesis test-ing research presented here is the first effort to investi-gate training programs. As such, its purpose is to stimu-late the orthopaedic and the educational research communityto do further studies indicated by the trends and relation-ships uncovered by the present Study.

f

74

t

t

-17-

INTRODUCTION

One of the first tasks confronting the Advisory Com-mittee and professional staff was the selection of the Studysamples. The objectives of the Study (sec page 2 & 3) suggestthat data collection from the introduction of innovations bedivided among three board classifications of programs.

1. Population: All programs would be asked to supplycertain information, such as, program length, aims,and other characteristics (see Supplement for alist of all data collection instruments and by whomthey were completed).

2. Survey Study Sample: A limited number of chiefs ofprograms (38) wore invited to participate in datacollection efforts which would be more extensivethan in the population at large (see Supplement).Of those invited, 31 accepted the invitation toparticipate in the Study. The resulting group ofprograms was called the Survey Study Sample.

3. Intensir1 Study Sample: Sixteen program chiefs wereinvited to participate in an intensive study oftheir programs; all accepted. This group agreed topermit data gathering, not only about its programand institutions, but about residents and attendingsas well. They also agreed to participate in thedevelopment and trial of innovative methods of in-struction which were developed by the Study staff.

Both of the samples (2 and 3 above) were selected by theAdvisory Committee in conjunction with the Study staff in orderto reflect a similar ratio of a number of characteristicsbelieved to influence the quality of orthopaedic education aswould be found in the total resident training program population.

It was intended to provide two samples that would reflect,among others, the variations in program size, geographiclocation, institution type (university, university-affiliated,or independent), numbers and types of affiliated institutions,program mission, patient composition, and program length sim-ilar to those found in the total group or population.

fs5

-18-

The extent to which programs cooperated can be seenfrom the rate of return for the various data collectioninstruments (see Supplement).

II - NATURE AND VARTA.TIONA - PROGRAMS

In the Final Report of the Phase I of the OrthopaedicTraining Study, four areas were suggested for a descriptionof program variation in orthopaedic training: programorganization, program objectives, program operation, andprogram evaluation. These areas will serve as a basis forthe following description of the nature and variation oforthopaedic training programs presently existing in theUnited States. In addition, ar attempt will be made to char-acterize a typical university, university-affiliated, orindependent program.

The same Program Questionnaire.(see Supplement) wasadministered twice to directors of all the 187 approvedorthopaedic residency programs. There were 150 responses in1969 and 131 in 1971. In addition, there were 116 responsesto the Institutional Description Form from approximately 200institutions associated with the 51 programs included in theOrthopaedic Training Study sample. The Program Questionnaireswill be the main source for information in this section of therer.irt. However, the information from the 131 Program Question-naires returned from the 1971 administration will be reportedin this analysis on:1y when they reveal apparent differencesbetween the earlier 1969 results and the 1971 results.

In the main, however, there did not seem to be muchchange in orthopaedic training from 1969 to the present timeas reported 1; these questionnaires. This is not particularly.surprising as institutional change traditionally occurs slowly.It is imoortant to note, at this juncture, thav where :hangerare indi, ated, the interpretation of these results must bespeculative in nature. Nineteen fewer programs responded tothe administration of the Program Questionnaire in 1971, andit may be that the apparent changing results are a function ofnon-response rather than of actual sample differences. Withthis reservation, where changes are apparent, they will beindicated in this analysis.

PROGRAM ORGANIZATION

This section of the report deals with the personnel, fa-cilities, and resources of the Orthopaedic Training Programsand their associated institutions in the sample. Unless other-wise specified, thr. statistics presented in this section of thereport refer to all programs in the Study sample.

-19-

-20-

It should be noted that this section of the report isconcerned with a description of a typical orthopaedic pro-gram, without regard to program type differences. The finalpages of this section detail the differences among types oforthopaedic programs in the hopes of further highlightingvariations in residency programs.

GENERAL PROGRAM CHARACTERISTICS

Approximately 40 percent of all approved programs fromthe 1969 sample and 47 percent from the 1971 sample identifythemselves as university programs.1 Approximately 29 per-cent of the programs identify themselves as affiliated witheither a medical school or a university,2 while 29 percent ofthe programs in 1969 and 23 percent in 1971 identify them-selves as independent programs.3 (It should be noted thatthese percentages are based upon the description of the pro-gram type by the chief of the program.) It is somewhat morecommon for university programs to be organized as a divisionwithin the department of surgery than as a separate depart-ment; while independent programs are more likely to identifythemselves as departments of orthopaedic surgery in non-uni-versity hospitals. In 70 percent of all the programs, thechief has academic rank, though not necessarily tenure. Thisindicates that almost all of the chiefs of university andaffiliated programs have academic appointments. Sixty per-cent of the chiefs have offices located in the principal

1Typically, university programs are those orthopaedic pro-grams which are an integral part of a university with theattending staff having university appointments.

2Typically, university-affiliated programs are those

orthopaedic programs which are connected to a university ormedical school through a variety of formal arrangements. Theattending staff of these programs may hold universityappointments.

3Typically, independent programs are those orthopaedic

programs which have no formal connections with a universitynor are their staffs drawn from university faculties.

K S

-21-

institution of the program. In slightly over one-third ofthe programs, the bulk of the chief's income derives fromsalary; while in an almost equal number, the chief receivesvirtually no income from that source.

Eighty percent of the programs embrace several asso-ciated institutions usually within the immediate geographic

area. These associated facilities are likely to include aprivate community hospital (in 70% of programs reporting), a"crippled children's" hospital (in 62%), a Veterans' Admin-istration Hospital (in 44% of the cases in 1969 and 50% in1971), and a "charity" hospital -(in 43%).

Even in these multi-institutional programs, the chiefof the program often retains direct control over all aspectsof the program. However, in approximately half of the pro-grams, some of the control is delegated to chiefs of serviceof constituent institutions, and in approximately one-fourthof tho reporting programs, the chief of service is reportedto be completely independent of the program chief. Similarly,even though training is distributed through several institu-tions, the resident is likely to encounter substantially the

same attending staff in all associated institutions; onlyslightly more than one-third of the programs report a sub-stantially different staff in each associated institution.

With respect to the general characteristics of the asso-ciated institutions, almost one-third identify themselves asprivate hospitals. 4 Another third of the institutions areabout evenly divided in identifying themselves as universityhospitals and public hospitals of the city-county type. Thebalance of the institutions report being hospitals which rep-resent some combination of the preceding, and by military,VA, and other non-profit organizational hospitals.

The board of directors is wholly appointed for over 40percent of these institutions. In less than 10 percent of

40ne hundred sixteen institutions are included. in thisanalysis, which constitutes approximately 50 percent of theinstitutions associated with programs in the study sample.The Institutional Description Form was administered only once,in 1969.

-22-

the cases, the boards include members elected from the commu-nity-at-large. Almost three-fourths of the institutions employa professional hospital administrator; most of the rest havea physician administrator. The medical staff administrationis elected from the staff-at-large in 37 percent, and is lim-ited to chiefs of services in 34 percent of the institutions.

Most of the institutions associated with programs in thestudy sample rely on multiple sources of support. Indeed,only four percent of the institutions obtain total supportfrom private foundations and only 10 percent report that theyreceive no government funds for any purpose. Over one-thirdof the institutions report that more than 50 percent of theirsupport is obtained from governmental sources (federal, state,and local), while almost half report that over 50 percent oftheir budget is derived from patient fees. Four percent ofthe institutions report that over 50 percent of their supportcomes from university budgets.

PERSONNEL

While some programs have fewer than five residents andsix attendings and others more than 60 and 80 respectively,most reported between six and 12 residents in 1969. Thesenumbers increased to between seven and 16 residents in 1971.In 1969, the bulk of the programs reported having between fiveand 20 attending staff members, while the numbers increased in1971 to between seven and 21. In the "typical" program in 1969,there were eight residents, 11 attendings, and no research orclinical fellows, while in 1971, the "typical" program was com-prised of 11 residents and 12 attendings. 5 This increase innumber of residents in a typical program may be attributableto pressure on the medical community to develop more trainedphysicians as well as the increasing attractiveness of thespecialty of orthopaedics.

5Only 20 percent of the programs reported any fellowshipstaff and no program reported more than four on the fellowshipstaff.

-23-

In addition to a full range of medical specialistsavailable as professional consultants in virtually all pro-grams, the subspecialties most often represented by theorthopaedic attending staff are hand surgery, orthopaedicpathology, pediatric orthopaedics, spine, and rehabilitation.Table 1 presents information concerning the percentage ofprograms reporting at least one subspecialist in their pro-gram within a particular category. The table should beinterpreted in terms of programs indicating the presence ofsubspecialists, not in terms of numbers of subspecialistswithin a particular program.

TABLE II - 1

PROGRAMS REPORTING SUBSPECIALISTS(Percentages)

1969 1971

Rehabilitation 49.0 53.0Bio-mechanics 20.0 32.0Orthopaedic Pathology 59.0 63.0Hand Surgery 76.0 84.0Neurologic disorders 30.0 31.0Pediatric Orthopaedics 59.0 73.0Spine 53.0 58.0Other subspecialists 15.0 9.0None 9.0 5.0

Table 1 indicates that the trend toward specializationwithin the general field of medicine is also evidenced withinthe field of orthopaedics. This is interesting, since thisspecialization trend is noticeable within a space of only twoyears. Especially noteworthy is the increase in programsreporting bio-mechanics and pediatric orthopaedics as sub-disciplines represented by specialists on the attending staffs.

In addition to the subspecialists represented on theattending staff, these programs report that allied health per-sonnel afford the programs additional specialties. Ninety-ninepercent of the programs report physical therapists comprising

-24-

part of their allied health staff, while 95 percent of theprograms have social workers on the ancillary staff. Over80 percent of the programs report prosthetists, orthotists,and occupational therapists as comprising part of theirallied health personnel. Additionally, over 80 percent ofthe programs report psychiatrists as part of the programpersonnel.

Other supporting services within the institutions asso-ciated with the sample programs are most likely to include anursing supervisor for in-patient services (in 98% of theinstitutions reporting), for the operating suite (in 97% ofthe institutions), for the emergency room (in 65% of theinstitutions), and for the out-patient service (in 68% of theinstitutions. An administrative supervisor is slightly lessfrequently available. For both in-patient and out-patientservices, the administrative supervisor is available in 80percent of the institutions. This percentage drops to GO forthe operating room and 50 percent for the emergency room.Eighty percent of the out-patient clinics have a registerednurse, but less than half have practical nurses, orderlies,plaster technicians or social workers, assigned solely to theorthopaedic clinic. Additionally, one is least likely to findpsychological services (available in 43% of the institutions),counseling services for vocational rehabilitation (in 39% ofthe cases), and home nursing services (in 41% of the institu-tions) as part of the support services within the institutionsassociated with the programs in the Study sample.

FACILITIES

In addition to both the general orthopaedic and children'sclinics found in virtually all programs, most programs reportthe following out-patient subspecialty clinics as also avail-able in at least one institution associated with the program:fracture (in 87% of the reporting programs), amputation andprosthetics (in 77% of the programs), arthritis (in 60% of theprograms), and scoliosis (in 52% of the 1969 responding pro-grams and in 63% of the 1971 responding programs ). However,only 40 percent of the programs report a long-term follow-upclinic where patients treated five years in the past arebrought back for review.

-25-

With respect to the institutions associated with pro-grams in the Study sample, the types of facilities mostcommonly included as part of the institutional structure arethe following:

Out-Patient Services: About one-half of the institu-tions associated with the Study programs report orthopaedicsubspecialty clinics. The most frequently encountered clinicwas a hand clinic, with 72 percent of the institutions report-ing this subspecialty. The presence of amputation prostheticsand children's subspecialty clinics were reported by 70 and 65,..Dercent of the institutions respectively, while a specialarthritis clinic was reported by about one-half of theinstitutions.

In-Patient Services: The institutions associated withthe study programs reported that the average number of bedsavailable to all services is 511, and to the orthopaedicservice, 64. 6 While 44 percent of the institutions report thatthey have no private, single-bed rooms, such facilities accountfor over one-half of the orthopaedic beds in 40 percent of theinstitutions.

Seventy-six percent of the institutions report in-patientswith fracture problems; almost two-thirds report in-patientswith other trauma problems and hand problems. However, onlyslightly more than one-half report in-patients with rehabili-tation and arthritic problems.

In approximately one-third of the institutions, admissionfor both private and non-private patients are scheduled accord-ing to general bed availability. Only three percent of theinstitutions report that patients are screened for admissionstrictly according to the teaching value of their problems,though an additional 21 percent report less strict applicationof this screening criterion. In 58 percent of the reportinginstitutions, residents do not have authority to arrange foradmissions.

6To the extent that respondents failed to answer allquestions, these figures may represent an underestimate.

;13

-27-

The mean autopsy rate for these institutions is 48 per-cent and that for the orthopaedic services is 43 percent.

Surgical Facilities: The mean number of operating roomsin a sample institution is eight. The mean number of roomsassigned specifically to orthopaedics is one. In 67 percentof the reporting institutions, the individual surgeon sched-ules his own elective cases. In 84 percent, he schedules hisemergency cases, while in nine percent of the institutions,scheduling of emergency cases is done through a chief surgicalofficer. In 63 percent of the institutions, orthopaedics isassigned specific operating days. The operating room is underhospital control in half of these institutions, general surgerycontrol in 22 percent and individual specialty control in 12percent of the reporting institutions. Fifteen percent of theinstitutions assign nursing and technical personnel accordingto general availability. Approximately three-fourths of theinstitutions report having an adequate assortment of surgicalequipment and appliances readily available, though 11 percentrequire special purchase of endoprosthetic devices and threepercent require that the surgeon supply his own instruments.All institutions have x-ray facilities available in the oper-ating room, while 23 percent have this type of equipment andtechnicians assigned specifically to the operating roomroutinely. In 53 percent of the institutions, anesthesiologistsadminister all general anesthesia. Only nine percent of theinstitutions have nurse anesthetists who work without super-vision of an anesthesiologist. Finally, in 73 percent of theinstitutions, both the surgical and anesthesia services admin-ister regional anesthetics.

Supporting Services: Satisfaction with the quality ofsupporting facilities was most likely to be reported for med-ical and nursing administration (in approximately 90% of theinstitutions), and for x-ray, special laboratory, prostheticsand physical therapy facilities (in 80 to 85% of theinstitutions). Social service, occupational therapy andorthotics were viewed as adequate or superior in 60 to 70 per-cent of the institutions: while research, school facilitiesfor patients, convalescent, psychiatric counseling, and voca-tional rehabilitation were most often reported as unavailableor inadequate by 40 to 50 percent of the institutions.

-28-

Although approximately one-third of the institutionshave formal arrangements or affiliation with convalescentand custodial care facilities, the majority do not havesuch arrangements. In fact, at least three percent of theinstitutions are in communities where no such facilitiesexist.

Record Facilities: Over two-thirds of the reportinginstitutions have the capability of retrieving in-patientcharts according to diagnosis or treatment, whereas onlyone-third can retrieve out-patient records in this manner.

Library Facilities: Whereas 80 percent of the insti-tutions have a medical school library available, only aboutone-half have an orthopaedic departmental library availableto the residents for at least eight hours daily. More thanhalf of the libraries used by the residents are relativelysmall, fewer than one-sixth reporting collections of over75,000 volumes.

Secretarial Service: Approximately three-fourths ofinstitutions report that such services are available tohouse staff for in-patient records and activities, whiletwo-thirds report such facilities available for out-patientactivities. One-quarter of the institutions report clericalsupport for research endeavors. Almost one-third report liai-son personnel for community resources. Less than 10 percentreport no clerical support available.

RESOURCES

One of the vital resources of any orthopaedic trainingprogram is the patient population. In this section of thereport, we will attempt to summarize the variations in pa-tient population and the resultant implications of this vari-ation for residency programs.

Size of Patient Population: The average number of out-patient visits to orthopaedic clinics is 3,858 although the

-29-

number varies significantly from one institution to anotherwithin a program.?

In addition to the orthopaedic clinic, about half the

institutions include a separate arthritis clinic in whichthe main number of annual out-patient visits is 454. Finally,the average number of annual out-patient visits in the medicaland surgical clinics of responding institutions is 17,562 and15,767 respectively.

Clinical Variety: Despite differences among institutionsin the pattern of administrative responsibility for patientcare, a few widely applicable generalizations can be made.Approximately 80 to 90 percent of the institutions report thatthe orthopaedic department assumes administrative control overpatients with the following problems: fractures, musculo-skeletal tumors and amputations occasioned by other than pe-ripheral vascular disease. It is interesting to note that inapproximately one-fourth of the institutions, the orthopaedicdepartment also assumes administrative responsibility for

-patients with'lumbar disc, hand problems, chronic spinal cordinjuries, or rehabilitative cases requiring long-term care.Finally, in between 10 and 15 percent of the institutions,the orthopaedic department assumes administrative responsi-bility for patients with arthritis, cervical disc, acutespinal cord injuries, or for patients requiring amputationdue to peripheral vascular disease.

Most programs report that the most frequently encounteredproblem is trauma, representing up to 60 percent of allpatient problems in a few programs. Rehabilitation problemsare least common, representing less than 20 percent of allpatient problems in most programs.

Age Distribution: In hospital settings other than thoselimited to specified groups (military, veterans, or children),the most frequent pattern with respect to the age distributionof the patient population is that 80 percent are adults and20 percent are children. This picture is characteristic of

7To the extent that respondents failed to answer all

questions, these figures may represent an underestimate.

.36

-30-

over 40 percent of all institutions to which residents areassigned.

Residential Distri'Altion: Over one-third of the insti-tutions in the Study sample serve patients who reside inmetropolitan areas of over one million peons. Fewer. than10 percent of these institutions serve predominantly ruralpopulations of under 10,000 people, while most of theremaining serve patients from urban centers of between100,000 to 500,000 people.

Socio-Economic Status: Although the ratio between pri-vate and non-private :)atients varies significantly amongprograms, only eight percent of the programs report no "pri-vate" patients available in the entire program, and onlyone percent report no "non-private" patients.8 Secondly,whether the private or the non-private patient predominatesin a particv!lr program, that type usually constitutes nomore than 80 percent of the total patient population. Inshort, it would appear that most programs serve patients fromvarious socio-economic levels. However, the opportunity foran individual resident within a program to gain experiencewith patients of various socio-economic levels is dependentupon the specific rotations available. Thirty-seven percentof the institutions report that they deal predominantly withpatients in the lower socio-economic groups.9 Indeed, overa fifth of all institutions report that they serve predomi-nantly indigent populations and another seven percent servepredominantly Medicare patients. Of the remainder, 51 per-cent of all institutions report that more than half of their

8Private patients are, in general, patients who arebilled directly by their attending physician for servicesrendered. Non-private patients are, in general, patientswho are not billed directly by the responsible physician.

9This category was not defined in the InstitutionalDescription Form, and hence, it is not possible to statethe income boundaries of thi...; designation.

.37

private patients have care financed through insurance plc.ns,and only four percent of the institutions report that between20 to 60 percent of these patients finance their care whollyfrom private resources.

In addition to this economic diversity in the patientpopulation, 60 percent of the institutions report that the21,serve non-English speaking patients to some degree. However,there is an interpreter available in slightly more than one-half of these institutions reporting a substantial number ofnon-English speaking patients.

Approximately one-half of the institutions report Thatthere is a communications problem between patients andresidents. This may, in part, reflect the disparity betweenthe socio-economic and ethnic backgrounds of the patientpopulation and that of the resident staff.

It is clear from the foregoing that the hinds of cliniclproblems and the varieties of socio-economic groups to whichthe resident is exposed depends almost entirely on the natureand location of the institution in which he is trained. Lessthan onc-fourth of the institutions report that they doscreening for the educational value of the medical problem.

PROGRAM OBJECTIVES

Orthopaedic training programs have both medical andeducational objectives which are said to direct the function-ing of the program operations. Eighty-four percent of theprograms report that their primary objective is to "produceclinicians oriented to community practice". The productionof subspecialists or academicians is ranked as the primaryobjective by only 15 percent of the programs. The productionof research-oriented orthopaedists is regarded as of firstimportance by only one percent of the programs, while slightlyover one-half rank this objective as third in priority.

The most frequently encountered educational pattern.characteristic of almost one-half of all programs, cuasistsin four years of orthopaedics following internship, althoughapproximately one-third of the programs offer a throe yearorthopaedic residency following 12 months of general surgery.

-32-

Nearly half the programs offer complete training in ortho-paedics plus options for development of more specializedinterests. Virtually, all others offer complete trainingin orthopaedics as standardly defined.10

Two-thirds of the programs include a rotation in neuro-surgc,ry sometime during the four years and over one-halfinclude rotation in plastic su;:gery. The sample from 1969reported that, in 53 percent of the programs, a rotation in"trauma" was also offeree., while the 1971 sample reportedthis rotation available in 65 percent of the programs.

Typically, clinical teachirr, includes the followingtypes of ,-Livitics on a regul-Arly schcduled basis: didacticinstruction, bedside teaching rounds, group conferences, workrounds, surgical e%purience and out-patient and specialtyclinics supervised by an attnnding. In 77 percent of the

scHlee instruction is offered within thepr.cac-ar and this instruction is distributed throughout the

tour years. In appro:dmately one-third of the reporting pre-gr;:.; 1)69, the resident was required to submit a researchpaper, and in all but a few of these programs, he is expected

to carry on his investigation and clinical work concurrently.In ]971, the percentage of programs requiring a researchpaper iricreast'd to 45 percent.

In order to determine if this apparent increase in empha-ois on research was an artifact of the differences in samplesbctween the two years, a re-analysis of the results were under-taken using only programs which had responded to both of thequestionnaire administrations. The 117 institutions whichresponded to both the 1969 and 1971 questionnaires showed anincrease in emphasis on research. Thirty-six percent of theseprograms reported that a research paper was mandatory in 1969,while that figure is 42 percent in 1971. However, despite theincreased emphasis on the submizsion of a research paper, thereis not a corresponding increase in the percentage of programsallotting specific time for work on the paper.

10Standardly defined training involves complying withthe requirements of the Board.

9

-33-

In about two-thirds of thy: programs reporting in 1969and 78 percent reporting in 197, presentation of papersat national, meetings is actively encourar - These per-centages wore identical when the 1.1.7 programs respondingto the 1969 and 1971 questionnaire administration wereanalyzed.

Attendance at one (or more) annual national meetingwas mandatory in 53 percent of the progras in 1969 and in61 porcent of the programs reporting in 1971.

There appears, then, to be a trend toward increasedresident )articipation ins research activities in residencyprograms, even over the short time of two years. Thistrend may have implietious for objectives, further programdirections, and allocations of resources within the ortho-paedic community.

PROOPA.1`,1 OPEFATION AND EVALUATION

The variatiens in program operations arc undoubtedly afunction of the organi:!,ation and objectives of the program.As a consequence, this section of the report connects theprevious material with information about resident responsi-bility, in terms of teaching and patient care, and with feed-back and supervisbn mechanisms

Resident TeachincT Responsibilities: In most programs,the resident is expected to participate in the instructionnot only of other residents and interns, but also of medicalstudents in 60', of the programs reporting in 1969 and in 67%reporting in 1971), and of allied health professionals includ-ing nurses and physical therapists.

Resident Patient Care Responsibilities: Table 2 presentsdata illustratina the' responsibility of residents for patientcare over the length of the residency. Eighty-eight percentof the progrem chiefs report that residents begin pe_formingsimple orthopaedic operations in the first year of trainingP.nd progress systematically toward the more complex procedureswith increments in experience. It is clear, however, fromTable 2 that that first-year residents in some programs areexpected to assume a level of responsibility that fourth-year

40 .

-34-

residents in other programs never assume. Thus, for example,while residents in most programs can expect to assume fullmedical responsibility for frequently encountered types ofsurgical trauma sometime between late in their second-year andearly in their fourth-year, three percent of the chiefsreport that such responsibility is usually assumed by first-year residents and five percent of the chiefs report thatsuch responsibility is never assumed by residents.

It is evident, then, that there is a rate differentialamong programs in the level of responsibility allocated toresidents.

It is clear, also, from Table 2 that there has been anincrease in providing residents with more responsibilityearlier in their residency from 1969 to 1971. There are sub-stani:ial increases in resident responsibility displayed onthis table, such the increase from 39 percent in 1969 to59 percent in 1971 of programs reporting that first-year resi-dents are responsible for out-patient trauma cases. Thisincrease colild not be accounted for solely by idiosyncrasiesof the responding programs in the 1971 sample. In every case,without reversal, there was an increase in responsibilityassumed by residents at each type of orthopaedic procedure.

SUP:RVISION ANDFEEDBACK

In approximately two-thirds of the programs, chiefsreport that actual supervision of residents in the operatingroom is provided in accord with the needs of the individualresident. In the rest of the program, actual supervision insurgery is provided for all cases regardless of complexity.Finally, 89 percent of the chiefs report that residents areregularly advised of their rate of progress and their areasof deficiency.

SUMMARY

The preceding section has attempted to describe ortho-paedic training programs with major emphasis on the natureand variation of these programs in terms of facilities, per-sonnel, organization, and resources. Less information waspresented concerning the objectives, feedback mechanisms, and

41

-35-

evaluative procedures employed by the programs. It wouldseem that future studios along these lines might be mostproductive if those relatively unexplored aspects of train-ing programs were thoroughly investigated.

A PROGRAM TYPOLOGY

In order to more clearly depict the nature and varia-tion of orthopaedic residency programs, a typology of pro-grams was developed based on the affiliation dimension ofprogram orqanization. That is, a characterization of atypical university, university-affiliated, and independentprogram was created. Table 3 displays that characterization.

The university programs appear to have more availablefacilities and resources than the other two types of programs.There are more out-patient clinics, subspecialists, and asso-ciated hospital facilities available to university programsthan. university-affiliated and independent programs. Uni-versity programs also tend to supervise resident training moreclosely and wait longer to give residents patient care respon-sibility for simple orthopaedic procedures. It is also acharacteristic of university programs that residents are per-mitted to become involved in special interest areas and areencouraged to give papers at annual meetings.

University programs tend to be more similar to university-affiliated porgrams than are either of these types to indepen-dent programs. These differences are particularly apparentwith respect to general organization and administration. Thesedifferences tend to disappear with respect to resident trainingper se.

Independent programs are characterized by more authorityvested in the chief of the program than chief of staff, by anemphasis on internship performance as a criterion for admissionto the program, and by fewer additionally required rotationsfor residents. Additionally, independent programs have fewersupspecialists, associated facilities, and out-patient clinicsthan university or university-affiliated programs. Independentprograms permit residents to assume non-surgical patient careresponsibilities early in their residency but maintain closersupervision for surgery at all levels of competency. Finally,

/12

-36-

these programs are characterized by comparatively moreemphasis on attendance at: annual meetings and optionalresearch papers, and with higher pay for the resident staff.

It should be noted that, in general, the characteriza-tion of the three types of programs are quite similar. Thereare standard facilities and resources available irrespectiveof program type, admissions criteria which are accepted inall three types of programs, and educational objectives andmechanisms for accomplishing these objectives which appearto remain similar throughout orthopaedic residency programs.

43

TABLE II - 2

RESIDENT PATIENT CARE RESPONSIBILITY BY YEAR IN TRAINING*

(Percentages)

1Year in Training

23

45

Never

Frequently encountered types of

"out-patient" trauma (e.g., simple

39.0

41.0

3.0

1.0

0.0

1.0

metatarsal fracture)

(54.0)4"

(31.0)

(11.0)

(2.0)

(0.0)

(0.0)

Frequently encountered types of

"in-patient" but non-operative

19.0

40.0

30.0

9.0

1.0

1.0

trauma (e.g., fracture of pelvis)

(21.0)

(47.0)

(18.0)

(9.0)

(0,0)

(1.0)

I.,

I

44,

Frequently encountered types of

Wsurgical trauma (e.g., hip

3.0

17.0

50.0

23.0

2.0

5.0

I

fracture)

(2.0)

(27.0)

(43.0)

(20.0)

(3.0)

(2.0)

Simple reconstructive surgical

procedures (e.g., elective

3.0

31.0

45.0

16.0

1.0

5.0

bunionectomy)

(5.0)

(33.0)

(45.0)

(11.0)

(2.0)

(3.0)

Complex reconstructive surgical

0.0

0.0

16.0

65.0

4.0

12.0

proglems (e.g., hip arthoplasty)

(0.0)

(2.0)

(23.0)

(57.0)

(5.0)

(11.0)

*The chiefs on the program were to state the earliest

year of Residency in which

responsibility was first given to residents.

+1971 results

PROGRAM CHARACTERISTICS BY TYPE OF PROGRAM.

TA

BL

E I

I3 Type of Program

Proaram Characteristic

University

University-Affiliated

Independent

Program Status (l)*

Division of department of surgery

Department of Orthopaedic Surgery

Department of Orthopaedic

in a public university

in a non-university hospital af-

Surgery in a non-university

filleted with a public university

hospital

Associated hospitals (4)

Several hospitals

Several hospitals

Several

hospitals

Function of Program chief (5)

Partial control to chiefs of

Partial control to chiefs of

Direct ccntrol

service

service

10

Position of Program chief (7)

Academic rank and tenure

Academic rank

No academic rank

m

Location of Program chief (8)

Office in principal institu-

Office in principal institu-

A mile from princip

tion

tion

institution

1

(.4

Position of Chief of Service

Academic rank

Academic rank

Belongs to practice group

$Z

(9)

of chief of program

tiIncome limits on staff (11)

Limits

1:- limits

No limits

07 01 U S 0

Allied health personnel** (24)

Physical therapist, social worker, Physical therapist, social

Physical therapist, prosthe-

prosthetist, orthotist, psychia-

worker, prosthetist, orthotist,

tint, social worker, ortho-

trists, occupational therapists

psychiatrists, occupational

tist, psychiatrists, occupa-

therapists

tional therapists

Consultants (25)

Full range

Full range

Full range

Special required rotations**

Neurosurgery

(32)

Neurosurgery, plastic surgery,

Neurosurgery

trauma

Out-patient clinics** (33)

General, children's, fracture,

amputation, prosthetics, arth-

ritis, scoliosis

General, children's, fracture,

amputation, prosthetics

General, children's frac-

ture, amputation, pros-

thetics

*Question number from Questionnaire Program.

**Listed in order of frequency of response.

TA

BL

E I

I -

3 (C

ont i

d)PROGRAM CUARACTERISTICS BY TYPE OF PROGRAM

Program Characteristic

Type of Program

University

Univer:;ity-Affiliated

Independent

C.4

Staff Subspecialists** (35)*

0

Hand, pediatrics, ortho-pathology, Hand, pediatrics, ortho-pathology

spine, rel:abilitation

rehabilitation

Hand

Wvi

Facilities** (36)

Private. community hospitals,

?rivate community hospitals,

crippled childre: 's, VA, charity

crippled children's hospitals

(public) hospitals

Private corrmunity hospi-

tals, crippled children's

hospitals

C W

-.4

0, C

Attendings responsibility

(13)

Formal teaching and clinical

work

Formal teaching and clinical

work

IVI

C.:V

-V >

rnvt

.0 43

0

Objectives (29)

Community practice orthopaedics

and subspecialists, academicians

Community practice

0 V 4J

Admissions criteria** (19)

Medical school standing, intern

performance, intellectual ap-

proach to problems

Medical school standing, intern

performance, intellectual ap-

proach to problems

Program length (26)

Four years

Four years

Program continuity (28)

Training in more than one insti-

tution, but same staff

Training in more than one in-

stitution. with different staff

Clinical sources (30)

Non-private patients

Non-private eatients

Clinical variety** (31)

Basic science (38)

Trauma, adult, general, pediat-

rics, rehabilitation

Formal teaching

Community practice

Intern performance, intel-

lectual approach to prob-

lems

Four years

Training in more than one

institution, same stiff

Split between private/ncn-

private

Adult, general, trauma, nediat-'

Trauma, general, pediatrics,

rics, rehabilitation

rehabilitation

Integral

Integral

Integral

Resident's teaching respon-

sibility (39)

Medical student residents, in-

terns, allied health

Residents, interns, allied health,

Residents, interns, allied

medical students

health

*Question number from Questionnaire Program.

**Listed in order of frequency of response.

0

TA

BL

E I

I -

3 (C

ont '

d)FROG= CHARACTERISTICS BY TYPE O

PROGRAM

Type of Program

Program Characteristic

university

Univerity-Affiliated

Independent

Program design (27)*

Orthopaedics and options for

Orthopaedics

Orthopaedics

ci

0 .6)

yResearch paper (40)

cMeeting attendance (42)

.6)

special interests

Optional

Cptional

Optional

Encourage papers

Encourage papers

Meetings mandatory

Supervision training (43)

Close

When necessary

Close

Operative supervision (47)

According to need

Patient care responsibility

(46)

a

According to need

All cases

Systematic progression in per-

formance

Systematic progression in

Systematic progression in

performance

performance

Operative responsibility (45)

Complete by 4th year

Diagnostic report to residents

Advised of progress

(44)

Complete

by4th year

Complete by 4th year

Advised of progress

Advised of progress

.1.)

Resident first two year salary

$5,000 - 7,500

E0(20)

o .0 o

-1

uG

l Q I::: -I

Resident second two year

$5,000 - 10,000

asalary (20)

S7,500 - 10,000

$5,000 - 10,000

$7,500 or more

$7,500 or more

*Question number from Questionnaire Program.

**Listed in order of frequency of response.

II - NATURE AND VARIATIONB - RESIDENTS AND ATTENDINGS

The Typical Orthopaedic Resident

The typical orthopaedic resident identifies by theStudy is 30 years old, male, married, Caucasian, has oneor two children, comes from the Midwest or Middle Atlan-tic states, frequently has a father or uncle who is an M.D.,and often a wife, sister, or mother in a health-relatedoccupation, usually nursing. His educational background andhis family's economic status are well above average. One-third of the residents have fathers with a professional de-gree and one-fourth have mothers with a college degree;75 percent have fathers in executive or professional posi-tions; 52 percent have fathers who are self-employed. Theresident is among the intellectual elite; nearly half ofthe resident population in the Study sample were in the

upper five percent of their high school class; over two-thirds were in the upper quarter of their college class; andnearly half were in the upper quarter of their medical schoolclass.

A desire for independence, an interest in science, inpeople, and in the workings of the body, and the prestigeof the profession were strong influences in his selectionof medicine as a profession--a decision he probably madeabout the time of high school graduation. Of little impor-tance were books or articles, movies about medicine, orprevious vocational guidance. The nature of the specialtyand of the clinical material, a desire to perform surgery,an ability to work with his hands, and his experiences dur-ing the internship were strong determinants in his choiceof orthopaedics as a career--a decision he tended to delayuntil his internship year.

In general, reputation of the institution and its geo-graphical location, as well as financial considerations,played a major role in his choice of both a medical schooland a residency program. However, in selecting the latter,he was also influenced by his desire to work with a particu-lar person and by his associations or experiences in medicalschool. Over 80 percent of the residents report that theywere appointed to the program of their first choice; onlyfive percent report that they were appointed to a programof third choice or lower. Approximately 60 percent ofresidents indicated at the time of the survey that they aspire

-41-

Js

-42-

to private practice in a small orthopaedic clinic and thatthey hope to have some medical school affiliation and dosome teaching.

In order to gain further insight into the variationsof the background of residents, the following descriptionof an atypical resident is presented. All of the character-istics mentioned do not necessarily describe any one indi-vidual or set of individuals, but they exist in the popula-tion under study. These characteristics highlight the rangeof responses rather than the mean or average response onwhich the typical resident description is based.

The Atypical Orthopaedic Resident

The atypical orthopaedic resident could be 25 yearsold (the age of most senior medical students) or 37 yearsold in his sixth year of training (the age of two chiefsof orthopaedic programs). He could soon be among the morethan 60 residents who left orthopaedics or changed to an-other program during the last five years, more than halfof whom were dismissed for irresponsibility (moral orethical) or poor clinical judgment. His father may nothave completed high school, as was the case with almost20 percent of the residents, while his parental familyincome last year was probably less than $7,500, and hisfamily's economic position last year was lower than it was10 years ago. Seven percent of the residents have accumu-lated over $10,000 in educational debts while one percentearned less than $4,000 as a senior resident. The atypicalresident may be among the three percent who maintained astraight C average through high school, college, and medi-cal school, and did not take any biology courses in highschool (5%) or college (1%), still he was able to becomean orthopaedic resident (even though he was admitted to aprogram which was his third or lower choice (5%). Hecould have decided to study medicine at age 27 and ortho-paedics at age 34. He could have elected medicine at agesix and decided on orthopaedics as a college freshman atage 18. A majority have seriously considered another pro-fession. Consequently, our atypical resident might havebecome an educator (7%), business administrator or lawyer(each 4%), orester or aeronautical engineer (each 3%) or.math-ematician (2%). He may be among the 29 percent who read

49

medical literature for less than 10 hours a week and spendmore time than that in employment outside the residency(12%). In fact, almost one-third of the residents reportedthat they do work outside the residency in an area directlyrelated to medicine, while this practice is expressly for-bidden by most training programs.

Resident Attitudes

As shown in tables 7 and 8, between 70 and 85 percentof the residents whose attitudes were surveyed showed asatisfaction with many aspects of their residency programand with the staff and residents with whom they work. Theydo, however, have some complaints about hospital procedures.**Their responses can be summarized in three categories, asfollows:

1. Satisfaction with attendings as teachers.(See table 4) .

2. Acceptance of responsibility to teach studentsand the satisfaction with the personal benefitsof having to teach (improving technique withthe extra practice and preparation by extrareading). (See table 5).

3. Dissatisfaction with hospital procedureswhich waste resident's time ("scut work",excessive night duty, services which areinadequate or available only with much dif-ficulty).** (See table 6).

**Residents are equally divided on this issue, i.e., asmany are satisfiea as are dissatisfied.

50

-44-

TABLE II - 4

Satisfaction with Attendings As Teachers

1. The full-time faculty in my department understandsand appreciates the contributions of the resident.

2. I have sufficient opportunity to work closely with thesenior (attending) staff.

3. The attendings in my department exert a strong efforttoward making the resident's experience a valuable one.

4. My department is generally regarded as having a"stimulating" teaching staff.

5. It was known to the residents that this residencyprogram had been organized with specific objectivesin mind.

6. The full-time staff in my department places greatemphasis on the practical management of problems.

7. Attendings consciously strive to improve their per-formdnce as teachers.

8. Residr,nts are helped to understand the source ofimportant problems they may be facing.

9. When I began my assignment on this specialty, theresident's role was made clear to me.

10. Co:Istructive suggestions are offered to residents indealing with their major problems.

11. Evaluation of the residents goes on constantly.

12. If I had it to do over again, I would select thisresidency program.

13. The different skills found in the attendings are fullyutilized.

14. Most of what I have learned thus far has been super-vised to my satisfaction.

15. The attending staff is sufficiently aware of my per-formance to render an accurate evaluation of mywork.

5

TABLE II - 5

Resident Teaching Responsibilities

1. Students on my service do not hinder the performanceof my duties.

2. Having students arou.td causes me to "read up" morethal I might have done otherwise.

3. Having to teach does riot take time away from moreimpertant things I want to do.

4. The presence of students on my service makes it abetter "teach!ng service."

5. I now believe this residency would be worse withoutstudents.

6. Teaching students how to perform certain procedureshelps to improve my own toch,lique.

7. Residents play a major role in teaching.

8. The presence of students probably affects my learningone way or the other.

9. My service is student-oriented.

52

-96-

TABLE II - 6

Dins,, '-raction with Hospital Procedures

1. Time spent arranging to have laboratory tests donedetracts from time I could use on some more produc-tive z.reas.

2. Residents do too many Juenial tasl:s on my service.

3. Residents are exploited by the hospital and itsattending physicians.

4. I feel that the resident's assignments are determinedprimarily by a need to provide "service" to patients.

5. Paramedical personnel have too much control over aresident's activities.

6. I have too much night duty.

7. Clinical laboratory services are inadequate andavailable with too much difficulty.

B. Demands on the residents are so great they are al-most impossible to meet.

-47-

TABLE 11 - 7

Many of the residents who responded to the surveyAGREED with the following statements:

Eighty-five Percent AgLeement:

Regularly scheduled resident meetings are a valuableeducational activity.

Most residents here make an effort to keep up withthe current medical literature.

Residents are open and free about exchanging information.

The residents get along together socially.

Seventy Percent Agreement:

Residents play a major role in teaching on my service.

I have sufficient orTertunity to work closely withthe senior (attending) staff.

The attendings in my department exert a strong efforttoward making the resident's experience a valuableone.

Residents have the feeling that their work is an impor-tant activity.

The full-time staff in my department places greatemphasis on the practical management of problems.

There is a congenial relationship between most ofthe attendings and residents on my service.

Teaching students how to perform certain procedureshelps to imprave my own technique.

Constructive suggestions are offered to residents indealing with their major problems.

Evaluation of the residents goes on constantly.

04

f-48-

TABLE II - 7 (Cont'd)

Residents are willing to help out when a fellowresident has a great deal of work to do.

I have sufficient opportunity to develop and use myclinical skills during residency.

If I had it to do over again, I would select thisresidency program.

-49-

TABLE II - 8

More than 70 percent of the residents who respondedto the survey DISAGREED with the following statements:

Evaluation of a resident is based primarily onsurgical skill.

Students on my service hinder the performance ofmy duties.

Residents do too much research.

The morale of the resident staff is low.

The number of private patients on my service isinadequate.

Having to teach takes time away from more importantthings I want to do.

I now believe this residency would be betterwithout students.

Instruction from the full-time facu:.ty is tootheoretical or abstract.

Most of what I have learned so far in this resi-dency has been from paramedical personnel.

-50-

Residents' Work Week

For seven days each resident kept a daily log of hisactivities in each of seventeen categories (of which the 11most prominent are included here). These data were thencombined to describe an average work week for residents ineach of the four residency years. The findings are summa-rized in table 9.

The most striking item is the length of the averageworking day, 11 to 12 hours, seven days a week over a four-year period. With such a pattern established during train-ing, the continued stamina and drive which seem to charac-terize the orthopedic community should come as no surprise.Whether such intensity provides an optimal experience forlearning other attitudes and values, as well as the problem-solving skills and reflective thinking which represent resi-dency program goals, is in question. However, as the thirdcategory of resident attitudes indicates, residents areequally divided regarding complaints about the amount oftime spent in night duty, obtaining equipment and services,etc.

The essentials of an approved residency of the AmericanBoard of Orthopaedic Surgery, like the essentials of an ap-proved internship, state unequivocally that these trainingexperiences must first be designed for learning. In thecourse of this learning, medical services must be rendered,but service is relatively a secondary, rather than a pri-mary, outcome. While there may be differences of view aboutthe extent to which on-the-job training is primarily educa-tional, there would probably be fewer disagreements with .theprinciple that effective learning requires some designedsequence, and that learning critical judgment in patientmanagement demands graded and ever-increasing responsibility.The 11 time components reported in table 11-9 (p. 52) are di-vided into three elements arbitrarily labelled formal learn-ing_ (observing surgery, assisting surgery, attending learningactivities, professional reading, and research), executing(which also embraces learning by doing through in-patientcare, out-patient care, emergency room care, performingsurgery, and clerical work), and teaching. The resultingcategorical time commitments are summarized in table II-10.While there is a tendency toward decrease in formal learningactivities and increase in execution dnd teaching over the

57

-51-

four years, which may suggest some sequence and gradedresponsibility, none of these differences is statistic-ally significant and certainly do not appear great enoughto have educational significance. Perhaps most surprisingis the observation that research, which many would feelrepresents a culminating experience that is more likely tobe fruitful if based upon a steadily widening familiaritywith significant problums, appears to occupy far moretime in the first residency year than in the final year.However, since some programs completely devote the firstyear of residency to resident research, the average amountof research done by first-year residents may be inflatedby inclusion of these programs.

58

TABLE II - 9

Mean Hours per Week Residents Spent in Selected Activities

1st Year

2nd Year

3rd Year

4th Year

XSD

XSD

XSD

XSD

I.

In-patient care

17.0

7.45

17.8

6.45

15.3

5.52

15.7

5.23

II.

Out-patient care

9.3

5.21

11.7

5.74

10.3

5.75

11.7

5.80

III.

Emergency room

6.5

6.25

7.1

5.51

7.4

6.54

5.6

6.00

IV.

Observe surgery

1.1

2.01

1.1

1.52

1.0

1.39

1.3

1.90

vi

NJ

V.

Assist surgery

9.0

4.92

7.5

4.17

7.5

4.35

6.3

4.07

I

VI.

Perform surgery

4.4.

3.76

5.9

3.52

6.8

3.94

7.5

4.51

VII.

Teaching

3.3

3.27

4.7

3.24

7.6

4.52

8.7

4.95

VIII.

Attend scheduled

learning activities

6.5

3.10

7.6

3.47

6.8

3.46

7.5

3.52

IX.

Professional

reading

9.0

3.17

10.4

4.20

8.6

2.79

8.3

2.98

X.

Research

6.3

5.43

2.9

3.80

2.9

3.40

2.4

2.82

.XI.

Clerical work

4.9

2.83

5.4

2.06

5.5

2.62

5.7

3.06

WeeklyHours

per.Category

1stYear

2ndYear

3rdYear

4thYear

TABLE

II

- 10

Learning

31.9

29.5

27.6

27.8

CI)

Executing

42.1

47.9

45.3

46.2

Teaching

3.3

4.7

7.6

8.7

-54-

Operative Procedures

On the average, a resident spends 14.9 hours per week,775 hours per year, or approximately 3,000 hours during hisresidency in the operating room either observing, assisting,or performing surgery. We now turn our attention to whenresidents learned, from whom they learned, and how confidentthey are about what they learned in this time.

The residents are from quite varied training programs

which have their own characteristics in terms of patientpopulation, selection of cases for resident teachingand training, and precedence with regard to delegation ofresident responsibility and authority. (ee the next sec-tion for a discussion of delegation of authority.) However,clear patterns emerged which can describe most residentexperiences with certain operative and nonoperativeprocedures.

Residents responded to five questions about each proce-dure by checking one of the alternatives provided.* Thequestions were:

1. When did you first learn this procedure?

2. How many times have you performed it?

3. How well can you perform it?

4. Who first taught you?

5. If self-taught, what method(s) did you use?

A total of 53 procedures were selected to representvarying degrees of difficulty and did not attempt to berepresentative of procedures that all orthopaedists shouldknow.

It is interesting to note that residents at differentlevels of training respond differently when asked when theylearned a particular procedure. The percentage of residentswho report having learned a procedure before beginning

*See Resident Procedures Form, Supplement.

61

-55.-

residency is usually highest for first-year residents, witha steady decline in this percentage for second-, third-, orfourth-year residents. This finding would support the hy-pothesis that what the resident accepts as evidence oflearning changes as he progresses through his residencytraining; he may discover during residency that he reallydid not know all of the implications and possible complica-tions of a particular procedure as well as he thought hedid. This decline in perceived early competence is commonfor many of the procedures surveyed.

Previous Experience

There were 15 procedures that over 75 percent of theresidents stated they learned before entering orthopaedicresidency. These procedures consist primarily of two typesof experiences. First, those procedures commonly delegatedto students, interns, or first-year residents (e.g., inject-ing a painful joint). Although there can he a good deal ofdiscussion about how well they are accomplished, the tashsare generally considered to be relatively simple and havea wide margin of safety, and any errors wl.aich occur can bemodified fairly.easily when seen later by a more experiencedperson. The second types of experience are those procedureswhich are ordinarily emergent in nature, and, perhaps moreimportant, frequently quite urgent. A resident confrontedwith a patient requiring a tracheostomy and/or in shock,quite simply does not have time to seek consultation. Hemust undertake the immediate treatment and management untila more experienced person arrives. Such emergent proceduresmay have been learned earlier in the military or in theemergency room.

Orthopaedic Traini22

There were 34 procedures that fewer than one-fourth ofthe first-year residents reported they knew how to performprior to their orthopaedic residency. A large number of theseare more involved orthopaedic procedures that are less likelyto have been delegated to the student or intern(e.g., perform

62

-56-

a wrist synovectomy, repair a lacerated digital nerve, andinsert a medullary rod in the femur). Others are consideredmore esoteric in contemporary orthopaedic surgery, regard-less of the level of experience, (perform a needle biopsyof a lumbar vertebrae, insert Harrington rods, perform andinterpret an arthrogram). Some others are procedures invelich the risk of the situation is such that the attendingstaff feels compelled, or are required by contemporarysociety, to undertake themselves (e.g., manipulate a con-tracted joint, perform an innominate osteotomy, give experttestimony in a court of law).

No amerience

More than 10 percent of fourth-year residents reportedthat they never learned eight of the procedures, more than25 percent never learned five of the procedures, and morethan 5U percent: never learned six of the procedures. Thefailure of fourth-year residents to learn these proceduresmay be attributed to several ot the following causes or acombination thereof. These procedures are not common inany orthopaedic service (e.g., apply a minerva jacket, applya corrective cast for scoliosis, perform an innominate os-teotomy). Others are simply not done in some areas of thecountry or by some orthopaedic services because of localprecedence (e.g., insert Harrington rods, perform a lamin-ectomy and discectomy, perform and interpret an arthrogram).A few of these are procedures frequently done by a consult-ing service or a consulting surgeon (e.g., repair a laceratedblood vessel, perform and interpret a nerve conduction test,perform and interpret an EMS). Finally, there are thoseprocedures which ordinarily are not delegated because theyare medical-legal situations in which attorneys and compen-sation boards prefer to have already certified individualswith all credentials necessary to impress an arbitrator ora jury (e.g., give a legal deposition, or expert testimony).

If learning any of the procedures in this section iscritical to the complete training of an, orthopaedist, thensome organized method of assessing resident's exposure tolearning the procedures should be devised. Many programsrequire residents to keep a list of the operative procedureswhich they performed or assisted at. The list is reviewed

63

-57-

periodically with the chief or one of the attendings toassure complete exposure to and learning of importantprocedures.

The list of procedures which many fourth-year resi-dents have not learned, has already been employed by afew programs to assure the resident's exposure to thefollowing: giving a legal deposition, giving experttestimony in a court of law, and evaluating a disabilityand reporting those findings in a form acceptable to acompensation board. Attendings from these programsrealized that while these three procedures were ones thatthey perform very frequently in private practice, theresidents-in-training ir. hospital settings have no experi-ence with them. One program designed a rotation in medical-legal problems during which a resident wcy!ld write andevaluate disability reports, hear court testimony, and helpprepare legal depositions given by the attending staff.

What would possibly be considered more important bymany practicing orthopaedic surgeons is the fact that thereare residents completing formal training who have not donea laminectomy and discectomy, have not performed anr1 i ter -preted a myelogram, did not feel confident in their abilityto repair lacerated blood vessels, or have not performedor been exposed to anterior cervical fusion. This, however,is a reflection of contemporary practice and in the courseof designing programs one must constantly keep abreast ofwhat constitutes contemporary practice so that the gradu-ating resident is not only well-prepared to meet the demandsand challenges ordinarily placed upon him early in his prac-tice but is also exposed to procedures that may be in thedevelopmental stage when he is in training but may becomefar more common after he has been in practice five to tenyears.

Confidence without Performance

A number of residents reported confidence in theirability to perform without actually having done a procedure.This discrepancy between not having performed the proce-dure and having confidence in the ability to perform itseems more apparent than real. One has to realize thatthe practicing orthopaedic surgeon is constantly expectedto be able to perform procedures or certain applications

t.-58-

of those procedures that ho has never performed before.This implies that emphasis on basic surgical principlesand basic knowledge is extremely important during resi-dency, since it would not be reasonable or possible fora resident to be exposed to all the possible procedureshe may be required to do.

Summara

This Study indicated a wide variety of residenttraining among the orthopaedic residency programs surveyed.This variety of experience with the 58 selected proceduresexists for both residents beginning their training andthose about to enter practice. There were orthopaedic pro-cedures which over 75 percent of the residents beginningtheir orthopaedic residency had previously learnc:d. Therewere some orthopaedic procedures that over 55 percent ofthe residents about to complete their final year had notlearned. There were also orthopaedic procedures whichresidnts had never performed }alit had confidence in thcirability to perform when called on to do so.

The significance of these findings is in the exampleit provides for the method of gathering detailed infor-mation during residency programs about what residents arelearning. It may provide a. model upon which a core curric-ulum of basic procedures all qualified orthopaedic practi-tioners must be able to perform with confidence can bederived and evaluated. It may provide program chiefs witha means of evaluating the clinical variety available toeach of their residents. Also, it may form the means ofdiscovering continuing education needs. The format of theinstrument can provide information necessary for all theabove purposes.

-59-

ResponsibLatyLIJ22crative Procedures

Individual operative procedures were examined in theprevious section. The detailed steps in decision-makingabout the performance of general operative procedures isthe subject of this section.

Residents were first asked to define in their own wordsthe two most common types of patients in the hospital oftheir present rotation. They mentioned three categoriesmost often. They are, in the words of the resident, pri-vate, clinic, and third-party-payment patients.

Although residents were directed to respond to thequestions personally, it apprars that they responded asresidents in general, since responses of residents at alllevels of traininig were essentially the same.

Preoperative management of patients is most oftencontrolled by residents with all types of patients. Resi-dents have this responsibility most often for clinic pa-tients (83/,) and least often for private patients (48A.

Arranging for proper equipment in the OR is usuallyhandled by the resident in 70 percent of the cases, by theOR staff in 20 percent of the cases, and by the attendingin a little over 10 percent of the cases.

The attending's role at the operation is quite differ-ent depending upon the type of patient (see table 11). Theattending performs about half of the operations for theclinic and third-party-payment patients, observes one-quarter of them, and is not present in the OR but availablein the immediate area for the remaining one-quarter.

The entire operation is usually done by the attending,or by the resident with step-by-step guidance from the at-tending, for 62 percent of the private patients and 16 per-cent of the clinic patients. With comments, assistance, orobservation by the attending, the resident performs theoperation for 27 percent of the private patients and twicethat for the clinic and third-party-payment patients.

Ancillary procedures as a part of a larger procedure,such as obtaining grafts, are most commonly done by the

c

-60-

resident (ew). The resident is in complete charge whilethe attending may not 11.2 pre!lent in about half of thecases for clinic and third-party-payment patients, and inono-quarter of the cases for private patients the attendingmay not be physically present in the OR.

Closure of the wound is clone by the resident in 00 per-cent of all cases and tho resident is in complete charge inover half of all ca es with tho attending making comments inleF,s thon 20 perent of all eases and in the remaining 20percent the attending gives step-by-step guidance.

Aecoiding to the re:;ident5, who responded to this survey,in over three-gurters of all cas,,,,s the following arc dis-cussed with the attending!):

1. nature of actual or pptential complications

2. manage:!:ent of actual or poLonLial cac,rAicaLions

3. rehabilitation of the p.Atient

4. progoo:;is of the patient

Between 65 and 75 percent of the time, the followingarc discussed:

1. steps in the basic operative procedure employed

2. pathology of the case

3. degree of success of the case

When not specifically discussed with the attending, theresidents either had these matters enumerated by the attend-ing or they discussed them with other residents. In about10 percent of the cases these matters were not discussed.

The least discussed item among those listed was thebasic principles of instruments and equipment employed. In60 percent of th.:, cases it was discussed by attendings orresident groups and in 30 percent it was not discussed atall.

67

TABLE II - 11

At the Operation the Attending Usually:

Private

(%)

Third Party

(%)

Clinic

(CA)

N=138

N=123

N=46

Performs or assists the resident

80

50

39

CY

NObserves the operation

726

25

Is not in the OR but in the area

624

30

:11

a

-62-

The Attending Staff

Composition and Functions

The attending staff is likely to number at least fiveand not more than 20 attendings, the average being twelve.Typically, at least 10 of the 12 attendings derive all in-come from private practice and devote no more than 25 per-cent of their time to resident education. Fewer than halfof the programs report one or more full-time attendingstaff members whose total incomes are derived from salary.Typically, such a staff member devotes not more than one-third of his time to formal teaching activities, the bal-ance being allocated to clinical, administrative, andresearch duties. He differs from his part-time colleaguesin that he is more likely to function in all four areas,whereas at least a quarter of the part-time staff haveneither administrative nor research responsibilities.

For both the full-time and part-time attending staff,the activites related to resident education most commonlyinclude ward rounds, and clinic or operating room super-7ision. Almost as many attendings spend at least sometime in individual conferences with residents, and approxi-mately two-thirds report at least some participation ingrand rounds and special lecture.,2..

Tables 12 and 13 indicate that attending staff, liketeachers in other areas of education, support contemporarytheories of administration, planning and teaching, basedon educational psychology principles.

Values and Attitudes

With respect to the goals of residency training, mostattending staff feel that residents should be evaluated onattitudes as well as on achievement and that attitudes canbe modified during residency. They are divided on the ques-tion of the desirability of resident research, and mostdisagree (some strongly) that the development of surgicalskill is "the most important function of the residency."

t.-63-

Whi]e attendings tend to agree that staff shcild use aresident's previous experience in deciding what he shoulddo on his new rotation, over half disagree (some strongly)that the goals of the residency should be determined bythe resident's needs and inte'ests, and one-fourth disagreethat individual differences should be accommodated in plan-ning the overall program of each resident. While they alsotend to agree that "mature residents learn more on theirown initiative than under tight supervision," they 'are al-most divided on the question of whether "residents shouldhave more supervision than they get," whether they will"cover important material without some prodding " andwhether, if given a free choice of activities, "residentsgenerally select what is best for them." Finally, substan-tial numbers disagree (some strongly) that 'residents shouldbe given more freedom in designing their own programs," andabout three-quarters agree (some strongly) that "disciplineis necessary for maintaining resident productivity." Al-though most of tnem express the view that "effective teach-ing is enl:anced by getting to know the residents well," atleast half of the attending staff feel that they are mosteffective when they maintain a "proper professional dis-tancc,." between themselves and residents; at least one-fourthbelieve that an attending staff member who becomes involvedin the personal problems of residents loses his effective-ness as a teacher.

In summary, the results show that attendings are quitedifferent in'their perceptions about methods of learningand strategies for organizing and administering an ortho-paedic residency program. The disparity is typical ofeducators in general and indicates a need for discussionamong the attendings in each program about the principalsof learning and instruction to be used in their program.The staff development procedures discussed elsewhere inthis report provides a media for such discussion in anorganized series of topic areas.

-64-

TABLE II - 12

More than 85 Percent of the Attendings Res?ondingto the Attending Attitude Survey Agreed the following$tatments:

1. The most effective teacher has a personal interestin the progress cf each resident.

2. Skillfull teaching can increase a resident's interestin orthopaedics.

3. The residents' service responsibilities are usefulfor learning orthopaedics.

4. Residents learn research procedure best by doingresearch.

5. Attendings should make allowance for individualdifference when teaching residents

6. Attendings who respect residents encourage partici-pation in planning patient management

7. Good teaching cases relate isolated problems withoverall goals of orthopaedic management.

8. The contrasting skills and interests of residentsmakes team work a profitable experience.

9. Attendings should have an overall plan of programobjectives to use in their work with residents.

10. Resident morale is maintained when open communica-tion is possible.

11. Demonstrated interest in a resident's progressincreases his productivity.

12. Follow-up is as important a learning experience forresidents as is acute management.

13. Teaching residents is a valuable activity.

-65-

TABLE 11 - 12 (Cont'd)

14. Residents should be evaluated on attitudes as well ason achievement.

15. Attendings should be prepared to demonstrate the rela-tionship between basic information and clinical problems.

16. Residents learn bast in an atmosphere of mutual respect.

17. Discipline is necessary for maintaining residentproductivity.

18. A good residency program integrates the basic sciences

with clinical problems.

19. Attendings should participate in the teaching programif they wish to have residents working on their service.

20. Patient management experience aids residents in thedevelopment of suitable attitudes toward patients.

21. Residents learn most efficiently by being required toassume responsibility for patient care.

-66-

TABLE 11 - 13

More than 70 Percent of the Attendings Respondingto the Attending Attitude Survey Disagreed with the follow-ing statements:

1. Teaching on ward rounds is usually impractical.

2. Teaching residents is best done by giving organizedlectures.

3. The quality of the teaching program has no effect onthe residents' interest in orthopaedics.

4. Attenchngs adequately meet their teaching responsibil-ities just by providing patients.

5. The major function of the resident is to assist ingetting the work done.

6. All an attending needs, to he a good teacher, is anextensive knowledge of orthopaedics.

7. The resident service responsibilities interferewith their learning.

8. Developing surgical skill is the most importantfunction of a residency program.

93

III - ANALYTIC STUDIES

Data Definition

There are two examinations mentioned often in thefollowing sections of the report. To prevent duplica-tion they will be explained once here.

The Orthopaedic In-Training Examination (OITE) is ad-ministered yearly to residents in orthopaedic residencytraining programs. The examination consists of multiplechoice questions (M/C) and written simulations ofclinical encounters with patients, also called by twoother descriptive terms (patient management problems(pup) and erasure examination). Erasure examination isa term used commonly by the resident examinees and isdescriptive of the format of the'questions. Informationgathering and decision making involved in solving theproblems are done by erasing areas on the examinationbooklet, thus obtaining information needed to proceedthrough the examination. The examination results arereported as follows:

Multiple choice questionstotal scoregeneral orthopaedics )

adult orthopaedics ) - Achildren's orthopaedics )

trauma )

anatomy )

physiology-biochemistry )psdaology ) - Bbiomechanics )

hand )

rehabilitation )

recall )

interpretation ) - Cproblem-solving )

Patient management problemstotal scorediagnosistreatmentrecallinterpretation,problem-solving

-67-

74

-68-

In A, each multiple choice question is categorizedinto one of the subject matter areas listed. In 13 , M/Cquestions are categorized on other content and a ques-tion may appear in more than one category. The defini-tion of the scores in C and D is as follows:

recall items which may be answered on the basisof remembered factual information.

interpretation items which may be answered byproper interpretation of verbal or visuall,:-presented findings.

problem-solving - items requiring clinical judgmentor employment of problemsolving strategiesin addition to factual information andinterpretive shills to answer.

The two PMP scores are defined as follows:diagnosis the culmination of history, physical exa-

ination, laboratory, x-ray diagnosis, andconsulatation in which the examinee is invitedto select as many as he would like in thediagnosis of patient disorders

treatment - the culmulative decisions implementingtreatment of patients

The above definitions are taken from reports of theAmerican Academy of Orthopaedic Surgery.

The second examination mentioned in this research isthe Orthopaedic Certification Examination (OCE). Thisexamination consists of the same types of questions andscores as the OITE in addition to a number of cral exam-inations which are not discussed =n this research.

Prior to the administration of the OITE, the chiefsof residency training programs were requested to provideinformation about resident's achievement in selectedareas. This Resident Evaluation Form consisted of 12 scales.For each scale, the chiefs were requested to compare eachresident to the average of all residents at that level of

113

-69-

training that he has known. The scales range from 1 to 12

with 6.0 called average. (See Supplement for sample) .

Interp,-0-ation of rorrelations

A correlation is an index of association betweentwo variables or measures. When evaluating "the worth"of any correlation -..oefficient, there are two matters to

consider. The first is the significance level of thestatistic. In order to put any faith in the predictiveusefulner.s of a correlation coefficient it must be largeenough to be statistically significant. A statisticallysignificance correlation is one in which there is a highprobability that the observed association is not due tochance. Statents such as p< .05 and .01 indicatethat the prob:)bilitv of the correlation co:.!ff icient. beingdue to a chance asf;cciation between the two variables isless than 5 in :L00 and 1 in 100 respectively.

The second matter to be considered is the educationalsignificance or usefullness of the calculated correlationcoefficient. Educational significance is a matter of howwell the coefficient can serve as a predictor of performance.A coefficient less than .2 indicates only a slight rela-tionship between the two variables and is of little use as apredictor. In the research under consideration here arestudies of characteristics of residency .programs andresidents, related to performance scores on the OITE. In

many cases statistically significant correlations below .2

were found. Such findings indicate a slight relationshipbetween the characteristic and performance but certainlydo not indicate that these characteristics taken individuallyare useful for prediction of performance. Finally, itshould be noted that although variables that have a cause-and-effect relationship to each other certainly will beexpected tc; be correlated, the fact thattwo variables are'correlated can never be used as evidence that a cause-and-effect relationship exists. COrrelation only indicatesan association between two variables.

-70-

Sample vs PcIpulation

In Phase II of the Orthopaedic Training Study, twosamples of residency training programs were selected fromall U.S. programs. The first consisted of 280 residentsfrom 16 residency training programs from which extensivedata collection and site-visits to observe the program inaction were to be made. This group of programs was titledthe intensive study sample. The second consistedof 504 esidenLs from 35 programs and was titled thr-survey study sample since the residents from this group ofprograms were only to complete several questionnaires.Together these samples are called the Study sample compri-sing 784 residents from 51 residency tYaining programs.(See supplement) .

In order to determine if achievement of the Studysample could be said to be repreentative of the populationof all U.S. programs from which this sample was drawn, astudy was conducted which compared the yearly Orthopaedic

Lin:4tion (OITE) -colts of the Study samplewith those of the population.

Statistically, this procedure entailed constructing aconfidence interval about the population mean using thestandard error of measurement of the mean, a function ofthe standard deviation of the population, and the sample size.

As Table 1 indicates, the study sample is significantlydifferent from the population for nine of the 11 scores ofthe OITE. The two non-significant scores were within .2 ofbeing significant. Therefore, the study sample cannot beconsidered a random sample form the population but mustrather be considered to be comprised of residents aboveaverage in the orthopaedic knowledge and abilities measuredby the OITE. Since the OITE, along with ratings of resi-dents by their chiefs of training were to be used as acriterion of achievement success in the residency trainingprogram, this re-ult has great importance. Care must betaken when generalizing test score characteristics of

individuals in the Study sample to the population. However,.the standard deviations of the Study sample and the populationwore nearly equal, and comparisons of characteristics ofindividual residents variation from other residents at any

'77

-71-poin.: above or below the sample mean

represents much

the same range ofdifferences as is

represented in thepopulation.

Correlations, orcomparisons of high scor-

ing and low scoringgroups from the Study sample cnn

therefore begeneralized to the

population.

r

TABLE III - 1

Population-Study Sample Comparison

Score

Lower

Limit

Population

Mean

Upper

Limit

Study

Sample

Mean

Total multiple choice (m/c)

127.4

129.4

131.4

134.5**

M/C general orthopaedics

29.9

30.5

31.1

31.5**

M/C adult ortho,aedics

46.9

47.6

48.3

49.1**

M/C childrens orthopaedics

36.6

37.4

38.2

38.6**

M/C trauma

13.6

13.9

14.2

14.1

M/C recall

104.0

105.7

107.4

109.3**

M.", interpretation

15.5

15.9

16.3

16.4**

M/C problem-solving

7.7

7.9

8.1

8.0**

"Patient management problem total

85.0

86.9

88.8

90.0**

PMP diagnostic

29.2

29.9

30.6

30.7**

PMP treatment

55.4

57.0

58.6

58.5

** p <.01

-73-

%(,;irl v Cow: .; t-cmcv of Chi r Itat4 nci Form------

This study was conducted to compare various ratingsgiven re:;identn each year, by their chiefs of training inconjunction with the Or In-Training E7:aminationpro.jram. The hypothcsis was that tile correlations Letweenyearlv rating:.; given residents would decrease as the timebetween the ratingr. increa!;es, (i.e. all resident:: wouldno increar,e in their abilities equally as they progressedthrou.jh the renidency).

An Table 3 indicates, the mean rating for all factorsFroT year one to '.'car cr.lr by about one pointfro :n 6.0 to A.G, on a 12 point scale. The chiefs

in:;Lructt:d to cel.)are eac'l resident- to all residf?nts atthat 1 ''v '1. of training that Iv.: has known. Assuming thatthe followed the instructions exactly and that the

rf,:; ;,1z.:nt5 were about

in rated, the avera,ie ratingn at each yearin tr:Lin;aj should be the aJera(je ratin of 6.0 on the 12

oc,.17.. It apparn tlblt the pro:!ent. first-yci.rdent. is alout average (6.0 of a possible 12) consideringall the first-year residents that the chief has ) :sown; buthis s(::ond-year residents are about eight percent betterthan all second-year residents he has known i.e. 7.0 of 12or 58 percent, 8 percent better than average) , the third-year residents are about 17 percent better aad his fourth-year residents are rated as about 25 percent better thanall the fourth-year resAent.r he has known. Another pos-sible explanation is that the chiefs are rating eachresident as compared to first year residents. In thin case,first-year residents are average aLd residents improve about8 percept per

When we tarn our attention to the hypothesis underquestion, we find that for thre'_ of the five factors thehypothesis is supported. (see "able 2) . The exceptions werethe factors of overall competence as an orthopaedic surgeonand the know] edge of basic science as related to ortho-paedic :; which are significantly correlated in both compari-son year pairings. The significant correlation of thesetwo factors in rating the same residents over tme indicates

-74-

that, as rated by their chiefs, residents gain sitililaramounts in their basic science Imowledge and overallcmiptence. For the chiefs' ratings of knowledge ofclinical orthopaedics, ability to use information tosolve proLle:i.s, and judgm,:!nt in deciding appropriate"arc, and lreatent, although the average rating increasedby about OHL point from year Lo year, the correlationsbetween the ratings decreased as the time between the rat-ings increa!-.ed. Also, for those three factors, the first-second vear correlations arc statistically signiCicanL,while t.fl first-thild year correlations are not statis-tically significant. It is assumed that the same chiefstrade tho ratings for each resident at each year.

The fact. that correlations Letween rating of LI1CSr!factors q:..A.asecl over time indicates that, as perceived

thy chl(:,::;, all ro::Idents de !lot inca!,:e in thLir achiL:ve-went by 11D! ItAitic amount. If all residents were to increasethe r achlevcm:.'nt the sare amount each year, the corrola-ti)n 1.):::tri yearly ratings would be very high. It appears,therefo,e, that the chiefs of training are aware of changesof differnt. degrees in the individual residents that theyrate and that they do not merely consider that a year ofezperiunce has the same effect on all of his residents.

cia

TABLE III - 2

Correlations of Chiefs'

Ratings of Residents

at the FirsU, Second

and Third -Years of

Traininga

Cbiefs' Rating of:

Year 1 vs.

Year 2

N = 100

Year 1 vs,

Year 3

N =

48

Knowledge of clinical

orthopaedics

Knowledge of basic

science as

related tc orthopaedics

.36***

.41***

.33

.42**

Abili`y to use

information to

solve problems

.23**

.10

Ju'gment in deciding

appropriate

care and treatment

.40***

.26

Overall competence

as an orthopaeic

surgeon

.39***

.34**

* *p <.01

***ID <,.001

a -,:oo few first vs. fourth-yea. L. ratings were

available

for useful correlations

TABLE III - 3

Mean Chiefs' Ratings for Each Year in Training

Chiefs' Rating of:

Year in Traininc

12

34

Knowledge of clinical orthopaedics

5.7

6.7

7.9

8.8

Knowledge of basic science as

related to orthopaedics

5.9

6.7

7.5

8.2

Ability to use information to

solve problems

6.5

6.9

7.6

8.9

Judgment in deciding appropriate

care and treatment

6.3

7.0

7.7

8.8

Overall competence as an orthopaedic

surgeon

6.0

6.9

7.8

8.9

-77--

Correlation of Chief's Ratings and OTTE Scores

This study was conducted to determine the relation-ship between selected che's ratings and scores on theOITE. it is generally thought that correlations betweensubjective measures, like the chief's ratings, and objectivetest scores will he low be'-ause of the expected unrelia-bility of subjective ratings (i.e., differences in the criter-ion employed by various raters), especially wich the ratingsmade by 51 chiefs. If scores on the OITE are accuratemea!,ures of certain achievements of residents and if inter-rater reliability is high, correlations with the chief'srating of similar achievements would be expected.* It washypothesized that there would be significant correlationsbetty :on the following rating factors and examination scores:

aating Factors Scores

Knowledge of orthopaedics, Multiple choiceclinical an1 basic science total score

Problem solving ability PMP di tnosis

Judgment in deciding on treatment PMP treatment

Overall competence PMP total score

Overall competence Multiple choicetotal score

The results were analyzed for residents at each levelof training. There were 152 first-year residents, 393second-year residents, 441 third-year residents, and 382fourth-year residents in the study.

The results indicate that the total multiple choicescore was significantly correlated with all of the selectedchief's ratings of residents at each of the four levels oftraining. These correlations ranged from .42 to .20. The

See table 4 (III), p. 81.

R4

-7E3-

patient management problem (PNP) to4.:Al score wasElignificantly correlated with all selected chief's ratingsfor second and third-year residents only. These correla-tions were lower and ranged From .20 to .13.

Of the particular scores expected to correlated highlywith the chief's ratings of similar acilievements, five ofthe six were significantly correlated (see Table 4) . Asexpected, significant correlz-,Lions were found at all fourlevels of trainLIg between Hie knowledge of cllnicalpacdics, knowledge of basic science as related to ortho-p,ledics, the overall competence as an orthou,aedic surgeonrating::; and the total multiple choice score. For three ofthe four years in trainj.ng significant correlations werefound between the judgment in deciding appropriate careand treatment rating and the 1-reatment PMP score. Also,for three of the four years in 1-raning significant cor-reletio!es wore fount 7 between the ove!.all competence ratingand the PMP total score.

In general, the OITE scores which summarize the resi -dent.s' performance on each of the two sections of the exam,multiple choice total score and PMP total score were thebest predictors of all the chiefs ratings. Knowledge ofbasic science as related to orthopaedics, knowledge ofcli_nical orthopaedics, and the overall competence ratings,in that order, are most associated with the multiple ci-.oieescores. Since the multiple choice questions on the OITEare composed of orthopaedic basic ience and clinicalapplication questions, Many of w , call for recall ofinformation, it is logical that the chief's ratings ofknowledge in these areas were n.ost h-ghly correlated withthe multiple choice score. Since many of the multiplechoice questions, besides requiring knowledge of facts,mere designed to tap areas involving the interpretationof, judgment w.th, and application of facts, it wasexpected that the total multiple choice score would be signi-ficantly correlated with the chiefs rating of overall com-petence as an orthopaedic surgeon.

-79-

Knowledge of clinical orthopaedics and overall com-petence as an orthopaedic surgeon, in that order, weremost associated with the PMP total score. Since the PMPwas composed co!: similated clinical encounters, iL was notsurprising that the chief's rating of isnowledge of clinicalorthopaedics had the highest correlations with the PMP totalscore. Similarly, there lJa the ezpected gn,s, 4 41--; cant cor-

relations between the PMP torei_ment score and the chief'srating yf judgment in deciding appropriate care and treal-ment of patients. Clinical judgment involvc the abilit!to decide upon and carry out a treatment plan and it isthis ability that the PMP treatment score purports tomeasure. This evidence indicates that it deals with Bijlarconcepts. The expected re]ationshi) between the ratingof the residents ability Lo uE: information to solve problTosand the PMP diagnostic' score was not apnarent. This cor-relation was very low for three of the four 1 evels of t:i:ain-ing. This a,pparent disreponey can possibly be exlainedby the manner in which the diagnosis section of the Pt11:' onthe OITE is scored. It is not scored on the basi5, oi

wrely obtaining the correct diagnosis but rather on the.11a.F.is of thoroughness of inquiry. This distinction isevidenced by the fact that practioners score lower thanresidents on the diagnosis section only. Therefore, thediagnosis PMP etore is perhaps more an evaluation ofthoroughness of diagnosis and should not be expected tocorrelate highly with the chiefs rating of ability to useinformation.to solve problems.

A comparison of the scores and ratings of similarachievements indicated the expected significant relation-ship. These results provide evidence for two conclu..ions.First, there is some evidence of chiefs' consistency inrating resident's achievement. If there were great inter-rater variety in the criterion used by the 51 chiefs inmaking the ratings, the ratings would not correlate withany other measures of similar achievement. Second, there isevidence that the OITE scores have some validity (i.e., theymeasure what they are intended to measure). Consistentwith expectations, correlations were Found between ratings

86

t.-80--

and scores of similar achievements. This findingprovides partial evidence of the concurrent validity (or,correlation with other independent measures of the sameability) of these scores. In general, it can at least besaid that chief's ratings and OITE scores are both evalua-tions of similar }::finds of achievement.

TABLE -III - 4

Selected Correlations between Chiefs'

Ratings and DI=

Scores for each Year in Training

Factors - Test Score

Year in Training

12

34

Knowledge of clinical orthopaedics

rating and total multiple choice

score

.26***

.36***

.39***

.28***

Knowledge of basic science as re-

lated to orthopaedics rating and

total multiple choice srcre

.30***

.42***

.40***

.35***

Ability to use information to solve

problems rating and diagnostic

PMP score

.03

.15***

.03

.02

Judgment in deciding appropriate

care and treatment rating and

treatment PMP score

..L8**

.09

.14**

.15**

Overall competence as an orthopae-

dic surgec- rating and PMP total

score

.14*

.16***

.14**

.10

Overall com:etence as an orthopae-

dic surgeon rating and total

multiple choice score

.23**

.26***

.27***

.22***

*p <.05

**

p <.01

***

p < .001

4,

-82-

CharacterisLie7s or Posidencv cm-4:: and Pesiden1-sCerrelated v.,1_11 Residents f;:amiliation Scores

1. Orthopaedic Training Prorani Characteristics Relatedto Resident's rerforance oil the OTTE.

In order to determine which cl)aracteristics oforthopaedic piogir.: are associated with residenL per-formance on the OITE, a correlation study was conducted.Only fourth-ver reHdents from the Stud./ sample weleemployed in the analysis, since effects of the progr:::.,chanicteristics would Lend to be more apparent in theirperformance than in that of residents at other levelsof training. There were 42J; J:ourth-year residerl:7included in the study.

wa,; a significant correlation between themultiple choice total score on the 07TE and each ofthe progrz::, characteristics listed. These cerrelationsranged from .20 to .13. Higher multiple choice scoreswere associated with residents from drogiams with thefollu.ing characteristics:

Chief1. the chirf's major office was in or very near

the principal institution of the program2. the chief had much of his total income from

salary3. the chiefs of service held academic rank

Resident Selection

4. resident selection was din primarily on thebasis of a candidates academic standing inmedical school, his intellectual approach toproblems, and his recognized interest inresearch

5. resident selection was Pot primarily done on_the basis of technical aptitude or emotionalmaturity

-83-

6. applicants for residency who were vulnerable formilitary induction duri.ig the residency wereacceptable

7. preference was given to graduate:1 of the affiliatedmedical school (if there was one

Objective!-,

8. the primary objee!ive of the projraT was Cie pro-duction of the commun:',/ actice oriented ortho-paedist-s

9. the primal,,- objeOive was not the production ofort.hopadic subz:p-cialists or academicians

Pesid(2nts10. there were many resid-2nts in ths: procvam11. residents im,tructed mcdi,.-7a1 students12. resident:. were required to write a ca_inic:11 o-

laboratory re.;earch 1,7,per

13. after copleting the program rosi:dents did not stayon in supervisory positions

14. some, but not many, residents have faild to com-plete the program in the last give yeurs

15. residents who failed to complete the protfram werejudged unacceptable becaur;e of poor clinical judgment

16. during the first two years of residency, residentssalary was below $7500 per year

17. there was supervision provided the resident inthe operating room according to the needs of theindividual resident rather than supervision in allcases

18. there was a fellowship staff

Program Facilities19. clinical material was more often adult and pedia-

tric, as opposed to trauma20. a biomechanics subspecialist was on the ortho-

paedics staff

-84-

21. a neurologic diso,-der;-, subspecialisL war; on theorthopacOic staff

22. there was a veteran. ' hospital associatprlthe program

The converse e:ALh of the procfram characteristicsis associal('.d wit!' lower multiple 21)( ice total scorr2s, forexalc,ple:

1. Chief's major office was not ner,c t

Lion of the prorxr

4-5. resident selection was not (Hue 1:ri:c.atily an thebaF.is of academic stanOing in medical school, etc..but dc.no on the 1,af-;is of techni.ea'.

aptiturin ,T,71 maLuriry10. there Wr in Lhe] 9. mItc2rial wi.s more often Lrau'k than adult

aid pcdiat..ric

Characteristics :Jot. thentioned such as the number orattendirirls who teach residentc, kinds of clinics availr,!):the ute of a follo-up clinic etc. were not signi ficanLlyrelated to resident's OITE scores.

2. Resident background relat,:d to OITE score=

This study was conducted to determine relationshipsbetween the bachund of orthopaedic residents andtheir performance on examinations. q:ere were 299fourth-year residents included in the study. Signifi-cant erirlelations ranged from .19 to .15. Residentswith '_he following characteristics tended to have highmultiple choice OITE scores:

1. younger than average2. decided on orthopaedics as a specialty at an early

age

91.

-85-

3. had a high grade point average in high school,college, and medical schr

4. had a high rank in class in high school andcollege, and medical school

5. liked mathematic; in both high school and collegeG. selc.ted their orthopaedic program because of its

ropu ation and not because of any offer offinancial support

7. was satisfied with the general reputation of thetraining program

8. was dissatisfied with t opportunities forsar(4] al experience to date

9. was dicsatisfed with the ammnt of responsibilitythey are given in thq t.t,msment of patients andwish they had more responsibility

1n. marital problems were not an interference withwork in the residency program

On the other hand, low scol;ng residents can D saidto terd in the opposite directions on all of the .7haracteris-tics lasted above. For example, the residents who scor,:,dlow on the multiple choice section of the OITE tended tobe older, had a lower grade point average and rank in class,were ':ot satisfied with the general reputation of the pro-gram, and maratal problis inierfered with thei- work inthe residency program.

Characteristics not w ntioned, such a:: size of hometown, parental occu:oatio, amoun'.. of fawi'v income, sizeof high schol, college, or medical school, ,ype of inter-ship, etc. ane others listed on the background questionnaire.(see supplement) were not significantly related to multipleelloice scores.

3. Resident attitudes related to OITE scores

The significant correlations between fourth-yearresidents' attitudes and the multiple choice total scorewere few and negative. A negative correlation indicatesthat agreement with the attitude statement is related tolow multiple choice score and disagreement is related tohigh multiple choice scores. There were 2G5 fourth-yearresidents in this study.

TABLE III - 5

Significant Correlations between Resident Attitude Statements

and OITE Score

Attitude Statement

Correlations

with multiple choice

scores

1.

There are too zany ward (charity

indigent)

patients admitted on my service

-.21***

2.

There are not enough private patients on

my service

3.

There is not ,nlolgh elective time provided

4.

Teaching takes too muci time away from n;J:e

important things I want to

.o

5. The are so many dem.L:As on a resident that

they are almost impossible to meet

6.

I have sufficient oppc-tu-ity to use my

operative skills

_.17**

**p < .01

***p < .001

-87-

The negative correlation of multiple choice t: cores withthe first_ five statuT,ents in Table 5 indiL:Ite that lowscores were ass.-7iated with residents who ccmplain abou.types of patients on their service (items 1 and 2) andabout the large number of demands upon them with toolittle time to fulfill thorn (items 3, 4, and r;).

Low multirlc choice scores were also as withresident's satisfaction with thci r opportunity to in:etheir operat vo shills (iteP 6) . Conversely, high s(:or-ing residents wore likely to state that they have not hadsuch opportunity. Supplementary data was found in Chel'ackground questionnaire. ,hen as), A to evaluate certainaspects of their residency, high scoring residents againtended to be ,3issatisfied with the opportunities the.! havehad for surgical exporJ,rice, while low scoring residentl!;.were satisfied.

Interl>>eti!I-1on of Pn:7,11.1_

In the results reported, the ,-orielat;ons ranged Crow.20 to .13, and were therefore a!1 itat'ticallv signifi-cant. The educational significance of the results can bequestioned since ':he :ire low in terms of theability to predict OITE scores by obsering one of thecharacteristics repo":ed in a part:ifalar p-ogran. It shouldbe noted that the mere presence of any characteristic doesnot guarantee that residents will have high sores, andvice versa. Although statis-ic711.y significant these cor-relations are low and indicate ('lily a slight relationshipbetween the characteristics and scores. It may also bethat the high scores are associated not with the particularcharacteristics listed but with other characteristics associa-ted with them. For example, a first and second-year residentsalary of less than $7500 was associated with high scores.However such salaries are also associated with public insti-tutions, as university programs usually are, which have beenshown in past studies to be associated with higher scores.

The results surely-indicate the multi-faceted natureof program effectiveness in the sense that there are manyand varied individual characteristics of programs whichwere all related to program effectiveness. A further studyof these program characteristics is described later inthis report which employs many characteristics of the

prograp. to detc.r7.ioe if am' cof.1)ir.,1.1on of them p10-

vide0 a gond bi!_!i!: for prcdicti,,n of OlT neoren. Thevanie or 'Ile prent nfold- in that it indicalen tff

ot proclix'in t1,z,1 can he incnided in theprediction

chrtrae! f 1rr, c:, d r"; rd ri nn

ATong itn TynininijStudy won deni'intd "to pIoviC a mod] of individwlliedgradi,aly dtcaf ;on in toedicioe ;o which 111.2 deTonnra-t Of (.11-t 1 (I't rlicr.', 1 Z11 lta!I 'r ru 1 f 1 1 1-

tr,('11t. of r iclid reyrkii I. -111; tit'. r:1A 'of foil

tr, tir:s Cp., :3!

fio, the 1 ni,n4;ive profjlicn Wr...re! gi-en the

opporLnnity to participate ia the certiricalion prJcen of-tho An.eric;,n roi.ird of Orthop.edic Surf)(ri Vlen thee... andtheir chit:`. arecH the- vier nrevin-f,d fo

it'.(. Zi.'. I

pu,nu-e or ti. 1:; stnc."4 was to der'!.ino ir th;dt..ncy Board candidFtt( n (70).FiCs ) di ff:(1' from

their fellow rosidentr. Am' diffornr:ef.; found could beused to idcntify and other potent al ARBCn in resi-dc.,,-}. wii' an.attend:nt reducton in trairinq period -Such process would '.ncrease Cie supply of orthopaedicmanpc,:er avcilaple in the health care delivery s',IsLem.

In 1970 and 1971 a total of 23 residents from amongthe 331 residents in the 16 expermental programs exercisedthe option of taking the certification exam early. Of theseARBCs, nin( were fourth-year resident-s and 14 were third-year residents at the time of the exam. No special pro-vision for, or special designation of these men was madeduring the certification process. Examines were onlyaware that ARBCs might be examined. Al]. 23 attained scoresconsidered by the American Board of Orthopaedic Surgery assatisfactory for certification.

Data from the background survey indicate that most ofwhat has been said about the typical orthopaedic resident:

-89-

(see resident background section) also applies to theARBC group. However, the ARDCs were slightly younger,have fewer relatives who were physicianr; and .fewerrelatives in the health-related fields. They came fromlarger towns than their peers, their parents were slightlybetter educated, had higher incomes and more of them wereself-employed than the parents of other residents in theStudy sample. NGile fewer of the ARBCs considered occupa-tions oth,_r than medicine, their choice of medicine as acareer was made for similar reasons; desire for indepen-dence, inLerost in people, curiosity about the human body;but the ABCs reported far greater interest in the profit,as a reason for entering mcW.einn, L.han did other resi-dent: surveyd. A larger porccfntage of ATMCs receivedappoint!f,ents in the residency program of their first choicebut were more anious about their appointments than otherresidents. AT-ICs perceived far more competition in collegethan their peers and slightly more competition in medicalschool and in residenev, although both c_Joups report thatresidnc'y ,1-1)out. as dif.acult as they anticipated. Dur-ing the pro:jraiA, ARBCs differ from their peers in reportingless interference in training program atmosphere from re17!-t bus with atthding:::; considerably more satisfaction withthe academic enthusiasm of their program, and with the in-terest.of residents with resident morale. They also re-perted slightly lower educational debts than did other resi-dents; but they anticipated making five to 10 thousand dol-lars less per year after five years of practice. Thisestimation of income was possibly a realistic expectationin view of the fact that many of them report an interestin an academic career in medicin-,' rather than straightprivate practice.

The OITE was conducted to provide multiple choicescores in many disciplines and taxonomy scores, and scoreson patent managemenL problems designed to simulate theencounter of the physician with the patient in a problem-solving situation. Results indicated that the ARBCs per-formance on all aspects of this exam was superior to theirpeers at each year in training.

r

WI

TABLE

III -

6AveragePercentile

Rankof

ARBCsComparedtoResidents

at theSameYear

ofTrainingon theMultiple

ChoiceSectionofthe

CITE,

As

1stYr.

Residents

As

2ndYr.

Residents

As

3rdYr.

Residents

As

4thYr.

Residents

1970ARBC

Recall

(a)

(a)

90

83

N = 9

Interpretation

(a)

(a)

87

79

Problemsolving

(a)

(a)

68

71

Totalmultiplechoice

(a)

91

91

85

1971ARBC

Recall

78

80

77

(a)

N =

14

Interpretation

89

65

-

65

(a)

Problemsolving

53

52

68

(a)

Totalmultiple

choice

83

78

73

(a)

a -Datawas not

reported

-91-

IL should be noted that while ARRCs maintained theirsuperiority over the average resident: in each of the ,ears,their percentile rank decreases as they progress Ulroughthe residenc This indicates that the. peers were catch-ing up slowly, since they had more improve:lient to make.This pheno.lionon is also known statistically to occuroften in multiple measurements of any group andcalled regression towards the mean.

The 1970 ARWs have higher pereenLile ranks than the1971 ATMCs. This may be a result of the cautiousness andanxieties of the chiefs and residents in 1970, the firstyear residenf:s have ever had the opportunitY to take theOCE earlier than usual. The success al the 1970 AW3Cs mayhave lowered the criterion, and certainly lowered theanxiety involved in the decision of chiefs and residentsto partici.paLe in this experimental program in 1971.

The ARnCs continued to demonstrate their superiorperfeymnr,ce! cc the Orthopaedic CertificaLion Exam (OCE).On the major Lest scores, total multiple choice and PMPtotal as well as all sub-test scores, the percentile rankof the .7,P.12,Cs riln9ed from the 54th to the 74th with themajority above the 70th percentile. In general, it appearedthat the ARP,Cs began their residency by demonstrating theirsuperior ability on the OITE examination and this abilitycontinued to be evidenced throughout the residency to andincluding the certification examination.

Prior to the annual OITE the chief of each trainingprogram submits a rating scale evaluating each resident inhis program on overall competence and performance factors.

'

After a consolidation of similar ratings, the comparison ofthese yearly ratings is presented in the table which follows.

TABLE

- 7

AverageChief's

EvaluationofARBCs

andAll

otherResidents

intheSameYear

inTraining

Year

inTraining

2nd

3rd

4th

ARBC

Peer*

ARBC

Peer

ARBC

Peer_

KnowledgeofOrthopaedics

8.1

6.4

8.4

7.7

8.4

8.6

Clinicalability

9.0

7.2

8.7

8.2

9.1

8.9

Surgical

skill

8.5

6.5

8.4

8.2

9.4

8.9

, o N)

Personal

relationship

9.4

8.1

9.6

8.9

9.2

8.9

Long-termcare

8.7

7.5

9.2

8.5

10.2

9.2

Emergencyaction

9.2

7.6

9.3

8.8

9.5

9.1

Moral

and

ethical

responsibility

9.6

9.2

10.5

9.7

10.6

10.1

Overall

competence

8.6

7.0

8.6

8.2

9.2

8-.3

Note:

Ratingof 1

=Unacceptable,

12=Outstandingperformance

*Contains

all

residents

at

that

levelof

training

forwhich

ratingsweremade

-03-

It app(!ars that the chiefs of residency training p,:ogrmshave been sensitive 'wen in the' firnt year to tho ARBcr,superior performance. ARrir,:s ar( superior as rated by cMcfsand in scores on th:: OITE.

ARrc nu-sl-ionnairc,

The fine] portion of this report is bacd on aporsonal survey questionnAre copleted by each of the APBCsconcerning their career plans and p.,7oparation for thc C.N;1111-ination which was distributed after. the certification (..amwas given. These data are beimj reported separaLrAy bcuu:;ethey concern projections about: future activities ratherthan present or pa:zt situation. Also, they are descrip-tive rather than comparative.

Of the 23 ARBC's 25 percent attended review coursesprior to the OCX; of these, only one felt the wsnot holprnl. All ARDC's reported rr.2viewing at least fiveyears of the Bone and joint Crournal and reading two or morespecialized texts in orthopaedics or -.?1?Itea basjc sclenecs.

Only one man reported the OCE to be mTre difficult thanexpected and 25 percent indicated that it was somewhateasier than had been anticipated.

Despite the fact that the ART1Cs were Board certifiedand not required by the Board to complete their residency,all but three chose to complete it. At the time of thesurvey, two of these indicated that they would enter privatepractice, and a third would utilize his remaining residencyyear in special fellowship training. The fact that mostresidents have a contractual obligation with a hospital didnot seem to be an interfering factor in the decision makingprocess reported by the residents. The primary reason reportedby 20 of the 23 ARBCs for remaining in the residency isrelated to a feeling that training was still incomplete, a

desire for special training, an obligation to the program,or the desire to complete the best rotation (usually chiefresident).

Practice plans of the ARBC's are varied, with most (10)anticipating entering a single specialty practice group.

-94-

Since ARBCs had attached greater importance to the proEitmotive as a factor in their carer choLce, one would expectthat anticipated practice income would be higher than otherres:I.dents. Instead, their anticipated incom,,s are con-siderably less. This i;, porhaps, a realistic expectationsince 22 of the 23 ARBCs indi2ated a desire to do somcteaching and four indicated a choice of full-time academics,which is probably not as monitarily rewarding as communitypractice.

It would appear that the major difference between ARBCsand their peers is, not in their g,:noral background, buttheir performance, both on examinations and in the medicalcare system. They have demonstrated superiority throughouttheir training period.

Such information, however, should rot obscure the factthat 23 men were able to successfully complete the certificationexamination as much as two years earlier. This, coup1,7,d

with the data reported, suggests three conclusions:

First, the results support the rationale behind theoriginal objective of individualized instruction, i.e.,apparently individuals learn at varying rates and, havingestoblished a criteria of competence, some can be expectedto attain that level of required competence sooner thanothers, as indicated by the fact that they all passed the OCE.

The second conclusion is related, in that once competencerather than time remains the variable factor, significantmanpower gains can be realized through early certification.It may also mean that thetrend toward increased super-specialization may be accommodated without significantlyincreasing the length of time a man spends in training.The very talented may complete regular training early andmove into super-specialty during the period which would other-wise be occupied by the basic program.

The fact that 20 of the 23 ARBCs chose to completeresidency training, may not be as much of an argumentagainst early certification (i.e., the man is going to stayaround until he fulfills his training time anyway), as much

-95-

as an example of the need .7or planning and counseling resi-

dentFl. It also points out the need '-or some means by whichpotential AUCs can be identified early in their trainingso that career plans, expectations, modifications andhlternatives can be develop-!d with these potential candi-dates. Such planning may well include d:Lscussions of theresidents' services.

Third, apparently the chies or residency training pro-grams have been sensitive to superior performance early inthe resHency, as indfcated by the ratinT; on the residentevaluation form. Vihm this is coupled with the fact thatin all 23 cases the judgment of the chief and resident wasvindicated, i.e., all 23 men who were thoug, to be capableof passing did so, considerable weight is given to the ideathat the chief is an excellent jud,p of resident performance.

The Study suggests, also, that both the OTTE and theResident Evaluation Form appear Lo be consIstent predictorsof potential success and provide effective bench marks bywhich early candidates may be identified. Obviously, agreat deal of additional study needs to be done to determinewhat other devices for prediction are available.

It seems rather clear from this study that carefullyselected individuals, obviously different from their peersin performance on examinations and abilities as evaluatedby the chief of their program, can profit from an opportunityto complete the certification process prior to the normaltime requirements. The Study also seems to suggest thatthe estimation of ability and prognosis for success, whenmade jointly by the chief and the resident, is highly accurate.

Orthopaedic In Training and Certification ExaminationMultiple Choice Scores

The designs for both the OITE and the OCE came fromPhase I of the Orthopaedic Training Study. Both exams employa series of multiple choice questions in ten areas oforthopaedics, patient management problems, and ratings bythe chief of the resident's training program. In addition,the OCE contains several more ratings by associates of the

-96-

canddnto and a nure):-.,r of individually adninistered oral.examintion:;. Thl'3 study e%amined the correlalLion betweensimllar scores on Odell ex.anination. Only the multiplechoice scores were examin:l since PMP scoring was differenton the L%yo e,:am:; and could not be directly compared.

Although Lii:! de.:; gn of t11,. two exams was similar,.clffercns in construction, resdnLs preparation, andadminisLration e%ist. The OITE is con3tructed by a com-mitte:, of the American Academy of Orthopaedic Surgery andth OCE is constructed by a committee of thr Am.:,rican Boanlof OrLhooadie Surgery. The Acadmy constructs the OITEas a d!.agnostir: instrument m:,ant to indi_caLe residents'relative strongns and i;onhnosqes. It is therefo,7e composedof question,,., with a wide ra-Ig2 of difficulty so that one can

clearly thstinjuish between high ability and low abilityresidnts. The Board constructs the OCE as a certificationinstrum-nt and it is therefore criterion oriented. Knowledgeand abiliti-2s tho,Ig'IL essential for any cmp:,tent ortho-pacdit a):e inclod :d and all candidates are expected toobtain a certain minimum score. For the OITE residentsoften make no advance preparation and therefore test scoresarr reflective of their long term knowledge. On the otherhand, many resident candidates for the OCE attend coursesdesigned to prepare them for the examination. The candidatesoften review past years of the Bone and Joint Journal andread through a text on orthopaedics. The OITE providesprecise directions regarding administration of the exam butthere is evidence that these directions are not followed asclosely as they arc for the OCE, since there are manydifferent examination settings for the OITE.

Considering all the differences between these examina-tions it was significant correlations betweensimilar scores on each exam were tound. The OCE scores onthe 1971 exam were matched with the fourth year OITE scoresfor 321 residents. A correlation of .14 is significantat the .01 level.

TABLE

III

- 8

CorrelationsbetweenCITE andCCE

Scores

Scores

Correlation

Total

multiplechoice

(M/C)

.58**

M/C

adultorthopaedics

.41**

N/C

children's

orthopaedics

.42**

N/Ctrauma

.35**

1

M/Cgeneral

orthopaedics

.32**

4

M/C

anatomy

.34**

11/C

physiology-biochemistry

.31**

M/Cpathology

.31**

14/Cbiomechanics

.19**

M/C

rehabilitation

.20**

M/C

hand

surgery

.24**

**D

<01

t iv PREDICTION fWODY

This final section of the ::eport is an attempt toprodic".: resident. performance on the multiple choice andPMP ,.ieetions of the OITE from hnowlcdge of residentbackground characteristics, his attitudes and perceptionsabout his residency projram, and characteristics of thatprogram. The signii:icance of this analysis lies in theinvestigation of the interplw. 1)etween the personal char-acteristic,; and previous hi;, tor or the resident and thesubsequen projram structures and e:Tcrienees which heWI].]. encounter in residency. Essentially, tnis sectionattempts '_(.) examine the program environment for which arenjd(,nt is most suited, wh...:ro most suited is deLined interms of OTTE performance only.

The tre of OTTE scores as a critrion for judingresident pn:C7orance has certain limitations. The actualsurgical competence of a resident: is virtually impossil)leto tesl- by timed paper and pencil Lasts and Is c(inse-quently not considel:ed in this analy:;is. Secondly, aLest_ is a somewhat artificial situation in which somedifferent shills fruit those utili7.ed in the practice oforthopaedics arc employed. The ability to function undertime restrictions, to recall information from memory, andto choose from presented alternatives are advantages inthe OITE which Co not necessarily play a role in the prac-tice of orthopaedics. Despite these limitations, thefourth year OTTE scores were felt to yield the most validevaluation of orthopaedic education, and were the most ob-jective criteria upon which to assess resident education.

Sample.

All residents in the fourth year of residency fromthe Study sample for whom scores were reported on the mul-tiple choice and PMP parts of the OITE were included inthis analysis. There was a total of 419 residents withthe following program designations:

-98-

105

-99-

1. 373 Residents from University programs (8V)1

2. 13 Residentn from University-affiliatedprograms (3;1

3. 33 Rec,idr:nts from independent program4 (LW)

An iJ-; apparent from these figures, this anz.lyniswill be hravily influenc(.d by re:71id(,:nt..; front unirsityprograms and this should be heft. in mind in inLerprvtingthe subcquent rerailL:-:. In order Le compnnnaLe for thruniversity program emphilni, separate analyses will a] sobe includod ' :'Mich arc re:Ariel( d Lo rc:lidunts from inde-pendent programs.

Method

Itemq from the various Orthopaedic Training Studyquentionr!airel werc re]ecLed as po:;sible seurce re-lated to variation in performance on the OTTfl. The iintof its" ::;: u:;cd and the questionnaires from which theywere Laken is presented in Table 1. (The Supplementcontains the questionnaire from which the items weretaken). If a resident had not responded to items whichwere used in this analysis, the average response forthat: item replaced the missing information.

The statistical technique used in this analysis ismultiple regression, which is essentially a correlationbetween a combination of variables and a criterion. Aswith the previous analytic studies, a correlation indi-cates the strength of an association between variables,and in the case of regression, the strength of the associ-ation between a comhinaWD of variables and a criterion.

'The definitions of these terms is presented in thesection of the Final Report which is concerned with Pro-gram Characteristics. The definitions appear as the firstthree footnotes of that section.

TABLE :IV77;(1.1,.d:11 iti '1111: PC1 olcT t. c1:104'

I I c' r, Item Souteo

tfoital f.LALus hAd.olound f,tivoyAqo haeLgroendPaLliria, Lineation Loy,.] lide!.opound Sureyrotlir'r'n 1,hr:ation11 L Doehlionnd SurvoyParental 1n,...0,, haaground SurveyIlioh lichoal Grad,- Point Lvera.lo ilach:Jroun0 SurveyCollo'e Gral- Av4r.c: hael;...4round Survey

1%vora..t. itacLqrounl Survey44

Aye a! vilich Ir ' ot, rr.licjiao BacLaronnd Survey4.44

Age at w1,ic!1 los14:(nt. .1,.t.ide.1 on orth paedien Macliqround SurveyAny othe, oecu1)ati4:: llaltriround ;111-vcy

Percept ion of r(: co. p...1 it ion liaclf,jround Survey

c'urv,.".,

1,),11(1 114- At Li t Sul vey

Sufficient o; ;,or Ltd, i t y to (Icy, 1(.1, and -Asoo:r.rat i Ow Atli t ude Survey

f; is in.telutte ALI i fed F,urvey

'10a many 41 pat 1, nt s ',ti fly i jc( Attitud, Survoy1:14

hi,:,, the rprci'ic cl.;ceti.n.s of tiv.!proqra:1; Attitedo Survey

rr, A00 litllo Oh try ALLittic:o Survey

Tc;v2Litvi away from ether amvertantactivities Attitude Survey

D-..mno.n on are irpossibla to tmet r,uveyIyn.a10 :41 11:i!.te.Nu ".;,-) of 1 z.n op.n .rocture Procn-ntri.s FormCori i' .noo in dthlit! at' Lptn lir.. Aire ProcoOorLn PormNul,l,or of 1 internally fi:iet1

fractur,.1 hip Procc:11.-cs Form

v.COhtionoo in internally fiity.; a r...Cluce'i

0 frael.urc:: hip

Nu..H.r of Lo.:.e 1 f :fc:mod a hip alliroplanty,Lon.,con:c. in p,.rioliong a hip arth opl4n..ti)niti.l :o-4], docision4.

4, r :'= -c;, 1..,l_ i

Clf,. prrecdneIln(il an 4 of

Po-t-o.ylativerriertoiley Lotrn Nani..4.14:-ic

cp..nt_ in L;e':.:ncy Room

Time pc.,t_ in pf,rformini uaraery:n To4.ching

t e),,01 It ;-,rn

'op., 01 i.rL-tram P1ogram Qt.ostionnaircorflaniiiation of the 1 re9ram Profit ttio Questionnaire

Location of ehief, or: Progra't QuorticrwiptroNimbor of ati.yncling I:_:'!1: r :. 1'roar.:7. Qur:sticnnaireAttinclin time an clinical vor Progro:1 (!uostionnaircAttonLIfnin time in to..hinq PrOgrat Questionnaire0:-ientntion of the attenditt:j staff Progro..1 Ou:stionnaire

0u Ntmhor of progrx:i residents Progra.i Questionnaire

0 Proyz!... QuestionnairePro.ilLm Tanath ProcfraL Questi.onnaire

0.11 ,1 CI) jr.Ct 1.,pcc Prorjr..'. Questionnaire.Stlit.t of 1,,,!N.:; of ont-paticrof clinics Proqra- QuestionnaireNo of t,:pt_s of sub-!Tvcialt -s on staff Progrtri QuestionnaireNumhor oi typ:a o' fncil.;tien Progwl QuestionnaireBasic Scinee cur i,-nlum Progrn QuestionnaireT achina Prourn...1 Qt.:stionnaireLe:.eaLch pnpor Projra..: Questionnaire11

Procecluin 1 orin

Proc.-% LormProot;P:rc:.. Form

Lval ILA ion of operative Prock-cluresEva) o it ion of veLvaluA ion of Opt...rati 1,roco:-.Itu.e.-;

(,:*1 of oy' ra ivo ProccflurosEv..111,:itien of Qp:,rative Proc.:lures

12valu-Lton of Opz:rative Proct..cluref,

Time Lo,,

Tim, Loll

Time Lo..1

Tim(' Lc.:

Pap:tn at Que:;LiennaireLiainin; ,:oierviston Pro,Jtom 011.-:5tionnaire

Foe.IhnL to renidont Pro9/.119 QuestionnairePatint Caro re:n.i.u.ihilitaos Proro.i. QuestionnaireSurgic;)1 supervi..ion .1.G7 Progr=3 QuestionnaireNum'Jer of tn the proqra.a Proyran Questionnaire

-300-

-101-

It should be remembered, however, that associatiun amongvariables is not indicative of causal relationships.Also, in this analysis, there will be no significancelevels plsentod because all reported variables and theircombinations were significant: at the .05 level. That isto say, that the probability that the associations ob-served among these variables was due to chance was lessthan five percent.

Regression has the advantage over correlationalanalysis of permitting an assessment of variables incombination as predictors of a criterion. Briefly, thetechnique determines which variable is iaost highly relatedto the criterion, then the variable which is next mosthighly related given the influence of the previous vari-able, etc. In this way, unique combinations of variablescan be abstracted which allow more precise estimation u.rprediction of an individual's performance on the criterion.

Results

TOTAL MULTIPLE CHOICE SCORE

A standard score on the total multiple choice sec-tion of the OITE was computed for all fourth-year residentswho has taken the examination. T1 sample had a mean of50.0 and a standard deviation of 10.0. The mean was 52.1,with a standard deviation of 9.5 for the 419 residentsincluded in this analysis. With this measure as a cri-terion, seven variables were found to be optimally func-tional for prediction of performance on the OITE multiplechoice section. Table 2 presents these items in descendingorder of their relative power for prediction, along withthe cumulative prediction power of the preceding variablesin the analysis. That is, the ability to predict increasesas more variables are included in the analysis and thethird column of each table presents this cumulativeinc.rease.

-102 -

TABLE IV - 2

INTERACTION OF RESIDENT AND PROGRAM CHARACTERISTICSIN PREDICTING MULTIPLE CHOICE SCORES

Item 2 Positive ItemResponse

CumulativePredictive

Power

Admissions Criteria-Technical AptitudeType of ProgramToo many ward casesSurgical SupervisionCollege Grade PointAverageAdmissions Criteria-Intellectual Approachto problems

Not enough elective time

Less Important 7.26University 11.69Disagree 15.06Wheil needed 18 01

High 20.72

More ImportantDisagree

22.9924.87

This table shows several trends. First of all, withthe exception of college grace point average, all ems whichare significant in the prediction of total multiple choicescore are program variables or attitudinal sets which aredeveloped in response to a program environment. Secondly,residents in programs which seem to emphasize technical apti-tude tend to perform less well on multiple choice sections,while residents in programs stressing intellectual endeavorsperform better on the CITE. This may be due to the relativeimportance of orthopaedic surgical competence in the formerprograms in compar2 to :_hc importance or -book" learning,whicn is she thrust of the OITE. Finally, in programs whereresidents perceive strong service demands, there tends tobe a lower score on the multiple choice part of the OITE.This finding may be attributable to an actual lack of timeto devote to book learning or it may be that these attitu-dinal items are measuring underlying dissatisfaction withthe programs or with orthopaedics in general.

2All items are included in the Supplf.:ment.,

I-103-

Having now investigated the interaction between charac-teristics of the resident and program influences, the rela-tionship of program characteristics,when the background ofthe resident was controlled for in the analysis, was examined.That is, abstracting out the influence of any personal char-acteristics of the resident, such as, college grade pointaverage, the most salient influences of the programs on OITEperformance were investigated. Table 3 presents the resultsof this analysis.

TABLE IV - 3

PROGRAM CHARACTERISTICS RELATED TO MULTIPLECHOICE SCORE PREDICTION

Item Positive ItemResponse

CumulativePredictive

Power

Admissions Criteria-Technical AptitudeType of ProgramToo many ward casesSurgical SupervisionAdmissions Criteria-Intellectual Approach toproblemsNot enough elective timeUse of Operative Skills

Less Important 7.26Uliversity 11.69Disagree 15.06When needed 18.01

More ImportantDisagreeDisagree

20.1321.9223.21

It is clear from this analysis that, despite the uniquepersonal characteristics of the resident, program structuresand resident attitudes formed in response to those struc-tures are predictive of OITE performance. This result, may,however, be slightly artifactual, due to the heavy weightingof university program isidents in this analysis. That isto say, the policy of surgical supervision as needed and thede-emphasis on technical aptitude as an admissions criterionare characteristics of university programs in general andthese characteristics may be expected to give separate re-sults within this sample. Further substantiation for thi,/argument is the relative importance of university progr7,as predictive of resident OITE performance, as indicatthis table.

109

-104-

In order to dissect further this apparent strong influ-ence of program characteristics on.OITE performance and toinvestigate the possible artifactual nature Of these rela-tionships, an analysis was performed separately for residentsin university and independent programs. i The 373 residentsin university programs had an average OITE total multiplechoice score of 53, and a standard deviation of 9. Table 4presents the significant influences on OITE performance foruniversity residents.

TABLE IV - 4

CHARACTERISTICS OF RESIDENTS AND PROGRAMSRELATED TO OITE MULTIPLE CHOICE SCORES

(University Programs)

Item

Admissions Criteria-Technical AptitudeCollege Grade PointAverageToo many ward casesSurgical SupervisionImpossible to meetdemands

Positive ItemResponse

CumulativePredictive

Power

Less Important 6.62

HighDisagreeWhen Needed

11.0714.8816.53

Disagree 20.40

It is interesting to note that, when the type of pro-gram is held constant, the influence of a background char-acteristic of the resident, his college grade point average,is more strongly.predictive of subsequent performance themare program characteristics which are influential when thesample was more heterogeneous. There are at least two pos-sible interpretations of these results. The environmentsof university programs may be sufficiently homogeneous thatany differences between them are unimportant in predicting

3University-affiliated program residents were excludedfrom a separate analysis because there were not a sufficientnuml:er of them.

eF

120

-105-

subsequent performance. That is to say, it may be that theonly relevant program characteristic is the overall type ofprogram rather than the variations within a specific typeof program.

On the other hand, it may be that university programsare able to be more selective in their recruiting practicesthan are orthopaedic training programs in general due to thegreater demand for positions in university programs. Thiswould allow university programs to choose residents who wouldbe more suited for success in their programs. As a conse-quence, the prior achievement of the residents would becomethe most relevant predictor of future performance, with theunique configuration of program structures among these pro-grams being less relevant. If admissions requirements aremore competitive, it would probably follow that the residentschosen would be more well-suited for the demands of the pro-grams and consequently, prior achievement would be most impor-tant in predicting future achievement.

The 33 residents from independent programs had an aver-age OITE multiple choice score of 45 and a standard devia-tion of 10. The most significant characteristics in predict-ing their OITE scores are displayed on Table 5.

TABLE IV - 5

CHARACTERISTICS OF RESIDENTS AND PROGRAMS RELATEDTO OITE MULTIPLE CHOICE SCORES

(Independent Programs)

CumulativeItem Positive Item Predictive

Response Power

Papers at annual meetings Yes 38.73Perceived residency com-petition Less 57.55Full-time attendingstaff in clinical work Greater 64.90

Time spent in performingsurgery Less 69.11

Admissions Criteria-Technical Aptitude More Important 72.95

-106-

Before discussing possible explanations of the resultsfrom this table, it is important to note that the increasein variance accounted for in this analysis is partially afunction of the smaller number of subjects in independentprograms. That is, the statistical manipulation enablesbetter prediction when sample size is small, rather than anecessarily better understanding of the multi-faceted natureof performance within independent programs.

ResIdents who are in independent programs which areless competitive, more clinical and research-oriented, em-phasize technical aptitude, and de-emphasize the performanceof surgery receive higher OITE multiple choice scores thanresidents in other types of independent programs. The back-ground characteristics of the resident do not seem to influ-ence his subsequent performance. Rather, the environment ofthe program and his perception of that environment appear

be the strongest predictors of OITE performance for resi-dents in independent programs.

There are three additional informative findings fromthese separate analyses which should be noted. First, resi-dents, on the average, in university programs perform at astatistically higher level on the OITE than do residents inindependent programs. Secondly, the only program influencecommon to the prediction of OITE scores between the twotypes of programs is the importance placed on technical ap-titude in the selection process. However, with respect tothis influence, the import of this selection criterion isreversed for the two program types. Finally, the lack ofperceived competiticn in independent programs is positively re-lated to subsequent performance. Conversely, an argument couldbe made that in university programs, the more competitiveenvironment, in terms of selection, results in a greateremphasis on prior achievement as predictive of residencysuccess, as measured by the OITE.

TOTAL PATIENT MANAGEMENT PROBLEMS SCORES

As with the multiple choice scores, a standard scorewas computed for all fourth-year residents. The group hada mean of 50.0 with a standard deviation of 10.0, whilethe 419 residents in the analysis had a mean PMP total

1.k

-107-

score of 51.0, with a standard deviation of 9.0. Table 6displays the influences most strongly related to performanceon the PMP section of the OITE:

TABLE IV - 6

INTERACTION OF RESIDENT AND PROGRAM CHARACTERISTICSIN PREDICTING PMP SCORES

ItemCumulative

Positive Item PredictiveResponse Power

Outpatient Trauma Respon. Earlier 1.69

Admissions Criteria-Technical Aptitude Less Important 4.14Surgical Supervision When needed 6.29Affiliated Program No 7.84School Learning More time 9.02

It is clear from this table that personal character-istics of the residents do not influence the prediction ofPMP total score. Rather, the extent of independence andresponsibility available to a resident are the most signif-icant estimators of problem solving ability as measured bythe PMP's. It is also significant that two of the itemswhich were predictive of multiple choice scores were alsopredictive of PMP scores, technical aptitude and surgicalsupervision. This may not be particularly-surprising asthe two tests scores are positively correlated (.32), in-dicating that certain program structures may be mutuallybeneficial in influencing both "book learning" and problemsolving ability. That is, as there is a positive associa-tion between PMP score and multiple choice score, it may bethat some of the same program characteristics that are asso-ciated with high performance on the PMP are also associatedwith high performance on the multiple choice section.

As none of the items related to personal characteris-tics of the residents aided the prediction of PMP perfor-mance, it was unnecessary to control for these influencesin a subsequent analysis. Instead, the analyses were per-formed separately for university and independent programs

-108-

to determine if the ability to predict PMP score would beimproved by examining the two types of programs separately.The results of the analysis for the residents in universityprograms are presented on Table 7.

TABLE, IV 7

CHARACTERISTICS OF RESIDENTS AND PROGRAMS RELATEDTO OITE PMP SCORES(University Programs)

Item

Surgical SupervisionAdmissions Criteria-Technical AptitudeComplex ProceduresHigh School Grade PointAverageTeach Allied HealthPersonnel

CumulativePositive Item PredictiveResponse Power

When needed 3.18

Less Important 5.61Earlier

Higher

8.01

9.23

No 10.31

Again, it is interesting to note that program influ-ences appear to be more significant in predicting performancethan personal characteristics of the residents. This istrue, even when the program type is held constant. On theother hand, when the same analysis was performed with totalmultiple choice score as the criterion, the influence ofcollege grade point average was the second most relevantpredictor of performance (see Table 4). In this analyses,high school grade point average was the fourth most relevantpredictor of PMP performance. This is interesting both be-cause its predictive power is considerably less than was col-lege grade point average for the corresponding multiple choiceanalyses and because this is the first time that the import-ance of high school grades have been observed in any of theanalyses. It is also clear that the ability to predict PMPperformance is considerably weaker than the ability to pre-dict multiple choice performance because the total cumulative

-109-

prediction power with PMP as a criterion is 10.31 as comparedto 24.87 with multiple choice score as a criterion.

Table 8 displays the results of the analysis for resi-dents in independent programs. Again, a word of caution isnecessary in interpreting these results, due to the smallnumber of residents included in the analysis.

TABLE Iv - 8

CHARACTERISTICS OF RESIDENTS AND PROGRAMS RELATEDTO OITE PMP SCORES

(Independent Programs)

Item

Specific Objectives knownImpossible to meet demandsResident training super-visionBoard requirements are toorigid

CumulativePositive Item PredictiveResponse Power

Agree 23.62Agree 38.79

Close 46.82

Disagree 54.42

With the exception of the supervision of residenttraining, the attitudinal dispositions of the residents inindependent programs seem to be most influential in pre-dicting of PMP performance. If program objectives are knownand Board requirements are deemed sufficiently flexible,residents tend to perform well on the PMP section of the OITE.

The findings with respect to supervision and work de-mands are suggestive in that within an independent programenvironment these findings may be a reflection of the cali-ber of the program. If residents perceive impossible demandsand if the program reports close supervision, then the educa-tional environment for these residents is atypical of inde-pendent programs. That is, the typical independent programis less concerned with teaching and educational opportunitiesas their service demands consume most of the attending staff

4 n

-110-

time. If, on the other hand, residents arc being providedwith educational opportunities superimposed on patient careobligations, the educational caliber of the program is en-hanced, The PMP performance of these residents may be sug-gestive of this type of program environment.

Discussion and Summary_

The ability to determine the relative influences onperformance indicates that the types and specific character-istics of residency programs are clearly significant ineducating orthopaedists. It appears that, once an internhas been accepted into a program, the unique program struc-tures determine subsequent performance to a greater extentthan the characteristics with which the resident enteredthe program. This is undoubtedly due, in part, to the pre-selection of residents which results in a relatively homo-geneous group of people. Having followed the same progres-sion from college to medical school, to surgery, andfinally, into orthopaedics, it is not particularly surpris-ing that the personal characteristics of the residents areless influential in prediction of criterion performancethan are program characteristics. To the extent that thesepersonal characteristics are still operating to influenceperformance, the question becomes one of fitting the pros-pective resident and program together for the optimalbenefit of each.

With respect to significant program characteristics,the suggestive trend indicates that subsequent performanceis more associated with selection criteria and the educa-tional opportunities than it is with the size or professionalorientation of the attending staff, or with the structuringof the curriculum. Rather, it appears that, with respectto OITE performance, residents in programs with selectioncriteria which emphasizeintellectual endeavors, de-emphasizetechnical aptitude, and a program which allows residentseducational and surgical opportunities perform better onthe OITE tests. Thus, it appears that the attitudinal setof the attending staff with respect to educational experi-ences is a better predictor of subsequent achievement thanthe size and specific structures of the program.

r.

It also seems evident from this analysis that the influ-ences on "book learning" performance arc more discerniblethan those on problem solving ability. This is probably noLsurprising, given the relatively structured nature of themultiple choice section of the OITE as contrasted with therelatively less structured interpretive PIN section of theexamination. Clearly, the more defined and concrete theperformance criterion, the more. precise the prediction ofthe performance. However, it does not necessarily followfrom this that one ability is more desirable than the other,only that there are differential influences on the two abili-ties. Thus, in evaluating the success of an orthopaedicprogram, both of these components of orthopaedic competenceshould be considered, as well as attempts to assess the sur-gical competence of the residents.

1:: 7