RESOURCES +* RESSOURCES

CertificationExaminationof the College ofFamily Physiciansof Canada

Part 2. Conduct and generalperformance

RICHARD HANDFIELD-JONES, MD, CCFP

JUDITH BELLE BROWN, PHDPAUL RAINSBERRY, PHD

CARLOS A. BRAILOVSKY, MD, MA(ED)

mhis paper is the second in a

series of four examining theCertification Examination of theCollege of Family Physicians ofCanada.' The College, founded in1954, has as one of its main respon-

sibilities to ensure that family physi-cians in Canada maintain a highstandard of practice. In recognitionof the acquisition of the knowledge,skills, and attitudes that reflect thisstandard, the College has conferredcertification on Canadian familyphysicians since 1969.A family physician can become

a certificant through a two-stageprocess that can be accessed two

ways: by successfully completingan accredited 2-year family medi-cine residency program and pass-

ing the certification examination;and by the practice-eligible route

designed for practising familyphysicians that requires evaluationof their practices by the College'sPractice Assessment (PASS)Program before they can sit forthe examination.2'3 In 1985, theCollege articulated the four princi-ples of family medicine.4* The doctor-patient relationship is

central to family medicine* The family physician is a skilled

clinician

* Family medicine is communitybased

* The family physician is a resourceto a defined practice population

The examination is designed toassess candidates' ability to practiseaccording to these principles.

Certification examinationWhen the first family medicine resi-dency programs in Canada wereestablished in 1966, the Collegeplaned to have an examinationready for the first graduates in1969. The inaugural examinationwas written by 13 candidates; thereare now more than 700 candidatesannually.

The examination has always hadboth written and oral components.'It has continually evolved in responseto feedback from both candidatesand examiners and to improvedunderstanding of examination tech-nology and psychometrics.5

At present, the written compo-nent, consisting of 40 to 45 short-answer management problems(SAMPs) is held on Friday and theoral part, consisting of five simulat-ed office orals (SOOs), is held onSaturday or Sunday depending oncandidates' schedules. The exami-nation is considered a single evalua-tion tool with separate butintegrated and complementarycomponents. Each component hasbeen developed to evaluate a partic-ular range of educational objectiveswithin the context of the four prin-ciples of family medicine.

Multiple-choice questions(MCQs) were used on the examina-tion until 1993. They are effectivewhen testing factual recall and haveconsistently had the greatest relia-bility. However, despite MCQs'ability to test a range of knowledge,they cannot adequately assess the

more sophisticated knowledge andadvanced skills required of certifi-cant family physicians.6 Successfuluse of SAMPs allowed MCQs to bephased out in the early 1990s.

Short-answer managementproblemsSimilar to the modified essay ques-tions (MEQs) on the Royal Collegeof General Practitioners of GreatBritain's membership examina-tion,7'8 SAMPs were first used bythe College in its emergency medi-cine examination in 1982. UsingSAMPs on the certification exami-nation during the last 4 years hasdemonstrated their relevance andsuitability to family medicine. Theyare effective for assessing much ofthe knowledge and many of theproblem-solving skills required forfamily practice. Positive feedbackfrom candidates also supports theiruse. The College has created a bankof more than 90 SAMPs; new onesare produced each year.

Each SAMP consists of a briefdescription of a clinical case fol-lowed by three to six questions. Thecase is described within a specificfamily practice context and includessome demographic, medical, andpsychosocial information about thepatient. The questions proceedthrough diagnosis and managementof the clinical situation. Candidates'brief written responses are scoredby comparing them with lists ofacceptable answers.

Simulated office oralsThere are five simulated office orals(SOOs) on the examination, eachconsisting of a 15-minute interviewwith a patient role-played by a cer-tificated family physician. TheSOOs present typical clinicalencounters in family physicians'offices because they are based on

1188 Canadian Family Physician I.L Medecin defamille canadien . VOL 42: JUNE .JUIN 1996 FOR PRESCRIBING INFORMATION SEE PAGE 1220 --),-

RESOURCES * RESSOURCES

real patients seen by members ofexamination subcommittees. Usingeffective interviewing and commu-nication skills, candidates areexpected to elicit the patient's histo-ry, to identify the patient's concerns,and to reach agreement with thepatient on a management plan.

Several procedures are followedto ensure standardization.Using standardized patientsand an objective scoring tech-nique, all candidates inter-view five equivalent patients.The examiners receivedetailed documentationdescribing the case and aretrained in their patient roleson the day before the exami-nation. Every examiner foreach case across Canadastudies videotapes of sampleencounters and practises therole with first-year familymedicine residents.

Candidates' performancesare scored using standardizedand pretested markingschemes by the physicians whoplay the patients. Scoring isbased on the Patient-CentredClinical Method described bythe Patient-Doctor Communi-cation Group of the Centrefor Studies in FamilyMedicine at the University ofWestern Ontario.9"0 Six scor-ing items are grouped underthe headings Problem identifi-cation, Social and develop-mental context, Management,

Tabbexaj

PERSOI

PHYSIC

Bc...........

Sit...........

w

asn...........

SA

SCSC

...........

SC...........

Pr(fix

...........

Tc

NONPH...........

Ac...........

Se...........

UC''''ci

...........

Al...........

SP

TO1

and Interview structure and orga-nization. Scoring criteria for thevarious performance levels arebased on objective observablebehaviours that are mutually exclu-sive. Scores reflect candidates'demonstration of both clinicalknowledge and patient-doctorcommunication skills.

Administrative structureTwo national College committeesplan and coordinate the examina-tion process. The Committee onExaminations produces the twoannual examinations by coordinat-ing the work of eight regional sub-committees. Members of thesubcommittees come from both

1. Personnel required for springrnination

VNEL PER SITE TO

:IANS...........................................................................................

ard of Examiners representative 1...........................................................................................

te coordinator 1...........................................................................................

rritten examination coordinator 1...........................................................................................

'ritten examination isistant coordinator

LMP marking coordinators 1-2 :...........................................................................................

DO coordinators 5 4...........................................................................................

DO assistant coordinators 5 4...........................................................................................

)O examiner/role players 15, 20, or 25 14...........................................................................................

actice candidates 5 4rst-year residents)...........................................................................................

)tal physicians 35-46 34

YSICIANS

dministrative coordinator I

cretaries 3-5 '

shers 4-5 4

Ierk 1

adiovisual technicians 1-2

LMP markers 6-19 14

tal nonphysicians 16-33 2:

rAL 51-79 55

academic and community practiceand from both urban and ruralareas. The national committee com-prises a chairperson, the chairs ofall regional subcommittees, a resi-dent representative, the College'sDirector of Education, and theexamination coordinator. The com-mittee meets over 11 months to

construct the examination. TheCommittee on Examinations is alsoresponsible for analyzing examina-tion results with the goal of improv-ing the examination instruments ordeveloping new ones. The Board ofExaminers, the senior policy-mak-ing committee, assumes responsibil-ity for pass-fail decisions and

addresses appeals from fail-ing candidates.

Having two examination

TAL bodies working independent-ly yet in a coordinated fash-

. ion helps increase the9 content validity of the exam-

iination process by providing........... checks and balances. The9......... Committee on Examinations9 can challenge a Board deci-

sion to change a policy, and13 the Board reviews all exami-.5 nation materials developed...... by the Committee. At times,.5 when major changes have60 been proposed (eliminating45 an instrument, adding a new

one, or changing an existing44 one, such as when the new

SOO scoring system was.......... introduced in 1992), it has

.proven valuable to have the39 two committees deliberate44 separately and approve......... change independently of9........ each other.10

01 Logistics12. wo examinations are held

each year. Most candidates,>6 approximately 720, sit the

spring examination that is heldsimultaneously across Canada in sixEnglish-speaking and three French-speaking centres. The nine sitesvary in size, accommodating32 candidates (Sherbrooke, Que) to188 candidates (Hamilton, Ont). Inthe fall, another examination is heldin both English and French at two

1190 Canadian Family Physician. LeMidecin defamille canadien oVOL 42: JUNE JUIN 1996

RESOURCES *+ RESSOURCES

sites mainly to accommodate prac-tice-eligible candidates and thosewho fail the spring examination.The two examinations are identicalin structure but different in content.The examination's annual bud-

get is approximately $725 000.Table 1 shows the personnelinvolved in the spring examination.Across Canada 344 physicians and212 support personnel areinvolved. The physicians receivethe College's usual per diem fee of$75 for each day of the examina-tion and are able to claim continu-ing medical education credits fortheir participation. The Collegerecognizes that this is a token reim-bursement for their time. Most ofthose involved do so out of a senseof commitment to the certificationprocess.

Examination performanceDuring the last 10 years, the overallpass rate has varied from 84.9% to92.0%. Table 2 gives mean scoresand reliability coefficients for thethree instruments.

Descriptive statistical analysis ofresults of the 1992 and 1993 exam-inations, conducted on various sub-groups of candidates, showed, inmost instances, no differences inperformance (Table 3). An excep-tion was practice-eligible candi-dates, who had a somewhat higherfailing rate than residency candi-dates. Analysis of the results ofgraduates of the 16 Canadian fam-ily medicine residency programsrevealed a difference in total scoresamong the programs at theextreme ends, but not for the groupof programs collectively and notenough to influence final pass-faildecisions.

Pearson correlations betweenthe scores of the three instrumentsobtained by all candidates and

VOL 42: JUNE .JUIN 1996 * Canadian Family Physician * Le Midecin defamille canadien 1191

RESOURCES +* RESSOURCES

partial correlation analysis areshown in Table 4. Results correlatesignificantly with the total score onthe examination. However, thecorrelations between scoresobtained with the three instru-ments are relatively low, indicatingthat they measure complementaryelements of competence.

Feedback is always sought fromcandidates by means of writtenquestionnaires. Candidates are con-sistently dissatisfied with the MCQsand very satisfied with the SAMPsand SOOs. Overall, they havethought the examination appropri-ate to family medicine.

DiscussionWith rapidly changing requirementsin medical licensure in Canada, therole and function of examinations isunder growing scrutiny. Largely dueto the increased use of standardizedpatients in objective structured clini-cal examinations (OSCE), examina-tions are becoming more complexand more logistically challenging."Well-constructed OSCE examina-tions can be more reliable and validfor evaluating medical trainees thanprevious formats.'2 Also, knowledgeof the psychometric properties ofexaminations is becoming moresophisticated, allowing for specificrefinement of the instruments. 13Because of these factors, examina-tions are enjoying a resurgence ofrespect and prominence in medicaleducation and from licensingbodies.'4 Unfortunately, introductionof more complex examinations isvery costly, and costs are oftenpassed on to candidates. To justifysuch expense, all examining bodiesshould look critically at the purposeand quality of their examinations.

Evaluation. The certificationexamination has been regularly

evaluated over the years.'5'8 Thisseries of papers reports on the mostrecent analysis of its performance.Despite outstanding features, theexamination still does not adequate-ly evaluate the complex knowledgeand skills necessary for practisingfamily medicine.One way to increase validity of

an examination is to use severaldifferent evaluation instruments.The Royal College of GeneralPractitioners in Great Britain usesfour different instruments (MCQs,MEQs, a critical reading paper, andan oral examination)'8 and theAustralian College of GeneralPractice uses eight.'9 The Collegehas always used a combination ofinstruments.The different instruments used

by the College over the years havedemonstrated acceptable reliability.Reliability coefficients of theMCQs and SAMPs compare wellwith published reliability statisticsfor other medical examinations.20

Because MCQs cannot effec-tively evaluate the complex knowl-edge and skills requisite for familypractice, their validity is greatlyreduced6; SAMPs are an excellentalternative. The SAMPs are reli-able because acceptable answersare standardized and are morevalid than MCQs because theydemand more than simple factualrecall. Presenting more than 40cases allows adequate sampling.For SOOs, the a coefficientsappear to be relatively low, proba-bly due to the small number ofcases used and because they evalu-ate complex, multidimensionalskills including patient education,communication, management, andestablishing effective patient-doctorrelationships.The challenge faced by all exam-

ination bodies is to design and

implement highly reliable and validexaminations; this requires exten-sive financial and personnelresources." Although SOOs offerhigh validity by simulating officeencounters, too few cases can bepresented to ensure reliability.Having even five SOOs is labour-intensive and expensive. Genera-lizability analysis (optimization orD-studies) has demonstrated thatbetween 15 and 20 SOOs would berequired to achieve a reliabilitycoefficient of 0.8.22 A reliability ofaround 0.6 is considered acceptablein a multidimensional examination.Recent introduction of a newmarking scheme has added to theSOO's reliability and validity. Thenew scheme is being evaluated, andfurther improvements shouldincrease reliability.

Evaluation of content validity isdifficult without performing a struc-tured retrospective or prospectivestudy with a randomly selected groupof judges. However, a degree ofvalidity is built into the examinationin the way it is constructed.Experienced family physicians whopractise in different areas and settings,including academic teaching units,construct the examination. MostSAMPs and all SOOs are based onactual patient encounters. TheSAMP scoring coordinators, SOOrole-players, and candidates them-selves have consistently reported thatthe examination accurately reflectsfamily practice and fairly evaluatesthe competence of family physicians.Despite this feedback, construct valid-ity should be researched to confirmthe assertions.23

Is an examination necessary?Though it has been shown thatresults of in-training evaluation ofresidents rarely predict their perfor-mance on terminal examinations,

1 192 Canadian Family Physician . Le Midecin defamille canadien * VOL 42: JUNE .JUIN 1996

RESOURCES + RESSOURCES

this finding might support thehypothesis that the two approachesare complementary.24

For residents, the certificationdecision ideally should be basedsolely on their performance duringtraining. For practice-eligible can-didates, quality of practice shoulddetermine their worthiness for cer-tification. The recommendationsof the College's Task Force onIn-Training Evaluation haveestablished the framework forhigh-quality in-training evalua-tion.25 However, tools for evaluat-ing residents would have to meetthe same rigorous standards ofreliability, validity, and standard-ization as examinations, and suchtools do not yet exist. Some toolsfor assessing the quality of medicalpractice, such as the College'sPASS program and the McMasterUniversity Physician Review andEnhancement Program (PREP)exist,26 but these cannot draw asufficient number of different clin-ical encounters to be both reliableand valid. The reality is, therefore,that perfect tools do not exist,either for practice evaluation orfor examinations, which, at best,can only indirectly evaluate actualpractice behaviours.

There are other, more indirectreasons to conduct examinations.Though examinations should bedesigned to evaluate the attainmentof educational objectives, it is truethat they often affect curriculum.In 1990, Quebec introduced anOSCE component as part of thelicensing examination for familymedicine that tests, among otherthings, physical examination skills.Subsequently, the four Quebec uni-versity departments of family med-icine increased teaching in thisarea.21 An examination can, there-fore, be used to reinforce the

importance of certain educationalobjectives.

Other reasons are unscientificbut carry important symbolic value.Licensing bodies need to beaccountable to the public that theyare delegated to protect.Examinations continue to providean important measure of a stan-dard of practice on which the pub-lic can rely until the effectiveness ofa comprehensive in-training orpractice assessment process can bedemonstrated and society acceptsdecisions based on such a process.Peers and institutions recognize suc-cessfully passing a terminal exami-nation as achieving an importantqualification. Finally, candidatesgain confidence to enter into inde-pendent practice.

ConclusionThe certification examination pro-vides a valid and largely reliablemethod for evaluating the knowl-edge and skills requisite for practis-ing family medicine. Innovationsand improvements are introducedregularly as the understanding ofpsychometrics improves and as theCollege continues to refine its under-standing of what is required to prac-tise family medicine effectively. m

Dr Handfield-Jones is Director ofContinuing Medical Educationfor the Collegeof Family Physicians of Canada andpractisesfamily medicine in Oakville, Ont. He wasChair of the College's Committee onExaminations from 1990 to 1994.Dr Brown is an Assistant Professor in theDepartment of Family Medicine at theUniversity of Western Ontario and is based atthe Thames Valley Family Practice ResearchUnit in London, Ont. She serves as consultantto the College's Committee on Examinations.Dr Rainsberry, the College's Director ofEducation, is responsible for developing andconstructing the Cert4fication Examination in

Family Medicine. Mr Brailovsky isDirector of the Centre d'evaluation des sciencesde la santi de l'universite Laval and is aProfessor in the Department of FamilyMedicine at Universite Laval in Qebec Ci_y.He is Coordinator for Measurement andEvaluationfor the College.

Correspondence to: Dr P Rainsberry,Director of Education, College of FamilyPhysicians of Canada, 2630 Skymark Ave,Mississauga, ON L4W5A4

References1. Handfield-Jones R, Rainsberry P.Certification examination of the Collegeof Family Physicians of Canada. Part 1:History and implications for the presentand future. Can Fam Physician 1996;42:957-69.

2. Borgiel AEW, WilliamsJI, AndersonGM, Bass MJ, Dunn EV, Lamont CT,et al. Assessing the quality of care in fami-ly physicians' practices. Can Fam Physician1985;3 1:853-62.

3. Borgiel AEW, WilliamsJI, Bass MG,Dunn EV, Evensen MK, Lamont CT,et al. Quality assessment of family physi-cians: does training have an effect?Can Med Assocj 1989;140:1035-43.

4. The College of Family Physicians ofCanada. General information and regulations onprogram accreditation and examinations.Mississauga, Ont: The College of FamilyPhysicians of Canada, 1993.

5. Handfield-Jones R, Hollingworth GR.CFPC's certification examination. Anoverview. Can Fam Physician 1990;36:2069-74.

6. Neufeld VR. Written examinations. In:Neufeld VR, Norman GR, editors.Assessing clinical competence. Springer series onmedical education. Vol 7. New York: SpringerPublishing Co, 1985.

7. WalkerJH, Stanley IM, Venables TL,Gambrill EC, Hodgkin GKH. TheMRCGP examination and its methods.IV MEQpaper. JR Coll Gen Pract 1983;33:804-8.

VOL 42: JUNE .JUIN 1996* Canadian Famiy Physician .L Medecin defamille canadien 1193

RESOURCES +* RESSOURCES

8. Lockie C, editor. Occasional paper 46: exam-

inationfor membership in the Royal College ofGeneral Practitioners. Exmouth, UK: Vineand Gorfin, 1990.

9. Stewart M, BrownJB, Weston WVW,McWhinney IR, McWilliam CL, FreemanTR. Patient-centered medicine: transforming theclinical method. Thousand Oaks, Calif: SagePublications, 1995.

10. LevensteinJH, McCracken EC,McWhinney IR, Stewart MA, BrownJB.The patient-centred clinical method. 1. Amodel for the doctor-patient interaction infamily medicine. Fam Pract 1986;3(1):24-30.

11. Grand'Maison P, LescopJ, RainsberryP, Brailovsky CA. Large-scale use of anobjective structured clinical examinationfor licensing family physicians. Can MedAssocj 1992;146: 1735-40.

12. Brailovsky CA, Grand'Maison P,LescopJ. A large-scale multicenter objec-tive structured clinical examination forlicensure. AcadMed 1992;67(Suppl):S37-9.

13. Van der Leuten CPM, Swanson DB.Assessment of clinical skills with standard-ized patients: state of the art. Teach LearnMed 1990;2:58-76.

14. Grand'Maison P, LescopJ, BrailovskyCA. Canadian experience with structured

clinical examinations. Can Med Assocj1993; 148:1573-6.

15. CorleyJB. The College certificationexamination: a preliminary study. Can FamPhysician 1975;21:1 18-24.

16. Lamont CT, Hennen BK. The use ofsimulated patients in a certification exami-nation in family medicine. 7Med Educ1972;47:789-95.

17. Rainsberry RP Oral examinations in theassessment of family physicians in Canada.ProcAnnu Conf Res Med Educ 1986;25:345-6.

18. Sawa RJ. Assessing interviewing skills:the simulated office oral examination.J Fam Pract 1986;23(6):567-7 1.

19. Hays RB, Fabb WE, van der LeutenCPM. Reliability of the fellowship exami-nation of the Royal Australian College ofGeneral Practitioners. Teach Learn Med1995;7:43-50.

20. Brailovsky CA, Bordage G, Allen T,Dumont H. Writing vs coding diagnosticimpressions in an examination: short-answer vs long-menu responses. Res MedEduc 1988;77:201-6.

21. LescopJ, Grand'Maison P, BrailovskyCA. L'examen du permis en medecine defamille, une evaluation systematique de lacompetence clinique. Ie Medecin de Quebec1994;29:1 15-9.

22. CardinetJ, Tourneur Y Assurer lamesure, guide pour les eitudes degeneralisabilite'. Berne: Editions PerterLang SA, 1985.

23. Brailovsky CA, Grand'Maison P,Lescop J. Construct validity of theobjective structured clinical examinationused in the Quebec licensing examina-tion (QLEx). In: Rothman Al, Cohen,editors. Proceedings of the Sixth OttawaConference on Medical Education; 1994 June26-29; Toronto. Toronto: University ofToronto Bookstore Custom Publishing,1995: 373-4.

24. Brailovsky CA, Grand'Maison P,LescopJ. Residency directors' predic-tions of candidates' performance on a

licensing examination. Acad Med 1995;70:410-4.

25. The College of Family Physicians ofCanada. The report of the Task Force onIntraining Evaluation of the Section of Teachersof Family Medicine. Mississauga, Ont: TheCollege of Family Physicians of Canada,1993.

26. Norman G, Davis DA, Lamb S, HannaE, Caulford P, Kaigas T Competencyassessment of primary care physicians aspart of a peer review program. JAMA41993;270: 1046-5 1.

C* ... ......,gev .'f.;~e .....r'..sCollege offersnewopp esNew educational softwareContinuing a tradition of educationaiexcellence, The College of FamilyPhysicians of Canada and ClinidataInc are pleased to announcethe launch of Self LearningInteractiveThm (Auto apprentissageinteractifTm). Called the "evolution ofSelf Evaluation," Self LearningInteractive"m contains questions com-posed by the writers working with theSelf-Evaluation Committee. The soft-ware is Windows compatible and is

approved for 30 MAINPRO-MIcredits yearly.

Self .Learning. Interactive.' alsouses the power of the Internet. With.-one. click of the mouse,. subscriberscan launch their o.wn Internet soft-ware to further their research. Forthose who are not on-line, theCollege has negotiated a specialpackage with NETCOM Canada, anational Internet service provider.Induded in. this offer is a 30-day freetrial for College members..For informa.tion and orders,

contact Sarah Beer, MarketingCoor4ator, at.The C.llegje ofFamily Physicians of Canada: phone

1-800-387-6197 or (905) 629-0900,or fax (905) 629-0893.

Sprint CanadaRecently, all members receivedinformation from The College ofFamily Physicians of Canada advis-ing of a new long-distance savingsoffer available through SprintCanada. The College and SprintCanada are building on a long-termrelationship by extending savings toour members. We hope this initia-tive will aid yoiu in reducing yourlong distancet fees and help to mini-mize the high costs of maintainingyour practice. u

'* FOR PRESCRIBING INFORMATION SEE PAGE 1236 VOL 42: JUNE*JUIN 1996* Canadian Family Physician . Le Midecin defamille canadien 1195