RESIDENTS’ EVALUATION SCHEME - pogsinc.orgpogsinc.org/files/annual_trainors/Residence...

RESIDENTSRESIDENTS’’EVALUATION SCHEMEEVALUATION SCHEME

ScopeScope

Background Background FormulationFormulationPilot study Pilot study

(some) Reaction to reactions(some) Reaction to reactionsAxioms in educationAxioms in educationYour experiencesYour experiences

RESIDENCY EVALUATION SCHEMERESIDENCY EVALUATION SCHEME

Formulated by 2004 PBOG in order Formulated by 2004 PBOG in order (1) To emphasize the competencies in the (1) To emphasize the competencies in the

development of knowledgeable, skilled development of knowledgeable, skilled and compleat specialists and and compleat specialists and

(2) To standardize the evaluation process (2) To standardize the evaluation process across accredited institutions.across accredited institutions.

NEEDS ASSESSMENTNEEDS ASSESSMENT

EVALUATION IN RESIDENCY TRAININGEVALUATION IN RESIDENCY TRAININGUniform format Uniform format Objective meansObjective meansValid toolsValid toolsMonitor progressMonitor progress

CONTEXTCONTEXT

Highest quality standardsHighest quality standardsCompetencyCompetency--based programbased programCore curriculumCore curriculumReadiness for certificationReadiness for certification

ProductionProduction

2004 Annual 2004 Annual trainorstrainors’’ meeting: meeting: workshops: knowledge, skills, attitude workshops: knowledge, skills, attitude

Output studied and refined by smaller Output studied and refined by smaller groups of PBOG membersgroups of PBOG members

OUTPUT: OUTPUT: RESIDENTS EVALUATION RESIDENTS EVALUATION SCHEMESCHEME

MASTER SHEET (summary)MASTER SHEET (summary)EVALUATION TOOLS (REF 1EVALUATION TOOLS (REF 1--4)4)

SCHEMESCHEME

COMPETENCYCOMPETENCY--BASEDBASEDFREQUENT ASSESSMENTFREQUENT ASSESSMENTMULTIPLE EVALUATORS MULTIPLE EVALUATORS FORMATIVE and SUMMATIVEFORMATIVE and SUMMATIVECHANGING WEIGHTSCHANGING WEIGHTS

COMPETENCIESCOMPETENCIES

KNOWLEDGEKNOWLEDGETheoretical baseTheoretical baseClinical competenciesClinical competencies

TECHNICAL SKILLSTECHNICAL SKILLSSurgical skillsSurgical skillsResearchResearch

ATTITUDEATTITUDEMotivation to learn, work ethic, social Motivation to learn, work ethic, social orientationorientation

EVALUATION METHODSEVALUATION METHODSWritten examinationWritten examinationOral examinationOral examinationPractical examinationPractical examinationChart reviews Chart reviews Observational rating Observational rating

SPECIFIED FREQUENCYSPECIFIED FREQUENCY

RATING SCALESRATING SCALES

CLINICAL COMPETENCIESCLINICAL COMPETENCIESATTITUDE ASSESSMENTATTITUDE ASSESSMENTSURGICAL SKILLSSURGICAL SKILLSRESEARCHRESEARCH

Total score / Perfect score = % Total score / Perfect score = %

PURPOSEPURPOSE

FormativeFormativeSummativeSummative

GRADINGGRADING

WEIGHTEDWEIGHTEDYEARYEAR--LEVELLEVEL--BASEDBASEDCHANGING RATIOS CHANGING RATIOS

MULTIPLE EVALUATORSMULTIPLE EVALUATORSSUMMARY SHEET PER RESIDENTSUMMARY SHEET PER RESIDENT

BACKGROUNDBACKGROUND

Consists of Consists of rating sheets to evaluate the residents rating sheets to evaluate the residents knowledge, skills and attitude and knowledge, skills and attitude and master sheet for each resident, containing master sheet for each resident, containing the summary of the residentthe summary of the resident’’s ratingss ratings

BACKGROUNDBACKGROUND

Dr. Erlinda G. Germar, 2004 PBOG Chair Dr. Erlinda G. Germar, 2004 PBOG Chair ––initiated the workshop, led refinement of schemeinitiated the workshop, led refinement of schemeDr. Rosendo Roque, 2004 BOT President Dr. Rosendo Roque, 2004 BOT President ––received outputreceived outputDr. Corazon T Lim, 2005 PBOG Chair Dr. Corazon T Lim, 2005 PBOG Chair -- urged urged the BOT early in 2005 to conduct a pilot study on the BOT early in 2005 to conduct a pilot study on the said schemethe said schemeDr. Evelyn Palaypayon, 2005 BOT president Dr. Evelyn Palaypayon, 2005 BOT president --created an ad hoc committee to conduct the pilot created an ad hoc committee to conduct the pilot study.study.

AD HOC COMMITTEEAD HOC COMMITTEE

Dr. Virginia R. de JesusDr. Virginia R. de JesusDr. Rey H. Delos Reyes Dr. Rey H. Delos Reyes Dr. Joseline FerrolinoDr. Joseline FerrolinoDr. Erlinda G. GermarDr. Erlinda G. Germar

Dr. Lilia P. LunaDr. Lilia P. LunaDr. Marietta R. Siongco Dr. Marietta R. Siongco

Dr. Maria Lourdes B. Coloma, Dr. Maria Lourdes B. Coloma, CHAIR CHAIR

Tenure Tenure -- three years or until the study is completedthree years or until the study is completed

REPORTREPORT

PILOT STUDYPILOT STUDY

BACKGROUNDBACKGROUNDOBJECTIVESOBJECTIVESSCOPE & LIMITATIONSSCOPE & LIMITATIONSMETHODOLOGY METHODOLOGY RESULTS & ANALYSISRESULTS & ANALYSISRECOMMENDATIONSRECOMMENDATIONS

PILOT STUDY : miniPILOT STUDY : mini--version of fullversion of full--scale scale implementationimplementation

TASK of the committeeTASK of the committeeto hold the study in selected institutions to hold the study in selected institutions with a view to uncover the schemewith a view to uncover the scheme’’s flaws s flaws and problems that can be dealt with before and problems that can be dealt with before the actual implementationthe actual implementation

OBJECTIVESOBJECTIVES

1. To ascertain these qualities regarding the 1. To ascertain these qualities regarding the schemescheme

FEASIBILITYFEASIBILITY -- the likelihood that it will work or the likelihood that it will work or be useful in attaining the end desiredbe useful in attaining the end desired

PRACTICABILITYPRACTICABILITY -- assess that can be assess that can be effected by available means or under current effected by available means or under current conditionsconditions

CLARITYCLARITY -- that the items in and methodology that the items in and methodology of the scheme can be easily and of the scheme can be easily and unequivocally understood by usersunequivocally understood by users

2.2. To assess its To assess its acceptability acceptability as a valid as a valid method to monitor learning progress in method to monitor learning progress in residency trainingresidency training

3.3. To determine what To determine what resourcesresources such as such as finance or staff are needed for the actual finance or staff are needed for the actual implementationimplementation

4.4. To detect To detect local problemslocal problems that might affect that might affect the actual processthe actual process

SCOPE SCOPE ……..

The pilot study willThe pilot study willinclude include

a trial run of the a trial run of the scheme scheme collection of data collection of data reporting of results reporting of results analysis of outcomeanalysis of outcome

From the information From the information gathered, gathered,

formulate formulate recommendations recommendations submit these to the submit these to the Board of Trustees. Board of Trustees.

……. & LIMITATIONS. & LIMITATIONS

The amendment of the scheme pursuant The amendment of the scheme pursuant to the recommendations and subsequent to the recommendations and subsequent procedures are beyond the scope of the procedures are beyond the scope of the pilot study.pilot study.

The validity and reliability of the evaluation The validity and reliability of the evaluation scheme should be realized in the actual scheme should be realized in the actual implementation phase. implementation phase.

the units of evaluation are periods the units of evaluation are periods covering a month, an entire year level and covering a month, an entire year level and over the entire training period of four over the entire training period of four years. years.

long pilot phase could make the evaluation long pilot phase could make the evaluation scheme unfamiliar, and possibly scheme unfamiliar, and possibly uninteresting, to later BOTs and PBOGs uninteresting, to later BOTs and PBOGs ––if not inappropriate, for much later time.if not inappropriate, for much later time.

METHODOLOGY METHODOLOGY

selected institutions from 4 different selected institutions from 4 different categories and geographic locationscategories and geographic locationsHave them use the RES forms to evaluate Have them use the RES forms to evaluate their residents starting July 2005their residents starting July 2005

to note positive as well as negative to note positive as well as negative experiences and any problems associated experiences and any problems associated with the usage.with the usage.

IMPLEMENTING PARTNERSIMPLEMENTING PARTNERS

Cagayan Valley Cagayan Valley Medical CenterMedical CenterWest Visayas State West Visayas State UniversityUniversityDavao Medical Davao Medical CenterCenterPhilippine General Philippine General HospitalHospital

Batangas Regional Batangas Regional HospitalHospitalVicente Sotto Vicente Sotto Memorial Medical Memorial Medical CenterCenterNorthern Mindanao Northern Mindanao Medical CenterMedical CenterJose Fabella Medical Jose Fabella Medical CenterCenter

IMPLEMENTING PARTNERSIMPLEMENTING PARTNERS

DLSU Medical DLSU Medical CenterCenterCebu DoctorsCebu Doctors’’HospitalHospitalUSTUSTMedical City (backMedical City (back--up)up)

LocationLocationCategoryCategory

LUZONLUZON VISAYASVISAYAS MINDANAOMINDANAO NCRNCR

GovernmentGovernment: academic: academic

Cagayan Cagayan Valley Valley Medical Medical CenterCenter

West West VisayasVisayasState State UniversityUniversity

Davao Davao Medical Medical CenterCenter

Philippine Philippine General HospitalGeneral Hospital

Private: Private: academicacademic

DLSU DLSU HospitalHospital

Cebu Cebu DoctorsDoctors

GovernmentGovernment: non: non--academicacademic

Vicente Vicente SottoSotto

Northern Northern MindanaoMindanao

Fabella Memorial Fabella Memorial HospitalHospital

Private: Private: nonnon--academicacademic

The MedicalThe MedicalCityCity

ACTIVITIESACTIVITIES

June 11, June 11, 20052005

Orientation held during Annual Orientation held during Annual TrainorsTrainors’’ Meeting in United LabMeeting in United LabImmediate echo to DepartmentsImmediate echo to Departments

JulyJuly Start of implementation in pilot sitesStart of implementation in pilot sites

OctoberOctober Survey among evaluators and Survey among evaluators and residents using questionnaireresidents using questionnaire

NovemberNovember Responses shared and discussed Responses shared and discussed among institutions during Annual among institutions during Annual ConventionConvention

October 26,October 26,December 12,December 12,20052005February 1, February 1, 20062006

Review of responses, statistical Review of responses, statistical analysisanalysisAnalysis of resultsAnalysis of resultsFormulation of recommendationsFormulation of recommendations

SURVEY QUESTIONNAIRESURVEY QUESTIONNAIRE

Two sets of survey questionnaires were Two sets of survey questionnaires were made: made:

Survey Form for Evaluators (SFE) Survey Form for Evaluators (SFE) Survey Form for Residents (SFRSurvey Form for Residents (SFR

SFESFE

Please indicate your Please indicate your agreement/disagreement with the agreement/disagreement with the statements on the first column by statements on the first column by checking the appropriate boxchecking the appropriate box

1.1.The scheme covers all areas of The scheme covers all areas of OBOB--Gyn training (knowledge, skills, Gyn training (knowledge, skills, attitude)attitude)2.1 The items in the survey forms 2.1 The items in the survey forms are clearly described and are easy to are clearly described and are easy to understand.understand.2.22.2 The methodology on the survey The methodology on the survey forms is easy to followforms is easy to follow3.1 The frequency of the evaluation 3.1 The frequency of the evaluation is adequateis adequate3.23.2 The number of evaluators is The number of evaluators is sufficientsufficient

Checks that Checks that -- the RES is the RES is comprehensivecomprehensive and meets and meets its objective of its objective of emphasizing the emphasizing the competencies competencies ((appropriateness)appropriateness)-- terms and procedures terms and procedures involved will be understood involved will be understood uniformly (pilot study uniformly (pilot study objective 1objective 1--c. c. clarity clarity of of scheme)scheme)-- fairnessfairness is enhanced with is enhanced with the multiplicity of the multiplicity of evaluations and evaluators evaluations and evaluators

Please answer the following Please answer the following questions:questions:

4. If you do not agree with statement 4. If you do not agree with statement 3.1, how many evaluations do you 3.1, how many evaluations do you think is adequate?think is adequate?

5. If you do not agree with statement 5. If you do not agree with statement 3.2, how many evaluators do you think 3.2, how many evaluators do you think is sufficient ?is sufficient ?

6. In your opinion, who is in the best 6. In your opinion, who is in the best position to evaluate the residents with position to evaluate the residents with regards performance? regards performance?

--How the local setHow the local set--up can up can cope with the demands of cope with the demands of the RES (objective 1athe RES (objective 1a--b, b, feasibility and feasibility and practicability).practicability).--Countercheck reply for Countercheck reply for items 3.1items 3.1--2.2.Should reflect what they Should reflect what they are comfortable with or are comfortable with or amenable to.amenable to.-- countercheck reply to countercheck reply to item 1; item 1; -- acceptability as valid toolacceptability as valid toolfor monitoring residents for monitoring residents progress (pilot study progress (pilot study objective #2)objective #2)-- anticipate resourcesanticipate resourcesneeded in actual needed in actual implementation; probe for implementation; probe for difficulties/deficiencies, difficulties/deficiencies, possible drawbackspossible drawbacks in in actual implementation actual implementation (pilot study objectives 3(pilot study objectives 3--4)4)


7. Which aspects of resident7. Which aspects of resident’’s s performance in training are not performance in training are not covered in the RES?covered in the RES?

8. How well does the scheme reflect 8. How well does the scheme reflect attainment of PBOG objectives for attainment of PBOG objectives for residency trainingresidency training

9. what problem/s did you encounter 9. what problem/s did you encounter in using/administering the evaluation in using/administering the evaluation tools of the scheme.tools of the scheme.

10. Other comments and suggestions10. Other comments and suggestions

--

SFRSFR


A.A.How does the new scheme differ How does the new scheme differ from the one used in your from the one used in your institution?institution?B.B.How have you been assessed in How have you been assessed in the last 3 months on cognitive skills, the last 3 months on cognitive skills, technical skills and attitude? Mention technical skills and attitude? Mention method and frequency.method and frequency.C.C.How did you rate in the 3 domains?How did you rate in the 3 domains?D.D.How has new scheme affected you How has new scheme affected you as a resident?as a resident?E.E.Did you have access to the results Did you have access to the results of the assessment?of the assessment?

--Items A&B check if RES Items A&B check if RES was was implemented as implemented as proposedproposed

--Items C &D see if Items C &D see if residents get feedback on residents get feedback on their performance (RES their performance (RES applications applications –– formative formative and summative)and summative)

--Items EItems E--I. How do I. How do residents viewresidents view the the proposed scheme? Probe proposed scheme? Probe for for difficulties/deficiencies, difficulties/deficiencies, possible drawbackspossible drawbacks in in actual implementation actual implementation (Pilot study objectives #2 (Pilot study objectives #2 & 4)& 4)

SFRSFR

F. To what extent does the new F. To what extent does the new scheme approximate your scheme approximate your expectation about assessment in expectation about assessment in training?training?

G. Where does the scheme fall short of G. Where does the scheme fall short of your expectations?your expectations?

H. Which has made better assessment H. Which has made better assessment of your performance, the new or of your performance, the new or the old; Why?the old; Why?

I. How have you used the results of I. How have you used the results of the new scheme so far?the new scheme so far?

J. How has your department made J. How has your department made use of the results?use of the results?

K. Other comments and suggestionsK. Other comments and suggestions

DATA COLLECTION & ENCODINGDATA COLLECTION & ENCODING

The questionnaires were sent to the participating The questionnaires were sent to the participating institutions during the ininstitutions during the in--service examinations. Each service examinations. Each questionnaire was accompanied by a cover letter questionnaire was accompanied by a cover letter explaining the purpose of the survey. explaining the purpose of the survey. Accomplished forms were sent back to the POGS Accomplished forms were sent back to the POGS secretariat. (All of these were bound and submitted as secretariat. (All of these were bound and submitted as attachment to this report). attachment to this report). Each SFE or SFR was assigned an identifying number. Each SFE or SFR was assigned an identifying number. Responses were classified to allow encoding and Responses were classified to allow encoding and frequency count. A biostatistician counterchecked the frequency count. A biostatistician counterchecked the result using SPSS v.10result using SPSS v.10Because of the ID number, it was possible to trace Because of the ID number, it was possible to trace discordant responses or frequencies and to correct discordant responses or frequencies and to correct accordingly. accordingly.

RESULTS RESULTS

The pilot study was not carried out at the UST The pilot study was not carried out at the UST Hospital as the period coincided with Hospital as the period coincided with organizational changes at the institution. The organizational changes at the institution. The Batangas Regional Hospital dropped out due to Batangas Regional Hospital dropped out due to lack of consultants.lack of consultants.

SFESFE

53 respondents to the survey forms for 53 respondents to the survey forms for evaluators (SFE): evaluators (SFE): 7 residency training officers, 43 consultants, 2 7 residency training officers, 43 consultants, 2 department chairs, and one who listed as department chairs, and one who listed as member of a residency training committee. member of a residency training committee. Among them 28 were from government hospitals Among them 28 were from government hospitals and 25 from private hospitals; 26 were from and 25 from private hospitals; 26 were from teaching hospitals and 27 from nonteaching hospitals and 27 from non--teaching teaching ones. ones. Sixteen came from 3 hospitals in NCR, 9 from 2 Sixteen came from 3 hospitals in NCR, 9 from 2 other hospitals in Luzon, 20 from 2 hospitals in other hospitals in Luzon, 20 from 2 hospitals in the Visayas and 8 from 2 hospitals in Mindanao. the Visayas and 8 from 2 hospitals in Mindanao.

SCHEME COVERS ALL AREAS OF OBSCHEME COVERS ALL AREAS OF OB--GYN GYN TRAININGTRAINING

q1 q1 | Number Percent Cum.| Number Percent Cum.------------------------++--------------------------------------------------------------------------------------------S. disagreeS. disagree | 1 | 1 1.89 1.89 1.891.89

AgreeAgree | 37 | 37 69.81 71.7069.81 71.70S. agreeS. agree | 15 | 15 28.30 28.30 100.00100.00

------------------------++--------------------------------------------------------------------------------------------Total Total | 53 | 53 100.00100.00

ITEMS ARE CLEARLY DESCRIBED/EASY TO ITEMS ARE CLEARLY DESCRIBED/EASY TO UNDERSTANDUNDERSTAND

q2_1 | Number Percent Cum.q2_1 | Number Percent Cum.------------------------++----------------------------------------------------------------------------------------------S. disagree | 1 S. disagree | 1 1.89 1.89 1.891.89

Disagree | 9 Disagree | 9 16.98 18.8716.98 18.87Neither | 5 Neither | 5 9.43 28.309.43 28.30Agree | 32 Agree | 32 60.38 88.6860.38 88.68

S. agree | 6 S. agree | 6 11.32 100.0011.32 100.00------------------------++----------------------------------------------------------------------------------------------

Total | 53 100.00Total | 53 100.00

ADEQUACY IN THE FREQUENCY OF ADEQUACY IN THE FREQUENCY OF EVALUATIONEVALUATION

q3_1 | Freq. Percent Cum.q3_1 | Freq. Percent Cum.------------------------++----------------------------------------------------------------------S. disagree | 3 5.66 S. disagree | 3 5.66 5.665.66


S. agree | 10 18.87 100.00S. agree | 10 18.87 100.00------------------------++----------------------------------------------------------------------

Total | 53 100.00Total | 53 100.00-->>

NUMBER OF EVALUATION DONENUMBER OF EVALUATION DONEq4 | Freq. Percent Cum.q4 | Freq. Percent Cum.

------------------------++----------------------------------------------------------------------As proposed | 28 52.83 As proposed | 28 52.83 52.8352.83Quarterly | 19 35.85 88.68Quarterly | 19 35.85 88.68

Every 6 mos. | 4 7.55 96.23Every 6 mos. | 4 7.55 96.23Cannot tell 2 3.77 100.00Cannot tell 2 3.77 100.00------------------------++----------------------------------------------------------------------

Total | 53 100.00Total | 53 100.00

IF NUMBER OF EVALUATORS IS ALREADY IF NUMBER OF EVALUATORS IS ALREADY SUFFICIENTSUFFICIENT

q3_2 | Freq. Percent Cum.q3_2 | Freq. Percent Cum.------------------------++----------------------------------------------------------------------S. disagree | 1 1.89 S. disagree | 1 1.89 1.891.89


S. agree | 9 16.98 100.00S. agree | 9 16.98 100.00------------------------++----------------------------------------------------------------------

Total | 53 100.00Total | 53 100.00

ASPECTS IN RESIDENTASPECTS IN RESIDENT’’S PERFORMANCE IN S PERFORMANCE IN TRAINING TRAINING NOTNOT COVERED BY EVALUATION COVERED BY EVALUATION SCHEMESCHEME

. q7 | Freq. Percent Cum.. q7 | Freq. Percent Cum.------------------------++----------------------------------------------------------------------

0 | 51 96.23 0 | 51 96.23 96.2396.23Response | 1 1.89 98.11Response | 1 1.89 98.11

No No respresp. | 1 1.89 100.00. | 1 1.89 100.00------------------------++----------------------------------------------------------------------

Total | 53 100.00Total | 53 100.00

Not covered by the scheme: Not covered by the scheme: teachingteachingentrepreneurship and salesmanship, entrepreneurship and salesmanship, attendance and punctuality attendance and punctuality extraextra--curricular activities curricular activities subspecialty areas subspecialty areas inin--service PBOGservice PBOG--administered administered

examinationexamination

ATTAINMENT OF PBOG OBJECTIVES IN ATTAINMENT OF PBOG OBJECTIVES IN RESIDENCY TRAININGRESIDENCY TRAINING


Well | 29 54.72 Well | 29 54.72 54.7254.72Falls short I 1 1.89 56.60Falls short I 1 1.89 56.60Cannot tell | 2 3.77 60.38Cannot tell | 2 3.77 60.38No response | 17 32.08 92.45No response | 17 32.08 92.45Impertinent | 4 7.55 100.00Impertinent | 4 7.55 100.00------------------------++----------------------------------------------------------------------

Total | 53 100.00Total | 53 100.00

Contrary comments Contrary comments the process being tediousthe process being tedioussubjective bias in evaluationsubjective bias in evaluationheavy weight on subspecialty areas in heavy weight on subspecialty areas in

the inthe in--service service

PROBLEMS ENCOUNTERED IN PROBLEMS ENCOUNTERED IN USING/ADMINISTERING THE EVALUATION USING/ADMINISTERING THE EVALUATION TOOLS OF THE SCHEME TOOLS OF THE SCHEME

Process Process forms complex or (specific to forms complex or (specific to

patient satisfaction form) difficult patient satisfaction form) difficult to explain to patient to explain to patient

repetitious questions repetitious questions ratings scales having narrow range ratings scales having narrow range

PROBLEMSPROBLEMS

Evaluators Evaluators ––availabilityavailabilitylack of time to prepare quizzeslack of time to prepare quizzescannot monitor residents 24/7cannot monitor residents 24/7

PROBLEMSPROBLEMS

Institutional conditions Institutional conditions ––private hospital policies private hospital policies too many patients too many patients too many residents too many residents residentresident--patient ratio patient ratio residentresident--consultant ratioconsultant ratiodepartment SOP on trainingdepartment SOP on training

SPECIFIC MENTIONED:SPECIFIC MENTIONED:

It should be simplified, short and conciseIt should be simplified, short and conciseToo many forms to fill up, too taxing for busy Too many forms to fill up, too taxing for busy schedules, schedules, Lack of consultants to evaluate.Lack of consultants to evaluate.Patient evaluation form Patient evaluation form –– the private hospital set the private hospital set up limits encounters which would allow residents up limits encounters which would allow residents to to ““present all optionspresent all options”” and therapeutic and therapeutic modalities or even difficulties diagnosis to modalities or even difficulties diagnosis to patients.patients.Time constraints in preparing exam and quizzesTime constraints in preparing exam and quizzesJust time management, hard to gather residents Just time management, hard to gather residents when so many on going cases.when so many on going cases.

Lack of residents based on the number of Lack of residents based on the number of admissions per 24 hour duty, residents to patient admissions per 24 hour duty, residents to patient ratioratioThere are occasions that contact is more with There are occasions that contact is more with the senior residents so evaluation is the senior residents so evaluation is compromised with regards to junior residentcompromised with regards to junior residentMonthly evaluation is too much and time Monthly evaluation is too much and time consumingconsumingDifficult to explain to patientsDifficult to explain to patientsGetting it doneGetting it done

Clinical competence in behavioral terms: What is Clinical competence in behavioral terms: What is missing?missing?

Ability to communicate to the patient, his/her Ability to communicate to the patient, his/her assessment of the patientassessment of the patient’’s present & potential problems s present & potential problems and management option.and management option.

Other skills to be evaluated: Leadership, Teaching, Other skills to be evaluated: Leadership, Teaching, Entrepreneurship & salesmanshipEntrepreneurship & salesmanship

The above were listed but NO specific questions/evaluation The above were listed but NO specific questions/evaluation scheme was made. Suggest create outcome indicators & scheme was made. Suggest create outcome indicators & matching scoring system so that these can be matching scoring system so that these can be measured. Suggest also give us a copy of POGS measured. Suggest also give us a copy of POGS Objectives of Residency Training so we can see how it Objectives of Residency Training so we can see how it fits into it.fits into it.

Independent decision making may be used/applied for Independent decision making may be used/applied for clinical & administrative matters but a distinction needs clinical & administrative matters but a distinction needs to be made. Rationale: some are okay in clinical matters to be made. Rationale: some are okay in clinical matters but needs improvement in administrative/leadership but needs improvement in administrative/leadership roles we eventually affects patient care & interpersonal roles we eventually affects patient care & interpersonal relations to peers/superiors/subordinates.relations to peers/superiors/subordinates.Residents are not seen 24/7 some complaints of Residents are not seen 24/7 some complaints of inefficiencies or misbehavior cannot be graded inefficiencies or misbehavior cannot be graded adequately,adequately,Should also focus on code of conduct as in Should also focus on code of conduct as in govgov’’tt service service where interactions are categorized as minor or gross and where interactions are categorized as minor or gross and penalized accordingly. penalized accordingly.

Too complicated at first but on further analysis Too complicated at first but on further analysis the tool is reasonable considering the the tool is reasonable considering the comprehensiveness of parameters being asked comprehensiveness of parameters being asked to be evaluated.to be evaluated.Rating scale too narrow if 1Rating scale too narrow if 1--4. It would be better 4. It would be better if the scale could be broader for choice, 1if the scale could be broader for choice, 1--1010Questions in different areas seem to be Questions in different areas seem to be repetitive/redundant.repetitive/redundant.The evaluation skills will work best if a certain The evaluation skills will work best if a certain evaluator is assigned for residency training level evaluator is assigned for residency training level since these residents are limited to their define since these residents are limited to their define of knowledge & skills & clinical judgment.of knowledge & skills & clinical judgment.

SFRSFRYEAR LEVEL OF RESIDENT RESPONDENTS YEAR LEVEL OF RESIDENT RESPONDENTS

YR LEVEL | Freq. Percent Cum.YR LEVEL | Freq. Percent Cum.------------------------++----------------------------------------------------------------------

1 | 28 31.11 1 | 28 31.11 31.1131.112 | 22 24.44 55.562 | 22 24.44 55.563 | 23 25.56 81.113 | 23 25.56 81.114 | 17 18.89 100.004 | 17 18.89 100.00

------------------------++----------------------------------------------------------------------Total | 90 100.00Total | 90 100.00

LOCATION | Freq. Percent Cum.LOCATION | Freq. Percent Cum.------------------------++----------------------------------------------------------------------

Luzon | 20 22.22 Luzon | 20 22.22 22.2222.22Visayas | 15 16.67 38.89Visayas | 15 16.67 38.89Mindanao | 15 16.67 55.56Mindanao | 15 16.67 55.56

NCR | 40 44.44 100.00NCR | 40 44.44 100.00------------------------++----------------------------------------------------------------------

Total | 90 100.00Total | 90 100.00

OWNER OF INSTITUTION OWNER OF INSTITUTION | Freq. Percent Cum.| Freq. Percent Cum.

------------------------++----------------------------------------------------------------------Government | 53 58.89 Government | 53 58.89 58.8958.89

Private | 37 41.11 100.00Private | 37 41.11 100.00------------------------++----------------------------------------------------------------------

Total | 90 100.00Total | 90 100.00

CATEGORY OF INSTITUTIONCATEGORY OF INSTITUTION| Freq. Percent Cum.| Freq. Percent Cum.

------------------------++----------------------------------------------------------------------Academic | 46 51.11 Academic | 46 51.11 51.1151.11NonNon--acadacad | 44 48.89 100.00| 44 48.89 100.00

------------------------++----------------------------------------------------------------------Total | 90 100.00Total | 90 100.00

DIFFERENCE OF NEW SCHEME FROM THE USUAL DIFFERENCE OF NEW SCHEME FROM THE USUAL SCHEME USEDSCHEME USED

-- Based on descriptionBased on description

qAqA | Freq. Percent Cum.| Freq. Percent Cum.------------------------++----------------------------------------------------------------------New scheme | 46 51.11 New scheme | 46 51.11 51.1151.11Old scheme | 21 23.33 74.44Old scheme | 21 23.33 74.44Cannot tell | 8 8.89 83.33Cannot tell | 8 8.89 83.33No response | 13 14.44 97.78No response | 13 14.44 97.78Unrelated | 2 2.22 100.00Unrelated | 2 2.22 100.00------------------------++----------------------------------------------------------------------

Total | 90 100.00Total | 90 100.00

Only 56.67% said they had access to the Only 56.67% said they had access to the results of such evaluation using the new results of such evaluation using the new scheme and 65.66% shared what their rating scheme and 65.66% shared what their rating was.was.

ACCESS OR NO IDEAACCESS OR NO IDEAqCqC | Freq. Percent Cum.| Freq. Percent Cum.

------------------------++----------------------------------------------------------------------Yes | 51 56.67 Yes | 51 56.67 56.6756.67

No idea | 30 33.33 90.00No idea | 30 33.33 90.00Cannot tell | 9 10.00 100.00Cannot tell | 9 10.00 100.00------------------------++----------------------------------------------------------------------

Total | 90 100.00Total | 90 100.00

RATING IN THE THREE DOMAINSRATING IN THE THREE DOMAINSqDqD Freq. Percent Cum.Freq. Percent Cum.

------------------------++----------------------------------------------------------------------Gives rate | 59 65.56 Gives rate | 59 65.56 65.5665.56

No idea | 16 17.78 83.33No idea | 16 17.78 83.33Cannot tell | 9 10.00 93.33Cannot tell | 9 10.00 93.33No response | 5 5.56 98.89No response | 5 5.56 98.89Impertinent | 1 1.11 100.00Impertinent | 1 1.11 100.00------------------------++----------------------------------------------------------------------

Total | 90 100.00Total | 90 100.00

HOW MUCH NEW SCHEME HOW MUCH NEW SCHEME APPROXIMATES EXPECTATION ABOUT APPROXIMATES EXPECTATION ABOUT ASSESSMENT IN TRAININGASSESSMENT IN TRAINING

Only 26 (28.89%) respondents gave a Only 26 (28.89%) respondents gave a quantitative answer to the question quantitative answer to the question ““To To what extent does the scheme approximate what extent does the scheme approximate your expectation about assessment of your expectation about assessment of your performance?your performance?”” Of these, only 10% Of these, only 10% said the new scheme was up to their said the new scheme was up to their expectation of an evaluation process. expectation of an evaluation process.

““To what extent does the scheme approximate your To what extent does the scheme approximate your expectation about assessment of your expectation about assessment of your performance?performance?””

qFqF I Freq. Percent Cum.I Freq. Percent Cum.------------------------++----------------------------------------------------------------------Approx. Approx. ExpectExpect’’nn | 9 10.00 | 9 10.00 10.0010.00Falls short Falls short | 8 8.89 18.89| 8 8.89 18.89Not at all Not at all | 9 10.00 28.89| 9 10.00 28.89Cannot tell Cannot tell | 11 12.22 41.11| 11 12.22 41.11No responseNo response | 23 25.56 66.67| 23 25.56 66.67Impertinent Impertinent | 30 33.33 100.00| 30 33.33 100.00------------------------++----------------------------------------------------------------------TotalTotal | 90 100.00| 90 100.00

EXTENT BY WHICH NEW SCHEME FALLS EXTENT BY WHICH NEW SCHEME FALLS SHORT OF EXPECTATIONSHORT OF EXPECTATION

According to 21 (27.6%), there is no area According to 21 (27.6%), there is no area where the scheme falls short of their where the scheme falls short of their expectation. Twentyexpectation. Twenty--four (31.5%) did not four (31.5%) did not write any response; one (1.3%) write any response; one (1.3%) ““could not could not say as of nowsay as of now””. The rest commented off. The rest commented off--tangent.tangent.

IN WHAT ASPECT THE SCHEME FALL SHORT IN WHAT ASPECT THE SCHEME FALL SHORT OF EXPECTATION?OF EXPECTATION?

Subjective/we cannot eliminate biasesSubjective/we cannot eliminate biasesIt can make or break you means that one It can make or break you means that one mistake will affect the rest of your good deedsmistake will affect the rest of your good deedsPeer evaluation is more credible than that of the Peer evaluation is more credible than that of the consultant since 80% of the time we work is with consultant since 80% of the time we work is with fellow residentsfellow residentsIt cannot see everything that you do as a It cannot see everything that you do as a residentresidentHard to accomplish due to time constraintHard to accomplish due to time constraint

Time consuming/too long/too tediousTime consuming/too long/too tediousToo frequent quizzes/sometimes unrealisticToo frequent quizzes/sometimes unrealisticThe very busy schedule of our consultant makes The very busy schedule of our consultant makes it difficult for us to comply with the number of it difficult for us to comply with the number of consultantconsultant--assisted activitiesassisted activitiesFor number of residents we have, it is difficult to For number of residents we have, it is difficult to reach quotas of consultantreach quotas of consultant--assisted proceduresassisted proceduresSome of the tools not valid such as patientSome of the tools not valid such as patient’’s s evaluation in Englishevaluation in English

Scoring system is too narrow, should have wider rangeScoring system is too narrow, should have wider rangeToo structuredToo structuredLimited to 4Limited to 4--point system there are no in betweenspoint system there are no in betweensProcedures learned during 1st year SVD, curettage, CS Procedures learned during 1st year SVD, curettage, CS is carried over and is added to the required procedure is carried over and is added to the required procedure during the next levelduring the next levelCannot say where as of nowCannot say where as of nowDifficult to accomplish because of lack of timeDifficult to accomplish because of lack of timeToo frequent quizzes/sometimes unrealisticToo frequent quizzes/sometimes unrealisticThe very busy schedule of our consultant makes it The very busy schedule of our consultant makes it difficult for us to comply with the number of consultantdifficult for us to comply with the number of consultant--assisted activitiesassisted activities

USE OF RESULTS BY THE HOSPITALUSE OF RESULTS BY THE HOSPITAL’’S OBS OB--GYN DEPTGYN DEPTto focus the training program in one way or other. to focus the training program in one way or other.

qJqJ | Freq. Percent Cum.| Freq. Percent Cum.------------------------++----------------------------------------------------------------------Positively | 33 36.67 Positively | 33 36.67 36.6736.67Not at all | 6 6.67 43.33Not at all | 6 6.67 43.33Cannot tell | 31 34.44 77.78Cannot tell | 31 34.44 77.78No response | 19 21.11 98.89No response | 19 21.11 98.89Impertinent | 1 1.11 100.00Impertinent | 1 1.11 100.00------------------------++----------------------------------------------------------------------

Total | 90 100.00Total | 90 100.00

COMMENTS AND SUGGESTIONSCOMMENTS AND SUGGESTIONS

While many comments were not pertinent While many comments were not pertinent to the questions, some of them were to the questions, some of them were nonetheless significant as they gave nonetheless significant as they gave insight to how programs are run insight to how programs are run

““The very busy schedule of our consultants make it The very busy schedule of our consultants make it difficult for us to comply with the number of consultantdifficult for us to comply with the number of consultant--assisted activitiesassisted activities””. .

Other comments and suggestionsOther comments and suggestions

How about an evaluation tool for consultants to be done How about an evaluation tool for consultants to be done by the residents so that the consultants will also know by the residents so that the consultants will also know their deficiencies.their deficiencies.Can we not make an evaluation form which is shorter but Can we not make an evaluation form which is shorter but still covers the 3 domains of cognitive, technical & still covers the 3 domains of cognitive, technical & attitude.attitude.Monthly quizzes may be too tedious and may not be Monthly quizzes may be too tedious and may not be given attention to, carefully by both evaluator and given attention to, carefully by both evaluator and resident.resident.Presently WVSUMC residents are given 3 long exams:Presently WVSUMC residents are given 3 long exams:InIn--service exam given by the PBOG and 2 long exams service exam given by the PBOG and 2 long exams given every six months to all residents in all hospitals in given every six months to all residents in all hospitals in Iloilo City. I feel that there is more than enough.Iloilo City. I feel that there is more than enough.For number of evaluators, I think evaluation of the For number of evaluators, I think evaluation of the training officer and service consultant will suffice.training officer and service consultant will suffice.

Giving of monthly exam is over kill. Quarterly exam and InGiving of monthly exam is over kill. Quarterly exam and In--service exam are enough.service exam are enough.SemiSemi--annual exam is okayannual exam is okayExclude points in evaluation & replace hospital personnel e.g. Exclude points in evaluation & replace hospital personnel e.g. nursesnurses

As per your plan As per your plan ““ The POGS will help in the developing of the The POGS will help in the developing of the skill of the Chairs & T.O. skill of the Chairs & T.O. ““ How & when will you do this? What How & when will you do this? What particular skills can we count on to help develop?particular skills can we count on to help develop?Suggest: Training of Suggest: Training of TrainorsTrainors –– creative ways of creative ways of training/educating using techniques in adult education that training/educating using techniques in adult education that work to be effective in training/educating residents in OBwork to be effective in training/educating residents in OB--GYNEGYNEDevelop ways by which POGS training to Chairs & T.O. can be Develop ways by which POGS training to Chairs & T.O. can be reached to the rest of the staff who share in the responsibilityreached to the rest of the staff who share in the responsibilityto train/educate the future generation of OBto train/educate the future generation of OB--GYNES. Teaching GYNES. Teaching is a skill that can be learned and taught and there are is a skill that can be learned and taught and there are institutions that teach people how to teach (e.g. UP, NTTC, institutions that teach people how to teach (e.g. UP, NTTC, etc.)Itetc.)It might help us improve our teaching skills if there is a might help us improve our teaching skills if there is a conscious program directed towards that.conscious program directed towards that.

In describing 1st to 4th year competencies In describing 1st to 4th year competencies suggest use the phrase suggest use the phrase ““Demonstrates skills in Demonstrates skills in the performance of or able to perform ___ the performance of or able to perform ___ instead of knowledge of __ because knowledge instead of knowledge of __ because knowledge may be in theory performance is in actual may be in theory performance is in actual practice.practice.

Research Research –– should be graded monthly or should be graded monthly or periodically so residents would not cram come periodically so residents would not cram come deadline date. Chart should be followed. E.g. by deadline date. Chart should be followed. E.g. by January January –– 10% of the data collected and this is 10% of the data collected and this is graded then by March 30 % of data collected graded then by March 30 % of data collected and so onand so on……

More weight should be given for the knowledge More weight should be given for the knowledge part because this is an objective assessment. part because this is an objective assessment. The others (clinical, technical skills & attitude), The others (clinical, technical skills & attitude), no matter how standardize dour evaluation no matter how standardize dour evaluation forms should be, there would always be a forms should be, there would always be a ““subjectivesubjective”” influence on the part of the influence on the part of the evaluator.evaluator.

I suggest that the inI suggest that the in--service exam be included service exam be included in the evaluation of the residents. Many in the evaluation of the residents. Many residents donresidents don’’t prepare for this since they know t prepare for this since they know it will not be included in their evaluation.it will not be included in their evaluation.

ANALYSIS OF RESULTSANALYSIS OF RESULTS

The proposed scheme meets the The proposed scheme meets the more salient requirements of an more salient requirements of an evaluation process.evaluation process.

The methodology and tools assess the The methodology and tools assess the content of the core curriculum in the domains content of the core curriculum in the domains of knowledge, skills and attitude, this towards of knowledge, skills and attitude, this towards the fulfillment of the objectives of the training the fulfillment of the objectives of the training program. program.

..

The component items are clearly described, The component items are clearly described, are easy to understand and to follow. An are easy to understand and to follow. An exception is the tool on patient satisfaction. It exception is the tool on patient satisfaction. It is in English and needs translation into the is in English and needs translation into the dialects for use in the regions. dialects for use in the regions.

The main resistance to acceptance of The main resistance to acceptance of the new scheme stems from the the new scheme stems from the frequency of evaluation that it frequency of evaluation that it requires. requires.

It appears increased to those whose It appears increased to those whose programs have not evaluated their residents programs have not evaluated their residents as frequently, and therefore threatens as frequently, and therefore threatens increased workload. From this standpoint, the increased workload. From this standpoint, the scheme is viewed as being unduly complex scheme is viewed as being unduly complex and tedious. and tedious.

??respondents know that educational principles respondents know that educational principles underlie the multiplicity of evaluation tools and underlie the multiplicity of evaluation tools and frequency of evaluation.frequency of evaluation.

Many residents took to the detailed Many residents took to the detailed scheme and more frequent evaluation scheme and more frequent evaluation positively, welcoming these as increased positively, welcoming these as increased opportunity for specific feedbacks and as opportunity for specific feedbacks and as very specific approach to their strengths very specific approach to their strengths and weakness. and weakness.

RECOMMENDATIONSRECOMMENDATIONS

The ad hoc committee for the pilot study of The ad hoc committee for the pilot study of the RES recommended the following to the RES recommended the following to the Board of Trustees:the Board of Trustees:

1.1. Adoption of the proposed residency Adoption of the proposed residency evaluation scheme as the standard in evaluation scheme as the standard in accredited institutions, with modifications:accredited institutions, with modifications:

Translation of the patient satisfaction formTranslation of the patient satisfaction formLess frequent evaluation for technical skills, Less frequent evaluation for technical skills,

researchresearchUtilization of inUtilization of in--service exam results in the evaluation ofservice exam results in the evaluation of

the knowledge domainthe knowledge domain

2. Orientation of evaluators and residents 2. Orientation of evaluators and residents through:through:

SeminarSeminar-- workshop on the residentsworkshop on the residents’’evaluation schemeevaluation schemeManual to accompany RES kitManual to accompany RES kit

3. Periodic assessment of the Residents3. Periodic assessment of the Residents’’Evaluation Scheme once fully implementedEvaluation Scheme once fully implemented

4. Establishment of a residency training committee 4. Establishment of a residency training committee as a standing committee in the POGS to as a standing committee in the POGS to oversee training programs.oversee training programs.

REACTIONS, REACTIONSREACTIONS, REACTIONS

WHY ASSESSWHY ASSESS

to help a student learn (FORMATIVE) to help a student learn (FORMATIVE) InIn--course, not at the endcourse, not at the endIncludes Includes detaileddetailed feedback on learnerfeedback on learner’’s s strengths and weaknesses strengths and weaknesses –– what area, what area, (rubrics)(rubrics)

to assess competence (SUMMATIVE) to assess competence (SUMMATIVE) End of course/level End of course/level Readiness for next levelReadiness for next level

ASSESSMENT DRIVES LEARNING.ASSESSMENT DRIVES LEARNING.

It can be used to the instructorIt can be used to the instructor’’s advantage s advantage to ensure that students direct their learning to ensure that students direct their learning activities to the programactivities to the program’’s objectivess objectives

HOW TO ASSESSHOW TO ASSESS

Assessment should be based on Assessment should be based on performance standardsperformance standards where students where students can measure their accomplishments can measure their accomplishments based upon based upon established competencies.established competencies.

•• By so doing, instructors can better guide a studentBy so doing, instructors can better guide a student’’s learning based s learning based upon weaknesses identified in particular areas, and the student upon weaknesses identified in particular areas, and the student can can monitor his or her own progress in an efficient, effective mannemonitor his or her own progress in an efficient, effective manner. r.

WHAT TO ASSESSWHAT TO ASSESS

Axiom: Knowledge and performance are not Axiom: Knowledge and performance are not separable.separable.

IT TAKES KNOWLEDGE TO PERFORM.IT TAKES KNOWLEDGE TO PERFORM.(You cannot do it, if you do not know how (You cannot do it, if you do not know how to do it)to do it)

Performance measures are a supplement Performance measures are a supplement to knowledge measuresto knowledge measures

TaxonomyTaxonomy

Knowledge (Recall)

Comprehension (Describe)

Application

Analysis

Synthesis

Evaluate

Axiom: There are no general cognitive, Axiom: There are no general cognitive, affective or psychomotor skills.affective or psychomotor skills.

Performance on one problem tells very little Performance on one problem tells very little on how student will perform on another on how student will perform on another problem.problem.

Hence for reliable assessment, do multiple Hence for reliable assessment, do multiple samplingsampling

WHY MULTIPLE SAMPLINGWHY MULTIPLE SAMPLING

Reduce effect of biasReduce effect of biasAdjust for circumstancesAdjust for circumstancesConsultants not available 24/7Consultants not available 24/7

NO to global ratingNO to global rating

EndEnd--ofof--course ratingcourse ratingLow interLow inter--rater reliabilityrater reliabilityRater is required to integrate episodes Rater is required to integrate episodes over several weeks/months over several weeks/months –– impossible impossible memory loadmemory loadPersonality rather than performance ratingPersonality rather than performance ratingSubjective, difficult to defend Subjective, difficult to defend avoid low avoid low ratings for fear of being asked to defend ratings for fear of being asked to defend judgmentsjudgments

UPDATE ON METHODUPDATE ON METHOD

CLINICAL WORK SAMPLINGCLINICAL WORK SAMPLINGA.k.a. mini clinical exams (miniA.k.a. mini clinical exams (mini--CEX)CEX)Use many encounters, scheduled or occurring Use many encounters, scheduled or occurring in the normal course of training, as evaluation in the normal course of training, as evaluation eventseventsReliability of 0.7Reliability of 0.7--0.8 with 80.8 with 8--12 encounters12 encounters

TURNBULL ET AL 2000TURNBULL ET AL 2000NORCINI ET AL 2003NORCINI ET AL 2003

RATING SCALE RATING SCALE

Suggested equivalentSuggested equivalent1 1 –– unsatisfactory unsatisfactory –– 70702 2 -- needs improvement needs improvement –– 75753 3 –– satisfactory satisfactory –– 80804 4 –– very good very good –– 8585

No middle ground!No middle ground!Wider range not advantageousWider range not advantageous

SummarySummary

RESRES

Formulation of schemeFormulation of schemePilot study Pilot study Underlying educational principlesUnderlying educational principles

THANK YOU

RESIDENTS’ EVALUATION SCHEME - pogsinc.orgpogsinc.org/files/annual_trainors/Residence...

Documents

Transcript of RESIDENTS’ EVALUATION SCHEME - pogsinc.orgpogsinc.org/files/annual_trainors/Residence...