The Assessment of School Psychologists in Practice Through Multisource Feedback.PDF

http://cjs.sagepub.com/School PsychologyCanadian Journal of

http://cjs.sagepub.com/content/25/4/328The online version of this article can be found at:

DOI: 10.1177/0829573510373585June 2010

2010 25: 328 originally published online 15Canadian Journal of School PsychologyJac J. W. Andrews and Claudio Violato

FeedbackThe Assessment of School Psychologists in Practice Through Multisource

Published by:

http://www.sagepublications.com

On behalf of:

Canadian Association of School Psychologists

can be found at:Canadian Journal of School PsychologyAdditional services and information for

http://cjs.sagepub.com/cgi/alertsEmail Alerts:

http://cjs.sagepub.com/subscriptionsSubscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

http://cjs.sagepub.com/content/25/4/328.refs.htmlCitations:

by Widiastuti Pajarini on April 29, 2012cjs.sagepub.comDownloaded from

What is This?

- Jun 15, 2010 OnlineFirst Version of Record

- Nov 23, 2010Version of Record >>


Canadian Journal of School Psychology25(4) 328 346

2010 SAGE PublicationsReprints and permission: http://www. sagepub.com/journalsPermissions.nav

DOI: 10.1177/0829573510373585http://cjs.sagepub.com

The Assessment of School Psychologists in Practice Through Multisource Feedback

Jac J. W. Andrews1 and Claudio Violato1

Abstract

In this article we provide an overview of the nature and scope of multisource feedback (MSF) and provide empirical evidence of its reliability, validity, and feasibility in one of the health professions. The overall internal consistency reliability (Cronbach alpha) of MSF instruments is generally greater than .96 for self and informants such as patients, coworker, and colleague surveys. Generalizability coefficients for the assessors across persons are approximately 0.80. There is also substantial evidence of content, criterion-related and some evidence of construct validity of the MSF instruments applied in the health professions. Based on these findings, we recommend the development and use of a MSF system for practicing school psychologists, present information about how MSF instruments can be constructed, and provide examples of what these instruments could look like.

Rsum

Cet article vous offre un aperu de la nature et de la porte des multisource feedback (MSF) et vous fournit des preuves empiriques de sa fiabilit, de sa validit et la faisabilit dans une des professions de la sant. La fiabilit de la cohrence interne globale (Cronbach alpha) des outils de MSF est gnralement plus grande que, 96 pour des sondages effectus sur soi-mme, sur des individus tels que des patients, des collgues de travail ou collaborateurs. Les coefficients de gnralisibilit pour les valuateurs travers les personnes sont denviron 0,80. Il y a aussi des preuves substantielles du contenu, les critres lis et certains lments de preuve de la validit des outils MSF appliqus dans les professions de la sant. Sur la base de ces rsultats, nous recommandons llaboration et lutilisation dun systme de MSF pour la pratique de la psychologie en milieu scolaire; prsenter des informations sur la faon dont les outils MSF peuvent tre dvelopps et fournir des exemples de ce que ces outils peuvent tre.

1University of Calgary, Calgary, Alberta

Corresponding Author:Jac J. W. Andrews, Division of Applied Psychology, University of Calgary, 500 University Drive N. W., Calgary, AB T2N1N4, CanadaEmail: [email protected]


Andrews and Violato 329

Keywords

multisource feedback, school psychologist assessment, 360-degree evaluation, professional development

Introduction

Although it has been recommended that multisources of information (360-degree feed-back) be utilized in the assessment of client outcomes from therapy (Cone, 2001) and that 360-degree feedback be applied to psychology training (Falender & Shafranske, 2004), a multisource feedback (MSF) system has yet to be adopted as a systematic way to assess the practice of psychologists in Canada. MSF systems have been widely used both in the health professions and industry for assessing professionals (Lockyer & Clyman, 2008). The major purpose of the present article is to describe the nature and scope of MSF instruments that have been developed and utilized by some of the health professions in Canada and to exemplify some empirical information with respect to the use of these instruments in one of the health professions. A secondary purpose of this article is to recommend the development and use of an MSF for practicing school psychologists, present information about how MSF instruments can be constructed, and provide examples of what these instruments could look like.

Nature and Scope of Multisource FeedbackMSF (sometimes referred to as 360-degree assessment) involves the creation of instru-ments to assess and compare observable behaviours of professionals. Typically, this involves a self-assessment and an assessment from others (e.g., colleagues, coworkers, clients). The use of multiple sources of data to evaluate performance has been con-sidered highly desirable in business (Sala & Dwight, 2002) as well as in medicine (Lockyer, Violato, & Fidler, 2003; Norcini, 2003; Ramsey et al., 1993) and other health professions (e.g., occupational therapy; Violato, Worsfold, & Polgar, 2009). It has been widely adopted in industry as a way of providing feedback to employees to guide self-directed learning and improve workplace performance (Sala & Dwight, 2002). In industrial settings, supervisors, peers, and occasionally clients provide feedback. MSF is frequently used in settings in which the staff person works in a team and/or cannot be directly and easily supervised by managers (Church, 1997). In medical application of MSF, physicians complete a self-assessment instrument and receive feedback from medical colleagues (peers), nonmedical coworkers (e.g., nurses, psychologists, phar-macists), and patients (Violato, Lockyer, & Fidler, 2008a).

Violato and his colleagues began the development and testing of MSF instruments in Canada in 1996 (Hall et al., 1999; Violato, Marini, Toews, Lockyer, & Fidler, 1997). At that time, the instruments were developed to assess family physicians, obstetri-cians, internists, and paediatricians. This work led to the adoption of MSF as a manda-tory requirement for continued licensure in the province of Alberta and the Physician Assessment Program (PAR) of the College of Physicians and Surgeons of Alberta (see


330 Canadian Journal of School Psychology 25(4)

www.par-program.org). PAR assesses the performance of physicians and is intended for quality improvement in medical practice by providing structured feedback and identifying physicians for whom a more detailed assessment of practice performance is needed. Subsequently, instruments have been developed for assessing surgeons, medical, paediatric and psychiatry specialists (Lockyer & Violato, 2004; Violato, Lockyer, & Fidler, 2003) as well as anaesthesiology and episodic care physicians. These instruments assess the areas that both American and Canadian specialty organ-izations have identified as critical areas for attention (American Council Graduate Medical Education, 2004; Levine, 2002; Societal Needs Working Group, 1996).

MSF systems have been used for both formative and summative assessment of persons and as a way to promote change of practice of persons assessed (Lockyer & Clyman, 2008). Typically, the assessed person as well as his or her colleagues, peers, and clients complete questionnaires examining core competencies. This feedback sys-tem provides a more global perspective that can be provided by one or a few sources alone (Bracken, Timmreck, & Church, 2001). MSF systems can assess important characteristics of health professionals such as their clinical skills, personal communi-cation, and client management as well as facilitate changes in their practice. Fidler, Lockyer, Toews, and Violato (1999) in a study of family physicians, for example, reported that 66% made at least one change in their communication with patients and support of patients based on multisource feedback.

An important issue with the use of a questionnaire-based MSF system is whether or not it can provide adequate reliability, evidence of validity, and demonstrate utility or feasibility. Some evidence of these psychometric properties has been reported over the past decade with instruments developed and analyzed for a number of health profes-sionals, for example, family doctors (Violato et al., 2008b), psychiatrists (Violato et al., 2008a), and pathologists and laboratory medicine physicians (Lockyer, Violato, & Fidler, 2008; Lockyer, Violato, Fidler, & Alakjia, 2009). As an example, in a study that focused on the development and psychometric assessment of a questionnairebased MSF for quality improvement for occupational therapists, Violato et al. (2009) developed surveys for the assessment of 238 occupational therapists by clients, coworkers, and themselves, respectively, using 5-point Likert-type scales and an unable to assess category. The com-petencies assessed by these surveys included professional responsibilities, practice knowledge, critical thinking, communication, professional development, and manage-ment of the practice environment. The number of respondents for the coworker question-naire was 2,621, and for the client questionnaire it was 2,881. The mean ratings ranged from 4 to 5 for each item on each scale. All of the instruments full scales had very high Cronbachs alphas > .92. The factor analysis revealed a 7-factor solution (66.3% of the total variance) for the coworker survey, and a 4-factor solution for the client question-naire (73.2% of the variance).The factors (professional responsibilities, practice knowledge, utilizes practice process, critical thinking, communication, professional development, manages practice environment) were found to be theoretically meaningful and cohesive, providing some evidence of the construct validity of the instruments.

On a follow-up survey with the occupational therapists, 90% of those surveyed found the self, coworker, and client instruments easily readable and easy to understand.



Moreover, the occupational therapists found it was easy to participate in the system and recruit the required number of informants (coworkers and clients). In summary, the results demonstrated that the developed instruments and procedures had high reli-ability, validity, and feasibility.

Assessment of Competence in PsychologyOver the past few years, there has been support for a shift to the assessment of profes-sional competence in professional psychology that involves a continual and systematic competency-based assessment approach from training to retirement (Kaslow et al., 2007). There has been long discussion, however, and a lack of consensus in the field about how to define and measure competence in professional psychology (Fantuzzo, Sisemore, & Spradlin, 1983; Kaslow, 2004; Shaw & Dobson, 1988) throughout the past years. Coinci-dentally, licensing and regulatory bodies as well as graduate training programs also vary in their definitions, procedures, and instruments for assessing competence in the profes-sion of psychology (Roberts, Borden, Christiansen, & Lopez, 2005).

Currently, there is some consensus among many of the leaders in the field of professional psychology that assessment of competence become more internalized and institutionalized along the career path of the professional psychologist (Roberts et al., 2005). Thus it has been recommended that assessment processes should be developed and implemented that result in feedback to assist professionals in contin-ual learning throughout their careers (Roberts et al., 2005), that assessment of com-petence include multiple perspectives, for example, by way of MSF that integrates input from multiple sources (i.e., self, colleagues, clients; Atkins & Wood, 2002; Fletcher & Bailey, 2003; Maurer, Mitchell, & Barbeite, 2002), that methods of assessing competency meet criteria for validity, reliability, and feasibility (Roberts et al., 2005), and that, among other considerations, core competencies be identified and benchmarks for performance be established (Kaslow et al., 2007). For core competencies, domains key to effective functioning have been by and large identi-fied and agreed on (Rodolfa, 2005; Rodolfa, Bent, Eisman, Nelson, & Ritchie, 2005; Stern, 2006). Kaslow et al. (2007), for example, have identified foundational domains (e.g., professionalism, reflective practice/self-assessment, scientific know-ledge and methods, relationships, ethical and legal practice, individual and cultural diversity, interdisciplinary systems) and functional domains (e.g., assessment, diagnosis, and conceptualization, intervention, consultation, research, assessment, supervision, training management, administration).

It is our contention that an MSF system can be developed to assess the core com-petencies (such as those noted above) of school psychologists that is reliable, valid, and feasible. Moreover, we suggest that such a system could not only identify strengths and weaknesses of school psychologists of their core competencies but also provide useful information and guidance for their professional development. In this regard, we are not suggesting that MSF is specific to school psychologists. Indeed, MSF can be used with counselling psychologists, clinical psychologists, as well as with a plethora of other health professionals (see, for example, Connors & Munro, 2001; Griffin,



Sanders, Craven, & King, 2000; Violato et al., 2008a, as viable applications of MSF with other health professionals).

Competence-based Assessment for School PsychologistsA competency-based assessment system for school psychologists can potentially pro-vide feedback to psychologists about their performance and improve their practice as evidenced with other health professionals. The development and implementation of MSF instruments designed to assess school psychologist in practice would seem to be a timely and valuable undertaking within and across the provinces and territories of Canada. A MSF system that incorporates self-assessment along with peer, coworker, and client assessment could provide information that is not only useful to practicing psychologists but also useful for national and provincial psychology associations and provincial and territorial psychology regulatory boards relative to their oversight and governance of psychology. In this regard, MSF has been and is currently used, by a wide variety of regulatory bodies in the health professions. Some of these regulatory bodies include the College of Physicians and Surgeons of Alberta, College of Phys-icians and Surgeons of Ontario, College of Occupational Therapists of Ontario, and the British Columbia College of Nurses (Violato et al., 2003; Lockyer, Violato, & Fidler, 2003). Medical regulatory bodies have also implemented MSF in the United Kingdom (e.g., Archer, Norcini, & Davies, 2005) and the United States (e.g., Bennett, Gatrell, & Packham, 2004). Importantly, the purpose of MSF is to provide systematic feedback to practitioners for formative purposesto monitor and improve their prac-tice. Regulatory bodies typically have the time, resources, moral obligation, and legal requirements to conduct ongoing professional review of their members of practice as part of their quality assurance program and this is detailed in provincial legislation across Canada (see, for example, Bill 25, Health Professions [Regulatory Reform] Amendment Act 2008 and Health Professions Act 2009).

To systematically develop and implement MSF instruments for use with school psychologists, a number of steps need to be undertaken: (a) formation of a MSF advis-ory group, (b) development of a table of specification for the MSF instruments, (c) cre-ation of items, and (d) formatting and pilot testing the instruments for evidence of reliability, validity, and feasibility of instruments with school psychologists.

Formation of a Multisource Feedback System Advisory GroupThe first step in developing a MSF system for school psychologists is to get the support of one or more of the provincial psychology associations and provincial psychology regulatory bodies within the Provinces of Canada for the development and trial of instruments. Once support of the project is given by a psychology association and regu-latory body, members for an advisory group need to be selected. Participants of the advisory group would include members of the psychology association and regulatory body as well as members deemed as experts in the practice of school psychology as



well as experts in the development and use of MSF systems. The roles and responsibil-ities of the advisory group would include (a) recruiting members for the advisory group who could provide further content expertise for the instruments, (b) identifying roles and competencies as integral to the professional practice of school psychologists, (c) participating in the development of the instruments, (d) providing information to school psychologists in the province(s) regarding the development and use of the instruments, and (e) overseeing the distribution and return of instruments from partici-pating school psychologists involved in the trial and follow-up use of the instruments.

Development of a Table of Specifications for the Multisource Feedback InstrumentsThere are different ways of specifying what MSF instruments should measure. A two-way chart called a table of specifications is typically used to organize and make decisions about the nature and scope of the instrument content, however. The content information within the table of specifications is typically created from reviewing the relevant literature and from the expertise of the advisory group members. Essentially, the preparation of the two-way chart serves as a guide for item construction.

Creating ItemsEach member of the advisory group would be responsible (under the guidance of experts in the construction of MSF systems) to create items that reflect the core com-petencies of school psychologists. After members are given time to create items, they would be reviewed, modified, and/or replaced. Decisions regarding the final content of each item are typically arrived at by using a consensus procedure. Once all of the items are developed for the instruments and judged satisfactory by the advisory group, it is necessary to design the format of the instruments and pilot them.

Formatting and Pilot Testing the InstrumentsTypically, a Likert-type scale response format is recommended for MSF instruments (for example, 1 = strongly agree, 2 = agree, 3 = neither agree nor disagree, 4 = disagree, 5 = strongly disagree, 6 = unable to assess/does not apply). Once the formatting of the instrument is complete, it is typically reviewed and potentially modified by members of the Advisory group. After the formatting is approved it is ready to be pilot tested. The instruments would be distributed to a representative sample of school psychologists who would typically complete the self-assessment instrument and distribute respective instruments to a predetermined number of clients, colleagues, and coworkers. Once all of the instruments are returned, they would be analyzed for reliability, validity, and feas-ibility. Often, based on the results of the first pilot test, instruments are revised and sent out for a second pilot test to confirm and better evaluate their reliability and validity. In addition, norm tables can be constructed to compare future respondents results with the original sample and make further adjustments if necessary. Norm tables allow for



standard scores to be created and used to interpret and compare the performance of a persons scores from the instruments. From this information, reports can be created to provide feedback to school psychologists regarding their performance according to their self-assessment and in relation to others judgment of their performance.

Example of Multisource Feedback Instruments for School PsychologistsThe following examples of instruments have been modified from instruments developed and used to assess the performance of medical doctors (i.e., PAR). These modified instruments only serve as an example of what they might look like for use with school psychologists. They have not been constructed or reviewed by representatives of prac-ticing school psychologists or by members of provincial psychology associations or regulatory boards. They have been created by the authors of this manuscript for the purpose of showing the readers what they might generally look like.

(Examples of the surveys are given in Appendices A-D)

ConclusionBy using the expertise of a broad range of psychological association members, and regulatory groups, the profession of school psychology can develop a broadly applic-able MSF system that can meet the current assessment standards of reliability, validity, and feasibility. This system could be developed relative to the functioning of school psychologists in their numerous roles. The profession of school psychology has an obli-gation to ensure the public that its members are competent and be able to offer evidence of this competence through proper assessment procedures. We suggest that this can be done by the development and use of MSF instruments that involves a self-assessment and an assessment from others (e.g., colleagues, coworkers, clients). Finally, although our examples of instruments can be informally used/modified by school psychologists to aid in their self-evaluations or be used/modified by supervisors of school psychology students in training, we believe that MSF would be much better utilized as a systematic and standardized procedure in the assessment of school psychologists in practice.

Appendix APsychologists Assessment of Performance

Self-Assessment Questionnaire

Name _____________________

How to Answer:

Beside each question there is a row of numbers ranging from (1) to (6)

(continued)



Appendix A (continued)

Make your response by circling the number that is closest to the way you want to respond

Compared to other psychologists you know, rate your performance for each statement

1. Among the worst2. Bottom half3. Average4. Top half5. Among the best6. Unable to assess

Example:

I communicate effectively with clients.

If you rate your performance among the worst, circle

1 2 3 4 5 6

If you rate your performance in the Top Half, circle

1 2 3 4 5 6

Thank you for your cooperation

RESPONSE CODE

1 2 3 4 5 6

Among the worst Bottom half Average Top half Among the best Unable to assess

Answer all questions by circling one of the numbers in the right hand column referring to the response code at the bottom of the page

(continued)

1. I communicate effectively with clients 1 2 3 4 5 62. I communicate effectively with clients families 1 2 3 4 5 63. I communicate effectively with other health care professionals 1 2 3 4 5 6



4. Within the range of services provided by me, I perform technical procedures skilfully

1 2 3 4 5 6

5. Within the range of services provided by me, I demonstrate appropriate judgment

1 2 3 4 5 6

6. I select psychoeducational tests appropriately 1 2 3 4 5 67. I critically assess diagnostic information 1 2 3 4 5 68. I make the correct diagnosis in a timely fashion 1 2 3 4 5 69. I select the appropriate recommendations 1 2 3 4 5 610. I maintain quality psychological records 1 2 3 4 5 611. I obtain informed consent for my services 1 2 3 4 5 612. I inform clients of the limits of confidentiality 1 2 3 4 5 613. I report any circumstances that affect the reliability, validity, or

interpretation of assessment results1 2 3 4 5 6

14. I limit my practice to areas of competence in which proficiency has been gained through education and training

1 2 3 4 5 6

15. I refer clients in an appropriate manner 1 2 3 4 5 616. I provide clear understanding about who is responsible for

continuing care of the client1 2 3 4 5 6

17. I communicate referral information to clients 1 2 3 4 5 618. I recognize biological aspects of psychological disorder 1 2 3 4 5 619. I make appropriate use of community resources for biological

aspects of care1 2 3 4 5 6

20. I manage clients with complex psycho-educational problems 1 2 3 4 5 621. I coordinate care effectively for clients with other health

professionals1 2 3 4 5 6

22. I show compassion for clients and their families 1 2 3 4 5 623. I respect the rights of clients 1 2 3 4 5 624. I collaborate with other psychologists 1 2 3 4 5 625. I am involved with professional development 1 2 3 4 5 626. I accept responsibility for my professional action 1 2 3 4 5 627. I manage psychological care resources efficiently 1 2 3 4 5 628. I manage personal stress 1 2 3 4 5 629. I am aware of my own shortcomings 1 2 3 4 5 6

Appendix BPsychologists Assessment of Performance

Psychologist Colleague Questionnaire

Assessed psychologists Name_______________________

Your name _____________________

Appendix A (continued)

(continued)



Appendix B (continued)

How would you describe your professional relationship to the psychologist (select one)?

1. Peer (similar practice)2. Consultant3. Referring psychologist

How well do you know this psychologist (mark one)?

1. Not at all2. Not well3. Somewhat4. Well5. Very well

Interpretation of the Rating Scale

Rate your colleague on the performance statements according to the following guidelines and scale

How to Answer:



Compared to other psychologists I know, this one is:


Example:

Communicates effectively with clients.If you rate the psychologists performance among the worst, circle

1 2 3 4 5 6

(continued)




If you rate the psychologists performance in the Top half, circle

1 2 3 4 5 6

Thank you for your cooperation

RESPONSE CODE

1 2 3 4 5 6

Among the worst

Bottom half Average Top half Among the best Unable to assess


1. Communicates effectively with clients 1 2 3 4 5 62. Communicates effectively with clients families 1 2 3 4 5 63. Communicates effectively with other health care professionals 1 2 3 4 5 64. Within the range of services provided by this psychologist, he or she performs technical procedures skilfully

1 2 3 4 5 6

5. Within the range of services provided by this psychologist, he or she demonstrates appropriate judgment

1 2 3 4 5 6

6. Selects psychoeducational tests appropriately 1 2 3 4 5 67. Critically assesses diagnostic information 1 2 3 4 5 68. Makes the correct diagnosis in a timely fashion 1 2 3 4 5 69. Selects the appropriate recommendations 1 2 3 4 5 610. Maintains quality psychological records 1 2 3 4 5 611. Obtains informed consent for services 1 2 3 4 5 612. Informs clients of the limits of confidentiality 1 2 3 4 5 613. Reports any circumstances that affect the reliability, validity, or

interpretation of assessment results1 2 3 4 5 6

14. Limits his or her practice to areas of competence in which proficiency has been gained through education and training

1 2 3 4 5 6

15. Refers clients in an appropriate manner 1 2 3 4 5 616. Provides clear understanding about who is responsible for

continuing care of the client1 2 3 4 5 6

17. Communicates referral information to clients 1 2 3 4 5 618. Recognizes biological aspects of psychological disorder 1 2 3 4 5 619. Makes appropriate use of community resources for biological

aspects of care1 2 3 4 5 6

(continued)



20. Manages clients with complex psychoeducational problems 1 2 3 4 5 621. Coordinates care effectively for clients with other health

professionals1 2 3 4 5 6

22. Shows compassion for clients and their families 1 2 3 4 5 623. Respects the rights of clients 1 2 3 4 5 624. Collaborates with other psychologists 1 2 3 4 5 625. Is involved with professional development 1 2 3 4 5 626. Accepts responsibility for his or her professional action 1 2 3 4 5 627. Manages psychological care resources efficiently 1 2 3 4 5 628. Manages personal stress 1 2 3 4 5 629. Is aware of his or her own shortcomings 1 2 3 4 5 6

Appendix CPsychologists Assessment of Performance

Family Questionnaire

Sex Male Female

Age of child who received psychological services

1. 5 or under2. 6 to 103. 11 to 154. 16 and above

Over the last 3 years how often have you seen this psychologist?

1. Once2. 2 to 3 times3. More than 3 times

Your most recent visit was mainly for:

1. New concern2. Ongoing concern

Psychologists name__________________________________

Interpretation of the rating scale

(continued)




Appendix C (continued)

Answer questions about this psychologist according to the following guidelines and scale

How to answer:



1. Strongly disagree2. Disagree3. Neutral4. Agree5. Strongly agree6. Not applicable (NA)

Example:

The psychologist explained your childs problems to you thoroughly

If you strongly disagree with this statement, circle

1 2 3 4 5 6

If you agree with this statement, circle

1 2 3 4 5 6

RESPONSE CODE

1 2 3 4 5 6

Strongly disagree Disagree Neutral Agree Strongly agree Not applicable (NA)

(continued)




Based on the MOST RECENT VISIT with the Psychologist

1. The psychologist explained your childs problems to you thoroughly 1 2 3 4 5 62. The psychologist adequately explained your childs treatment choices

1 2 3 4 5 6

3. The psychologist adequately explained how treatment was going to be monitored

1 2 3 4 5 6

Based on ALL OF YOUR VISITS with this psychologist, how do you feel about the psychologists attitude and behaviour toward you?

This psychologist:4. Spends enough time with me 1 2 3 4 5 65. Shows interest in my childs problems 1 2 3 4 5 66. Answers my questions well 1 2 3 4 5 67. Treats me with respect 1 2 3 4 5 68. Talks to me about my childs needs 1 2 3 4 5 69. Selects appropriate recommendations 1 2 3 4 5 610. Informs me about the limits of confidentiality 1 2 3 4 5 611. Obtains informed consent for services 1 2 3 4 5 612. Informs me of his or her qualifications 1 2 3 4 5 613. Explains all test results on my child well 1 2 3 4 5 614. Demonstrates good knowledge and skill 1 2 3 4 5 615. Arranges appointments with specialists when necessary 1 2 3 4 5 6General16. The psychologist follows up with me about my child 1 2 3 4 5 617. I would go back to this psychologist 1 2 3 4 5 618. I would send a friend to this psychologist 1 2 3 4 5 6

RESPONSE CODE

1 2 3 4 5 6


Appendix DPsychologists Assessment of Performance

Coworker Questionnaire

Assessed psychologists Name________________________

Appendix C (continued)

(continued)



Appendix D (continued)

Your name _____________________

Interpretation of the rating scale

Rate this psychologist on the following performance statements according to the fol-lowing guidelines and scale.

This form is used by a variety of psychologists coworkers (e.g., teachers, educational specialists, therapists)

How well do you know this psychologist (mark one)?

1. Not at all2. Not well3. Somewhat4. Well5. Very well

How to answer:



Compared to psychologists I know, this one is:


Example:

Communicates effectively with clients.

If you rate the psychologists performance among the worst, circle

1 2 3 4 5 6

(continued)



Appendix D (continued)

If you rate the psychologists performance in the Top half, circle


Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research and/or authorship of this article.

1 2 3 4 5 6

RESPONSE CODE

1 2 3 4 5 6


1. Communicates effectively with clients 1 2 3 4 5 62. Communicates effectively with clients families 1 2 3 4 5 63. Verbal communication with other health care professionals is effective 1 2 3 4 5 64. Written communication with other health professionals is effective 1 2 3 4 5 65. Is courteous to coworkers 1 2 3 4 5 66. Respects the personal knowledge and skills of coworkers 1 2 3 4 5 67. Makes appropriate use of community resources for psychoeducational problems

1 2 3 4 5 6

8. Shows compassion to clients and their families 1 2 3 4 5 69. Is nonjudgmental of clients and their families 1 2 3 4 5 610. Is courteous to clients and their families 1 2 3 4 5 611. Accepts responsibility for professional actions 1 2 3 4 5 612. Accepts responsibility for client care 1 2 3 4 5 613. Collaborates well with coworkers 1 2 3 4 5 614. Is available to clients and their families 1 2 3 4 5 615. Maintains confidentiality of clients 1 2 3 4 5 616. Is available for consultation about mutual clients 1 2 3 4 5 6



References

Accreditation Council for Graduate Medical Education Outcome Project. (2004). Accredit-ation Council for Graduate Medical Education web site. Retrieved November 2, 2004, from http://www.acgme.org/Outcome/

Archer, J., Norcini, J., & Davies, H. A. (2005). Peer review of paediatricians in training using SPRAT, British Medical Journal, 330, 1251-1253.

Atkins, P. W. B., & Wood, R. E. (2002). Self- versus others ratings as predictors of assessment center ratings: Validation evidence for 360-degree feedback programs. Personnel Psychol-ogy, 55, 871-904.

Bracken, D. W., Timmreck, C. W., & Church, A. H. (Eds.). (2001). The handbook of multisource feedback: The comprehensive resource for designing and implementing MFS processes. San Franscisco: Jossey-Bass.

Bill 25, Health Professions (Regulatory Reform) Amendment Act 2008, 4th Session, 38th Parliament of the Legislative Assembly, British Columbia, Third Reading (2008, May 27).

Church, A. H. (1997). Do you see what I see? An exploration of congruence in ratings from multiple perspectives. Journal of Applied Social Psychology, 27, 983-1020.

Cone, J. J. (2001). Evaluating outcomes: Empirical tools for effective practice. Washington, DC: American Psychological Association.

Connors G. P., & Munro, T. W. (2001). 360-degree physician evaluations. Healthcare Execu-tive, 27(5), 58-59.

Falender, C. A., & Shafranske, E. P. (2004). 360-degree Assessment applied to psychology in training. Manuscript in preparation.

Fantuzzo, J. W., Sisemore, T. A., & Spradlin, W. H. (1983). A competency-based model for teaching skills in the administration of intelligence tests. Professional Psychology: Research and Practice, 14, 224-231.

Fidler, H., Lockyer, J., Toews, J., & Violato, C. (1999). Changing physicians practices: The effect of individual feedback. Academic Medicine, 74, 702-714.

Fletcher, C., & Bailey, C. (2003). Assessing self-awareness: Some issues and methods. Journal of Managerial Psychology, 18, 395-404.

Griffin, E., Sanders, G., Craven, D., & King, J. (2000). A computerized 360 feedback tool for per-sonal and organizational development in general practice. Health Informatics Journal, 6, 71-80.

Hall, W., Violato, C., Lewkonia, R., Lockyer, J., Fidler, H., & Toews, J. (1999). Assessment of Physician performance in Alberta: The physician achievement review project. Canadian Medical Association Journal, 161, 52-57.

Health Professions Act 2009, Alberta Legislature, Alberta Queens Printer.Kaslow, N. J., Rubin, N. J., Forrest, L., Elman, N. S., Van Horne, B. A., Jacobs, S. C., et al.

(2007). Recognizing, assessing, and intervening with problems of professional competence. Professional Psychology: Research and Practice, 38, 479-492.

Kaslow, N. J. (2004). Competencies in professional psychology. American Psychologist, 59, 774-781.

Levine, A. M. (2002). Medical professionalism in the new millennium: A physician charter. Annals of Internal Medicine, 136, 243-246



Lockyer, J., Violato, C., Fidler, H., & Alakija, P. (2009). The assessment of pathologists/Labora-tory medicine physicians through a multisource feedback tool. Archives of Pathology and Laboratory Medicine, 133, 1301-1308.

Lockyer, J., & Clyman, G. S. (2008). Multisource feedback (360-degree Assessment). In E.S. Holmboe, & R. E. Hawkins (Eds.), Practical guide to the Assessment of clinical competence (pp 75-85). Philadelphia: Mosby.

Lockyer, J., Violato, C., & Fidler, H. (2008). Assessment of radiology physicians by regulatory Authority. Radiology, 247, 771-778.

Lockyer, J., Violato, C., & Fidler, H. (2003). Likelihood of change: A study assessing surgeon use of multi-source feedback data. Teaching, Learning, and Medicine, 15, 168-174.

Lockyer, J., & Violato, C. (2004). An examination of the appropriateness of using a common peer Assessment instrument to assess physician skills across specialties. Academic Medi-cine, 79, 5-8.

Maurer, T. J., Mitchell, D. R. D., & Barbeite, F. G. (2002). Predictors of attitudes toward a 360-degree feedback system and involvement in post-feedback management development activity. Journal of Occupational and Organizational Psychology, 75, 87-107.

Norcini, J. J. (2003). Peer assessment of competence. Medical Education, 37, 539-543.Rodolfa, E. E. (Ed.). (2005). Competency education-competent professionals-ethical practice

[Special Issue]. Professional Psychology: Research and Practice, 36, 347-375.Rodolfa, E. R., Bent, R. J., Eisman, E., Nelson, P. D., & Ritchie, P. (2005). A cube model for

Competency development: Implications for psychology educators and regulators. Profes-sional Psychology: Research and Practice, 36, 347-354.

Ramsey, P. G., Wenrich, M. D., Carline, J. D., Inui, T. S., Larson, E., & Logerfo, J. P. (1993). Use of peer ratings to evaluate physician performance. Journal of American Medical Asso-ciation, 269, 1655-1660.

Roberts, M., Borden, K., Christiansen, M., & Lopez, S. (2005). Fostering a culture shift: Assess-ment of competence in the education and careers of professional psychologists. Professional Psychology: Research and Practice, 36, 355-361.

Sala, F., & Dwight, S. A. (2002). Predicting executive performance with multirater surveys: Whom you ask makes a difference. Consulting Psychology Journal: Practice and Research, 54, 166-172.

Shaw, B. F., & Dobson, K. S. (1988). Competency judgments in the training and Assessment of psychotherapists. Journal of Consulting and Clinical Psychology, 56, 666-672.

Societal Needs Working Group Report. (1996). Skills for the New Millennium: CanMEDS 2000 Project. Annals of Royal College of Physicians and Surgeons of Canada, 29, 206-221

Stern, D. T. (Ed.). (2006). Measuring medical professionalism. Oxford, UK: Oxford University Press.

Violato, C., Worsfold, L., & Polgar, J. M. (2009). Multisource feedback systems for quality improvement in the health professions: Assessing occupational therapists in practice. Jour-nal of Continuing Education in the Health Professions, 29(2), 111-118.

Violato, C., Lockyer, J., & Fidler, H. (2003). Multisource feedback: A method of assessing sur-gical practice. British Medical Journal, 326, 546-548.



Violato, C., Lockyer, J., & Fidler, H. (2008a). Assessment of psychiatrists with multisource feedback. Canadian Journal of Psychiatry, 53, 525-533

Violato, C., Lockyer, J., & Fidler, H. (2008b). Changes in performance: A 5-year longitudinal study of participants in a multi-source feedback program. Medical Education, 42, 1007-1013.

Violato, C., Marini, A., Toews, J., Lockyer, J., & Fidler, H. (1997). Feasibility and psychometric properties of using peers, consulting physicians, co-workers, and patients to assess phys-icians. Academic Medicine, 72, 82-84.

Bios

Jac J. W. Andrews is a professor in the Division of Applied Psychology at the University of Calgary. His current research interests are in the areas of childhood psychopathology, psycho-logical assessment, and clinical supervision. He has published many scientific articles in journals such as Adolescence, Journal of Psychoeducational Assessment, British Journal of Educational Psychology, and Canadian Journal of Counselling.

Claudio Violato is a professor in the Faculty of Medicine at the University of Calgary. A research focus is the development and validation of performance measures for health professionals. He has published many scientific and technical articles and reports in journals such as the Canadian Journal of Surgery, Educational and Psychological Measurement, Academic Medicine, Medical Education, Canadian Journal of Psychiatry, and British Medical Journal and Pediatrics.


The Assessment of School Psychologists in Practice Through Multisource Feedback.PDF

Documents

Transcript of The Assessment of School Psychologists in Practice Through Multisource Feedback.PDF