Holistic Assessment of Social Work Students’ Professional Practice Marion Bogo Joint Social Work...

Holistic Assessment of Social Work Students’ Professional Practice

Marion Bogo

Joint Social Work Education Conference and Social Work Research Conference

University of Hertfordshire2010

Overview

• Link between social work education and practice

• Conceptualization of professional practice

• Assessment– Cognition– Interpersonal– Contextual

Social Work Education and Practice

The purpose of social work education is to prepare social work practitioners

Professional Work: Dimensions

• Thinking• Performing• Acting with integrity(Lee Shulman, 2005)

Signature Pedagogy

“…pedagogical norms with which to connect and integrate theory and practice.”

(Lee Shulman, 2005)


• United States: Council on Social Work Education, Educational Policy and Accreditation Standards 2008

• England: Social Work Task Force 2009• Canada: Canadian Council of Social Work

Regulators; Project - Social Worker Competency Profile 2010

Conceptualization of Professional Practice

• Critique of competency as dimensions and skills – a checklist approach

• How to capture holistic nature of practice and performance– not mechanistic– context– professionalism– knowledge competencies, internal cognitive

processes for judgment, decision making

“In their own words…”

Design a practice-based evaluation tool

• Reflect practice teachers’ tacit or implicit conceptualization of student competence – dimensions they use

• In practice teachers’ language

Holistic Competence

Meta-Competencies

• Higher order overarching abilities and qualities - of a different order and nature than procedural or operational behaviors and skills.

• Related to individuals’ ability to:– use discrete behaviors in a purposeful, integrated,

and professional manner.– learn the specific role competencies of particular

professions.

Meta-competencies

Conceptualization of practice

Relational capacity/Intentional use of self

Learning and growth as a professional

Procedural competencies

Collaborative relationship

Conducts assessment and intervention

Behavior in the Organization

Professional Communication

14DIRECT PRACTICE

Meta-competencies

Critical Thinking and Analysis

Leadership

Learning and growth as a professional

Procedural competencies

Planning

Project management

Behavior in the Organization/Relationships

Professional Communication

15MACRO PRACTICE

Challenges to Assessment

Levels of Performance• High agreement on individual rankings of 20

vignettes- interclass correlation coefficient: 0.83

• Five categories generated– Exceptional - - - - - - - - - - - - - - - - - 5– Ready for practice - - - - - - - - - - - - 4– On the cusp - - - - - - - - - - - - - - - - - 3– Needs more training - - - - - - - - - - -2– Unsuitable for the profession - - - - 1

From Descriptions to a Tool

• Dimensions of holistic competence– Learning and growth– Behavior in the organization– Conceptualizing practice – Collaborative relationships– Assessment and intervention– Professional Communication

From Descriptions to a Tool

• For each dimension • Extracted descriptors from the ranked

vignettes for each of 5 levels of performance • In practice teachers’ language.

Behavior in the OrganizationEXEMPLARY

READY TO PRACTICE

ON THE CUSP MORE TRAININGUNSUITABLE

Work in a responsible and experienced manner within the agency.

Show respect to other staff and work well within the agency’s culture.

Understand the student role and maintain professional boundaries with other team members.

Respected and valued by agency staff. Other staff regard her/him as a valued colleague.

Highly flexible and adaptable.

Relate to agency staff with humility and respect.

Viewed as co-workers by other members of the team.

Work well with colleagues, but can be intimidated by members of other professions.

Follow agency procedures and are accepted by the team.

Tend to work well in the student role.

Regarded as a student/learner by other members of the team.

Experience difficulty in building relationships with organizational staff.

Struggles with the student role, as she/he wants more clients and responsibility, but does not possess or demonstrate the needed skills and abilities to take on this extra responsibility.

Viewed primarily as a student.

Lack appreciation of the student role and self-awareness about interactions with organizational staff.

No respect for the agency and its procedures.

Experience difficulty in fitting into the agency and engage in problematic behaviours, such as being highly critical of and argumentative with other staff and the field instructor, and/or becoming involved in agency politics.

Demonstrate an overall inflexibility in the workplace.

Competency-Based EvaluationIntervention Planning and Implementation

Identify and confront a reluctance to recognize viable options.

1 2 3 4 5 N

Appropriately challenge the client system when required.

1 2 3 4 5 N

Give feedback on above in a manner, which conveys respect and understanding.

1 2 3 4 5 N

Results

• New PBE tool was not more effective at discriminating students than older CBE tool.

CBE Tool PBE Tool

Mean 4.08 4.20

Std Dev 0.53 0.63

Pct <3.00 2.4% 4.8%

Using all Tools

• Recall most recent student• 20 vignettes ordered randomly

– Select vignettes “similar” to their student.– Select from “similar vignettes” one or two

vignettes that are “most similar” to their student.

• Evaluate same student using – Practice-Based Evaluation (PBE) Tool – Competency-Based Evaluation (CBE) Tool

Distribution of Student Scores for the Three Measures

0

1

2

3

4

5

6

7

1

1.2

5

1.5

1.7

5 2

2.2

5

2.5

2.7

5 3

3.2

5

3.5

3.7

5 4

4.2

5

4.5

4.7

5 5

Student Scores

Nu

mb

er

of

Stu

de

nts

CBE Tool

PBE Tool

Vig Match

Scales

• Holistic impression of students

• Deconstruct

• Relate to a skill

• Convert to a point on a scale

Reflections on Assessment

• When values collide

• Student response and the inter-personal relationship

• Negotiating

• Where is ‘the school?’

Conclusion

• In instructors’ own language – holistic, global, authentic - reflects their conceptualization of practice competence.

• A tool without numbers. • Approaches that empower the practice

assessor/field instructor and are not negotiated – exercise her expertise.

An Online Tool

Dimensions of Practice

• Six dimensions– Learning and growth– Behavior in the organization– Clinical relationships– Conceptualizing practice– Assessment and intervention– Professional Communication

• For each dimension, descriptors provided for each of 5 levels of performance.

Levels of PerformanceFive categories

–Exceptional - - - - - - - - - - - - - - - - - 5–Ready for practice - - - - - - - - - - - - 4–On the cusp - - - - - - - - - - - - - - - - - 3–Needs more training - - - - - - - - - - -2–Unsuitable for the profession - - - - 1

Conceptualizing PracticeEXEMPLARY READY TO

PRACTICEON THE CUSP MORE TRAINING UNSUITABLE

Think about and understand practice on multiple levels from instrumental tasks to deeper, therapeutic interventions.

Possess strong critical thinking skills.

Competent in using numerous theories in practice, and apply them in a flexible manner.

Understand and carry out the mandate of the organization, while remaining sensitive to clients’ situations and needs.

Maintain an ethical practice, but are not rigid or inflexible.

View practice on multiple levels, but in some cases this develops only with field instruction.

Knowledgeable about theory and are capable of applying it to practice. Some students, however tend to know one approach very well and work primarily from that framework.

Understand and maintain professional, ethical practice.

Some ability to think critically.

View practice as simply the process of collecting factual or affective information and demonstrate only a limited ability to link affect, behaviour, and cognition.

Experience difficulty integrating theory, but some growth is observed.

Understand and maintain an ethical practice.

Give attention primarily to the concrete aspects of clients’ situations.

Tend to think in the here and now and experience difficulties in making deeper connections.

Able to identify and discuss ethical matters.

Work well within established frameworks, but demonstrate inflexibility in using different practice approaches.

Struggle with understanding the difference between personal relationships and worker-client relationships.

View practice at a concrete level only and experience extreme difficulty in going beyond this level.

Have difficulty applying theoretical understanding to practice situations.

Do not understand ethical practice and client-worker boundaries. Some students have the potential to be harmful to clients.

Demonstrate rigidity in her/his thinking and belief systems.

Struggle in understanding social work values.

Personal issues and difficulties are not kept out of her/his work and practice.

Conceptualizing PracticeDescriptor RankAble to begin to identify and discuss ethical matters. 2

Competent in using numerous theories in practice, and applying them in a flexible manner. 5

Culturally competent and actively incorporates factors relevant to diverse clients. 5

Culturally competent, understands and begins to incorporate factors relevant to diverse clients. 4

Demonstrates rigidity in his/her thinking and belief systems 1

Demonstrates some ability to think critically. 4

Displays some understanding of the impact of cultural factors. 3

Gives attention primarily to the concrete aspects of clients' situations. 2

Has difficulty applying theoretical understanding to practice situations 1

2008-9 Comparisons between CBE Tool (480) and PBE Tool (152)

OSCE: Objective Structured Clinical Examination

• Developed in medicine and used in health professions (Harden et al., 1975)

• Students interview series of standardized, simulated “clients”

• “Clients” are trained actors• Students are given brief information – role,

presenting issues, time frame and goal.

Adapted for Social Work

• Holistic competence - in social work two components:– interview for 12-15 minutes for procedural

competencies– followed by a structured reflective dialogue for

12-15 minutes to elicit meta-competencies

• Both observed and rated by a trained rater.

Iterative Process to Develop OSCE

The Examination

Reading Instructions

One of Five “Clients”

Hospital Social Work

Child Protection

Rater and Widowed Mother

Initial Study: Participants

• 11 students completed Year 1 of the MSW Program

• 7 recent graduates of the MSW Program• 5 experienced social workers with 5-7 years of

experience

Results

• Developed reliable tools – dimensions in each tool were assessing a common ability

• The two aspects – performance and reflection were related but conceptualized differently by raters.

Results

• Moderate reliability across stations : Cronbach’s alpha of 0.55 for Practice and 0.48 for Reflection

• Some evidence for generalizability of individual scores across stations

• Need more stations, better stations, better raters?

Results

• 30% of variance on performance– Experienced practitioners had significantly higher

scores

• 23% of variance on reflection– Marginally significant effect of experience

Assessing Performance and Reflection

Range of student ability to engage in meaningful reflective dialogues about:

• Conceptualizing practice and using frameworks about diversity

• Intentional use of self and own reactions• Integrating learning into practice.Performing well and in depth reflection not

necessarily at same level in individual students.

Is OSCE in our Future?

• Cautious optimism about the merits of using OSCE adapted for social work

• Appears useful for evaluation of both performance and critical thinking, conceptualization, reflection, self-assessment

• Scores not at high range as in field evaluations• Participants and students very enthusiastic.

Concluding Thoughts

• Need approaches that assess holistic competence

• Need to assess at different points in time• Need multiple assessment methods-

authenticity and standardized• In university program and in practice setting

Concluding Thoughts

• Need to study assessment methods and links to pedagogy

• Social work education researchers• Institutional arrangements with practice

settings

AcknowledgementsThese studies were funded by the Social Sciences and Humanities Council of Canada.

Co-investigators: Cheryl Regehr, Professor, Factor-Inwentash Faculty of Social Work and Vice Provost, University of Toronto. Glenn Regehr, Professor and Associate Director, Research, Centre for Health Education Scholarship, Faculty of Medicine, University of British Columbia.

Researchers: Maria Mylopoulos, Educational Researcher, SickKids Learning Institute, Hospital for Sick Children, Assistant Professor, Department of Paediatrics, Faculty of Medicine, University of Toronto

Carmen Logie and Ellen Katz, doctoral candidate, and Rachael Walliser, research assistant, Factor-Inwentash Faculty of Social Work, University of Toronto.

Selected References

About Field EducationBogo, M. (in press) Achieving competence in social work through field education.

Toronto, ON: University of Toronto Press.Bogo, M. (2005). Field instruction in social work: A review of the research

literature 1999-2005. The Clinical Supervisor, 24(1/2), 163-193.Bogo, M., & Vayda, E. (1998). The practice of field instruction in social work:

Theory and process (2nd ed.). New York, NY: Columbia University Press.Fortune, A. E., & Kaye, L. (2002). Learning opportunities in field practica:

Identifying skills and activities associated with MSW students' self-evaluation of performance and satisfaction. The Clinical Supervisor, 21, 5-28.

Fortune, A. E., McCarthy, M., & Abramson, J. S. (2001). Student learning processes in field education: Relationship of learning activities to quality of field instruction, satisfaction, and performance among MSW students. Journal of Social Work Education, 37(1), 111-124.

Homonoff, E. (2008). The heart of social work: Best practitioners rise to challenges in field instruction. The Clinical Supervisor, 27(2), 135-169.

Shulman, L. S. (2005a). Signature pedagogies in the profession. Daedalus, 134(3), 52-59.

Selected ReferencesAbout Competence and Holistic CompetenceBogo, M., Regehr, C., Woodford, M., Hughes, J., Power, R., & Regehr, G. (2006). Beyond

competencies: Field instructors' descriptions of student performance. Journal of Social Work Education, 42(3), 191-205.

Cheetham, G., & Chivers, G. (2005). Professions, competence and informal learning. Cheltenham, UK: Edward Elgar.

Eraut, M. (1994). Developing professional knowledge and competence. London, UK: Falmer Press.Kelly, J., & Horder, W. (2001). The how and why: Competences and holistic practice. Social Work

Education, 20(6), 689-699.Schon, D. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass.Skinner, K., & Whyte, B. (2004). Going beyond training: Theory and practice in managing learning.

Social Work Education, 23(4), 365-381.

Selected ReferencesAbout Evaluating CompetenceBogo, M., Regehr, C., Hughes, J., Power, R., & Globerman, J. (2002). Evaluating a measure of

student field performance in direct service: Testing reliability and validity of explicit criteria. Journal of Social Work Education, 38(3), 385-401.

Bogo, M., Regehr, C., Power, R., Hughes, J., Woodford, M., & Regehr, G. (2004). Toward new approaches for evaluating student field performance: Tapping the implicit criteria used by experienced field instructors. Journal of Social Work Education, 40(3), 417-426.

Bogo, M., Regehr, C., Power, R., & Regehr, G. (2007). When values collide: Providing feedback and evaluating competence in social work. The Clinical Supervisor, 26(1/2), 99-117.

Regehr, G., Bogo, M., Regehr, C., & Power, R. (2007). Can we build a better mousetrap? Improving the measures of practice performance in the field practicum. Journal of Social Work Education, 43(2), 327-343.

About OSCEHarden, R. M., Stevenson, M., Downie, W. W., & Wilson, G. M. (1975). Assessment of clinical

competence using objective structured examination. British Medical Journal, 1(5955), 447-451.

Rating ScalesBogo, M., Katz, E., Logie, C., Regehr, C.,& Regehr, G. (2008). OSCE for social work: Practice

performance rating scale. Unpublished scale. Factor-Inwentash Faculty of Social Work, Toronto, ON.

Bogo, M., Mylopoulos, M., Katz, E., Logie, C., Regehr, C., & Regehr, G. (2009). Reflective dialogue rating scale. Unpublished scale. Factor-Inwentash Faculty of Social Work, Toronto, ON.

Website

Factor-Inwentash Faculty of Social WorkResearch Institute for Evidence-Based Practice

Competency for Professional Practicehttp://www.socialwork.utoronto.ca/research/initiatives/competency.htm

Knowledge building – studies, articles, presentationsKnowledge dissemination – fact sheets, assessment and evaluation tools.

http://www.socialwork.utoronto.ca/research/initiatives/competency.htm

Holistic Assessment of Social Work Students’ Professional Practice Marion Bogo Joint Social Work...

Documents

Transcript of Holistic Assessment of Social Work Students’ Professional Practice Marion Bogo Joint Social Work...