Post on 28-Mar-2015
Faculty of Medicine and Health Sciences
Exploring new approaches to assessing attitudes and behavioursDr Sandra GibsonDr John Winpenny
6th February 2012
Faculty of Medicine and Health Sciences
The plan
• Introductions• Discussion of attitudes and behaviours• Item types used in medical training• Transferability• Lunch • Item writing• Future plans• Closing discussion
Faculty of Medicine and Health Sciences
Outcomes
To discuss the theory and application of assessing attitudes and behaviours
To discuss the methods of assessing behaviours used in medical training and their transferability to other disciplines
To try writing assessment items in chosen disciplines
To develop a plan of action to follow up questions/ideas raised during the workshop
Faculty of Medicine and Health Sciences
Why are attitudes and behaviours important to you?
• Take a minute and write down why measuring attitudes and behaviours is of interest/importance to you
Faculty of Medicine and Health Sciences
What do we mean by ‘attitude’?
‘a psychological tendency that is expressed by evaluating a particular entity with some degree of favour or disfavour’
Eagly and Chaiken (1993)
Faculty of Medicine and Health Sciences
Attitudes
• How an attitude is expressed (or not) can be influenced by– Context (specificity)– Reinforcement (behavioural control)– External drivers- guidelines (subjective norm)– Reflection (Schön)
Attitudes can change
Faculty of Medicine and Health Sciences
Attitudes vs. Behaviours
A student I supervised repeatedly mentioned she held very strong personal and religious views against the termination of pregnancy.
I knew she had just been placed in a family planning clinic.
Faculty of Medicine and Health Sciences
Our approach
• Attitudes do not determine behaviour
• Students who hold suitable attitudes will experience less cognitive dissonance
• Seletion procedures are increasingly incorporating the assessment to try and assess attitudes/behaviours.
Faculty of Medicine and Health Sciences
Tommorow’s DoctorsThe graduate will be able to behave according to ethical
and legal principles. The graduate will be able to:
(c) Be polite, considerate, trustworthy and honest, act with integrity, maintain confidentiality, respect patients’ dignity and privacy, and understand the importance of appropriate consent.
(d) Respect all patients, colleagues and others regardless of their age, colour, culture, disability, ethnic or national origin, gender, lifestyle, marital or parental status, race, religion or beliefs, sex, sexual orientation, or social or economic status.
Faculty of Medicine and Health Sciences
Why are attitudes and behaviours important to you?
• Before I describe how we measure behaviours, lets revisit your interest in measuring them attitudes and behaviours.
Faculty of Medicine and Health Sciences
Medicine
• Fostering professional attitudes
• Measuring Professional behaviours• Observed Structured Clinical Exams (OSCEs)• Situation Judgment tests• Script concordance tests
Faculty of Medicine and Health Sciences
Dealing with uncertainty• Studies with Foundation Year Doctors are
consistently finding that dealing with uncertainty is an issue that they are concerned about.
• Use of these item types can be helpful in preparing and reassuring students about their behaviour in situations where decision making is not clear cut.
Faculty of Medicine and Health Sciences
Observed Structured Clinical Exam (OSCE)
• Sit back and enjoy the movie.• osce.mp4
Faculty of Medicine and Health Sciences
Observed Structured Clinical Exam (OSCE)
• Greater reliability that traditional clinical assessments.• The more realistic the better• Feasibility can be an an issue• Generalisibility Analysis used to optimise design
Faculty of Medicine and Health Sciences
Observed Structured Clinical Exam (OSCE)
• The mark scheme– Global judgment used to set boundary between
minimum competence and non competence– Can be used to assess professional behaviour
• Dealing with difficulty patients• Honesty• Dealing with uncertainty/ambiguous situations
– Multiple mini interview
Faculty of Medicine and Health Sciences
Situation judgment tests
• Another assessment.
• For further examples of SJT’s see the Medical School Council- Assessment Alliance Project page
• Improving Selection to the Foundation Programme• http://www.isfp.org.uk/SJT/Pages/Example-
questions.aspx
Faculty of Medicine and Health Sciences
Situation Judgment Tests
• Vary widely in format and application• Format can have an impact on what they test.• Scoring be complex• Experts can score similarly to novices, but with smaller
standard deviation.• Faking
Faculty of Medicine and Health Sciences
Situation judgment Tests
• Used in General Practitioner (GP) selection• Will be used in Foundation Year selection in the near
future• May be introduced into MBBS selection (UKCAT)
Faculty of Medicine and Health Sciences
Script concordance
• Another test to try.
Faculty of Medicine and Health Sciences
Script concordance
• Based on script theory• Ddesigned to assess decision making in ambiguous
situations.• Studies confirm acceptable reliability and validity• Novices consistently less well than experts• Case specificity and generalisabiity need to be
considered• Importance of reference group
Faculty of Medicine and Health Sciences
Script concordance
• Currently used in Masters in Oncoplastic Breast Surgery on a formative basis
• Plan to introduce in undergraduate course for formative assessment
Faculty of Medicine and Health Sciences
Questions
Faculty of Medicine and Health Sciences
Writing workshop
• Ask group what they want to do• Split into groups each try each item type, or groups
doing different types?• Spend half time facilitating writing, then presenting
items to other groups for review.
Faculty of Medicine and Health Sciences
The critical incident
• To help identify potential areas to write items in, no matter what the format, it can help to start with a critical incident/significant event.
• Take a few minutes and jot down an event were you felt that your judgment felt uncertain.
Faculty of Medicine and Health Sciences
Planning
• What have you got out of the workshop?• Are there any areas that we could take forward as a
group/subgroup?• Do you have any suggestions for future workshops?
Faculty of Medicine and Health Sciences
Thank you
• Thank you to the HEA for supporting this workshop• Feedback forms will be sent to you individually after
the session. • Contact details
– s.gibson@uea.ac.uk– john.winpenny@uea.ac.uk
Faculty of Medicine and Health Sciences
ReferencesBland AC, Kreiter G & Joel A (2005). The psychometric properties of five scoring methods applied to the script
concordance test. Academic Medicine 80:395-9
Charlin B, et al. (2000). The script concordance test: a tool to assess the reflective clinician. Teach Learn Med 12: 189-95
Creighton P, Scott N. (2006). An introduction to situational judgement inventories. Sel Dev Rev 22:3-6
Eagly HA & Chaiken S, (1993). The Psychology of Attitudes, Fort Worth, TX: Harcourt
Feltovich, PJ.et al (1984). LCS: The role and development of medical knowledge in diagnostic expertise. In W.J. Clancey & E.H. Shortliffe (Eds.), Readings in medical artificial intelligence: The first decade (pp. 275-319). Reading, MA: Addison Wesley
Fournier JP, et al (2008). Dcript Concordance Testis: Guidelines for construction BMC Midecal Informatics and Decision Making 8:18
Gagnon R (2005). Assessmet in context of uncertainty: how many members are needed on the panel of reference of a script concordance test? Med Educ 39: 28
Humbert AJ, Besinger B & Edward JM (2011). Assessing clinical reasoning skills in scenarios of uncertainty: convergent validity for a script concordance test in an emergency medicine clerkship and residency. Academic Emergency Medicine 18: 627-634
Mc Daniel MA, & Nguyen NT (2001). Situational Judgment Tests: A Review of Practice and Constructs Assessed. International Journal of Selection and Assesment 9(1-2): 105-113
McDaniel MA, Hartman NS & Grubb W L (2003). Situational Judgment Tests, Knowledge, Behavioral Tendency, and Validity: A Meta-Analysis. Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology. Orlando.
McDaniel MA, & Whetzel DL (2005). Situation Judgment Tests, An IPMAAC Workshop.
Motowidlo S. Borman W & Schmit M (1997). A theory of individual differences in task and contextual performance. Human Performance 10(2): 71-83
Schön D. (1983) The reflective practitioner: how professionals think in action. New York: Basic Books
Faculty of Medicine and Health Sciences
Exploring new approaches to assessing attitudes and behavioursDr Sandra GibsonDr John Winpenny