PERSPECTIVES IN COMPETENCY ASSESSMENTDebra Sibbald, Executive Director, Assessment ProgramsBScPhm RPh ACPR MA (Adult Education) PhD (Curriculum Teaching & Learning)

February 2015

KEY DECISION-MAKING IN ASSESSMENT DESIGN:

EXPERT LENSES

Thus far you have heard opinions fromexpert authoritative sources on competency assessments:

how best practices should inform key decision making...

Best Practices: Identify Important Design Principles

• Candidates• Outcomes• Competencies• Levels

• ‘Fit for purpose’ aligned blueprint• Methods• formats• Scales• Subject matter experts

• Writing• Validating• Standard setting• Iterative reflection and refinements

HARMONIZATION OF COMPETENCY ASSESSMENTS:

Collective CommunityIndividual

Agents

EXPERTS

INDIVIDUAL AGENTSlikely perceive the problem relative to their context

COLLECTIVE COMMUNITYA Collaboration

Individual Agents:

• Relevant decision-makers /organizations• Major Stakeholders (directly affected and held accountable)• Experts in their contexts • Others (groups/individuals who provide commentaries)

• HARMONIZATION PROCESS: • EXTERNAL EXPERTS (TOUCHSTONE)• Consider each agent’s vested interest and what that group wants• ‘Fit-for-purpose’ alignment • Acceptable to all• Understood and acted upon by all

EXAMPLE:OPTOMETRY EVALUATING EXAM FOR INTERNATIONAL OPTOMETRY GRADUATES

Major Stakeholders

1. Candidates – International Optometry Graduates

2. College of Optometrists of Ontario (Registration Committee)

3. International Optometric Bridging Program (IOBP)

4. CEO / ECO (Canadian Examiners Optometry) / (Examinateurs Canadiens en Optométrie)

5. Practicing Optometrists (from Canadian schools or from schools in North America accredited by the Accreditation Council on Optometric Education (ACOE)

6. Practicing Optometrists who are graduates of IOBP

7. University of Waterloo

Major Stakeholders8. Pan Canadian Provincial Jurisdictions (FORAC – CORA)

9. Office of the Fairness Commissioner

10. TOUCHSTONE• Executive Director of Assessment• Director Health Professional Affairs• Psychometricians

11. TOUCHSTONE: Board of Directors• Chief Executive Officer• Directors Finance/Resources• Managers – Operations and Standardized Patients

12. Blueprint Subject Matter Experts:• Participants for EE from Optometric Practice

Other Stakeholders

13. Evaluating Exam Chief Examiners

14. Subject Matter Experts Writers

15. Evaluating Exam Examiners

Major Stakeholder Major Stakeholder Minor Stakeholder

Candidates from IEO Pan Canadian Provincial Jurisdictions (FORAC – CORA)

Evaluating Exam Chief Examiners

College of Optometrists of Ontario(Registration Committee)

Office of the Fairness Commissioner

Subject Matter Experts Writers

International OptometricBridging Program (IOBP)

TOUCHSTONE-Director of Assessment-Director Health Professional Affairs-Psychometricians

Evaluating Exam Examiners

CEO / ECO(Canadian Examiners Optometry) / (Examinateurs Canadiens en Optométrie)

TOUCHSTONE Board of Directors- Chief Executive Officer- Directors - Finance/Resources- Managers – Operations/SPs

Practicing Optometrists from ACOC schools

Blueprint Subject Matter Experts:Participants for EE from Optometric Practice

Practicing Optometrists who are graduates of IOBP

Harmony, balance, sustainability

WHO ARE THE AGENTS IN OUR AUDIENCE?Turn on your clickers!

What is your main focus of interest?A. RegulatoryB. EducationC. ClinicalD. AssessmentE. Administrative

From what viewpoint is the domain of your concern?

A. Economic forcesB. Social/sociological structures and practicesC. Political structures and policiesD. Educational affordances and constraintsE. Cultural beliefs or supports

TODAY YOU HAVE BEEN PRESENTED WITH A GREAT DEAL OF INFORMATION FROM VARIOUS DIMENSIONS.

i-Clicker Polling - CommunityMicrophone -

Agents

It’s time to pause and engage you in a guided reflection

Recall the issues and perspectives presented(s)

How did you react during the presentation?

Why did you feel this way?

Future changes based on these insights?

Reflection & Feedback on This Morning

Identify

Change

Analyze

Reflect

Identify:

The most surprising piece of information that you learned today

Or

The perspective or view which is not aligned with your current practices or beliefs

What was your primary Reaction?

Why do you have this reaction or concern? The domain your view arises from is :A. Economic

forcesB. Social/sociological

structures and practicesC. Political

structures and policiesD. Educational

affordances and constraintsE. Cultural

beliefs or supports

What are your main issues?

Considering the reasons for these surprises or differences, will you change your beliefs or practices? A. YES

B. No

If yes, what will be your main goal?

What will be likely implications of this decision ?

You identify your primary challenge as

A. Problems created by your own decisions and behaviours

B. Problems created by external forces

What is the primary characteristic of your challenges?

A. Problems can be solved in whole or in part

B. Problems beyond our controls

On what step towards change has today’s discussion had impact, if any?

A. You recognize you now face an important decision

B. You now more clearly or accurately can identify alternatives

C. You now are better equipped to evaluate alternatives

D. You are now ready to act on the best alternative

E. No impact on steps toward change

Audience has the floor!

You Say….. I Say….