Intra and Inter-Rater Reliability and Convergent Validity ...
Inter-rater Agreement In Kansas
description
Transcript of Inter-rater Agreement In Kansas
![Page 1: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/1.jpg)
INTER-RATER AGREEMENT IN KANSASSummer Principals AcademyJuly 22-24, 2014Abilene, KS
![Page 2: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/2.jpg)
2
What is Inter-Rater Agreement?• Consistent
understanding• Consistent
agreement• Consistent
process• Consistent
credibility
The degree 2 raters using the same scale give the same rating in identical situations.
![Page 3: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/3.jpg)
3
Where is Inter-rater Agreement Useful??
Inter-Rater Agreement is useful in all areas of administrative practice including:
• Kansas Educator Evaluations• Kansas Accreditation • Personnel Decisions• Family Engagement Programs• Budget Management
![Page 4: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/4.jpg)
4
What are the Goals of Inter-rater Agreement?
• The Inter-Rater Agreement Statistical Goal is 75% - agreement with no rating more than one level apart.
• The Goal is to have an administrative Common Frame of Reference for your district such as:• Agreement with no rating more than one level apart. • Consistent understanding• Consistent agreement• Consistent process• Consistent credibility
![Page 5: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/5.jpg)
5
How is Inter-rater Agreement Helpful??
• What is a Common Understanding?
• How does Inter-rater Agreement help administrators??
• Facilitates Quality• Provide Consistent Messaging• Impacts Increased Student Growth• Enhances Trust • Facilitates Quality Feedback• Legally Defensible • Assists Accurate Data Collection• Allows for Increase Cooperation
over Time • Passes Public Scrutiny
![Page 6: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/6.jpg)
6
District-wide Agreement about External Criteria
• Agreed Upon External Criteria for:• Observations• Artifacts• Units of Study• Processes• Program Development• Time Limits • Walk-Throughs• Protocols
![Page 7: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/7.jpg)
7
Pragmatics/ConsiderationsPotential Landmines
• Discuss with an Elbow Partner Possible Issues that may cause Rater Error• Leniency: Benefit of the
Doubt• Central Tendency: Safety in
the Middle• Comparing One with
Another: Stick to the Rubric• Relying on Gut too Much:
Stick to the Rubric• Evaluating Absent Evidence:
Stick to What is Observed
![Page 8: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/8.jpg)
8
How to Train Administrators in using Inter-rater Agreement
• Inter-Rater Agreement Training – Things to Consider: • Synchronous and Asynchronous • Experience of the Participants• How to Ensure Fidelity • Use of Video Examples• Use of Case Studies• Bias Exists• Single versus Multiple Observations• Professional Judgment• Degree of Familiarity
![Page 9: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/9.jpg)
9
Getting Started in YOUR District• Getting Your Bearings• Developing a Guidebook• Developing a Training Protocol
![Page 10: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/10.jpg)
10
I. Getting Your BearingsPilot with Administrative Team1. Pilot Approach: Start with Safe Example:
• Use a video from YouTube• Assess Specific Area of Teaching
• Student Engagement• Teacher Activity• Classroom Environment
2. Review current issue/goal/vision/needs3. Review current practices/instrumentation
• Rubrics• Protocol• Training
![Page 11: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/11.jpg)
11
II. Developing a Guidebook• Paper and Electronic: Synchronous and Asynchronous• EXCELLENT Administrative Council Endeavor –
Ultimately to be used during Professional Learning Times with the Staffs
• Pieces and Parts• Terms/Definitions: Frame of Reference/Common Understanding
• “Demonstrates”• “Frequently”• “Usually”
![Page 12: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/12.jpg)
12
II. Developing a Guidebook• Pieces and Parts (continued)
• Examples and Rubrics• Areas to Observe
• Establishes/maintains a culture of safety, respect and rapport
• Knows how to make content accessible to all learners
• Plans standards-based instruction and formative assessments
• Uses a variety of instructional strategies to meet needs
• Deepens understanding through critical thinking, problem-solving and student reflection
• Analyzes student performance and provides feedback
• Collaborates to support student learning
![Page 13: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/13.jpg)
13
II. Developing a Guidebook• Pieces and Parts (continued)
• Exemplars: Video and Artifacts• Master-Coded Videos: Leads to high level of
agreement• Example Artifacts
• Classroom Rules• Lesson Plans• Parent Note Home
• Lesson Plan for Adults: Andragogy
![Page 14: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/14.jpg)
14
III. Developing a Training Protocol• Step-by-Step Procedure1. Initial Training: Goals/Terms
• Practice Interpreting Rubrics• Differentiate between Different Performance Levels with Examples
2. Calibration of the Players• Video and Artifacts
3. Review of the System4. Adjustments (If needed)5. Implementation6. Evaluate/Reflect/Adjust7. Re-Calibrate…
![Page 15: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/15.jpg)
15
• Start Slowly• Start with
“Engagement” with a Small Group
• Get Agreement• Gain Confidence
Over Time• Write and Refine in
Teams
Take Aways – To Do List
![Page 16: Inter-rater Agreement In Kansas](https://reader035.fdocuments.us/reader035/viewer/2022062323/56816606550346895dd937c0/html5/thumbnails/16.jpg)
16
• Start with the End in Mind• Develop Common
Understandings• Use Groups to Ensure
Buy-In• Build Measurable Steps to
Reach Each Goal
Final Thoughts