Building a Peer Evaluation Program
2
Housekeeping
S O ME G E N E R A L H O U S E K E E P I N G R U L E S
o A copy of the slides is available in the resources panel
o The webinar will be recorded and made available ON-Demand
o To keep the webinar short questions will be answered via email later today.
Speaker
Sarah MarrsPRINCIPAL CONSULTANT, PEOPLE INSIGHTS
Sarah has dedicated her career to designing and running feedback programs for over 7 years. She joined Qualtrics in April 2015, after working in-house at Tesco overseeing their global colleague insights survey. At Qualtrics, Sarah works with clients to design feedback solutions that drive real value, through assessment design, sampling methodologies, communication programs, or interpreting results.
She’s a big believer in the power of listening to and acting on feedback, and always energised to talk about new and innovative ways to collect and act on that information.
4
Agenda
TO D AY W E W I L L B E C O V E R I N G
o What is a peer evaluation programo How to create a peer evaluation program
1. Designing a competency framework and items
2. Designing your reports3. Designing your workflow
5
What is peer evaluation? A peer evaluation program collects,
quantifies, and reports observations of a student by their classmates or faculty.
C O MM O N LY I N D I V I D U A L S A R E R AT E D B Y:
o Themselveso Cohort / classmateso Tutor / Professoro Potentially course directoro Potentially academic advisor
6
What is peer evaluation? A peer evaluation program collects,
quantifies, and reports observations of a student by their classmates or faculty.
7
What is peer evaluation? A peer evaluation program collects,
quantifies, and reports observations of a student by their classmates or faculty.
8
What is peer evaluation? A peer evaluation program collects,
quantifies, and reports observations of a student by their classmates or faculty.
9
What is peer evaluation? A peer evaluation program collects,
quantifies, and reports observations of a student by their classmates or faculty.
Peer evaluation is ultimately designed to drive
lasting behaviour change
10
What is peer evaluation? W HAT D O S T U D E N T S G E T F R O M P E E R E VA L U AT I O N?
o Self-awareness (no.1 leadership trait!)o Improvement in learningo Teaching how to give feedbacko Teaching how to take feedback
W H AT D O E S A T U TO R G E T F R O M P E E R E VA L U AT I O N
o A feedback instrument that works for different student styleso An instrument that allows them to get a gauge on student
dynamicso An instrument that allows them to get data-driven feedback
aggregation
11
What is peer evaluation? T H R E E C O M MO N U S E S F O R P E E R E VA L U AT I O N
1. Measure contributions and teamwork at the end of a specific project to feed into project grades
2. Mid-year measurement of a student’s behaviours to provide developmental feedback
3. End-of-placement review of a student work placement to get feedback from both professional and academic sides
12
What is peer evaluation?
YOU MUST KNOW YOUR PEER EVALUATION PURPOSE BEFORE MOVING ON TO DESIGN YOUR PROGRAM
13
How to create a peer evaluation program
T H R E E S T E P S TO C R E AT E Y O U R P E E R E VA L U AT I O N
1. Design your competency framework and items2. Design your reports3. Design your workflow
14
C O MP E T E N C Y A
I T E M ( B E H AV I O U R ) 1
I T E M ( B E H AV I O U R ) 2
I T E M ( B E H AV I O U R ) 3
C O MP E T E N C Y B
C O MP E T E N C Y C
I T E M ( B E H AV I O U R ) 4
I T E M ( B E H AV I O U R ) 5
I T E M ( B E H AV I O U R ) 6
I T E M ( B E H AV I O U R ) 7
I T E M ( B E H AV I O U R ) 8
I T E M ( B E H AV I O U R ) 9
STEP 1 - Design a competency framework & items
15
T E AM W O R KI N G S K I L L S
L i s t e n s t o c l a s s m a t e s
’ i d e a s
Ta k e s o n o t h e r ’s
f e e d b a c k
W o r k s w e l l w i t h
c l a s s m a t e s
C O N T R I B U T I O N TO W O R K L O A D
A t t e n d s a l l w o r k i n g s e s s i o n s
P a r t i c i p a t e s a c t i v e l y i n
w o r k i n g s e s s i o n s
I s a b l e t o c a r r y o u t t a s k s i n d e p e n d e n t l y
P L AN N I N G & O R G A N I S I N G
A r r i v e s o n t i m e t o w o r k i n g s e s s i o n s
M a n a g e s t h e i r s c h e d u l e w e l l
M e e t s p r o j e c t -r e l a t e d
d e a d l i n e s
W o r k s w e l l w i t h
c l a s s m a t e s
P R O - T I PFrequency scales are the best way to focus on the consistency of behaviours, e.g:
• Never• Rarely• Sometimes• Often• All the time
STEP 1 - Design a competency framework & items
16
G O O D I T E M S . . .
o Capture an important aspect of a key competency
o Focus on a single behaviouro Start with an action verb (e.g., motivates,
listens, etc.)o Are written in simple plain languageo Can be acted upon if they are identified as
weaknesses/ opportunities
B AD I T E MS . . .
o Loosely or fail to measure an important competencyo Focus on multiple behaviours (double, triple barreled)o Include too many unnecessary adverbs (e.g.,
efficiently, effectively)o Are too technical or hard to understando Do not match the rating scaleo Are culturally biased
STEP 1 - Design a competency framework & items
17
STEP 1 - Design a competency framework & items
O T H E R C O MP E T E N C Y A N D I T E M T I P S
o Avoid including too many items (generally 50-60 total items max)o Include a few (up to 3) open-ended items that ask students to provide
constructive and actionable feedback to their cohorto Remind evaluators not to include identifying information in open-ended
items to protect confidentiality
18
STEP 2 - Design your reports
T W O M A I N T Y P E S O F R E P O R T
1. The individual report (to go back to the student)
2. The aggregated report (to be seen by the tutor / professor / course director only)
19
STEP 2 - Design your reports
G O O D R E P O R T S :
AGGREGATE RESULTS WITHIN SOURCE – Aggregate ratings within source for each competency and item
20
G O O D R E P O R T S :
AGGREGATE RESULTS WITHIN SOURCE – Aggregate ratings within source for each competency and item
PROVIDE MEANINGFUL COMPARISONS – Compare source-level ratings to internal benchmarks
STEP 2 - Design your reports
21
G O O D R E P O R T S :
AGGREGATE RESULTS WITHIN SOURCE – Aggregate ratings within source for each competency and item
PROVIDE MEANINGFUL COMPARISONS – Compare source-level ratings to internal benchmarks
INCLUDE OPEN-ENDED COMMENTS – Present both positive and constructive comments
STEP 2 - Design your reports
22
STEP 3 - Design your workflow C O MM O N W O RK F L O W D E C I S I O N S
o Who chooses who should evaluate an individual?o Should the tutor / professor approve nominations for
evaluators?o Should evaluators have an opportunity to opt out?o Should the individual receive their report directly?
23
STEP 3 - Design your workflow S C E N A R I O 1 . MEASURE CONTR IBUT IONS AND
TEAMWORK AT THE END OF A SPEC IF IC PROJECT TO FEED INTO PROJECT GRADES
Tutor / professor adds all students in project groups
– and asks all students in group
to rate each other
All students need to complete their
project group evaluations
(no opting out)
Report goes back to tutor / professor
directly – for each individual. Aggregate report also created allowing them to
compare individuals.
Tutor discusses report with student and
feeds into project grades
24
STEP 3 - Design your workflow S C E N A R I O 2 . MID-YEAR MEASUREMENT OF AN
INDIV IDUAL ’S BEHAVIOURS TO PROV IDE DEVELOPMENTAL FEEDBACK
Individual is asked to
select 3-5 peers and up
to 3 tutors who they
would like to receive
feedback from
Individual will automatically
receive evaluations from their
course director and academic advisor
Peer evaluations
are opt-in – so not
mandatory for all classmates to complete
Report goes back to
individual student first, for them to digest and
later discuss with academic
advisor or tutor.
Course directors get aggregated feedback reports
25
STEP 3 - Design your workflow
Course director adds
placement students -
asking tutor, classmates
and professional coworkers to
evaluate them
Placement evaluations are ‘nudged’
but not mandatory
Individual student
report goes to each tutor for feedback discussion
Aggregate report goes to course director to
review placement
program as a whole
Course directors
reach out to company
placement teams to discuss
feedback / any red flags
S C E N A R I O 3 . END- OF-PL ACEMENT REV IEW OF A STUDENT WORK PL ACEMENT TO GET FEEDBACK FROM BOTH PROFESS IONAL AND ACADEMIC S IDES
In Summary
1. Define Purpose2. Define Competencies & Behaviours3. Define Report4. Define Workflow5. Launch evaluation6. Follow-up with students
27
Thank you
Top Related