Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

22
1 ExamSoft Assessment Conference | #ASSESS2015 ExamSoft Assessment Conference

Transcript of Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

Page 1: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

1ExamSoft Assessment Conference | #ASSESS2015

ExamSoft Assessment Conference

#ASSESS2015

Page 2: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

2ExamSoft Assessment Conference | #ASSESS2015

Assessing the Impact of Peer Instructor Generated Formative

Self-AssessmentsPresenters:

Melinda Lull Ph.D., Jennifer Mathews Ph.D. Wegmans School of Pharmacy

St. John Fisher College, Rochester, NY

Jane Marie Souza, Ph.D.University of Rochester, Rochester, NY

Page 3: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

3ExamSoft Assessment Conference | #ASSESS2015

Learning Objectives for the Session:

• Demonstrate the process of enrolling peer instructors as exam item writers.

• Demonstrate the use of the internal comments box as a means of providing feedback to peer instructors.

• Describe the process for analyzing ExamSoft data in the comparison of quiz and exam performance.

Page 4: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

4ExamSoft Assessment Conference | #ASSESS2015

Why use self-assessments?

• Self-assessments increase course performance. (Stewart et.al., 2014; West and Sadoski, 2011)

• Self-assessments increase self-directed learning. (Nicol et.al., 2006)

• Students achieve better long-term retention of material. (Nevid and Mahon, 2009)

• Students get more practice with the format of in-class exams.

Page 5: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

5ExamSoft Assessment Conference | #ASSESS2015

Why use peer instructors?

• Peer instructors increase student performance. (Santee and Garavalia, 2006)

• Students may feel more comfortable with peers than faculty. (Carrol, 1996)

• Peer instructors themselves benefit from reviewing material. (Haist et.al., 1997; Haist et.al., 1998)

Page 6: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

6ExamSoft Assessment Conference | #ASSESS2015

Tra

inin

g FacultyExamSoft Infrastructure

Enrolling Peer Instructors

StudentsQuestion Writing

ExamSoft

Imp

lem

enta

tion

Question WritingQuestion Editing

Feedback

Launching Quizzes

Ass

ess

me

nt

Utility

ImpactStudents

Peer Instructors

Page 7: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

7ExamSoft Assessment Conference | #ASSESS2015

• ExamSoft Infrastructure– Peer instruction requires separate folders

• Questions

• Assessments

• Course

Training FacultyExamSoft Infrastructure

Enrolling Peer Instructors

Page 8: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

8ExamSoft Assessment Conference | #ASSESS2015

• Enrolling Peer Instructors– Peer Instructor “Faculty” Account

• E.g., [email protected]

– With limited permissions• Create exam items only• Only access tutoring courses question folder• Can only save, not approve, questions

Training FacultyExamSoft Infrastructure

Enrolling Peer Instructors

Page 9: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

9ExamSoft Assessment Conference | #ASSESS2015

• Provide peer instructors with:– Detailed ExamSoft instructions– Resources on how to write good questions

with examples

• Considerations:– Have them provide rationales for all questions– Limit question types (e.g., essays)

Training StudentsQuestion Writing

ExamSoft

Page 10: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

10ExamSoft Assessment Conference | #ASSESS2015

• This is a learning process for peer instructors too!

Implementation Question Writing

Question Editing

Feedback

Page 11: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

11ExamSoft Assessment Conference | #ASSESS2015

• Faculty will ultimately be in control of:– Reviewing all questions– Assembling quizzes– Launching quizzes

• Aim for consistency – Across quizzes, courses, and with in-class exams

• Considerations:– Posting multiple versions or allow for multiple downloads– Enable secure quiz and review– Make passwords easily available (e.g., post on course

management site)– Ensure adequate time to access

Implementation Launching Quizzes

Page 12: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

12ExamSoft Assessment Conference | #ASSESS2015

Implementation Launching Quizzes

Page 13: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

13ExamSoft Assessment Conference | #ASSESS2015

• Data Available from ExamSoft– Number of exam takers– Aggregate performance on quizzes and

individual questions– Individual student performance on individual

questions– Category performance*

AssessmentUtility

Impact

Page 14: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

14ExamSoft Assessment Conference | #ASSESS2015

Assessment Utility

Number per semester 9-12

# quizzes per exam 2-3

Average # of students taking each assessment (% of class)

1st attempt: 53.9 (67.1%)

2nd attempt: 24.2 (30.2%)

% taking at least one quiz 95%

Average # of quizzes taken per student 8.5

Average of 6 semesters worth of data in 5 different courses

Page 15: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

15ExamSoft Assessment Conference | #ASSESS2015

• Comparisons using data mined from ExamSoft– Exam performance vs. quiz performance– Exam vs. quiz performance broken down by

course learning objective– Exam performance of quiz takers vs. non-quiz

takers

Assessment ImpactStudents

Peer Instructors

Page 16: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

16ExamSoft Assessment Conference | #ASSESS2015

Assessment ImpactStudents

Peer Instructors

* p<0.05** p< 0.001

Example: Quiz vs. Exam Performance by Exam

By Learning OutcomeBy Exam

Page 17: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

17ExamSoft Assessment Conference | #ASSESS2015

Assessment ImpactStudents

Peer Instructors

Student Perceptions of Tutoring Quizzes (n=37)

Question Average Score*

% SA/A

I feel that the tutoring quizzes are a valuable resource. 4.65 95%

I feel that taking the tutoring quizzes increased my confidence going into the exam.

4.43 92%

I feel that taking the tutoring quizzes increased my performance on the exam.

4.27 81%

I feel that the tutoring quizzes accurately reflected the material that was on the exams.

4.24 92%

I recommend continuing to offer tutoring quizzes in the future.

4.76 98%

Based on a Likert Scale of 1-5 (1= Strongly Disagree; 5= Strongly Agree); SA= Strongly Agree; A= Agree

Page 18: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

18ExamSoft Assessment Conference | #ASSESS2015

Assessment ImpactStudents

Peer Instructors

Peer Tutor Perceptions of Tutoring Quizzes (n=8)

Question Average Score* % SA/A

Online tutoring quizzes were a valuable resource for the students I was instructing.

4.63 100%

Writing online tutoring quiz questions increased my knowledge of the course topics covered.

4.38 88%

Learning to write quiz questions was a valuable learning experience.

4.63 100%

Based on a Likert Scale of 1-5 (1= Strongly Disagree; 5= Strongly Agree); SA= Strongly Agree; A= Agree

Page 19: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

19ExamSoft Assessment Conference | #ASSESS2015

Use of Self-Assessments in Pharmaceutical Science Courses

• Started with a pilot in 1 pharmacology course with 2 peer instructors

• Now implemented in 7 courses throughout the pharmaceutical sciences curriculum– 16 peer instructors involved

• All guided by same guidelines and using same resources and instructions

Page 20: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

20ExamSoft Assessment Conference | #ASSESS2015

Conclusions

• Creating the infrastructure for self-assessments in ExamSoft is similar to that of in-class assessments, and only requires a few minor modifications.

• Feedback demonstrates that students perceive a benefit from self-assessments.

• Quiz and exam performance indicate that students benefitted from self-assessments.

• Peer instructors perceived a benefit from creating self-assessments.

• Faculty perceive a benefit and more courses are utilizing self-assessments and peer instructors.

Page 21: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

21ExamSoft Assessment Conference | #ASSESS2015

References• Carroll M. Peer tutoring: can medical students teach biochemistry. Biochem

Educ. 1996; 24:13-15• Haist SA, Wilson JF, Fosson SE, Brigham NL. Are fourth-year medical students

effective teachers of the physical examination to first-year medical students. J Gen Intern Med. 1997; 12:177-181

• Haist SA, Wilson JF, Brigham NL, Fosson SE, Blue AV. Comparing fourth-year medical students with faculty in the teaching of physical examination skills to first year students. Acad Med 1998; 3: 190-200

• Nevid JS, Mahon K. Mastery quizzing as a signaling device to cue attention to lecture material. Teach Psychol. 2009; 36(1): 29-32

• Nicol DJ, Facfarlane-Dick D. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Stud High Educ. 2006; 31(2): 199-218

• Santee J and Garavalia L. Peer tutoring programs in health professions schools. AJPE 2006; 70: Article 70

• Stewart D, Panus P, Hagemeier N, Thigpen J, Brooks L. Pharmacy student self-testing as a predictor of examination performance. AJPE 2014; 78(2): Article 32

• West C and Sadoski M. Do study strategies predict academic performance in medical school? Med Educ. 2011; 45(7): 696-703

Page 22: Assessing the Impact of Peer Instructor Generated Formative Self-Assessments

22ExamSoft Assessment Conference | #ASSESS2015

THANK YOU!

CONTACT US: Mindy Lull: [email protected]

Jennifer Mathews: [email protected] Marie Souza: [email protected]