1ExamSoft Assessment Conference | #ASSESS2015

ExamSoft Assessment Conference

#ASSESS2015

2

Restructuring a capstone course based on longitudinal assessment of module exams

Karen L. Kier, Ph.D., M.Sc., R.Ph., BCPS, BCACPProfessor of Clinical Pharmacy

Director of AssessmentRaabe College of PharmacyOhio Northern University

k-kier@onu.edu

3

ONU and Capstone

• Ohio Northern University – Small private university– College of Pharmacy has 1000 of the 3300 students on

campus– Pharmacy is a direct entry 0-6 program

• Capstone course– Designed in 1998 as part of the transition from a BSPh to a

PharmD– Course was designed to prepare students for 1 year of

rotations (APPE)– Pretest/Posttest design for one measure of material retention

4

ONU Curricular Design

• 2 ½ year pre-pharmacy curriculum with a Profession of Pharmacy sequence each semester

• Spring semester 3rd year module introduction• 4th and 5th year practice modules• Modules designed as a team-based approach with

module coordinator• Integrated materials in module including medicinal

chemistry, pharmacology, pharmacokinetics, pharmacogenetics, and therapeutics

5

Module Assessment

• Exams• OSCE– Rubric assessment– Skill check-off

• Skill-based assessments– Rubric assessment– Skill check-off

• Case-based assignments– Rubric assessment

6

ExamSoft

• ExamSoft instituted in spring semester 2013 for the capstone course – Pretest/Posttest on ExamSoft – Categories and Strength & Opportunities (S&O) report

utilized with exams– Pretest results used to determine course materials as in the

past• Entire 4th and 5th year required to use ExamSoft within

all modules in fall 2013– First cohort to have entire module curriculum on ExamSoft

7

ExamSoft

• Category creation– OSU Medical School Modified Bloom’s taxonomy– Each exam categorized by Director of Assessment

for consistency– Categories include course, material subject,

writing professor, modified Bloom’s– Each question mapped to ACPE 2009 (now 2016),

CAPE 2013, ONU Ability Based Outcomes (ONU ABOs)

8

Topic Discussion

• Category Selection?

9

ExamSoft

• Strength and Opportunities Reports– Students trained to access reports– Rationale release is a primary incentive to look

• Rationales– Faculty development session on use of rationales

to promote a process of remediation– Students to not get exams back only rationales

10

OSU Modified Bloom’s

• 4 areas of modified Bloom’s– Factual– Conceptual– Procedural– Metacognitive

• ONU Exams– 25% Factual– 45% Conceptual– 5% Procedural– 25% Metacognitive

11

ONU Exam History

• Scantron technology used in the past– Item analysis run for every exam– Module teams review item analysis for adjustments– Review for potential questions that could be revised– Point biserial

• ExamSoft technology– Same methodology as scantron– Item analysis reviewed for each exam– Point biserial

12

Pre-test/Posttest Design

• Various methods have been employed for designing the exams

• These designs were intended to modify course materials to improve deficiencies

• Methods included:– PSAP questions– Module coordinators selected questions at

random– Implementation of ExamSoft Exams

13

Capstone

• Idea to use ExamSoft as a basis for pretest selection

• Had first complete cohort• Capstone coordinator and Director of

Assessment felt that ExamSoft could create a better evidence-based selection of questions

• Department chairs agreed• Held module coordinator meeting

14

Module Coordinator Meeting

• Director of Assessment provided ExamSoft metrics from each of their modules

• Item analysis available for every exam in every module

• Category reports by modules showing student performance by subtopics within module

• Category reports included modified Bloom’s for level of questions

• Capstone coordinator gave each module point values• Module coordinators selected questions

15

Module Coordinator Meeting

• What to do with results?• How do we remediate material?• How did this compare to past measures?• Decided to use a combination of Moodle and

ExamSoft to deliver materials and to track student performance

• Decided to use S&O report to require students to look at materials– Red areas were mandatory remediation– All students had access to all remediation materials

16

Module Coordinators

• Each module created a remediation site in Moodle to allow students to review materials

• Quizzes were available to test the review materials

• Module coordinators used ExamSoft results to design course materials (dynamic process) for the current capstone course in spring semester

17

Topic Discussion

• Selection of questions?• Categories?• Difficulty?• Weakness in subtopics versus missing one

question in an area?– Local anesthetics versus depression

18

Pretest

• Creation of pretest• Students took in January 2015 prior to

Pharmacy Administration module, Special Populations module.

• Students knew results in January with S&O report available to them. Rationales were released.

• Categories were simplified to major topic areas.

19

Pretest Results

• Class average 32% (compared to averages in the past of 55-63%)

• What does this mean? How to interpret results?

• Capstone coordinator and Director of Assessment went to discuss results with class in Feb 2015 and expectations for posttest

20

Remediation

• 100% of the students completed all Moodle assignments and quizzes for S&O red areas

• 93.7% of the students completed all Moodle assignments and quizzes regardless of S&O area achievement

• All module coordinators dynamically modified their in class materials based on results of the pretest exam

21

Topic Discussion

• Remediation topics? Methods?• How could ExamSoft analytics help in this

process?• What ExamSoft analytics could be used?

22

Posttest

• Posttest average 58%– The high score was an 89%– Analytics of posttest followed the analytics for student

performance on exams throughout the last 2 years– Tracking cohort performance demonstrated strong

students continued to excel and the weaker students still struggled

• Is this good or yikes???• Students below one standard deviation of the

mean had to remediate the posttest.

23

Curricular Summit

• COP had a curricular summit in May• Evaluating all aspects of the curriculum• Evidence from ExamSoft was used extensively

to think through our process• How to manage weak areas prior to a student

proceeding in curriculum?• Incentives for life-long learning

24

Curricular Summit

• Longitudinal assessment made more readily available to students

• Assessments prior to students moving to next year

• Put more active learning into curriculum to reinforce material prior to capstone

25

Topic Discussion

• Rationale Development• Helpful?• How do you make sure students are reviewing

rationales and learning from them?• Complement rationales with other content• Fair game questions for any module?? • Pull ExamSoft questions in to modules from

other modules?

26

Application Topic Discussion

• How could this be used at your institution?• How could these analytics change what you

do now?• What can we do better?

27

Conclusion

• ONU continues to evolve with this process• Faculty perceptions• Student perceptions• Provided some great evidence for our faculty on

how to improve our testing and curriculum process

• Provided great curriculum dialogue• Rewards• Barriers