© 2005 The University of Vermont

University of VermontCollege of Medicine

Online Exams

Embrace the opportunity, avoid the pitfalls

Copyright: The University of Vermont This work is the intellectual property of the University of Vermont. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors

© 2005 The University of Vermont

Testing

© 2005 The University of Vermont

Who we are

University of Vermont College of Medicine• Located in Burlington, Vermont• Founded in 1822• Nation’s seventh oldest medical school• One of 125 medical schools in the US • Enrollment

o 399 medical studentso 93 graduate studentso 12 MD/PhD students

• Facultyo 678 full time facultyo 383 part time facultyo 711 volunteer faculty

• Affiliated with Fletcher Allen Health Care (FAHC) o a level 1 trauma center

• Allied with Maine Medical Center (MMC)

© 2005 The University of Vermont

Why On-Line Exam Delivery

Vermont Integrated Curriculum (VIC)o Transforms medical education

• Learning defined by body system, disease, and skill set• Patient-based learning integrates with basic science

o Merges tradition and innovation• Lectures, labs, and seminars join case-based studies,

collaborative small groups, and virtual learning• The art and science of medicine are fused with an advanced hybrid

online learning system

o Assesses competencies• Questions linked to teaching• Statistical analysis• Competency tracking

o 12 courses over first 18 months, some with as many as 120 faculty; 8 clinicalexperiences in two states

© 2005 The University of Vermont

History

• Courses no longer resided in departments; now overseen centrally

• Needed to assess the efficacy of the new curriculum – was it working?

• Wanted single platform for assessment, and it had to produce detailed statistics for analysis

• National licensing exams were computerized• Wanted to embrace potential of technology

© 2005 The University of Vermont

Requirements

• Size/Volumeo Any one exam, up to 100 students at a time; up to 150 questions.o One large exam delivered simultaneously with other smaller exams

• Need for hybrid functions for pedagogical purposes – especially image-based disciplines

o Embedded interactive multi-media as part of question delivery.

• Valid student assessmento Alignment of exam questions to competencies

• Security >= paper based assessmentso Inability to do anything but take the examo Minimize the ability to look at another's worko Maximize exam/question reusability

• Grading/Statisticso Grading/Statistical analysis that at minimum matched existing bubble

sheet process.

© 2005 The University of Vermont

Technologies

• Blackboard is the core platformo Test Manager/Gradebook

o Force completion, single attempt, randomized, password protected

• Respondus for uploading questions.o Images, videos, sounds uploaded separately into Blackboard

• Baseline laptops o Ensure consistent delivery and behavior.

• Secure Exam browser and privacy screens o Help discourage cheating

• Fault tolerant platform/platform monitoring toolso Maximize availability.

• Application tuning, generous hardware and exam delivery process maximize performance.

© 2005 The University of Vermont

Technology considerations

• Benchmark and simulate exam delivery under load.o Exam size, randomization, number of students,

simultaneous initiation, and simultaneous exams are all key factors.

o Simulation tools valuable in predicting system performance. Microsoft Application Center Testing (ACT)

o Real world simulation validates entire process.

• Network connectivityo Wired vs Wireless

• Baseline clientso Reduce variables and unforeseen problems.o Have a spares/support strategy in placeo Beware of SMS packages/Windows updateso Teardown disruption

© 2005 The University of Vermont

Technology considerations

• Disaster planningo Develop backup strategieso Understand system behavior and recovery strategies

• New versionso Develop a test script, and be methodical evaluating

every releaseo Understand the types of questions and multimedia

being utilized in exams, o Run your load test.

• Future considerationso Bb 7 advertises improved performance. Validation in

process.o Tablet PCs possible for next yearo Testing throughout the country for 4th year courses

© 2005 The University of Vermont

Process – development/delivery

• Exam developmento New questions added via Blackboard or Respondus.o On-Line exam review by faculty members.

• Exam deploymento Exam scheduled for release 15 min prior to scheduled

exam start.o Password protected. Algorithm based on encrypted

exam name plus something we assign.

• Exam deliveryo Students get setup 5 – 15 min prior to exam starto Support team present during exam kickoff.o Launch ~ 16 students at a timeo Loaner laptops/spare parts (Network cables, power

bricks, mice etc) on hando Save often

© 2005 The University of Vermont

Process -- statistics

• Exam performance analysiso Blackboard provides the cumulative statisticso A custom developed tool extracts detailed

exam informationo Excel pivot-table generated reports include:

• Individual student performance

• Answer key

• Enables grade adjustment for questions determined to be statistically invalid

o SAS calculates performance based statistics• Discrimination index

• Difficulty index

• Intra-examination reliability index

© 2005 The University of Vermont

Process – analysis/adjustment

• Exam Results Analysiso Questions flagged if <60% correcto Difficulty and Discrimination indices – good

discriminators kept; poor ones reviewedo Poorly written questions or material not covered =

question droppedo Faculty development opportunity as they review

their question performance

• Exam Review (for students)o After exam adjustedo SQL Reporting Services for individual exam results.o Students see master question number they got

wrong, their answer and correct answero Question text projected on screen

© 2005 The University of Vermont

People

• COMET team o Exam deployment and delivery issues

• COMIS team o laptop help and network connectivity concerns

• Curricular proctors o Timing and accommodations

• Course director o Content oversight and delivery parameters

• Assessment director o Statistical analysis

© 2005 The University of Vermont

Demonstration

© 2005 The University of Vermont

Opportunities

• What do faculty want to achieve with exams?o Ease of grading? o New question types?o Rich media?

• What can faculty achieve with exams?o Incorporating video or soundo New question types

• Find the correct location• Multiple answer• Random from a question pool

o Case based or question by question examso Just in time exams o Testing as content has been taught

© 2005 The University of Vermont

Considerations

• Studentso Access to/familiarity with technology

• Facultyo Do they know what they want?o Will they make the commitment to work with the technology?o Are they prepared to change their testing paradigm?

• Processo What are the student & faculty expectations for exam

review?o Are summary statistics enough?o How will they drop questions?o Off site/Distance learning limitations

© 2005 The University of Vermont

Questions?

© 2005 The University of Vermont

Contact Us

Contact us:

Jill JemisonE-Learning ManagerJill.Jemison@uvm.edu(802) 656-0076

Jim HeintzSenior Technical Program ManagerJim.Heintz@uvm.edu(802) 656-9770