Electronic Course Evaluations

Presentation to Department Chairs January 31, 2013

Collaboration between USM Office of Academic Assessment and PCE • Susan King, Director of Academic Assessment • Dr. Monique LaRocque, Executive Director, PCE • Dr. Khusro Kidwai, Director of Online Teaching and Learning

How did the USM Electronic Course Evaluation Initiative begin? • Faculty requests for an online course evaluation option • Increasing number of online and blended courses

Agenda

• History of course evaluations at USM • Results from Summer 2012 Pilot • Comparison: Paper-based vs. Electronic • Next Steps

History of Course Evaluations

1980 USM course evaluations (Green Form) were designed by faculty

2003 CONHP adopted the SIR-II course evaluation

2006 SIR-II option available University wide

2008 Scanning software: mainframe-based --> PC-based

2012 Piloted electronic course evaluation

Issues with Current Paper-Based System

• Faculty are requesting a quicker turn around time, and additional breakdowns of

the data..…but the paper-based system depends upon the Opscan scanner equipment, which is outdated and archaic, scantools software is slow at processing reports.

• Faculty depend on their department personnel for the manual distribution,

collection, organization, and accurate completion of forms….errors on the header sheets creates errors in the output reports.

• Faculty are requesting an electronic option to use with all their courses, online

and face-to-face courses…for examining their own course improvement. • Faculty are not able to receive an overall compilation of their evaluation results

for peer review…if they teach in more than one department (the use of varied forms across campus, Green Form, SIR-II form cannot be aggregated).

Paper-Based Process & Annual Costs

Costs

45,000 Scantron forms (both SIR-II and Green) $18,000

Header sheets

$1,250

Scantools software and scanner maintenance $2,500

Update scanner equipment (5-6 years @$7,000)

$1,250

Total Personnel costs (prepare header sheets; distribute, collect, scan and process evaluation forms)

$20,000

Total Annual Costs $43,000

Electronic Course Evaluation Pilot

Summer 2012 (Faculty: 91 | Students: 1357)

Student Interface

Automated Email to Students

Dear [first] [last],

You are being asked to complete an online course evaluation for your Summer courses. By clicking on the link

below you will be taken to a list of courses for which you have evaluations to complete. Click on the title of the

course to access the evaluation. Each course evaluation should take no longer than 5 minutes to complete.

Click here to access the evaluation or login at https://usm.campuslabs.com/courseeval

Thanks for your participation. If you have questions about the course evaluation process, please contact the

office listed below.

Office of Academic Assessment

General phone: 780-4383

Email: susank@usm.maine.edu

Mobile Interface

Mobile Interface

Faculty Interface

Reporting

Outcomes of Summer ’12 Pilot

• Demonstrated infrastructure to support electronic evaluation system • Minor technical glitches/problems can be worked out • Faculty pleased to receive their reports immediately after final grades • Response rate: 46% (low, but double compared to UC online survey, summer 2011)

Benefits of Electronic Version

• Quick turn-around; several breakdown reports available for faculty • Departments can choose their own questions; updating instrument is

convenient. • Students can provide feedback over a 1-2 week period; surveys can

be completed outside of class on mobile devices.

Comparison

Paper-based Electronic

Annual Cost $43K+ $31K

Personnel 1 halftime + multiple AAs 1 halftime

Turn-around time 6‒14 weeks Immediate

Instruments * Few variants Multiple

Reporting Simple Custom-designed

Student Experience Bubble sheets + A4 pencils Digital; multiple devices

Submission Window One-time Two-week window

Response rate * 65-75% ?

Response Rates at Other Institutions

Semester Institution Paper-based Online: Year 1 Incentive Online: Year 1+

FA 04 MS State 88% 42% Bookstore Voucher 97%

FA 05 Harvard 89% 65% View grades early 96%

FA 06 U of Kent 82% 46% University event tickets 95%

Paper-based

Transition period

Electronic

Res

po

nse

Rat

e

Strategies for Improving Response Rates

Common strategies being used at campuses: • Frequent email reminders, and announcements posted on campus • Allowing students to view grades at an earlier date • Incentives: campus raffle, gift cards, bookstore vouchers, tickets

for campus events, coupons for food • Posting how the evaluation results are being used on campus • Students able to use a mobile device for completing their evals • More instructor involvement….emphasizing the importance of

student feedback for class improvement

Next Steps

Next Steps

Platform Instrument Development

Possible Model for an Electronic

Course Evaluation System

University-wide Items

Department Items Each department selects items, in addition to Green Form items

Using all or some items on the USM’s Green Evaluation Form

USM Course Evaluation Landscape

Green Form

University-wide Items (Green Form)

Department Items Department Items

Custom SIR II (proprietary)

{paper-based} {electronic}

Web-based App’s

Phase 1 – Department Chairs Phase 2 – Faculty

Two electronic options available: • Green Form only • Green Form with additional Dept. items

• Faculty will receive a link to the Phase 2 App

• Faculty will be able to create/choose/vote from a pool of items.

• The items are agreed upon by the Department

• Items are added to the electronic course evaluation

Next Steps

January

• Meetings with USM Department Chairs

• Department Chairs App distributed for discussion

February

• Departments make decisions about electronic form

• Faculty App distributed to Departments who want additional

items via electronic version

• Options for vendors explored and in-house options

March

• For those participating…Departments make decisions

about their items

• Vendors and in-house options selected

April

• Remaining Departmental responses collected to create a

draft electronic course evaluation format

• Begin planning the Summer ‘13 pilot

• If desired by Departments, further discussion about fall ‘13

course evaluation plans.

Discussion