Improving performance and the student assessment experience:

multi-choice testing which rewards partial knowledge increases satisfaction and reduces anxiety

without introducing gender bias

Dr Geertje van Keulen

College of Medicine

MCQ assessments in higher education

• MCQs are used to objectively measure:

o Factual knowledge

o Learning ability

o complex, high-level learning outcomes

• Often the sole assessment of theoretical knowledge for

level 1 UG students in large classes

• Variety of different types of mcq tests exist

Ben-Simmon et al 1997Bradbard et al 2004Ng & Chan 2009a

MCQ with Single (best) Answering method

Most commonly used MCQ test

Student marks only one option, the correct answer

Negative marking inhibits ‘pure guesswork’

Several issues:

o Lower student confidence in answering

o No chance to show partial knowledge

o Reduced test score reliability through ‘educated guesswork’

o Reduced student satisfaction

o Unfair? Gender bias?

MCQ with Elimination Answering method

Discriminate between all possible levels of knowledge (full, partial,

and absence of knowledge, fully and partially misinformed)

Students asked to eliminate up to 4 possible incorrect answers

Elimination of the correct answer incurs a penalty

Test score reliability improved as guessing is not required, guessing is

discouraged

Student confidence improved?

Improved tool for teaching staff?

Jennings & Bush 2006, Ben-Simon et al 1997,Coombs et al 1956, Dressel & Schmidt 1953

Fairness of MCQ testing?

Comparison of MCQ test performance in some subject

areas suggested introduction of gender bias (Ng & Chan

2009b)

However, introduction of gender bias not absolute but

appear to depend on:

o subject area

o instruction/scoring condition

o question difficulty

Data missing for life sciences subject area

Objectives – Student related

• To compare student performance in life sciences in

identically-worded single and elimination answer MCQ tests

with negative marking

• To assess whether single and elimination answer MCQ

tests in life sciences result in gender bias in performance

overall

• To assess student satisfaction and anxiety/stress levels

for and between each type of MCQ test

Approach

• MCQ tests were voluntary

• Students were examined on course material taught in the first

3 weeks of the module

• Tested in week 5 of the module

Approach

• Level 1 students consisted of the whole life science cohort

• Level 2 cohort of biochemistry and genetics students

• Student evaluation surveys

‐ Anxiety

‐ Stress

‐ Satisfaction

• Incentives

Answer methods and Scoring Mechanisms

• 1 test question sheet with 2 answer sheets

o Single answer method

o Elimination answer method

o Sheets were scanned, electronically read,

scored and input into Access database

• Scoring grid for MCQ tests with 5 possible answers:

Datasets

• 142 students (72%) participated in the Level 1 MCQ tests

‐ 138 paired answer sheets incorporating 25 questions

‐ 63 males and 75 females

‐ 138 unpaired student evaluation questionnaires

• 40 students (80%) participated in the Level 2 MCQ tests

‐ 40 paired answer sheets incorporating 25 questions

‐ 26 males and 14 females

‐ 40 unpaired student evaluation questionnaires

• Further data analysis with SSPSv19

MCQ scores distribution is normally distributed

• Student performance between the two types of tests is normally distributed for level 1 and level 2 students

• Kolgomorov-Smirnov test conducted (p>0.05 for all)

Not significantly different from the normal distribution

MCQ test P-value

L1 SA 0.2

L1 ET 0.08

L2 SA 0.2

L2 ET 0.2

Comparison of student cohort performance

• Cohort average scores are relatively poor

• Students perform statistically significantly better in Elimination tests than in Single Answer tests

• L1=rel. 7% better• L2=rel. 9% better

Paired t-testsL1: P<0.0005 andL2: P=0.023

Mean (±SE) of the performance

Single Answer Elimination33

33.534

34.535

35.536

36.537

37.538

MCQ Test

% S

core

single answer elimination41

42

43

44

45

46

47

48

49

MCQ test

% sc

ore

44.0

47.4

34.6

38.0

Does MCQ testing introduce gender bias in the life sciences subject area?

Independent t-tests were conducted to determine if gender effects performance:

L1: males slightly better in both tests, but not significantL2: females slightly better in both tests, but not significant

NO significant gender bias in Single Answer MCQ testingNO significant gender bias in Elimination MCQ testingNO significant gender bias in Level 1 Life Sciences student cohorts (9 degrees) NO significant gender bias in Level 2 Biochemistry & Genetics cohorts

Student surveys on experiences

• Immediately after sitting the MCQ tests, students completed an evaluation survey

• MCQ evaluation questions investigated:

o The students’ experience/attitude/emotion towards each type of MCQ test

o Comparison of the two answering options

• A second survey was completed by the students following the announcement of test results

Student experience survey questions

• The answering options were confusing• There is a high chance of getting answers right• I got distracted by thinking about the best tactics for getting a high mark• It makes you think more about your answers• It made me feel more relaxed, knowing that I can get a reasonable mark• I was scared to answer a question• I was confident to answer a question• It made me feel motivated• My stress levels were high• It is a fair test• Loosing marks for guessing detracted from the legitimate marks for

knowing the right answers to some questions• I prefer single answer testing• I prefer elimination testing• Many more....

• Students gave scores of: 1=strongly agree 2=agree

3=neutral 4=disagree5=strongly disagree 6=not applicable/don’t know

• One sample sign test performed with Null hypothesis: the median

of the scores was 3 or neutral

Student satisfaction surveys

Strongly

Agree

Agree

Neutra

l

Disagre

e

Strongly

Disagre

e

Don't Know/N

ABlan

k0

10

20

30

40

50

60

Q14 & Q32

L1 ETL2 ETL1 NCL2 NC

% R

esul

ts

Selection of student experience responses (significant p<0.05)

• Most students agreed they• prefered Elimination Testing over Single Answer testing• felt Elimination Testing better reflects their knowledge• felt both types of tests are fair• felt scared in Single Answer testing but were neutral for Elimination• enhanced their critical thinking skills by Elimination• felt confident to answer in both styles

• Most students were neutral • Towards loosing marks for guessing detracted from the legitimate

marks for knowing the right answers to some questions (except L2 for Single Answer)

• about the questions being easy to answer• about being motivated by both types of answering methods• About Single Answer method enhancing their critical thinking skills

Conclusions • Both student cohorts get significant advantage answering elimination

MCQ tests compared to single answer tests with negative marking (L1 – P <0.0005 and L2 - P=0.023)

• There is no significant difference in performance between genders in either cohort for either MCQ test (L1 - P=0.309 and L2 - P=0.779)

• Students prefer Elimination testing to Single answer MCQ and feel Elimination testing reflects their knowledge better

• Students feel scared to answer Single answer MCQs

• In general, the student learning experience can be improved through the incorporation of the elimination answering methods in MCQ tests

Acknowledgements

•Dr Liz Bond - Researcher

•Dr Owen Bodger – (bio)statistician

•Dr Hugh Jones - Examinations Coordinator

•Dr Colin Restall – Biochemistry Module Coordinator

• Dr Ed Dudley – Biochemistry Module Contributor

• Dr Jess Murtagh – Learning Technologist

• Support staff from Education Unit

• 7 further academic staff from Genetics and Biochemistry

• With many thanks to the tutors in Biosciences

Contact us at: g.van.keulen@swansea.ac.uk or

a.e.bond@swansea.ac.uk