How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience...
Transcript of How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience...
![Page 1: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/1.jpg)
How do we achieve accuracy in the content, presentation and
instructions of examination papers?
Sonya StewartRichard Greatbanks
University of Otago
![Page 2: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/2.jpg)
Overview
• Introductions
• Context – the examination experience
• Process – the exam paper system
• Analysis – data collection and analysis approaches
• Future improvement actions and further analysis
![Page 3: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/3.jpg)
Introductions
• Sonya Stewart, Group Leader – Student Administration
• Richard Greatbanks, Associate Dean (Academic) – OBS
![Page 4: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/4.jpg)
Context - the Examination Experience
• Examinations are integral to the university experience– Academically essential in our current university system
– Can account for up to 70 percent of a programme’s assessment• Often considered the only true reflection of a student’s individual capability
– Might become even more prominent in the future• Emergence of ghost writing sites and increased access to technology
– Exams are therefore a ‘key business process’ for a university
![Page 5: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/5.jpg)
Context - the Examination Experience
• From a student’s perspective they are integral too– They are often a ‘rite of passage’
– Examinations mark the ‘end of study’• A semester, a paper, an academic year, or even their final degree
– Examinations often the last interaction before graduating university• They can be stressful or emotional
• They are meant to test the student’s capability
– Will directly influence the class of degree a student obtains
– Will influence early career choices and opportunities
![Page 6: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/6.jpg)
Context - the Examination Experience
• As a university community, we therefore have an obligation to offer the best examination experience for our students
– Provide examination papers which are • Appropriate and relevant to the subject of study
• Marked fairly and consistently across the class
• Set out with clear accurate instructions
• Offered complete and without errors
• To any student, these would be considered basic expectations
![Page 7: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/7.jpg)
Process - the Examination Paper System
Department
Academic responsible
for exam
ExaminationsOffice
(PE errors)
Student examination(ED errors)
Exams office query
Department rework
Day of exam query to Dept/ Academic and response
![Page 8: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/8.jpg)
Analysis - the Examination Paper System
• Data collected by Sonya Stewart
– Number of errors in each examination paper submitted to the examinations office between S1 2015 and S1 2019• Number of papers with at least one error (defective papers)
• Number of errors per paper (defects)
• Type of error detected – admin or content
• Time when error detected – pre-exam (PE) or exam day (ED)
![Page 9: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/9.jpg)
Exam paper data analysis - 1
0
100
200
300
400
500
600
2015 2015 2016 2016 2017 2017 2018 2018 2019
Number of exam paper in each semester
0
20
40
60
80
100
120
140
160
180
200
2015 2015 2016 2016 2017 2017 2018 2018 2019
Number of exam papers - PE and ED errors
Note different scales
![Page 10: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/10.jpg)
Exam paper data analysis - 2
0
20
40
60
80
100
120
140
160
180
200
2015 2015 2016 2016 2017 2017 2018 2018 2019
S1 2015 to S1 2019 time based variationPE and ED errors – Number of papers
S2 2018 indicates 176 papers were identified with at least
one error at the PE stage
S2 2018 indicates 26 papers were identified with at least
one error at the ED stage
![Page 11: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/11.jpg)
Exam paper data analysis - 3
0
50
100
150
200
250
300
2015 2015 2016 2016 2017 2017 2018 2018 2019
PE Error type - Admin or ContentNumber of errors (Defects)
admin content
0
5
10
15
20
25
2015 2015 2016 2016 2017 2017 2018 2018 2019
ED Error type - Admin or ContentNumber of errors (defects)
admin content
Note different scales
![Page 12: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/12.jpg)
Exam paper errors - mean system performance
P1 Department
Academic responsible
for exam
P2 Examinations
Office
P3Student
examination
PE Error query
Department Rework 31%(145 papers)
ED Errors 3% (16 papers),of which 2% (9 papers) were initiallyconsidered error free at P2 PE check
100% all papers
Day of exam query to Dept/ Academic and response
![Page 13: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/13.jpg)
Process analysis – Rolled Throughput Yield
P1 Department100% Yield
P2 Exam Office69.3% Yield
P3 Examination96.7% Yield
Total Process RTY = 1.0 x 0.693 x 0.967
Total Process RTY = 0.670 or 67% yield
![Page 14: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/14.jpg)
Where as for S1 2016 the RTY is only 66.4%. We could expect 66.4% of all papers to pass through the process
without failing at any stage
Process analysis – Rolled Throughput Yield
2015 2015 2016 2016 2017 2017 2018 2018 2019
Rolled Throughput Yield S1 2015 to S1 2019
So for S2 2017 the RTY is 72.7% - that is we could expect 72.7% of all
papers to pass through the process without failing at any stage
RTY represents the capability of a system to produce good product through all process steps
![Page 15: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/15.jpg)
Future actions and opportunities
• Advanced Quality Planning (AQP)• Analysis of key business processes
– Ideally in advance of product or service delivery
– Develop measures which indicate system capability
– Ability to identify and minimise high risk factors
• Focus on placing responsibility for quality at source– Commination of PE and ED performance across divisions
![Page 16: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/16.jpg)
Development of Risk Priority Number• An RPN defines the likely levels of risk given known factors
– Severity of error • Missing page number = 1, incorrect exam duration = 5, missing formula = 10
– Ability to detect error • Spelling mistake = 1, incorrect formula = 10• Qualitative or discursive questions = 3, quantitative or science questions = 9
– Consequential effect - This could be class size or final year exams• Small class < 20 = 3, large class > 500 = 10• First year papers = 3, Final year papers = 10
– Each factor assessed out of 10 and multiplied to derive a total RPN out of 1000
• Calculation of RPN can then be used to identify higher risk combinations in advance of the exam paper being produced– Effectively offering a alternative process for high risk papers, subjects or departments
![Page 17: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/17.jpg)
Future improvement actions
• Improvement actions must emphasize placing responsibility with the academic team– Develop policy and standards to define expectations
– Offer training to academic staff to support better practice
– Establish an audit regime for high RPN papers, subjects and departments
– Continue to collect data and disseminate across departments
– Use RTY each semester as a measure of system capability
Department
Academic responsible
for exam
ExaminationsOffice
Student examination
Exams office query
Department rework
Day of exam query to Dept/ Academic and response
![Page 18: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/18.jpg)
Conclusions
• Ultimately, this requires a cultural change• Place responsibility with academic team
• Further analysis by division, department and subject
• Continue to collect data and measure system capability
• Potential to benchmark with other departments, divisions or institutions
• Voice of the Customer (VOC) data on exam experience
![Page 19: How do we achieve accuracy in the content, presentation ... · Context - the Examination Experience • As a university community, we therefore have an obligation to offer the best](https://reader031.fdocuments.us/reader031/viewer/2022041802/5e51f1257a1c6570aa43f2a9/html5/thumbnails/19.jpg)
Thank you for your attention
Questions?