La Casita de Mis Viejos (Handwritten and Typed)_Tango_Astor Piazzolla
Are typed and handwritten Just to let you know: exams...
Transcript of Are typed and handwritten Just to let you know: exams...
![Page 1: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/1.jpg)
2 Oct 2019: 07:00AM UTC/GMT Are typed and handwritten exams equivalent? Dr Liz Masterman & Dr Jill Fresen University of Oxford, UK
Justtoletyouknow:Bypar'cipa'nginthewebinaryouacknowledgeandagreethat:
Thesessionmayberecorded,includingvoiceandtextchatcommunica'ons(arecordingindicatorisshowninsidethewebinarroomwhenthisisthecase).Wemayreleaserecordingsfreelytothepublicwhichbecomepartofthepublicrecord.Wemayusesessionrecordingsforqualityimprovement,oraspartoffurtherresearchandpublica'ons.
Webinar Series
e-Assessment SIG
Webinar Hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Academic Lead: Digital Assessment, University of New South Wales m.hillier[at]unsw.edu.au
![Page 2: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/2.jpg)
Are typed and handwritten exams equivalent? (and if not, does it matter?)
Dr Liz Masterman & Dr Jill Fresen 2nd October 2019
Centre for Teaching and Learning https://ctl.ox.ac.uk
![Page 3: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/3.jpg)
Outline of today’s session ■ Context: trial of e-exams at the
University of Oxford
■ Methodology for the literature survey
■ Summary of findings: § Students: sitting an e-exam § Staff: marking an e-exam § Performance: higher or lower?
■ Implications of our findings
![Page 4: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/4.jpg)
Context: A trial of e-exams at Oxford ■ Invigilated essay-based examinations in which students
type their scripts either on their own laptops (BYOD) or on devices provided by the institution.
■ Face-to-face setting: e.g. exam hall
![Page 5: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/5.jpg)
Context: A trial of e-exams at Oxford ■ Summer 2017: ‘Landscaping task’ – literature on practical and
cognitive perspectives
■ April 2018: Trial § Inspera + Safe Exam Browser § 4 e-exams: >70 undergraduate and taught master’s students
■ Formative: Theology, Computer Science (coding) ■ ‘Mocks’: 2 modules in MSc Law & Finance
§ Typed only § BYOD and/or University-supplied computers
■ May 2018: Evaluation
![Page 6: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/6.jpg)
Are typed and handwritten exams equivalent? …the need for equivalence to be fully determined to ensure that overall performance outcomes are matched (Noyes & Garland, 2008, p. 1357) 1. Students: differences in
§ Cognitive process? § Length and style of scripts?
2. Markers: differences in § Perception and attitudes re scripts? § Cognitive process?
3. Marks: § Is there a difference? § If yes, is it important?
![Page 7: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/7.jpg)
Searching the peer-reviewed literature ■ Google Scholar ■ 2000 onwards
+ some older ■ Followed up references
and ‘related articles’ on journal websites
■ 40 relevant articles
![Page 8: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/8.jpg)
Students: Sitting an e-exam
![Page 9: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/9.jpg)
The process of writing: a conceptual framework ! Planning � Knowledge retrieval � Translating ideas into text
§ Generating text § Transcription
" Revising
Fluency in generating ideas and writing them down quickly frees cognitive resources to produce ‘reader-based prose.’ The exam environment can place pressure on these resources. (Peverley, 2006)
![Page 10: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/10.jpg)
Writing in an exam: process
■ Reliance on on self-reports by students: § Hand-writers spend more time planning (Lee, 2002) § Typists rough it out and expand later; also pause more (Lee, 2002)
■ Inconsistent or contradictory data between studies: § Typists revise more (Kohler, 2015) vs § Similar proportions of typists and hand-writers revise (Hillier, 2015b)
■ Differences: depend on the individual student (Lee, 2004)
![Page 11: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/11.jpg)
Writing in an exam: product
■ Length: typed = generally longer (various incl. Mogey et al., 2010)
■ Length of sentences; number and length of paragraphs: contradictory (various incl. Mogey & Hartley, 2012)
■ Lexical variation: typed = greater (Charman, 2014)
■ Lexical density: typed = less (Charman, 2014; Mogey & Hartley, 2013)
■ Style: typed = more informal (Charman, 2014; Mogey & Hartley, 2013; but disagreement e.g. Whithaus et al., 2008)
![Page 12: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/12.jpg)
Influence of the tool … lack of fluency in lower order cognitive processes such as keyboarding or handwriting constrains higher order cognitive processes such as planning and reviewing. (Kohler, 2015, pp. 140–141)
■ Typing proficiency has greater impact on performance than computer experience (e.g. Kohler, 2015; Mogey & Fluck, 2015)
§ But faster typists don’t necessarily type more (Mogey & Hartley, 2010)
■ Anxiety re technical failure à more constraints on cognitive processes? (e.g. Hillier, 2014)
■ Impact of an unfamiliar device on typing speed (e.g. Hillier, 2015b) § ‘IT proficiency for assessment’ (Walker & Handley, 2016)
NB inequities in handwritten exams too (Graham et al., 1998; Connelly et al., 2005)
![Page 13: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/13.jpg)
Sharing and discussion (i)
![Page 14: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/14.jpg)
Sharing and discussion (i) How far does your experience reflect the findings of research? For example:
■ Do you – or your students – think that typing affects the way they write their answer?
■ Do you – or your students – think that a typed answer differs in length and/or style compared with a handwritten answer?
■ Are students more (or less) anxious about an e-exam compared with a handwritten one? What reasons do they give?
![Page 15: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/15.jpg)
Staff: Marking an e-exam
![Page 16: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/16.jpg)
Markers’ perception of exam scripts ■ Legibility:
§ Negative impact vs ‘empathy’ re poor handwriting (Lee, 2004; Powers et al., 1994)
§ Errors more visible in typed scripts (various incl. Kohler, 2015)
■ Typed answers visually shorter (Powers et al., 1994) § Positive correlation between length of answer and mark awarded
(various incl. Augustine-Adams et al., 2001)
■ Possible expectation of higher quality in typed answers § Having the exams keyboarded seems to have shifted readers’
expectations away from first-draft writing toward higher expectations associated with texts that have been more thoroughly revised (Whithaus et al., 2008, p. 14; also various incl. Mogey et al.; 2012)
![Page 17: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/17.jpg)
OSM and the process of marking
Support for cognitive strategies (e.g. Shaw, 2008)
■ Skim-reading to extract salient themes and establish overall sense
■ Navigation within and between scripts ■ Both ‘public’ and ‘private’ annotation
![Page 18: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/18.jpg)
Sharing and discussion (ii)
![Page 19: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/19.jpg)
Sharing and discussion (ii)
■ Do you think that markers have higher expectations of typed exam scripts than handwritten ones? § Are your students concerned about this?
■ Does anyone have experience of on-screen marking? Does the tool make marking easier or harder than on paper? In what way(s)?
![Page 20: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/20.jpg)
Performance: Higher or lower?
![Page 21: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/21.jpg)
Students’ performance
■ Marks awarded: § Typed = higher (Augustine-Adams et al., 2001; Charman, 2014; McCann et al., 2002;
Whithaus et al., 2008)
§ Handwritten = higher (Bridgeman & Cooper, 1998; Kohler, 2015; Lee, 2004; Mogey et al., 2010; Powers et al., 1994)
■ Rubrics: § Holistic scoring à handwritten higher
Analytic scoring à typed higher (Lee, 2004; McCann et al., 2002)
■ Differences not statistically significant ■ Influence of research settings and methods
![Page 22: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/22.jpg)
Markers’ performance
…the need to analyze how the medium of reading an exam impacts the raters’ ability to apply assessment criteria (Whithaus et al., 2008, p. 14)
■ Severity: no significant difference (Whithaus et al., 2008; Johnson & various colleagues, 2009; 2012)
■ Accuracy: no significant difference (Johnson & various colleagues, 2009; 2012)
■ Inter-rater reliability: § may be better on typed scripts (Bridgeman & Cooper, 1998)
§ OSM may improve inter-rater reliability (Tisi et al., 2013)
![Page 23: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/23.jpg)
So, are typed and handwritten exams equivalent? The current findings do not indicate whether handwriting and keyboarding are significantly different cognitive processes or not. (Whithaus et al., 2008, p.17)
… the constructs measured in computer and paper modes are not the same. That is, the incorporation of computers into writing assessments involves a new way of thinking about composing processes … Inevitable sources of non-equivalence of the construct … might lead to differences in test performance to some extent. (Lee, 2002, p. 152)
![Page 24: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/24.jpg)
Non-equivalence: implications
… the constructs measured in computer and paper modes are not the same. (Lee, 2002, p. 152)
■ Rule out choice between handwriting and typing (or is students’ right to choose more important?)
■ Make e-exams compulsory and provide opportunities for students to develop typing proficiency (or should this be their responsibility rather than the institution’s?)
■ Rule out choice between marking on paper and on screen (or is academics’ freedom to choose more important?)
![Page 25: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/25.jpg)
Non-equivalence: what to do about it
■ Statistically insignificant differences can matter on grade boundaries – but NB longstanding inequities in handwritten exams
■ ? Adopt e-exams across the board § Use analytics to understand students’ writing behaviours and
academics’ marking strategies § Support students to develop IT proficiency for assessment
![Page 26: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/26.jpg)
Sharing and discussion (iii)
![Page 27: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/27.jpg)
To find out more…
■ E-exams project reports: § Landscape report: https://ctl.ox.ac.uk/project-reports § Evaluation report: please email us
■ Liz’s ASCILITE paper: § https://2018conference.ascilite.org/conference-proceedings/
(full paper #21)
■ Email addresses: § [email protected] § [email protected]
![Page 28: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/28.jpg)
Image credits Slide 1 (title): Liz Masterman Slides 2, 3: © Examinations & Assessments team, University of Oxford
Slide 7: CC BY NC Sebastian https://www.flickr.com/photos/bastispicks/2614766449
Slides 12, 17, 25: pencil parker https://pixabay.com/illustrations/picture-frame-banner-status-badge-3042585/ Slide 14: CC BY SA Alex France https://www.flickr.com/photos/alexfrance/3194662301 Slide 17: CC BY SA Sage Ross https://www.flickr.com/photos/ragesoss/2158796487
![Page 29: Are typed and handwritten Just to let you know: exams equivalent?transformingassessment.com/sites/default/files/files/TA... · 2019-10-02 · Do you – or your students – think](https://reader033.fdocuments.us/reader033/viewer/2022041714/5e49d6278b13cf1e1b0712d9/html5/thumbnails/29.jpg)
Webinar Series
Webinar Session feedback With thanks from your hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Academic Lead: Digital Assessment, University of New South Wales m.hillier[at]unsw.edu.au Recording available http://transformingassessment.com
e-Assessment SIG