11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007

BENEFITS AND OBSTACLES: FACTORS AFFECTING THE UPTAKE OF CAA IN UNDERGRADUATE COURSES

John Dermo,

Pathfinder Project Team, University of Bradford

j.dermo@bradford.ac.uk

2

First, a plea for your help….

“The idea being that authors use the conference to air their work and receive feedback to enable judgements to be taken on progress to date and how things may be taken forward in future.”

(extract from Submissions: 11th CAA Conference)

http://www.caaconference.com/submissions/index.asp

3

Background to the Research Project

University of Bradford

• e-Strategy

• VLE = Blackboard ™

• CAA = QM Perception™

4

Developments in CAA @ UoB so far include:

• Deciding on Questionmark™ Perception™ as our supported enterprise level software for online summative assessment

• Encouraging and supporting its use in formative assessment and feedback

• Centralising the administrative support for all summative assessments in our Examinations Office

• Implementing Questionmark™ Perception™ version 4.3 with a server configuration to ensure security and reliability

5

Pathfinder Project @ UoB

Investment in

1. CAA

2. Enhancing FYE

March 2007-April 2008

6

Pathfinder Project: CAA Aims and Objectives

to embed support processes for e-assessment

to develop the administrative and support systems to ensure reliable and secure large-scale implementation of CAA with first year students so that we can subsequently roll this out to all students

7

Pathfinder Project: CAA Deliverables

A system to support the effective implementation of formative and summative computer-aided assessment across the university, consisting specifically of:

• baseline study and needs analysis report on use of CAA at UoB• research data (quantitative and qualitative) on use of CAA in undergraduate

courses• official guidelines for CAA: admin and support functions to ensure resilient and

manageable online assessment routes• pedagogical guidelines (“best practice”) document for recommended use of

software• training sessions for instructors (Sept/Jan/May)• induction and training sessions for invigilators• in-house training materials and web-based resources for use of software• case studies of institutional implementation of CAA• banks of online assessment materials

8

Aims of my research project

• gather data on attitudes towards computer-assisted assessment

• to identify drivers and obstacles to the uptake of CAA

• test various widely-held assumptions about attitudes towards e-learning

• inform Pathfinder project• anticipate risks

9

Methodology

• descriptive

• survey opinions of comprehensive range of interested stakeholders:– students, academic staff, administrators,

technical and support staff, management and financial departments.

• primarily a qualitative approach

10

Phases of research

• Informal interviews, desktop research to identify key areas of interest

• Online questionnaires

• Follow-up interviews to investigate issues arising

11

Challenges: methodology

• Quantitative vs Qualitative

• Positivist vs Anti-positivist

• Survey vs Case study

How can we draw generalisable conclusions based on individual personal opinions?

12

Challenges: methods

• Mode of delivery (online vs paper)• Item design (open vs closed)• Subject groups and sampling• Piloting• Assuring confidentiality• How to survey opinion on something never

experienced?

13

Important Note:

Spend time at the outset to make sure that the research tools are reliable and valid before moving to data collection.

All too often researchers rush into the data collection stage, only to find that the data they are collecting are useless and any findings based on it equally so.

14

Initial findings and looking ahead

• research still at initial phase• key areas of interest being put to subjects to

inform the main survey questions to come• reviewing the literature• It is hoped that feedback from conference

delegates will be able to feed into this process!!

15

Drivers

savings in human and financial resources

improved reliability in marking

ease of production of results and item analysis data

ease of creation of different versions and randomised assessments

recycling assessments positive backwash effect on teaching and learning

appeal to “digital native” students possible benefits for recruitment and retention

potential of portfolio assessment accessibility issues

encouraging good assessment practise concerning item banking and item analysis

16

Obstacles 1

limited suitable task types inability to assess higher level skills in a valid way

high risk of technical failure initial outlay of time

steep learning curve for instructors high cost of software licenses and support plans

difficulty in convincing examination boards and QAA concerning issues of quality

anonymous submission of assignments

security issues – e.g. passwords / collaboration / collusion / cheating / impersonation

item banking requiring more effort and time

technical expertise required of instructors

lack of immediate technical support

17

Obstacles 2

difficulties for administrators difficulties for invigilators

training implications accessibility issues

health and safety issues difficulty of instructors in moving away from traditional task types

issues of task design threat that CAA will be used to justify increased class sizes or staff reductions

lack of an agreed and enforced institutional policy

discrimination against “non-digital native” students

limited availability of Internet-connected computers at home, in halls of residences, on campus

availability of large computer rooms for examinations

18

Issues to consider

• Does the cumulative effect of the barriers outweigh that of the drivers?

• How to interpret the data in a meaningful way?• To what degree can e-assessment add value to

the learning experience? • Do some factors work as drivers under some

circumstances but as obstacles in others?

19

Closing comment…

The researcher is keen to involve the input of other experienced practitioners and researchers in the field of Computer Assisted Assessment by means of this conference, and hopes to work this short paper up to a full paper submission for the next event in 2008.

20

References

Cohen, L., Manion, L., Morrison, K. (2003) Research Methods in Education. London: RoutledgeFalmer.

Conole, G. (2003) Understanding Enthusiasm and Implementation: e-Learning Research Questions and Methodological Issues in JK. Seale (ed) Learning Technology in Transition: from Individual Enthusiasm to Institutional Implementation, Abingdon: Swets and Zeitlinger, pp. 129-146.

Dillman, DA. (2007) Mail and Internet Surveys: the Tailored Design Method. Hoboken, New Jersey: John Wiley & Sons.

Frankfort-Nachmias, C. and Nachmias, D. (1996) Research Methods in the Social Sciences. London: Edward Arnold.

Oppenheim, AN. (2000) Questionnaire Design: Interviewing and Attitude Measurement. London: Continuum International.

Verma, GK. and Mallick, K. (1999) Researching Education: Perspectives and Techniques. London: Falmer Press.

21

Thank you

Contact: John DermoPathfinder Project,

Learner Support Services,

University of Bradford

Email : j.dermo@bradford.ac.uk