SCROLLA 16th June 2004

The Future of Computer Aided Assessment

Helen AshtonRuth Thomas

Heriot-Watt University

SCROLLA 16th June 2004

Why Assess?• Diagnosis• Support for Learning• Certification of Learning• Quality Assurance

SCROLLA 16th June 2004

Assess How?• Traditional exams, open book

exams, quiz(multiple choice/true-false), essays, reviews, reports, practical work, portfolios, presentations, vivas, performances, group work, self assessment, peer assessment...

SCROLLA 16th June 2004

Computerising – why?

• Reduction of teacher workload • Flexibility• Immediacy• Accessibility• Reuse

SCROLLA 16th June 2004

Computerising – what?

Which bits can be automated?– Delivery– Data Storage/Collection– Customisation/Reproduction– Marking– Feedback

SCROLLA 16th June 2004

PASS-IT• Online delivery of NABS• Online, incremental data collection• Randomisation for reuse• Some automatic marking

(including partial credit)• Immediate & post assessment

feedback

SCROLLA 16th June 2004

Process• Development with Subject Specialists

– aligning learning objectives• Validation & verification procedures

– marking schemes– usability

• Formative practice -> summative exam

SCROLLA 16th June 2004

“Like many innovations in their early stages, today’s computerised tests automate an existing process without reconceptualising it to realise the dramatic improvements that the innovations could allow”

Randy Bennett (1998)

SCROLLA 16th June 2004

• How can reflection be developed? (learner and teacher)

• Are there new opportunities to assess a wider range of learning outcomes (in CAA)?

• Are there approaches to assessment that were impossible/impractical?

SCROLLA 16th June 2004

Developing Reflection

• During assessment– Immediacy– Some Automation

• Post assessment– Immediacy– Repurposing (reporting, reflection)

• Example

SCROLLA 16th June 2004

SCROLLA 16th June 2004

SCROLLA 16th June 2004

SCROLLA 16th June 2004

Developing Reflection

• A wealth of information!– What do we record?– How do we report it?

• Further developments– Assessing features of answers to give

individualised feedback– Individual appropriate experiences

• Where else can we go?• How do we get to “the guide on the side”?

SCROLLA 16th June 2004

Higher Order Skills• Automation of existing techniques:

– Objective testing– Mostly of lower order skills

• What about– understanding,– application, – analysis, – evaluation and – creation.

SCROLLA 16th June 2004

Higher Order Skills• Simulations –

– exploratory environment, realism, interactivity.

• Aim total integration– 2 way communication– Configure simulation starting state– Simulation state available for

marking and storage.• Example

SCROLLA 16th June 2004

SolarGeom example• Randomisation of state• Learner must devise strategy to

determine latitude.– Day length, seasonal differences, etc.

• Hypothesis, prediction, test, revise• More than one skill involved

– Understanding, analysis, evaluation

SCROLLA 16th June 2004

Authentic assessment

• Realistic task – greater complexity and duration.

• Assessing in the same environment in which learning takes place.

• Example

SCROLLA 16th June 2004

Authentic assessment example

• Calculating the activation energy of a reaction

• The same simulation can be used with different experimental data

• Issues – Support– Feedback

SCROLLA 16th June 2004

Questions• What should we be assessing?• How do we integrate learning and

assessment?• Can CAA ever be as good as a 1:1

session with a teacher/expert?• By concentrating on automating

the current process are we strangling innovation?