CQF – QMT Meta- data standard

JISC PALS - R&D by eCom ScotlandProject: Blooming CQF

Project Aim

The project's primary aim is to specify a system

which supports the calculation of a SCQF level when supplied with a CQF/SCQF-compliant repository of questions which make use of Bloom’s Taxonomy (BT).

Blooming CQF-QMT (Credit Qualification Framework – Question Meta Tags)

Key Deliverables1. Develop a specification for a new metadata standard

CQF-QMT for questions held in assessment banks.

2. Produce a prototype web service based on Bloom’s Taxonomy and linked to SCQF level, to give a representative output on question levelling.

Why is this of interest?

• Rise in the use of randomisation of question banks for e-assessment - SCQF levelling requirement identified.

• Embedding of methodology into good practice for assessment authors.

Blooms taxonomy – Cognitive domains

SCQF - Framework

Web service

• A web service was developed to return a list of metadata in an XML format, utilising the Bloom’s query approach.

• Used to tryout assessment questions to check values of existing question banks.

• Items of interest were around how the output of the values from the blooms taxonomy could possibly be referenced the SCQF level.

QTI- Input /OutputQuestion Test Interoperability

• Provides a well documented content format for storing and exchanging items independent of the authoring tool used to create them.

• Supports the deployment of item banks across a wide range of learning and assessment delivery systems.

• It provides a well documented content format for storing and exchanging tests independent of the test construction tool used to create them.

QTI Input / Output

• It supports the deployment of items, item banks, and tests from diverse sources in a single learning or assessment delivery system.

• It provides systems with the ability to report test results in a consistent manner.

Learning Object Model used

Activity Diagram

Load Question

Get Keywords Apply Mapping

[Success]

Generate Metadata

[No Success]

CQF Level Generator

Use Cases

Load Question

Blooming CQF-QMT

IdentifyingMappings

Identify SCQF Level

Generate Metadata

Parse Question«extends»

Top Package::User

Top Package::System

Generating Keywords

Use Case Diagram.

Prototype – http://bloomingcqf.ecomscotland.net

Natural language results

Negative

• After evaluation of a range of electronic assessments across a number of curricular areas, it was determined that the relationship between the language used in assessment stems and SCQF / QCF levels could not be accurately or consistently identified through the proto-type software tool at present.

Reasons for difficulties • The following reasons were identified –

– There was often a limited amount of text within existing question stems for the software to parse and identify the level.

– The use of language within the question stems did not always reflect the learning domain / level being addressed and, as a consequence, provided no indication of the potential SCQF / QCF level.

– The language used in assessments at different SCQF / QCF levels was often similar / the same therefore providing no discrimination of SCQF / QCF level.

– As a consequence, the SCQF / QCF level could only be consistently and accurately identified through the tool and reference to programme / unit descriptors.

Positive

• Where assessments were well designed, it was possible, though parsing the question stem, to identify the Bloom’s Domain and associated Level and the relevant SCQF / QCF generic outcome / indicator using the prototype

Good Practice Guidelines

• Through discussion in practitioner focus groups and the delivery of workshops in assessment design, it was clear that there was a value to practitioners in identifying these characteristics to support assessment design, alignment with other aspects of the curriculum (such as programme and unit descriptions and learning materials) and to ensure that assessments are deployed to address specific learning objectives

Further information

• http://bloomingcqf.ecomscotland.net• http://www.jisc.ac.uk/whatwedo/progra

mmes/pals3/bloomingcqf.aspx