Open Standards and Specifications for Outcome-

based Assessment

Israel GutiérrezUniversidad Carlos III de Madrid

Madrid, April 12th 2010

Standards and Specifications

Computer-based assessmentReduce effort and costs: permit continuous assessmentAssessment re-use: interoperability needed

Variety of assessment formatsProprietary: MoodleXML, HotPotatoes, Blackboard Ad hoc: OpenMark IMS QTI

Open Standards and Specifications for Assessment

Outcome-based Assessment Universities developing an

outcome-based learning strategy Adaptation to European

Space for Higher Education Improve course structure for

accreditation (ABET)

Creation of new coursesLearning outcomesAssessment as a key process

Outline

Analysis of assessment standards and specifications

Usage of assessment formats in Europe

Interoperability issues

Next generation assessment scenarios

Best practices

Methodology

Analysis of Assessment Standards and Specifications

Metamodels (UML diagrams) from MDAUnderstand better studied

formatsAbstraction from syntax

(XML binding)Focus on content

Studied Assessment Formats

IMS QTI

OpenMarkSuML

FML

General purposeAssessment purpose

QAML

Qualitative ComparisonKey features

Multiple response allowanceCorrect response indicationMetadata capabilitiesHybrid question management

Question types (from FREMA)Constructed: essay, short answer, FIB, etc.Constrained: MCQ, MRQ, TF, matching, etc.Miscellaneous: simulations, practical

sessions

Assessment Formats Comparison

Assessment Formats Usage in Europethe current picture – methodology and sample description

• Data collection• Personal interview• In-depth report

• Qualitative analysis• Sample characteristics

• Higher education• Multicultural• Multilingual

Series10

2

4

6

8

10

12

14

Assessment Scenarios

Evaluation Scenarios

Tools & Services

Pilots

Sample size

Project results: WP6

Geographical Distribution

Assessment Formats Study: Resultsthe current picture – key conclusions

Yes22%

No78%

Re-use of assessement resources

• Scarce re-use of assessment resources• Scarce acceptance of IMS QTI specification• High technological diversity

– LMS, ad-hoc tools, etc.• Diversity of assessment methodologies

– formative/summative, various types of activities, face-to-face/blended/elearning settings, etc.

Interoperability Issues: IMS QTIScarce use of the specification

Incomplete support in LMS i.e. adaptive material

Flexibility handicaps

QTI profiles

Versions instability: 1.2.1 or 2.1?Versions are not backwards compatible

Assessment Interoperability

Moodle

.LRN Clix

QTIexport QTIMoodle

QTIlrn QTIclix

QTI 1.2.1

QTI 1.2.1

QTI 1.2.1

Next Generation Assessment Scenarios • Hints

• Adaptation• Instructional

design

Assessment Resources Interoperability ProposalAssessment resources

Objective: Gather and develop best practices about interoperable content as support to higher education and learning outcome oriented

Results• ICOPER Reference Model• Gathering of recommendations• Open ICOPER Content Space• One of the most successful eContent+ projects since 2004

Open ICOPER Content Space• Federated repository of educational resources

Outcome-based Assessment AppUse cases

Search for assessment resources in a repository Filter by intended learning outcomes Filter by assessment methods

Import/re-use of assessment resourcesPublish/share assessment resources

Online demo: http://mozart.gast.it.uc3m.es:9000/

Outcome-based Assessment AppUse cases (cont.)

Annotate assessment resources Peer review Teacher reflection Student feedback

Update learner’s profile

StakeholdersFaculty membersCourse developers

Online demo: http://mozart.gast.it.uc3m.es:9000/

Outcome-based Assessment App

Learning outcome-based search

Learning outcome profile update

Implementation for dotLRN based on

ICOPER LOM profile

Outcome-based Assessment App: EvaluationEngineering evaluation

ImplementersTool developers

End user evaluationFaculty membersCourse developers

Epistemological evaluation ResearchersStandardization bodies

ICOPER Assessment appAssessment resources should include information

about Intended learning outcomesRecommended assessment methodsAnnotations about resources

Usage of integrated environment for outcome-based assessment

Best Practices for Assessment

Thank you for your attention!

http://www.it.uc3m.es/igrojas/index.htmligutierrez@inv.it.uc3m.es