The Use of ICT for the Assessment of Key Competences

Irish Presidency Conference Better Assessment and Evaluation to Improve Teaching and Learning19-20 March 2013

Christine RedeckerYves Punie

IPTS: Part of Joint Research Centre of the EC: 7 Research Institutes across Europe

Mission: “to provide customer-driven support to the EU policy-making process by developing science-based responses to policy challenges that have both a socio-economic as well as a scientific/technological dimension”

IPTS- Who we are

Future Learning Spaces in 2020

2006 2007 2008 2009 2010 2011 2012 2013

Learning 2.0 in formal E&T Learning 2.0 in informal settings

Creativity & Innovation in schools in EU

eLearning in the EU10 MS- Digital Competence- Active Ageing & Learning

Tell-Net Teacher Networking

Review ICT Impact

IPTS Studies on ICT for Learning

The Future of Learning

ICT for the Assessment of Key Competences

Digital Competence

Mainstreaming Creative Classrooms

Open Education in Europe

1to1 Learning initiatives in Europe

Mapping Technologies for Learning

DC Follow up

Where we stand

Multiple Choice

Short Answer

Generation 1Automated administration

and/or scoring of conventional tests

Adaptive TestingGeneration 2 Generation 4

Intelligent Tutors

Automated Feedback

Personalised feedback and

tutoring

Learning 2.0

Generation 3Continuous integrated assessment

Data Mining & Analysis

Behavioural tracking

Generation Re-inventionTransformative strategy: use technology to

change test formats

Peer assessment

Collaborative multimedia learning environments

Online collaborationVirtual laboratories

Games

Simulations

Virtual Worlds

Replication of complex real life situations within a

test

Bun

ders

on, I

nouy

e &

Ols

en (1

989)

. The

four

gen

erat

ions

of c

ompu

teriz

ed e

duca

tiona

l m

easu

rem

ent.

In R

. L. L

inn

(Ed.

), Ed

ucat

iona

l mea

sure

men

t (Th

ird e

d., p

p. 3

67-4

07).

New

Yo

rk: M

acm

illan

Ben

nett,

R. E

. (20

10).

Tech

nolo

gy fo

r Lar

ge-S

cale

Ass

essm

ent.

In P

. Pet

erso

n, E

. Bak

er &

B.

McG

aw (E

ds.),

Inte

rnat

iona

l Enc

yclo

pedi

a of

Edu

catio

n (3

rd e

d., V

ol. 8

, pp.

48-

55).

ford

: El

sevi

er.

1990 1995 2000 2005 2010 2015 2020 2025Efficient testing Personalised learning

Learning Analytics

Computer-Based Assessment (CBA) Technology-enhanced learning

Current use of ICT in assessment in schools

Central recommendations on the use of ICT in pupil assessment in compulsory education in primary and general secondary education (ISCED 1, 2, 3), 2009/10Source: Eurydice (2011). Key Data on Learning and Innovation through ICT at School in Europe 2011, Education, Audiovisual and Culture Executive Agency.

There is scope for improvement

1st Generation CBA: Problem-Solving

http://www.worldclassarena.org/v5/flash/09_year_old/backtofront.swf

World Class Arena Sample Test: Problem Solving (9 years)

Examples

http://www.worldclassarena.org/v5/flash/13_year_old/pyramids.swf

Problem Solving 13 years

Generation Re-Invention: Problem-Solving

Examples

1st Generation CBA: Situational Judgement

EPSO sample situational judgement test questions; http://europa.eu/epso/discover/prepa_test/sample_test/index_en.htm

Examples

Generation Re-Invention: e-tray exercise

EPSO

sam

ple

asse

ssm

ent c

entre

e-tr

ay e

xerc

ise

http

://eu

ropa

.eu/

epso

/dis

cove

r/pre

pa_t

est/s

ampl

e_te

st/in

dex_

en.h

tm

Examples

Future eAssessment Strategies

Multiple Choice

Short Answer

Generation 1Automated administration

and/or scoring of conventional tests

Adaptive TestingGeneration 2

Generation 4

Intelligent Tutors

Automated Feedback Personalised feedback and

tutoring

Learning 2.0

Generation 3Continuous integrated assessment

Data Mining & Analysis

Behavioural tracking

Generation Re-inventionTransformative strategy: use technology to

change test formats

Computer-Based Assessment (CBA) Technology-enhanced learning

Peer assessment

Collaborative multimedia learning environments

Online collaborationVirtual laboratories

Games

Simulations

Virtual WorldsReplication of complex

real life situations within a test

Bun

ders

on, I

nouy

e &

Ols

en (1

989)

. The

four

gen

erat

ions

of c

ompu

teriz

ed e

duca

tiona

l m

easu

rem

ent.

In R

. L. L

inn

(Ed.

), Ed

ucat

iona

l mea

sure

men

t (Th

ird e

d., p

p. 3

67-4

07).

New

Yo

rk: M

acm

illan

Ben

nett,

R. E

. (20

10).

Tech

nolo

gy fo

r Lar

ge-S

cale

Ass

essm

ent.

In P

. Pet

erso

n, E

. Bak

er &

B.

McG

aw (E

ds.),

Inte

rnat

iona

l Enc

yclo

pedi

a of

Edu

catio

n (3

rd e

d., V

ol. 8

, pp.

48-

55).

ford

: El

sevi

er.

Learning Analytics

1990 1995 2000 2005 2010 2015 2020 2025Efficient testing Personalised learning

Towards Generation 3: Datamining

Signalshttp://www.itap.purdue.edu/tlt/signals/signals_final/index.htm

Similar: GPS at Northern Arizona University

http://resourceconnect.nau.edu/grade-performance-status.asp

Examples

Towards Generation 3: Behaviour TrackingExample: SNAPPSNAPP is a software tool that allows users to visualize the network of interactions resulting from discussion forum posts and replies. Teachers can rapidly identify patterns of user behaviour. SNAPP can extract all user interactions from various commercial and open source learning management systems, such as BlackBoard and Moodle.

Examples

Towards Gen 4: Intelligent Tutoring Systems

GnuTutor (http://gnututor.com/)OpenSource version of AutoTutorhttps://www.youtube.com/watch?v=k6HDEEnYuIA&feature=player_embedded

AutoTutorhttps://www.youtube.com/watch?v=aPcoZPjL2G8

Examples

Generation 1 Generation 2

Generation 4Personalised feedback and

tutoring

Learning 2.0

Generation 3Continuous integrated assessment

Generation Re-invention

Computer-Based Assessment (CBA) Technology-enhanced learning

Collaborative multimedia learning environments

Bun

ders

on, I

nouy

e &

Ols

en (1

989)

. The

four

gen

erat

ions

of c

ompu

teriz

ed e

duca

tiona

l m

easu

rem

ent.

In R

. L. L

inn

(Ed.

), Ed

ucat

iona

l mea

sure

men

t (Th

ird e

d., p

p. 3

67-4

07).

New

Yo

rk: M

acm

illan

Ben

nett,

R. E

. (20

10).

Tech

nolo

gy fo

r Lar

ge-S

cale

Ass

essm

ent.

In P

. Pet

erso

n, E

. Bak

er &

B.

McG

aw (E

ds.),

Inte

rnat

iona

l Enc

yclo

pedi

a of

Edu

catio

n (3

rd e

d., V

ol. 8

, pp.

48-

55).

ford

: El

sevi

er.

- mainstream use for high stakes testing- further development of innovative test formats- increased ease of use at school level - trend towards automated scoring

→ cultural change: increased acceptance of CBA

Automated administration and/or scoring of conventional tests

Adaptive Testing

Replication of complex real life situations within a test

- Multimedia collaborative learning environments support key competence

development.

- Peer and portfolio assessment evolve naturally.

- Data mining and analysis will make integrated assessment and tutoring

possible in the future.We are in a transitional period: Moving from the “Testing Paradigm” to the (future) “Tutoring Paradigm” in which explicit

testing could become obsolete

Technological trends1990 1995 2000 2005 2010 2015 2020 2025

Efficient testing Personalised learning

Key Questions

1. Which ICT tools, formats, environments, devices etc. are most suitable for the assessment of which Key Competences?

2. How can ICT be used to support competence-based assessment strategies that adequately capture the skills and attitudes dimensions of all eight Key Competences and their transversal nature?

Assessing Key Competences

F = Formative AssessmentD = Diagnostic AssessmentS = Summative Assessment

1 Communication in the mother tongue DS FD FS2 Communication in foreign languages DS FS3a Mathematical competence DS FD FDS FDS3b Basic competences in science and technology FS FDS4 Digital competence FDS5 Learning to learn FS FDS6 Social and civic competences FS7 Sense of initiative and entrepreneurship FS8 Cultural awareness and expression FS FS

CB

A

ePor

tfol

ios

Peer

A

sses

smen

t

Self

Ass

essm

ent

Qui

zzes

/ si

mpl

e ga

mes

Vir

tual

wor

lds

gam

es

Sim

ulat

ions

Inte

llige

nt

Tuto

rs e

tc.

Focus Areas

• PART 1

• INSPIRATION

Mother Tongue Literacy1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

Matching spoken and written languagehttp://www.familylearning.org.uk/sight_word_games.html

Foreign language learning2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

http

://w

ww

.nic

kjr.d

e/ga

mes

/107

-sw

iper

s-bu

chst

abie

r-buc

h

Mathematical games3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)

Mathematical games using simulation for secondary school educationSource: http://hotmath.com/games.html.

5 Learning to Learn CBA (F) Peer Assessment (FS) Self Assessment (FS)

http://spark.uts.edu.au/

Example: SparkPlus

Peer Assessment for Collaborative Work

• http://www.youtube.com/watch?v=Tp8v2UBbmN4

www.moodle.org

Moodle is a free web application that educators can use to create effective online

learning sites.

Moodle StatisticsRegistered sites: 77,213Countries: 229Courses: 7,116,466Users: 66,349,750Teachers: 1,292,731Enrolments: 56,306,274Forum posts: 118,413,272Resources: 63,891,942Quiz questions: 147,057,434

Peer Assessment using moodle

ePortfolios1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS)http://sites.google.com

/site/eportfolioapps/overview/m

odel-k-12

ePortfolios

ePortfolios1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS)

http://sites.google.com/site/eportfolioapps/overview

/levels

Level 1 - ePortfolio as Storage Collection: Creating the Digital Archive (regularly – weekly/monthly)A Focus on Contents & Digital Conversion

Level 2 - ePortfolio as Workspace Collection + Reflection: Immediate Reflection on Learning & Artifacts in Collection (regularly) A Focus on Process & Documentation of LearningLevel 3 - ePortfolio as Showcase Selection/Reflection/Direction (each semester? End of year?)A Focus on Product & Documentation of Achievement

→ A (formative/summative) assessment approach on the basis of artefacts → Focus on production and reflection

A game approach to arts8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS)

http://pheon.org; http://pheon.org/intro.html; http://americanart.si.edu/multimedia/games/pheon.cfm.

Pheon is a multimedia scavenger hunt that tests players’ ingenuity through exploring the collection of the Smithsonian American Art Museum, making objects, and texting through the galleries

ePortfolios meetVirtual reality

Science: Molecular Workbench

3b Basic competences in science and technology CBA (S) Virtual Worlds / games (FS) Simulations (FDS)

http://mw2.concord.org/public/part2/diffat/page3.cml

3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)

Simulations for mathematics

Sample task from US future New State Tests

(2014/15)

http://www.k12center.org/rsc/pdf/a-sea-change-in-assessment-letter-size.pdf

Cognitive Tutor (Carnegie)

3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)

http://www.youtube.com/watch?v=RR4HPrGlUz8&feature=related

Science: Quest Atlantis

3b Basic competences in science and technology CBA (S) Virtual Worlds / games (FS) Simulations (FDS)

Example: Quest AlantisQuest Atlantis (QA) is an international learning and teaching project that uses a 3D multi-user environment to immerse children, ages 9-16, in educational tasks. QA combines strategies used in the commercial gaming environment with lessons from educational research on learning and motivation.

Citizenship6 Social and civic competences CBA (DS) Peer/Self Assessment (FDS) Virtual Games / Simulations (FDS)

Educational game, designed to help teach concepts in citizenship, geography, and media. Developed by Serious Games International, it has detailed lesson plans and assignments for students.

Example: Child labour in Bangladesh:

http://www.youtube.com/watch?v=mf9Ju86IVYw

Global Conflicts http://www.globalconflicts.eu/

Examples• PART 2

• CRITICAL INSPECTION

Digital Competence4 Digital competence CBA (S) Simulations (FDS)

Internet and Communication Core Certification (IC3) exam sample questionhttp://www.certiport.com/Portal/common/htmllibrary/Sales-Video-Demos/demo_ic3.html

→ Focus on knowledge

CBA forDigital Competence

Digital Competence: IC34 Digital competence CBA (S) Simulations (FDS)

Inte

rnet

and

Com

mun

icat

ion

Cor

e C

ertif

icat

ion

(IC

3) e

xam

sa

mpl

e qu

estio

nht

tp://

ww

w.c

ertip

ort.c

om/P

orta

l/com

mon

/htm

llibr

ary/

Sale

s-V

ideo

-Dem

os/d

emo_

ic3.

htm

l

→ Focus on operational skills

CBA forDigital Competence

Entrepreneurship Questionnaire7 Sense of initiative and entrepreneurship CBA (DS) Peer/Self Assessment (FDS) Virtual Games / Simulations (FDS)

?https://

www.careeringahead.com.au/survey/Entrepreneurship/

Generic/questions.asp

→ Focus on personality/disposition as a static fact

Entrepreneurship Questionnaire

Conclusions

ICT offers powerful, engaging assessment formats, in particular: CBA ePortfolios

Self / Peer Assessment

Quizzes / simple games Virtual worlds/ gamesSimulations

Basic literacy(foreign/mother tongue)

Basic mathematical competence

Learning to Learn

Creative writing(foreign/mother tongue)

Cultural awareness and expression

Scientific competence

Digital competence

Social and civic competences

Educational software

Mathematical competence

CBA lends itself to simplified, knowledge-based assessment→ Pedagogy comes first! Careful selection of question formats

ICT Assessment approaches are underdeveloped for:

Learning to Learn Sense of initiative and entrepreneurship Social and civic competences

Caveat:

The Paradigm Shift

Technology

1990 1995 2000 2005 2010 2015 2020 2025Focus on: Efficient testing Focus on: Personalised learningEra of Computer-Based Assessment (CBA) Era of Technology-enhanced learning

PedagogyEfficient Testing

Validity of Scores

QuickFeedback

Authentic Contexts

CBA(Gen.1)

AdaptiveTests

(Gen.2)Automatic

ScoringGen. Re-Invention

Learning2.0

Immersive Learning

LearningAnalytics

IntelligentTutors

? ? ? ?→ Primacy of Pedagogy:

Pedagogical guidelines for intergrated assessment

• Thank you very much for your attention!

• Christine Redecker• Yves Punie

• Christine.Redecker@ec.europa.eu• Yves.Punie@ec.europa.eu

http://ipts.jrc.ec.europa.eu/publications/pub.cfm?id=5719

further information ……http://is.jrc.ec.europa.eu

Newsletters: http://is.jrc.es/pages/newsletter.html

Publications: http://is.jrc.ec.europa.eu/pages/Publications.html

Contact: jrc-ipts-is-NEWS@ec.europa.eu