A Practical Method for Courseware Evaluation

Aldo de Moor CommunitySense

PragWeb 2007

The Pragmatic Web of what?

Communal effective use of Information Functionalities

Courseware and the Pragmatic Web

Pragmatic Web = context How to evaluate web functionalities in their context

of use?

Courseware provides an interesting domain technological environments consisting of multiple

functionality components, together offering a complete system of info/comm services required for supporting course needs

Functionality evaluation needed Too much (costly) functionality Gaps between required and available functionality Conflicting functionality requirements

Functionality

Functionality A set of functions and their specified properties that satisfy

stated or implied needs Levels of granularity

Systems

Courseware environments Tools

Blackboard Modules

Announcements Functions

Post announcement

A context model of courseware evaluation

A practical courseware evaluation method

Portfolio methods Scores weighed by infrastructure/architecture of organization.

Bedell’s method for ICT investment selection Functionalities scored on both effectiveness and importance for the

activities to be supported.

Practical method Simplification of Bedell

No higher-order analyses

Actors (users in their roles) provide, interpret and decide upon scores

Context: courseware evaluation: Actors: students, software manager Tool system level: module

Two questions How well are course activities supported by various functionality

components? To what extent are the functionality modules used?

Scores

Elements I(a) = importance of activity

E.g. I(Information Collection) = 9 I(f,a) = importance of functionality f in supporting

activity a E.g. I(Virtual Chat, Information Collection) = 4

Q(f,a) = quality of functionality in supporting an activity E.g. Q(File Transfer, Submission of Results) = 8

Activity and functionality scores

Activity scores I(fi,a) * Q(fi,a), for all functionalities 1..i.

Usefulness of the combined technologies for a particular activity

Relevant for technology users (lectures, students)

Functionality scores I(aj) * I(f,aj) * Q(f,aj), for all activities 1..j

Usefulness of a particular functionality component for the combined activities

Relevant for technology developers/maintainers

Experiment: evaluating group assignment functionality

Two courseware tools: Blackboard, CourseFlow Goal: making group assignments Four activities, 11 functionality modules Actors: 2nd year Information Management students,

software manager 2002: 62 students, 16 groups 2003: 46 students, 12 groups

Questions Quality of tools for various group assignment activities? Usefulness of various functionality modules?

Activity scores 2002/2003

Avg. Activity Scores in 2002 and 2003

0

50

100150

200

250

300350

400

450

Info coll. Discuss. Subm. Feedback

Activity

Avg

. sco

re

2002

2003

Functionality scores 2002/2003

Avg. Functionality Scores in 2002 and 2003

0200400600800

100012001400

send

e-m

ail

disc

ussio

n boa

rd

virtu

al ch

at

stud

ent r

oste

r

disc

ussio

n boa

rd (g

roup

)

virtu

al ch

at (g

roup

)

file tr

ansf

er (g

roup

)

anno

unce

men

ts (in

fo)

cour

se in

form

atio

n (in

fo)

cour

se d

ocum

ents

(inf

o)

assig

nmen

ts (i

nfo)

Functionality

Avg

. sco

re

2002

2003

Activity Scores BB/CF (2003)

Avg. Activity Scores Blackboard and CourseFlow (2003)

050

100150200250300350400450

Info coll. Discuss. Subm. Feedback

Activity

Avg

. sco

re

Blackboard

CourseFlow

Functionality Scores BB/CF (2003)

Avg. Functionality Scores Blackboard and CourseFlow (2003)

0

200

400

600

800

1000

1200

1400

1600

send

e-m

ail

disc

ussio

n boa

rd

virtu

al ch

at

stud

ent r

oste

r

disc

ussio

n boa

rd (g

roup

)

virtu

al ch

at (g

roup

)

file tr

ansf

er (g

roup

)

anno

unce

men

ts (in

fo)

cour

se in

form

atio

n (in

fo)

cour

se d

ocum

ents

(inf

o)

assig

nmen

ts (i

nfo)

Functionality

Avg

. sco

re

Blackboard

CourseFlow

Conclusion

Practical method, shown to be useful for initial courseware functionality selection

Context-bound, in toto, ex-post evaluation Simple measures useful for quickscan and

discussion purposes Open source can perform just as well “E-learning organization”: continuous evolution /

evaluation needed