Relevance Ranking Metrics for Learning Objects

Xavier Ochoa, ESPOL, Ecuador

Erik Duval, KULeuven, Belgium

Agenda

• Why Relevance Ranking?

• What is Relevance?

• Relevance Ranking Metrics

• Do they work?

• What is next?

Economy of Abundance

Economy of Abundance

Put your LMS here!

Why Relevance Ranking?

Abundance

=

Difficult to find most relevant

Why Relevance Ranking?

Abundance

=

Difficult to find most relevant

Solution:Do it the Google way

Why Relevance Ranking?

Abundance

=

Difficult to find most relevant

Relevance RankingMEANINGFUL & SCALABLE

How we do it now?

• Manual rating:–Meaningful but not Scalable

• Text based Ranking Algorithms:–Scalable but not Meaningful

What is Relevance?

Relevance Ranking Metrics

Relevance Ranking

Relevance Ranking

MetricB

MetricA

MetricC

Context

Relevance Ranking Metrics

Relevance Ranking

Relevance Ranking

MetricB

MetricA

MetricC

Context

Meaningful

Relevance Ranking Metrics

Relevance Ranking

Relevance Ranking

MetricB

MetricA

MetricC

Context

Scalable

Topi

cal R

elev

ance

Personal Relevance

Situational Relevance

Do they work?

• Exploratory Study– 10 users (8 Prof. and 2 R.A.)– MIT OCW learning objects (34,640 LO)– Search and Select Objects to create 10

lessons (10 different topics on CS)– Web application was used

Ranking the Rankings

• Baseline Rank: TF-IDF similarity algorithm

• Re-rank according to Basic Metrics

• All rankings compared with manual relevance ranking

• Kendall Tau was used to measure similaritiy between lists

Results

Basic Topical Relevance MetricTrees

Basic Personal Relevance Metric

Net.

Basic Situational Relevance Metric

XML

Linear Combination

Linear Combination

Results

• Even basic metrics improve the ranking

• Linear Combination should be learnt

• Limited (and Synthetic) Study!

What is next?

• Implementation in ARIADNE NEXT

• Capture user behavior and learn from it for 3 months

• Natural (Real) evaluation of the metrics

• Develop BETTER metrics

Conclusions

• Meaningful and Scalable metrics are possible

• Cheap to implement • Provide significant improvement• While not optimal, they could be

use as a base-line for further research

Thank You! Dank U! Gracias!• Questions, Comments, Critics…

are all welcome!!

Xavier Ochoaxavier@cti.espol.edu.ec

http://www.cti.espol.edu.ec/xavier

Erik DuvalErik.Duval@cs.kuleuven.be http://www.cs.kuleuven.ac.be/~erikd