Can Social Information Retrieval Enhance the Discovery and Reuse

of Learning Resources?

Riina VuorikariKatholieke Universiteit Leuven, Department of Computer Science

European Schoolnet, Belgium

Doctoral Consortium, RecSys 2007

Outline of the presentation

● Context of the dissertation work

● Main research questions

● Experimental design

● First evaluations so far:– Multi-lingual use of tags– Levels of user engagement

Context of the dissertation work

Context

● European education, especially that of K-12 education, is inherently multilingual and multicultural.

● European teachers have access to multiple repositories of digital learning resources by – Educational Authorities, – publishers, – other teachers,..

EUN partners..image

Context

● Resources – In many different languages– For different national and regional curriculum– Contain metadata (e.g.title, keywords, language)– Of varying quality

● Repositories have formed federations to make resources available– Federated search based on metadata– Harvesting of metada

Challenge for users

● End-users (e.g. teachers) have difficulties to discover and find resources from educational repositories– Metadata does not always match search terms

● Locating content across linguistic and national borders within Europe has proven hard – Despite the use of a multilingual Thesaurus and

controlled vocabularies

Challenges for repositories

● Users become more demanding and expect services that are seen elsewhere (own collections, pedagogical hints, ..)

● European Schoolnet leading projects that build services on top of federation of European repositories– Social bookmarking tool– Tags– My networks

My Main Question

Can Social Information Retrieval Enhance

the Discovery and Reuse

of Learning Resources?

Social Information Retrieval (SIR)

● Refers to a family of techniques that assist users in obtaining information to meet their information needs by harnessing the knowledge or experience of other users.

● Examples of SIR techniques include: – sharing of queries, – collaborative filtering, – social network analysis, – social bookmarking, – subjective relevance judgements such as

tags, annotations, ratings and evaluations, etc.

What is SIR for education?

● Is education as a field of implementation that different from other fields (e.g. music, movies)?

● What are the domain specific requirements, where does the data come from and what are its semantics?

● What are objects of recommendation?

● SIR TEL http://ariadne.cs.kuleuven.be/sirtel/

● My audience are teachers. Metaphor: it's like recommending for DJs?

Context of this dissertation

To empower the social and contextual aspects of teachers' work

Digitalcontent

Education

Digitallibraries

Social Information Retrieval (SIR) methods

Information seeking theories

Main research questions

Main research questions 1

Teachers, tagging, languages:

● How do teachers tag and use social bookmarking in a multi-lingual environment?

● Are those bookmarks and tags useful for discovery of resources?

● How about tags in multiple languages?

Main research questions 2

SIR aspect:

● Can bookmarks and tags be used to connect like-minded teachers cross country and linguistic borders?

● ...and thus used for social information retrieval?

● What are the levels of user engagement with the system?

Main research questions 3

Information Seeking aspect:

● What are the main information seeking tasks that teachers have?

● What are the main SIR retrieval methods that they use for them?

● Can we match a task to a SIR method?

Experimental design

Data source 1

● Calibrate project (http://calibrate.eun.org), now to end of 2007

● K-12 digital learning resources

● Personal collections and tags (not shared)

● 78 pilot schools in Hungary, Austria, Estonia, Czech Republic, Lithuania and Poland

Implementation area and data source 2

● MELT project (http://info.melt-project.eu), from now to March 2009

● K-12 digital learning resources from a federation of about 10 repositories

● Implementation of a social bookmarking tool, annotations and my networks

● About 70 teachers from Austria, Belgium, Finland and Hungary

Data gathering

● Diverse data collection methods to allow triangulation of collected data.

– log files from the portals to see the grand lines, patterns, etc

– complimented by some questionnaires to understand groups or communities

– possible interviews, thinking alouds, observation, etc. on some few users to understand individual behaviour.

Experimental Design

IndependentCondition

Social Condition

● Salganik, M., Dodds, P., & Watts, D. Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market. Science, 311(5762), (2006), 854-856.

Experimental Design

SocialInformationRetrieval

Ranking ofresources

Social navigation based onbookmarks, tags, annotationsand my networks

IndependentCondition

Social Condition

Tag input No tags shown when tagging

Tags shownwithin usersspoken language

Tags shownin alllanguages

Some early analysis

Analysis of User Behavior on Multi-lingual Tagging of Learning Objects

● January 24 to April 21 2007

● 77 teachers /173 total participating

● 459 bookmarks

● 417 multilingual tags

● 320 different learning resources

Cross-border and language use

Tag 1 fi

Tag 2en

Tag 4 fr

LO 1in Fi

Tag 1 fi

Tag 3de

Tag 5 fr

LO 2in Fr

frde

fi fi

fi

Tag 4 fr

Language should not divide..

Tag 1 fi

Tag 2en

Tag 5 fr

Tag 1 fi

Tag 2de

fr defi fi

fi

LO 1in Fi

LO 1in Fi

LO 2in Fr

LO 2in Fr

LO 2in Fr

..but bring like-minded people together

Tag 1 fi

Tag 2en

Tag 1 fi

fr

de

fi fi

fi

Tag 2de

Tag 5 fr

LO 1in Fi

LO 2in Fr

Visualisation tool for cross-country use of bookmarks

● Prototype tool to visualise – Bookmarks (title, classification keyword, country)– Tags (language)– Users (name, country, language)–

● Wanna play around with it?

http://www.cs.kuleuven.ac.be/~hmdb/infovis/calibrate/calibrate.html

Distribution of bookmarks

● Average: 6 bookmarks● Wide distribution:

– 10% “Super users” more than 20

– 15% 20-6 bookmarks– 45% 6-2 bookmarks

– About 30% only experimented (1)

Language analysis

● Out of 417 tags many were with multiple terms, when separated we found 585 terms

● 1/3 in Hungarian

● 26% in English, even though none of the users were native English speakers

● 1/3 in German and Polish

Language analysis

● The language was right in about 70% of cases (from the interface), and found out that...

● ...users tag in many different languages:

– at the same time (e.g. Baum, arbre, tree)

– at different times (once in Pl, other times in En)

– use the interface in different languages (seems like not only to test)

Btw, what do others do?

● del.icio.us, Yahoo.fr, MyWeb.Yahoo.uk, blogmarks.net, MisterWong.de...

● Two different ways to deal with multiple languages can be observed;

– ones taken care of by users (i.e. crowd-sourcing”)

– others where the system supports multiple languages to certain extent

Does the language matter?

● Need for better ways to identify the language– Give rules (if the user first preferred languages is.., then..)– Automate the recognition of languages– Out-source it to users

Semantic analysis

● Factual tags 63%(Golder: item topics, kinds of item, category refinements)

● Subjective tags 29%( Golder: item qualities)

● Personal tags 3% (Golder: item ownership, self-reference, tasks organisation)

● 5% other

● Sen et al. (2006).

Why tag categories?

● In Sen et al. (2006) it was found that tags of different categories can be useful for different tasks

● In our case it is too early to say anything, but ...we'll have an eye on it!

“Travel well” tags

● About 13% of tags contain a general term, a name, place

● e.g. EU, Euroopa, Europa, europe, geograafia, Pythagoras, etc.

What's the point of travel well tags?

● If those tags need no translation or language filtering to be understood, and ..

● ..if they can be identified

● We can be sure to show at least some tags to users – whose language preferences we don't know, and – in which language there are no tags or keywords

available.

Do users find tags useful?

Usefulness of tags..

● Overall, the thesaurus terms performed better than the tags,

● However, it can be argued that tags, after all being produced with no outlay, showed an overall encouraging and potential gain in overall usefulness!

So what is needed?

● HIDE ALL BUT THE RIGHT STUFF!

● In the tagging interface (guided tagging)– Show tags in all languages?– Show only travel well tags?– Show only tags in users' preferred languages

● While viewing the tags– In a tag cloud– For social navigation (resource-user-tag) – Q: does the system translate tags or only when a

user-given translation exist?

Future studies

● Similar language and semantic analysis are planned for a more thorough data in 2008

● Moreover, our goals are to find out:– How do users use the tags (e.g. language and

tag convergence) ?– How are tags and the relation resource-tag-user

used for discovery? – Identify teachers information seeking tasks and a

best fit for a retrieval system.

User engagement

● Inspired by Yahoo!'s START– rating shows the first level of engagement;– then tags it;– user views a page; – forwards it to friends, – and finally writing a review

● How can this be used for recommending purpose?

User engagement

● In our case these look very different:– views the page – views metadata– bookmarks and tags– rates– actual use?

That's it for now!

http://www.cs.kuleuven.be/~riina

riina.vuorikari@eun.orgriina.vuorikari@cs.kuleuven.be