8/27/2011 www.insemtives.eu 1

Combining human and computational intelligence for

collaborative knowledge creationElena Simperl

Talk at the IEEE International Conference on Intelligent Computer Communication and Processing, Cluj-Napoca, Romania

Insemtives in a nutshell• Many aspects of semantic content authoring naturally rely on human

contribution.

• Motivating users to contribute is essential for semantic technologies to reach critical mass and ensure sustainable growth.

• Insemtives works on – Best practices and guidelines for incentives-compatible technology design.– Enabling technology to realize incentivized semantic applications.– Showcased in three case studies: enterprise knowledge management;

services marketplace; multimedia management within virtual worlds.

www.insemtives.eu 2

Incentives and motivators

• Motivation is the driving force that makes humans achieve their goals.

• Incentives are ‘rewards’ assigned by an external ‘judge’ to a performer for undertaking a specific task.– Common belief (among

economists): incentives can be translated into a sum of money for all practical purposes.

• Incentives can be related to both extrinsic and intrinsic motivations.

• Extrinsic motivation if task is considered boring, dangerous, useless, socially undesirable, dislikable by the performer.

• Intrinsic motivation is driven by an interest or enjoyment in the task itself.

Examples of applications

www.insemtives.eu 4

Examples (ii)

Mason & Watts: Financial incentives and the performance of the crowds, HCOMP 2009.

Examples (iii)

www.insemtives.eu 9

What is different about semantic systems?

• Semantic Web toolsvs applications. – Intelligent (specialized)

Web sites (portals) with improved (local) search based on vocabularies and ontologies.

– X2X integration (often combined with Web services).

– Knowledge representation, communication and exchange.

What do you want your users to do?

• Semantic applications– Context of the actual application.– Need to involve users in knowledge acquisition and

engineering tasks?• Incentives are related to organizational and social factors.• Seamless integration of new features.

• Semantic tools– Game mechanics.– Paid crowdsourcing (integrated).

• Using results of casual games.

http://gapingvoid.com/2011/06/07/pixie-dust-the-mountain-of-mediocrity/

Case studies

• Methods applied– Mechanism design– Participatory design– Games with a purpose– Crowdsourcing via MTurk

• Semantic contentauthoring scenarios– Extending and populating

an ontology– Aligning two ontologies– Annotation of text, media

and Web APIs

Mechanism design in practice

• Identify a set of games that represents your situation.• See recommendations in the literature.

• Translate what economists do into concrete scenarios.• Assure that the economists’ proposals fit to the concrete situation.

• Run user and field experiments. Results influence HCI, social and data management aspects.

8/27/2011 www.insemtives.eu 15

Factors affecting mechanism design

8/27/2011 www.insemtives.eu 16

Goal Tasks Social Structure

Nature of good being produced

Communication level (about the

goal of the tasks)

High

Variety of

High

Hierarchy neutral

Private goodMedium Medium

Low Low

Participation level (in the definition

of the goal)

High

Specificity of

High

Public goodMedium Medium

Low Low

Clarity levelHigh Identification

withHigh

HierarchicalCommon resourceLow

Low Required skillsHighly specific

Club goodTrivial/Common

More at http://www.insemtives.eu/deliverables/INSEMTIVES_D1.3.1.pdf andhttp://www.insemtives.eu/deliverables/INSEMTIVES_D1.3.1.pdf

Phase 3: OKenterprise annotation tool

4/14/11 www.insemtives.eu 17

Mechanism design for Telefonica

• Interplay of two alternative games– Principal agent game

• The management wants employees to do a certain action but does not have tools to check whether employees perform their best effort.

• Various mechanisms can be used to align employees’ and employers’interests

– Piece rate wages (labour intensive tasks)– Performance measurement (all levels of tasks)– Tournaments (internal labour market)

– Public goods• Semantic content creation is non-rival and non-excludable• The problem of free riding

• Additional problem: what is the optimal time and effort for employees to dedicate to annotation

4/14/11 www.insemtives.eu 18

Mechanism design for Telefonica (ii)

• Principal agent game– Pay-per-performance

• Points assigned for each contribution

– Quality of performance measurement

• Rate user contributions• Assign quality reviewers

– Tournament• Visibility of contributions by

single users• Search for an expert based on

contributions• Relative standing compared to

other users

• Public goods game– To let users know that their

contribution was valuable– The portal should be useful

• Possibility to search experts, documents, etc.

• Possibility to form groups of users and share contributions

– The portal should be easy to use

• Experiments– Pay-per-tag vs winner-takes-

it-all for annotation.

4/14/11 www.insemtives.eu 19

Knowledge engineering tasks• Granularity of ontology

engineering activities is toobroad; further splitting isneeded

• Crowdsource very specifictasks that are (highly) divisible– Labeling (in different

languages)– Finding relationships– Populating the ontology– Aligning and interlinking– Ontology-based annotation– Validating the results of

automatic methods– …

www.insemtives.eu 20

OntoGame API• API that provides several methods that are

shared by the OntoGame games, such as: – Different agreement types (e.g. selection

agreement).– Input matching (e.g. , majority).– Game modes (multi-player, single player). – Player reliability evaluation. – Player matching (e.g., finding the optimal

partner to play).– Resource (i.e., data needed for games)

management.– Creating semantic content.

• http://insemtives.svn.sourceforge.net/viewvc/insemtives/generic-gaming-toolkit

www.insemtives.eu 218/27/2011

OntoGame games

8/27/2011 www.insemtives.eu 22

Lessons learned• Tasks which can be subject to games

– Definition of vocabulary– Conceptualization

• Based on competency questions• Identifying instances, classes, attributes, relationships

– Documentation• Labeling and definitions• Localization

– Evaluation and quality assurance• Matching conceptualization to documentation

– Alignment– Validating the results of automatic methods

• But, the approach is per design less applicable because– Knowledge-intensive tasks that are not easily nestable– Repetitive tasks players‘ retention?

www.insemtives.eu 24

Lessons learned (ii)• Approach is feasible for mainstream domains, where a

knowledge corpus is available• Knowledge corpus has to be large-enough to allow for

a rich game experience– But you need a critical mass of players to validate the

results• Advertisement is essential• Game design vs useful content

– Reusing well-kwown game paradigms– Reusing game outcomes and integration in existing

workflows and tools• Cost-benefit analysis

elena.simperl@kit.edu

http://www.ontogame.orghttp://apps.facebook.com/ontogame

http://www.insemtives.eu/iswc2011-tutorial/