INSEMTIVES Tutorial ISWC2011 - Session1
-
Upload
insemtives-project -
Category
Technology
-
view
834 -
download
0
description
Transcript of INSEMTIVES Tutorial ISWC2011 - Session1
10/24/2011 www.insemtives.eu 1
Ten ways to make your semantic app addicted - REVISITED
Elena Simperl
Tutorial at the ISWC2011, Bonn, Germany
Executive summary • Many aspects of semantic content authoring naturally rely
on human contribution. • Motivating users to contribute is essential for semantic
technologies to reach critical mass and ensure sustainable growth.
• This tutorial is about
– Methods and techniques to study incentives and motivators applicable to semantic content authoring scenarios.
– How to implement the results of such studies through technology design, usability engineering, and game mechanics.
www.insemtives.eu 2
Incentives and motivators
• Motivation is the driving force that makes humans achieve their goals.
• Incentives are ‘rewards’ assigned by an external ‘judge’ to a performer for undertaking a specific task. – Common belief (among
economists): incentives can be translated into a sum of money for all practical purposes.
• Incentives can be related to both extrinsic and intrinsic motivations.
• Extrinsic motivation if task is considered boring, dangerous, useless, socially undesirable, dislikable by the performer.
• Intrinsic motivation is driven by an interest or enjoyment in the task itself.
Examples of applications
www.insemtives.eu 4
Extrinsic vs intrinsic motivations • Successful volunteer crowdsourcing is difficult
to predict or replicate. – Highly context-specific. – Not applicable to arbitrary tasks.
• Reward models often easier to study and control.* – Different models: pay-per-time, pay-per-unit, winner-
takes-it-all… – Not always easy to abstract from social aspects (free-
riding, social pressure…). – May undermine intrinsic motivation.
* in cases when performance can be reliably measured
Examples (ii)
Mason & Watts: Financial incentives and the performance of the crowds, HCOMP 2009.
Amazon‘s Mechanical Turk
• Types of tasks: transcription, classification, and content generation, data collection, image tagging, website feedback, usability tests.*
• Increasingly used by academia. • Vertical solutions built on top. • Research on extensions for complex tasks.
* http://behind-the-enemy-lines.blogspot.com/2010/10/what-tasks-are-posted-on-mechanical.html
Tasks amenable to crowdsourcing
• Tasks that are decomposable into simpler tasks that are easy to perform.
• Performance is measurable. • No specific skills or expertise are required.
Patterns of tasks* • Solving a task
– Generate answers – Find additional information – Improve, edit, fix
• Evaluating the results of a task – Vote for accept/reject – Vote up/down to rank
potentially correct answers – Vote best/top-n results
• Flow control – Split the task – Aggregate partial results
• Example: open-scale tasks in Mturk – Generate, then vote. – Introduce random noise to
identify potential issues in the second step
* „Managing Crowdsourced Human Computation“@WWW2011, Ipeirotis
Gene
rate
ans
wer
Label image
Vote
ans
wer
s
Correct or not?
Examples (iii)
www.insemtives.eu 10
What makes game mechanics successfull?*
• Accelerated feedback cycles. – Annual performance appraisals vs immediate feedback to
maintain engagement. • Clear goals and rules of play.
– Players feel empowered to achieve goals vs fuzzy, complex system of rules in real-world.
• Compelling narrative. – Gamification builds a narrative that engages players to
participate and achieve the goals of the activity.
• But in the end it’s about what task users want to get better at.
*http://www.gartner.com/it/page.jsp?id=1629214 Images from http://gapingvoid.com/2011/06/07/pixie-dust-the-mountain-of-mediocrity/ and http://www.hideandseek.net/wp-content/uploads/2010/10/gamification_badges.jpg
Guidelines • Focus on the actual goal and incentivize related
actions. – Write posts, create graphics, annotate pictures, reply
to customers in a given time… • Build a community around the intended actions.
– Reward helping each other in performing the task and interaction.
– Reward recruiting new contributors. • Reward repeated actions.
– Actions become part of the daily routine.
Image from http://t1.gstatic.com/images?q=tbn:ANd9GcSzWEQdtagJy6lxiR2focH2D01Wpz7dzAilDuPsWnL0i4GAHgnm_0hyw3upqw
What tasks can be gamified?* • Tasks that are decomposable into simpler
tasks, nested tasks. • Performance is measurable. • Obvious rewarding scheme. • Skills can be arranged in a smooth learning
curve.
*http://www.lostgarden.com/2008/06/what-actitivies-that-can-be-turned-into.html Image from http://www.powwownow.co.uk/blog/wp-content/uploads/2011/06/gamification.jpeg
What is different about semantic systems?
• It‘s still about the context of the actual application.
• User engagement with semantic tasks in order to – Ensure knowledge is
relevant and up-to-date. – People accept the new
solution and understand its benefits.
– Avoid cold-start problems. – Optimize maintenance
costs.
Tasks in knowledge engineering • Definition of vocabulary • Conceptualization
– Based on competency questions – Identifying instances, classes, attributes,
relationships
• Documentation – Labeling and definitions. – Localization
• Evaluation and quality assurance – Matching conceptualization to documentation
• Alignment • Validating the results of automatic methods
www.insemtives.eu 15
16
http://www.ontogame.org http://apps.facebook.com/ontogame
OntoGame API • API that provides several methods that are
shared by the OntoGame games, such as: – Different agreement types (e.g. selection
agreement). – Input matching (e.g. , majority). – Game modes (multi-player, single player). – Player reliability evaluation. – Player matching (e.g., finding the optimal
partner to play). – Resource (i.e., data needed for games)
management. – Creating semantic content.
• http://insemtives.svn.sourceforge.net/viewvc/insemtives/generic-gaming-toolkit
www.insemtives.eu 17 10/24/2011
OntoGame games
10/24/2011 www.insemtives.eu 18
Case studies
• Methods applied – Mechanism design. – Participatory design. – Games with a purpose. – Crowdsourcing via MTurk.
• Semantic content authoring scenarios – Extending and populating
an ontology. – Aligning two ontologies. – Annotation of text, media
and Web APIs.
Lessons learned • Approach is feasible for mainstream domains, where a
(large-enough) knowledge corpus is available. • Advertisement is important. • Game design vs useful content.
– Reusing well-kwown game paradigms. – Reusing game outcomes and integration in existing workflows
and tools.
• But, the approach is per design less applicable because – Knowledge-intensive tasks that are not easily nestable. – Repetitive tasks players‘ retention?
• Cost-benefit analysis.
Using Mechanical Turk for semantic content authoring
• Many design decisions similar to GWAPs. – But clear incentives structures. – How to reliably compare games and MTurk results?
• Automatic generation of HITs depending on the types of tasks and inputs.
• Integration in productive environments.
– Protégé plug-in for managing and using crowdsourcing results.
Outline of the tutorial Time Presentation 14:00 – 14:45
Human contributions in semantic content authoring
14:45 – 15:30
Case study: motivating employees to annotate enterprise content semantically at Telefonica
15:30 – 16:00
Coffee break
16:00 – 16:45
Case study: Crowdsourcing the annotation of dynamic Web content at seekda
16:45 – 17:30
Case study: Content tagging at MoonZoo and MyTinyPlanets
17:30 – 18:00
Ten ways to make your semantic app addicted - revisited www.insemtives.eu 22
10/24/2011 www.insemtives.eu 23
Realizing the Semantic Web by encouraging millions of end-users to
create semantic content.