Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Crowds

Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Crowds

Lora Aroyo

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

so9ware systems are ever more intelligent


but they don’t actually understand people

focus on human knowledge in machine-‐readable form


but there are types of human knowledge that can’t be captured by machines


in a form that machines can acquire & process

urge computer science research to push fronCers of human knowledge representaCon

classical AI involves human experts to manually provide training knowledge for machines


human expert-‐based ground truth does not scale for current demand for machines to deal with wide

ranges of real-‐world tasks and contexts


to provide an approach to capturing human knowledge in a way that is scalable & adequate to real-‐world needs

the key scienCfic challenge is

QuanCty is the new Quality


Human ComputaCon adopts human intelligence at scale to improve purely machine-‐based systems

humans accurately perform interpretaCon tasks


humans accurately perform interpretaCon tasks


can their effort be adequately harnessed in a scienCfically reliable manner that scales across tasks,

contexts & data modaliCes?


Research Projects


1. extracCng knowledge from social annotaCon

Research Projects



2. harnessing of non-‐expert human knowledge

Research Projects




3. nichesourcing of expert knowledge

Research Projects

4. capturing event knowledge




3. nichesourcing of expert knowledge

Research Projects

1. Human Knowledge from Social Tagging


Waisda? @ Beeld en Geluid – an online video tagging game, e.g. NCRV and VARA – power of the crowd for Web-‐scale enrichment of A/V collecAons

– significant accuracy for improving IR beyond the state-‐of-‐the-‐art of expert-‐based annotaAons

– sustainable means for engaging large volunteer crowds


1. Human Knowledge from Social Tagging


hZp://waisda.nl/ hZp://spotvogel.vroegevogels.vara.nl/

Engage with Games in Entertainment

Following the grandeur of Baroque, Rococo art is often dismissed as frivolous and unserious, but Waldemar Januszczak disagrees. […] The first episode is about travel in the 18th century and how it impacted greatly on some of the finest art ever made. The world was getting smaller and took on new influences shown in the glorious Bavarian pilgrimage architecture, Canaletto's romantic Venice and the blossoming of exotic designs and tastes all over Europe.

Rococo: Travel, pleasure, madness

A boarding school where boys from the Dutch East Indies receive a vocaAonal educaAon has been set up by the ministry of Social Affairs near Batavia. Trumpeter sounds the reveille; -‐ the boys get out of bed; -‐ the muster is held in front of the building; aIerwards the boys stand in line with a food bowl; they get rice pudding and eat this around long tables in the open air; -‐ the boys get instrucAons and pracAce a.o. drawing shapes, forging, metalworking, filing and woodworking;

News from Indonesia: youth care

hZp://vista-‐tv.eu hZp://dive.beeldengeluid.nl

Crowdsourcing for Video Analysis

2. Harnessing Non-‐expert Human Knowledge in Medical Domain


2. Harnessing Non-‐expert Human Knowledge in Medical Domain

Training IBM Watson – gathering higher quality ground truth data from the crowd than from experts

– open source human-‐machine computaAon framework for accurately measuring quality of resulAng data

– CrowdTruth.org


http://CrowdTruth.org http://data.CrowdTruth.org/ http://game.crowdtruth.org

Achieving Expert-Level Annotation Quality with the Crowd

The Case of Medical Relation Extraction


Crowdsourcing for Text Analysis


hNp://game.crowdtruth.org

Engage with Games in Sciences

3. Nichesourcing Expert Knowledge


3. Nichesourcing Expert Knowledge

Cultural Heritage – Different layers and domains of experAse –  Intrinsic moAvaAon of knowledgeable crowds – Accurator.nl @ Rijksmuseum


seman'c enrichment with LOD can be complemented with domain knowledge from niches of experts

there are massive niches of experts online …


hNp://annotate.accurator.nl/

Engage the Niche in Cultural Heritage

hNp://accurator.nl/

Engage with Games in Cultural Heritage

4. Capturing Event Knowledge


4. Capturing Event Knowledge

interpretaCon support framework Digital HermeneuCcs –  text, images and videos, e.g. newspapers, tweets, TV and radio programs, and images

–  events carry ambiguity, bias & wide range of perspecAves –  knowledge representaAon challenges, e.g informaAon granularity, temporal and provenance modeling

– Agora, DIVE+ @ KB, Beeld en Geluid, Amsterdam Museum –  Crowddriven.nl for TV show Game of Thrones


engaging users

through event

narratives

“Digital Hermeneu/cs: Agora and the online understanding of cultural heritage” In proceedings of Web Science Conference, (ACM: New York,

2011)

InterpretaCon Support for Scholars


Browsing support for wide audience


Engage with Games in Entertainment


understanding perspecCves – diversity of opinions from the crowd – mulAtude of contexts – ambiguity in language, images or videos –  independent interpretaCons – aggregated view – the big picture

– a new approach to understanding semanCcs by harnessing the power & diversity (disagreement on the correct interpretaAon) of the crowd

– human disagreement is essenCal in helping machines with semanCc interpretaCon

Goodbye to Truth

Hello Perspectives

Goodbye to Truth

Human Computation

making sense of human

semantics

Goodbye to Truth

Human Computation

making sense of human

semantics

http://lora-aroyo.org http://slideshare.net/laroyo

@laroyo

Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Crowds

Technology

Transcript of Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Crowds