★

•

•

•

•

http://CrowdTruth.org

•

•

http://CrowdTruth.org

• Annotator disagreement is signal, not noise.

• It is indicative of the variation in human semantic interpretation of signs

• It can indicate ambiguity, vagueness, similarity, over-generality, etc, as well as quality

http://CrowdTruth.org

★

Text Video

Sounds

Images

• Goals:

○ collecting a relation extraction gold standard

○ improve the performance of a relation extraction classifier

http://CrowdTruth.org

TASK

→

1 1 1

1 1 1

1 1

1

1 1

1 1

1 1

1

1

1

0 1 1 0 0 4 3 0 0 5 1 0

1

0 1 1 0 0 4 3 0 0 5 1 0

Unit vector for relation R6

Sentence Vector

Cosine = .55

• methodology • disagreement-aware

crowdsourcing to collect gold standard data

• metrics to capture disagreement

• software• online platform for

crowdsourcing task workflows and data analytics

• ground truth collection• medical relation extraction• salience in news and tweets• sound annotation

http://CrowdTruth.org

CrowdTruth.org

github.com/CrowdTruth

data.CrowdTruth.org