Effectiveness of Gamesourcing Expert Painting Annotations
-
Upload
myriam-traub -
Category
Science
-
view
58 -
download
1
Transcript of Effectiveness of Gamesourcing Expert Painting Annotations
Effectiveness of Gamesourcing Expert Painting Annotations
Are there features of images or subject types that can predict high or low agreement??
Start to play!Can a simplified version of an expert annotation task be carried out by non-experts? baseline #
imperfect # 200
400
600
800
1000
1200
num
ber o
f ann
otat
ions
(bar
s)
020
4060
8010
0
users
perc
enta
ge o
f cor
rect
ann
otat
ions
(dot
s) baseline % imperfect %
baseline # imperfect #
1
10
100
1000
num
ber o
f ann
otat
ions
(bar
s)
2 4 6 8 10
020
4060
8010
0
number of repetitions
perc
enta
ge o
f cor
rect
ann
otat
ions
(lin
es)
baseline % imperfect %
Do users learn to correctly label subject types of paintings?
?Can they apply what they have learned to new paintings of known subject types?
?
2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
baseline condition − aggregated annotations
96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
imperfect condition − aggregated annotations
48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations
291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
How do they compare with experts, both, individually and as a crowd??
Top players:
1. Myriam C. Traub 2. Jacco van Ossenbruggen3. Jiyin He4. Lynda Hardman
!Label paintings with subject types from the Art and Architecture Thesaurus!
Game over! Congratulations!
You found out that our results show a notable agreement between experts and non-experts, that users improve when playing on “perfect” data, and that aggregating annotations increases their precision. Future research will focus on peer-feedback and using judgements to improve the selection of candidates.
baseline # imperfect #
0
50
100
150
200
250
300
350
num
ber o
f ann
otat
ions
(bar
s)
sequence number of new images
perc
enta
ge o
f cor
rect
ann
otat
ions
(lin
es)
baseline % imperfect %
[1,20] (40,60] (80,100] (120,140] (160,180] (200,220] (240,260] (280,300] (320,340] (360,380]
020
4060
8010
0