Cost-Effective HITs for Relative Similarity Comparisons

Michael J. WilberCornell University

Iljung S. KwakUniversity of California, San Diego

Serge J. BelongieCornell University

Abstract

Similarity comparisons of the form “Is object a more simi-lar to b than to c?” are useful for computer vision and ma-chine learning applications. Unfortunately, an embedding ofn points is specified by n3 triplets, making collecting everytriplet an expensive task. In noticing this difficulty, other re-searchers have investigated more intelligent triplet samplingtechniques, but they do not study their effectiveness or theirpotential drawbacks. Although it is important to reduce thenumber of collected triplets, it is also important to understandhow best to display a triplet collection task to a user. In thiswork we explore an alternative display for collecting tripletsand analyze the monetary cost and speed of the display. Wepropose best practices for creating cost effective human in-telligence tasks for collecting triplets. We show that ratherthan changing the sampling algorithm, simple changes to thecrowdsourcing UI can lead to much higher quality embed-dings. We also provide a dataset as well as the labels collectedfrom crowd workers.

IntroductionRecently in machine learning (Tamuz et al. 2011; Jamiesonand Nowak 2011; van der Maaten and Weinberger 2012;McFee 2012), there has been a growing interest in collectinghuman similarity comparisons of the form “Is a more sim-ilar to b than to c?” These comparisons are asking humansto provide constraints of the form d(a, b) < d(a, c), whered(x, y) represents some perceptual distance between x andy. We will refer to these constraints as triplets. By collectingthese triplets from humans, researchers can learn the struc-ture of a variety of data sets. For example, the authors of(McFee 2012) were able to learn music genres from tripletcomparisons alone with no other annotations. Specificallyin computer vision, human similarity comparisons are use-ful for creating perceptually-based embeddings. In (Agarwalet al. 2007), the authors created a two dimensional embed-ding where one axis represented the brightness of an object,and the other axis represented the glossiness of an object. Inthis work we focus on creating perceptual embeddings fromimages of food.

For any set of n points, there are on the order of by n3unique triplets. Collecting such a large amount of triplets

Copyright c© 2014, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

Figure 1: Questions of the form “Is object a more similar tob than to c?” have been shown to be a useful way of collect-ing similarity comparisons from crowd workers. Tradition-ally these comparsions, or triplets, would be collected witha UI shown at the top. In this work we collect triplets using agrid of images and ask the user to select the two most similartasting foods to the food on the left. The grid UI, bottom, al-lows us to collect 8 triplets whereas the triplet UI, top, onlyyeilds a single triplet.

from crowd workers quickly becomes intractable for largerdatasets. For this reason, a few research groups have pro-posed more intelligent sampling techniques (Tamuz et al.2011; Jamieson and Nowak 2011). However, the difficultyof collecting a large number of triplets is also related to thetime and monetary cost of collecting data from humans. Toinvestigate this relationship more closely, we chose to studya triplet human intelligence task (HIT). In this work we pro-vide a better understanding of how the HIT design affectsnot only the time and cost of collecting triplets, but also thequality of the embedding, which is usually the researcher’sprimary concern.

arX

iv:1

404.

3291

v1 [

cs.C

V]

12

Apr

201

4

Traditionally, an MTurk task designed to collect tripletswould show crowd workers three images, labeled a, b, c.The worker is then asked to select either image b or imagec, whichever looks more similar to image a. See the top ofFig. 1 for an example. Although this is the most direct de-sign to collect triplets, it is potentially inefficient. Instead,we chose to investigate triplets collected from a grid of im-ages. In the grid format, a probe image—analogous to image“a” in the triplet representation—is shown next to a grid ofn images. The crowd worker is then asked to choose the kmost similar images from the grid. This layout allows us tocollect k images that are more similar to the probe imagethan the remaining n− k images, yielding k(n− k) tripletswith one screen to the user. We can change the number oftriplets per grid answer by varying n and k, but this also af-fects the amount of effort a crowd worker must exert to an-swer the question. We are not the first to realize that a grid ismore efficient for collecting triplets—such techniques werealso used by (Wah et al. 2014; Tamuz et al. 2011)—but webelieve we are the first to investigate more thoroughly the ef-fectiveness of triplets collected with a grid. This is importantbecause previous authors acknowledge neither the efficiencygain nor the potential drawbacks of the grid triplets they relyon.

This paper outlines several UI modifications that allowresearchers to multiply the number of triplets collected perscreen for perceptual similarity learning. We show that sim-ple changes to the crowdsourcing UI—instead of fundamen-tal changes to the algorithm — can lead to much higher qual-ity embeddings. In our case, using our grid format allows usto collect several triplet comparisons per screen. This leadsto much faster convergence than asking one triplet questionat a time. Researchers with tight deadlines can create rea-sonable embeddings with off-the-shelf algorithms and a lowcrowdsourcing budget by following our guidelines.

Our contributions are:

• A set of guidelines to use when collecting similarity em-beddings, with insights on how to manage the trade-offbetween user burden, embedding quality, and cost;

• A series of synthetic and human-powered experimentsthat prove our methods’ effectiveness;

• Evidence that each individual triplet sampled with a gridmay capture less information than a uniformly randomtriplet, but that their quantity outweighs the potential qual-ity decrease;

• A dataset of 100 food images, ingredient annotations, androughly 39% of the triplet comparisons that describe it, tobe made available upon publication.

Related WorkPerceptual similarity embeddings are useful for manytasks within the field, such as metric learning (Frome etal. 2007), image search/exploration (Ferecatu and Geman2009), learning semantic clusters (Gomes et al. 2011), andfinding similar musical genres and artists (van der Maatenand Weinberger 2012; McFee 2012). Our work is useful toauthors who wish to collect data to create such embeddings.

The common idea behind all of this work is that these au-thors use triplets to collect their embeddings.

In our work, we collect human similarity measurements ofimages in the form of triplets. The authors of (Heikinheimoand Ukkonen 2013) proposed an algorithm for collectingtriplets from humans as well. However in (Heikinheimo andUkkonen 2013), the triplets that were collected did not havea probe image. because they formulated the question differ-ently (Yi et al. 2013) focuses on estimating user preferencesfrom crowd sourced similarity comparisons. However (Yi etal. 2013) uses pairwise comparisons rather than triplets.

Figure 2: Top: An example cuisine embedding, collectedwith our 16-choose-4 grid UI strategy. This embedding costus $5.10 to collect and used 408 screens, but yielded 19,199triplets. It shows good clustering behavior with dessertsgathered into the top left. The meats are close to each other,as are the salads. Bottom: An embedding with 408 randomtriplets. This embedding also cost $5.10 to collect, but the re-sult is much dirtier, with worse separation and less structure.Salads are strewn about the right half of the embedding anda steak lies within the dessert area. From our experiments,we know that an embedding of such low quality would havecost us less than $0.10 to collect using our grid strategy.

Our work bears much similarity to Crowd Kernel Learn-ing (Tamuz et al. 2011) and Active MDS (Jamieson andNowak 2011). These algorithms focus on collecting tripletsone at a time, but sampling the best triplets first. The ideabehind these systems is that the bulk of the information inthe embedding can be captured within a very small num-ber of triplets, since most triplets convey redundant infor-mation. For instance, Crowd Kernel Learning (Tamuz et al.2011) considers each triplet individually, modeling the in-formation gain learned from that triplet as a probability dis-tribution over embedding space. Active MDS (Jamieson andNowak 2011) consider a set of triplets as a partial ranking

0

1

2

3

4

5

6

7How often do objects appear in triplet results? (Grid 16 choose 4)

1200 1400 1600 1800 2000 2200 24000

5

10

15

20

25

30(Random sampling)

Figure 3: Random triplets have a different distributionthan grid triplets. The top histogram shows the occurrencesof each object within human answers for “Grid 16 choose 4”triplets. The bottom histogram shows a histogram of sam-pling random triplets individually. 59520 triplets were col-lected for both histograms. Each object occurs in our an-swers about µ = 1785 times, but the variation when usinggrid triplets (top) is much wider (σ ≈ 187.0) than the vari-ation when sampling triplets uniformly (bottom, σ = 35.5).This effect is not recognized in the literature by authors whouse grids to collect triplets. We study its impact in our ex-periments.

with respect to each object in the embedding, placing geo-metric constraints on the locations where each point may lie.In our work we focus on altering UI design to improve speedand quality of triplet collection.

MethodInstead of asking “Is a more similar to b or c?”, we presenthumans with a probe image and ask “Mark k images thatare most similar to the probe,” as in Fig. 1. This way, witha grid of size n, a human can generate k · (n − k) tripletsper task unit. This kind of query allows researchers to col-lect more triplets with a single screen. It allows crowd work-ers to avoid having to wait for multiple screens to load, es-pecially in cases where one or more of the images in thequeried triplets do not change. This also allows crowd work-ers to benefit from the parallelism in the low-level human vi-sual system (Wolfe 1994). Since many of these observationsinvolve human issues, we conclude that the right way ofmeasuring embedding quality is with respect to human costrather than the number of triplets. This human cost is relatedto the time it takes crowd workers to complete a task and thepay rate of a completed task. Some authors (Wah et al. 2014;Tamuz et al. 2011) already incorporate these ideas into theirwork but do not quantify the improvement. Our goal is toformalize their intuitive notions into hard guidelines.

It is important to note that the distribution of grid tripletsis not uniformly random, even when the grid entries areselected randomly and even with perfect answers. To ourknowledge, no authors that use grids acknowledge this po-tential bias even though it deteriorates each triplet’s quality,as we will show in our experiments. Figure 3 shows a his-togram of how many times each object occurs in our tripletanswers. When using grid sampling, some objects can oc-

cur far more often than others, suggesting that the qualityof certain objects’ placement within the recovered embed-ding may be better than others. The effect is less pronouncedin random triplets, where objects appear with roughly equalfrequency. This observation is important to keep in mind be-cause the unequal distribution influences the result.

Synthetic ExperimentsWe aimed to answer two questions: Are the triplets ac-quired from a grid of lower quality than triplets acquiredone by one? Second, even if grid triplets are lower qual-ity, does their quantity outweigh that effect? To find out, weran synthetic “Mechanical Turk-like” experiments on syn-thetic workers. For each question, we show a probe anda grid of n objects. The synthetic workers use Euclideandistance within a groundtruth embedding to choose k gridchoices that are most similar to the probe. As a baseline, werandomly sample triplet comparisons from the groundtruthembedding using the same Euclidean distance metric. Aftercollecting the test triplets,we build a query embedding witht-STE (van der Maaten and Weinberger 2012) and comparethis embedding to the groundtruth. This way, we can mea-sure the quality of our embedding with respect to the to-tal amount of human effort, which is the number of workertasks. This is not a perfect proxy for human behavior, but itdoes let us validate our approach, and should be consideredin conjunction with the actual human experiments that aredescribed later.

Datasets. We evaluated our UI paradigm on threedatasets. First, we used MNIST1k, a handwritten digitdataset containing 1,000 random digits across 10 classes. Togenerate groundtruth comparison triplets, we use Euclideandistance between feature vectors. Second, we use the mu-sic similarity dataset from (van der Maaten and Weinberger2012) as a point of comparison. This set contains 9,107human-collected triplets for 412 artists. Finally, we presentresults on a subset of LFW (Huang et al. 2008), the La-beled Faces in the Wild dataset. We considered identitiesthat have between 32 and 77 images in the set, using theface attribute vectors extracted by (Kumar et al. 2009). Thisleaves us with a total of 938 73-dimensional feature vectorsfrom 20 identities. To generate groundtruth triplets, we againconsidered Euclidean distance. These three datasets provideus with a healthy balance of synthetic and real-world non-vectorial data.

Metrics. Our goal is not to build a competitive face orwritten digit recognizer; rather, we simply wish to evaluatethe quality of a perceptual embedding constructed with thehelp of synthetic workers. To do this, we evaluate each em-bedding’s quality using two metrics from (van der Maatenand Weinberger 2012): Triplet Generalization Error, whichcounts the fraction of the groundtruth embedding’s tripletconstraints that are violated by the recovered embedding;and Leave-One-Out Nearest Neighbor error, which mea-sures the percentage of points that share a category labelwith their closest neighbor within the recovered embedding.As pointed out by (van der Maaten and Weinberger 2012),these metrics measure different things: Triplet Generaliza-tion Error measures the triplet generator UI’s ability to gen-

103 104 105 106 107

Number of triplets

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80G

enera

lizati

on E

rror

Leave-1-out NN error

Grid 12, choose 3

Grid 12, choose 4

Grid 12, choose 5

Grid 12, choose 6

Random triplets

103 104 105 106 107

Number of triplets

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Genera

lizati

on E

rror

Constraint Error

Grid 12, choose 3

Grid 12, choose 4

Grid 12, choose 5

Grid 12, choose 6

Random triplets

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

Number of screens

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Genera

lizati

on E

rror

Leave-1-out NN error

Grid 12, choose 3

Grid 12, choose 4

Grid 12, choose 5

Grid 12, choose 6

Random triplets

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

Number of screens

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Genera

lizati

on E

rror

Constraint Error

Grid 12, choose 3

Grid 12, choose 4

Grid 12, choose 5

Grid 12, choose 6

Random triplets

Music dataset, 20 dimensions

Figure 4: Over the course of a synthetic experiment, we collect triplets, either randomly one at a time (thick black line) or inbatches using our grid UI (colored lines). When the embedding quality is viewed as the number of triplets gathered (top twographs), it appears that sampling random triplets one at a time yields a better embedding. However, when viewed as a functionof human effort, grid triplets create embeddings that converge much faster than individually sampled triplets. Here, quantityoutweighs quality as measured by Leave-One-Out NN Error (left graphs) and Triplet Generalization Error (right graphs). Seetext for details.

eralize to unseen constraints, while NN Leave-One-Out er-ror reveals how well the embedding models the (hidden) hu-man perceptual similarity distance function. We use thesemetrics to test the impact that different UIs have on embed-ding quality.

Results. Across all three datasets, our experiments showthat even though triplets acquired via the grid converge fasterthan random triplets, each individual grid triplet is of lowerquality than an individual random triplet. Figure 4 showshow the music dataset embedding quality converges with re-spect to the number of triplets. If triplets are sampled one at atime (top two graphs), random triplets converge much fasteron both quality metrics than triplets acquired via grid ques-tions. However, this metric does not reveal the full story be-cause grid triplets can acquire several triplets at once. Whenviewed with respect to the number of screens (human taskunits), as in the bottom two graphs in Figure 4, we now seethat the grid triplets can converge far faster than random withrespect to the total amount of human work. This leads usto conclude that “quality of the embedding wrt. number oftriplets” is the wrong metric to optimize because framingthe question in terms of triplets gives researchers the wrongidea about how fast their embeddings converge. A researcherwho only considers the inferior performance of grid tripletson the “per-triplet” metric will prefer sampling triplets indi-vidually, but they could achieve much better accuracy usinggrid sampling even in spite of the reduced quality of eachindividual triplet, and as we shall see in our human experi-ments, this translates into decreased cost for the researcher.In other words, efficient collection UIs are better than ran-dom sampling, even though each triplet gathered using suchUIs does not contain as much information.

Why does this happen? In all cases, the 12 images withinthe grid were chosen randomly; intuitively, we expect a uni-form distribution of triplets. However, because certain ob-jects are more likely than others to be within each grid’s“Near” set, certain objects will appear in the triplet moreoften than others. This leads to a nonuniform distribution ofcorrect triplets, as shown in Fig. 3. Here, we can see that thenon-uniformity creates a difference in performance.

The other two datasets—MNIST and Face—show verysimilar results so we do not report them here. In all cases,any size of grid UI outperforms random selection. However,we do see a small spread of quality across different gridsizes. As in the music dataset, the error is lowest when weforce our synthetic workers to select 3 close images out of12 as opposed to selecting the 4, 5, or 6 closest images. Thisdifference is more pronounced in the “Leave-One-Out NN”metric. This could be because selecting the 3 closest imagesallows the metric to be more precise about that image’s lo-cation in the embedding since it is compared to fewer neigh-bors. Our synthetic workers always give perfect answers; wedo not expect imperfect humans to reflect this effect.

Human ExperimentsThese synthetic experiments validate our approach, but theyhave several problems. In particular, there is no reason whyhumans would behave similarly to a proxy oracle as de-scribed above. Further, we must also consider the effort ofour workers, both in terms of the time it takes to completeeach task and how much money they can make per hour—metrics that are impossible to gather via synthetic means. Toverify that these approaches build better embeddings even

Figure 5: Example images from our dataset. The images inour dataset span a wide range of foods and imaging con-ditions. The dataset as well as the collected triplets will bemade available upon publication.

when humans provide inconsistent triplets, we ran Mechan-ical Turk experiments on a set of 100 food images sourcedfrom Yummly recipes with no groundtruth. The images werefiltered so that each image contained roughly one entree. Forexample, we avoided images of sandwiches with soups. Ex-ample images are shown in Fig. 5. For each experiment, weallocated the same amount of money for each hit, allowingus to quantify embedding quality with respect to cost. Uponpublication, the dataset as well as the collected triplets willbe available for download.

Design. For each task, we show a random probe and a gridof n random foods. We ask the user to select the k objectsthat “taste most similar” to the probe. We varied n across(4, 8, 12, 16) and varied k across (1, 2, 4). We ran three in-dependent repetitions of each experiment. We paid $0.10 perHIT, which includes 8 usable grid screens and 2 catch trials.To evaluate the quality of the embedding returned by eachgrid size, we use the same “Triplet Generalization Error” asin our synthetic experiments: we gather all triplets from allgrid sizes and construct a reference embedding via t-STE.Then, to evaluate a set of triplets, we construct a target em-bedding, and count how many of the reference embedding’sconstraints are violated by the target embedding. Varyingthe number of HITs shows how fast the embedding’s qualityconverges.

Baseline. Since we wish to show that grid triplets pro-duce better-quality embeddings at the same cost as ran-dom triplets, we should collect random (a, b, c) comparisonsfrom our crowd workers for comparison. Unfortunately, col-lecting all comparisons one at a time is infeasible (see our“Cost” results below), so instead, we construct a groundtruthembedding from all grid triplets and uniformly sample ran-dom constraints from the embedding. This is unlikely to leadto much bias because we were able to collect 39% of the pos-sible unique triplets, meaning that t-STE only has to gener-alize to constraints that are likely to be redundant. All eval-uations are performed relative to this reference embedding.

4 8 12 16

Grid size

0

2

4

6

8

10

12

14

Seco

nds

Timing tasks

Choose 4

Choose 2

Choose 1

Figure 6: We show the median time that it takes a hu-man to answer one grid. The time per each task increaseswith a higher grid size (more time spent looking at the re-sults) and with a higher required number of near answers(which means more clicks per task). Error bars are 25 and75-percentile.

ResultsTwo example embeddings are shown in Fig. 2.

Cost. Across all experiments, we collected 14,088 grids,yielding 189,519 unique triplets. Collecting this data costus $158.30, but sampling this many random triplets one ata time would have cost us $2,627.63, which is far outsideour budget1. If we had used the 16-choose-4 grid strategy(which yields 48 triplets per grid), we would be able to sam-ple all unique triplets for about $140—a feat that would costus $6737.50 by sampling one at a time.

Quality. As we spend more money, we collect moretriplets, allowing t-STE to do a better job generalizing tounseen redundant constraints. All embeddings converge tolower error when given more triplets, but this convergenceis not monotonic because humans are fallible and there israndomness in the embedding construction. See Fig. 7 fora graphical comparison of grids with size 4,8,12, and 16.When viewed with respect to the number of triplets, randomtriplets again come out ahead; but when viewed with respectto cost, the largest grid converges more quickly than oth-ers, and even the smallest grid handily outperforms randomtriplet sampling.

This time, we observe a large separation between the per-formance of various grid sizes. Grid 16-choose-4, whichyields 4 ·12 = 48 triplets per answer, uniformly outperformsthe rest, with Grid 12-choose-4 (at 4 · 8 = 32 triplets per an-swer) close behind. Both of these outperform 8-choose-4 (16triplets/answer) and 4-choose-2 (4 triplets/answer).

We also compare our performance with the adaptivetriplet sampling strategy of (Tamuz et al. 2011). CKL pickstriplets one-at-a-time but attempts to select the best triplet

1There are 100 · 99 · 98/2 = 485, 100 possible unique tripletsand each triplet answer would cost one cent. We additionally needto allocate 10% to Amazon’s cut and 20% of our tasks are devotedto catch trials.

102 103 104

Number of triplets

0.1

0.2

0.3

0.4

0.5

Genera

lizati

on E

rror

Grid 4 choose 2

Grid 8 choose 4

Grid 12 choose 4

Grid 16 choose 4

Random triplets

CKL

$0.00 $1.00 $2.00 $3.00 $4.00 $5.00

Total cost ($)

0.1

0.2

0.3

0.4

0.5

Genera

lizati

on E

rror

Grid 4 choose 2

Grid 8 choose 4

Grid 12 choose 4

Grid 16 choose 4

Random triplets

CKL

Human experiments on foods, 5 dimensional

Figure 7: Results of our human experiments on the food dataset. Left graph: Triplet generalization error when viewed withrespect to the total number of triplets. Right: The same metric when viewed with respect to the total cost (to us) of constructingeach embedding. The left graph implies that a randomly-sampled embedding appears to converge faster. However, when qualityis viewed with respect to cost, we find that an embedding generated using a 16-choose-4 grid cost $0.75, while an embeddingwith random triplets of similar quality costs $5.00. It is clear that the grid UI saves money; in this case, by over a factor of 6.

Grid n choose k Error at $1 Time/screen (s) Wages ($/hr)n: 4, k: 1 0.468 3.57 $10.09

k: 2 0.369 3.45 $10.45n: 8, k: 1 0.400 3.04 $11.85

k: 2 0.311 5.79 $6.22k: 4 0.273 7.65 $4.71

n: 12, k: 1 0.406 4.17 $8.64k: 2 0.294 6.78 $5.31k: 4 0.235 8.67 $4.15

n: 16, k: 1 0.413 6.72 $5.36k: 2 0.278 8.84 $4.07k: 4 0.231 9.59 $3.76

Random 0.477 – –CKL 0.403 – –

Table 1: Results of our actual Mechanical Turk experiments.We ask workers to choose the k most similar objects from agrid of n images. We invest $1 worth of questions, givingus 100 grid selections. When n and k are large, each answeryields more triplets. Large grids require more time to com-plete, but many of our tasks (bold) still pay a respectablewage of more than $6 per hour.

possible to ask by maximizing the information gain fromeach answer. In our experiments, it did not outperform ran-dom sampling; further analysis will be future work.

Though catch trials comprised 20% of the grid answerswe collected, we found that the results were generally ofsuch high quality that no filtering or qualification was re-quired.

Time. Fig. 6 shows how fast each human takes to answerone grid question. Our smallest task was completed in 3.5seconds ( ), but even our largest grid (16 choose 4) can becompleted in less than 10 seconds. Times varies widely be-tween workers: our fastest worker answered 800 questionsin an average of 2.1 seconds per grid task for 8-choose-1

grids.Worker Satisfaction. At our standard 1c|-per-grid/$0.10-

per-HIT rate, our workers are able to make a respectable in-come, shown in Tab. 1. The smallest tasks net more than$10/hour by median, but even our largest task allows half ofour workers to make $3.76 for every hour they spend. If thefastest, most skilled worker sustained their average pace in8-choose-1 grids, they could earn over $17 per hour.

Since there is a trade-off between grid size and worker in-come, it is important to consider just how far we can pushour workers without stepping over the acceptable bound-aries. Across all of our experiments, we received no com-plaints, and our tasks were featured on multiple HIT aggre-gators including Reddit’s HitsWorthTurkingFor sub-reddit and the “TurkerNation” forums as examples of boun-tiful HITs. Our workers did not feel exploited.

According to the HitsWorthTurkingFor FAQ 2,“the general rule of thumb . . . is a minimum of $6/hour.”Though HITs below this amount may be completed, the bestworkers may pass for more lucrative HITs. Being featured inforums such as HitsWorthTurkingFor gave us an ad-vantage since our hit was visible to a very large audience ofpotential skilled turkers. Though high payouts mean highercost, in our case, the benefit outweighed the drawback.

Guidelines and conclusionThroughout this paper, we have shown that taking advantageof simple batch UI tricks can save researchers significantamounts of money when gathering crowdsourced perceptualsimilarity data. Our recommendations can be summarized asfollows:

• Rather than collecting comparisons one-at-a-time, re-searchers should use a grid to sample comparisons in

2http://reddit.com/r/HITsWorthTurkingFor/wiki/index

batch, or should use some other UI paradigm appropriateto their task. However, researchers should not assume thatsuch “batch” comparisons are of identical quality to uni-formly random sampling—this is a trade-off that shouldbe considered.

• If cost is an issue, researchers should quantify their re-sults with respect to dollars spent. We found that usingour simple UI paradigm can creates embeddings of higherquality than those created using algorithms that pick thebest triplet one-at-a-time.

• Researchers should continuously monitor the humaneffort of their tasks, so that they can calculate an ap-propriate target wage and stand a better chance of beingfeatured on “Good HIT” lists and be seen by more skilledTurkers.

• When using grids to collect triplets, researchers shouldconsider the trade-off between size and effort. Considerthat an n-choose-k grid can yield

k(n− k) (1)

triplets per answer. Since this has a global maximum atn = 2k, one appropriate strategy is to select the largest nthat yields a wage of $6/hour and set k equal to n/2.

There are several opportunities for future work. First, weshould better quantify the relationship between n, k, andtask completion time to build a more accurate model of hu-man performance. Second, we should continue investigat-ing triplet sampling algorithms such as “CKL” as there maybe opportunities to adaptively select grids to converge fasterthan random, giving us advantages of both strategies.

AcknowledgmentsWe especially thank Jan Jakes, Tomas Matera, and EdwardCheng for their software tools that helped us collect gridtriplets so quickly. We also thank Vicente Malave for helpfuldiscussions. This work was partially supported by an NSFGraduate Research Fellowship award (NSF DGE-1144153,Author 1) and a Google Focused Research award (Author 3).

References[Agarwal et al. 2007] Agarwal, S.; Wills, J.; Cayton, L.; Lanckriet,

G.; Kriegman, D. J.; and Belongie, S. 2007. Generalized non-metric multidimensional scaling. In International Conference onArtificial Intelligence and Statistics.

[Ferecatu and Geman 2009] Ferecatu, M., and Geman, D. 2009. Astatistical framework for image category search from a mental pic-ture. IEEE TPAMI.

[Frome et al. 2007] Frome, A.; Singer, Y.; Sha, F.; and Malik, J.2007. Learning globally-consistent local distance functions forshape-based image retrieval and classification. In IEEE ICCV.

[Gomes et al. 2011] Gomes, R.; Welinder, P.; Krause, A.; and Per-ona, P. 2011. Crowdclustering. NIPS.

[Heikinheimo and Ukkonen 2013] Heikinheimo, H., and Ukkonen,A. 2013. The crowd-median algorithm. In First AAAI Conferenceon Human Computation and Crowdsourcing.

[Huang et al. 2008] Huang, G. B.; Mattar, M.; Berg, T.; andLearned-Miller, E. 2008. Labeled faces in the wild: A database

forstudying face recognition in unconstrained environments. InWorkshop on Faces in’Real-Life’Images: Detection, Alignment,and Recognition.

[Jamieson and Nowak 2011] Jamieson, K., and Nowak, R. 2011.Low-dimensional embedding using adaptively selected ordinaldata. In Allerton Conference on Communication, Control, andComputing.

[Kumar et al. 2009] Kumar, N.; Berg, A. C.; Belhumeur, P. N.; andNayar, S. K. 2009. Attribute and simile classifiers for face verifi-cation. In IEEE ICCV.

[McFee 2012] McFee, B. 2012. More like this: machine learningapproaches to music similarity. Ph.D. Dissertation, University ofCalifornia, San Diego.

[Tamuz et al. 2011] Tamuz, O.; Liu, C.; Belongie, S.; Shamir, O.;and Kalai, A. T. 2011. Adaptively learning the crowd kernel. InICML.

[van der Maaten and Weinberger 2012] van der Maaten, L., andWeinberger, K. 2012. Stochastic triplet embedding. In IEEEMLSP.

[Wah et al. 2014] Wah, C.; Horn, G. V.; Branson, S.; Maji, S.; Per-ona, P.; and Belongie, S. 2014. Similarity comparisons for interac-tive fine-grained categorization. In IEEE CVPR.

[Wolfe 1994] Wolfe, J. M. 1994. Guided search 2.0 a revised modelof visual search. Psychonomic Bulletin & Review 1.

[Yi et al. 2013] Yi, J.; Jin, R.; Jain, S.; and Jain, A. 2013. Infer-ring users preferences from crowdsourced pairwise comparisons:A matrix completion approach. In HCOMP.