A Dynamic MicroTask Scheduling Approach forSLO based Human-augmented ComputingKOUSHIK SINHA

SOUTHERN ILLINOIS UNIVERSITY, CARBONDALE

Outline

Introduction

Problem Statement

Proposed Approach

Results

Conclusion

Introduction

Human Computation

“Some problems are hard, even for the most sophisticated AI algorithms.”

“Let humans solve them ... ”

Introduction

Using Humans as Computers

A very old idea: Humans were the first “computers”

Halley’s comet orbit,1758

Astronomical almanac with moon positions, used for navigation, 1760

Logarithmic and trigonometric tables, 1794

Math Tables Project, unskilled labor, 1938

Grier, When computers were human, 2005Grier, IEEE Annals 1998

Introduction

Crowd: group of workers willing to do small duration and simple tasks on a crowdsourcing platform

Heterogeneous group

Members do not know each other and work independently

An individual member of such crowd is known as crowdworker or simply worker

Microtask: smaller, well defined sub-task derived from task decomposition

Can be done quickly by humans - few seconds/minutes of low cognitive load

Machine solution unsatisfactory

Either not solvable by machine algorithm

Or, poor quality

Or, would take significantly longer time than humans

Can solve independent of other microtasks derived from the same task

• Image tagging

• Image categorization

• Image digitization

• Text validation in images

• Object tagging in images

• Sentiment analysis of text

• Text classification

• Language translation

• Event detection in video

• Keyword spotting in audio

Microtask Examples

SpeechSpeech transcriptionSpeech translationKeyword spottingSentiment analysis

Document imagesTaggingCategorizationDigitization/OCRValidating OCR

VideoData collectionDescription/taggingEvent detectionObject recognition

TextCreation/Data collectionLanguage translationSentiment analysisCategorization

ImagesData collectionDescription/taggingObject locationObject recognitionDemographics

“I would rather eat a stone instead of

this cake”

“XYZ printers are not that bad”

Task

Is this a dog?o Yeso No

Workers

Answer: Yes

Task: Dog ?

Pay: $0.01

Broker

$0.01

Human Intelligence Task (HIT)

Microtask

Microtask

ReCAPTCHA – Using Human Computation to Improve OCR

200M+ each day by people around the worldUsed on 350,000+ sites, digitizing 100M words per day, equivalent of 2.5M books a year

Crowdsourcing Microtask - Many Platforms

Multiple Dimensions

Experience

Ease of Use

Quality of crowd

Satisfactory results (SLO)

Cost advantage

Privacy & security

Infrastructure

Source of crowd

Work Definition support

Work & Process Oversight

Results & Quality Management

Payment processing

Problem Statement

Design Human-Machine Hybrid Task Execution Environment for Unstructured Data Analytics

Orchestrate machine and human computing resources to meet:• Service level objectives (SLO) - budget,

turnaround time, accuracy

• Scalability – handle big data volumes

• Reliability – resilience against computing resources unpredictablity

Why this is hard• Providing SLO guarantees under dynamic resource

availability and quality conditions• Algorithms do not meet quality expectations for

unstructured data analytics

• Humans are unpredictable, slow, error-prone, required skills may not be available immediately

Problem Statement

No microtask-based crowdsourcing platform provides automated, runtime management of all three SLOs of

accuracy, time & budget

Complete task S with quality/accuracy of the results being at least A*, while ensuring that the total money spent is less than budget B* and the total time taken is less than T*

Budget

Time Accuracy

Human+Machine

Computing Agents

We assume data parallel tasks – similar but independent microtasks with different inputs

Proposed Approach

Introducing new System Control Points

• H-M Ratio (λ)

ratio of number of human tasks 𝒏𝒉 to number of machine tasks 𝒏𝒎

• Microtask-completion-rate (MCR)

ρ : rate of completing microtasks

H

M

nmicrotasks

nh

nm

• 𝑐ℎ 𝑡 = payment per microtask to each human worker at time 𝑡

• 𝑐𝑚 𝑡 = payment per microtask to each machine agent at time 𝑡

• 𝑘ℎ 𝑡 = number of workers per each human assigned microtaskat time 𝑡

• 𝑘𝑚 𝑡 = number of machine agents per each machine assigned microtask at time 𝑡

• 𝛿ℎ = estimated prior accuracy of human workers

• 𝛿𝑚 = estimated prior accuracy of machine agents

𝑐ℎ 𝑡 ≥ 𝑐𝑚 𝑡 and 𝛿ℎ ≥ 𝐴∗ ≥ 𝛿𝑚

System Model

Proposed Approach

Given 𝐴∗, 𝐵∗, 𝑇∗ SLO specifications, determine initial HM Ratio and MCR

𝑛ℎ 𝑡 =𝜆 𝑡

1 + 𝜆 𝑡𝑛(𝑡)

𝑛𝑚 𝑡 =1

1 + 𝜆 𝑡𝑛(𝑡)

𝜆 𝑡 =𝑛ℎ 𝑡

𝑛𝑚 𝑡

𝐴∗ − 𝛿𝑚𝛿ℎ − 𝐴∗

≤ 𝜆 𝑡0 ≤𝐵∗ − 𝑛𝑘𝑚 𝑡0 𝑐𝑚 𝑡0𝑛𝑘ℎ 𝑡0 𝑐ℎ 𝑡0 − 𝐵∗

𝜌∗ = 𝜌 𝑡0 =𝑛

𝑇∗

𝑛ℎ 𝑡0 𝛿ℎ + 𝑛𝑚 𝑡0 𝛿𝑚𝑛

≥ 𝐴∗

Initial Constraints

𝑘ℎ 𝑡0 𝑐ℎ 𝑡0 𝑛ℎ 𝑡0 + 𝑘𝑚 𝑡0 𝑐𝑚 𝑡0 𝑛𝑚 𝑡0 ≤ 𝐵∗

Dynamic Microtask Execution Control

DMTEC Algorithm

Divide total time 𝑇∗ into 𝐾 polling intervals of equal duration 𝜏𝑝𝑜𝑙𝑙

After every 𝜏𝑝𝑜𝑙𝑙, compute current values of

𝑛 𝑡 : number of remaining microtasks at time 𝑡

Current ideal MCR: 𝜌∗ 𝑡 =𝑛(𝑡)

𝑇∗−𝑡

Current MCR: 𝜌 𝑡 =𝑛−𝑛(𝑡)

𝑡

Current Accuracy: 𝐴 𝑡 =𝑐𝑜𝑟𝑟𝑒𝑐𝑡(𝑡)

𝑛

Current Budget: 𝐵 𝑡 = 𝐵∗ − 𝑐𝑜𝑠𝑡(𝑡)

Dynamic Microtask Execution Control

States requiring Corrective Actions

Case 1: 𝜌 𝑡 < 𝜌∗ 𝑡 but 𝐴(𝑡) ≥ 𝐴∗

tasks being completed slowly, but work quality (and hence workers) is satisfactory

Case 2: 𝐴(𝑡) < 𝐴∗ but 𝜌 𝑡 ≥ 𝜌∗ 𝑡

tasks being done at satisfactory rate but quality of work not good enough (workers probably not motivated enough)

Case 3: 𝜌 𝑡 < 𝜌∗ 𝑡 and 𝐴(𝑡) < 𝐴∗

tasks not being done within allotted time and hence accuracy falls or, workers not skilled/motivated enough

DMTEC Corrective Actions

Case 3 Actions: subject to 𝐵∗, determine best action between

change to mobile/captive expert workers to

improve both 𝐴 𝑡 and 𝜌 𝑡

package several HITS to increase quantum of payment per work hence microtaskpayment attractiveness

Case 1 Actions

Change 𝝀 𝒕 in favor of machines in order

to increase 𝜌 𝑡

Increase 𝒄𝒉, subject to satisfying 𝐵∗

Or, both

Case 2 Actions: subject to meeting 𝐵∗, determine best action between

Increase 𝒄𝒉 to improve attractiveness of the HITs

Increase 𝒌𝒉 to improve accuracy by getting more answers

Change worker group to more skilled workers

Change 𝝀 𝒕 in favor of humans to increase 𝐴 𝑡

Or, combination of the above four

Mobile/captive workers special class of workers • Microtask assignments can be ”pushed” • Will work on assigned micotasks with least average delay

among all worker types• Produces result quality comparable to expert workers

Experiment Results

Using data from Intention Analysis Experiments

• General Crowd = Amazon MTurk workers• cH = $0.02

• Used 𝑘ℎ = 3 and 𝑘ℎ = 5

• Machine Agent = HP AUTONOMY IDOL• cM = $0.001

• mean accuracy of trained classifier = 67%

• Simulate Mobile Expert Crowd using in-house expert labeled data• cH = $0.05

• Used 𝑘ℎ = 3

Used set of 250 tweets • Each tweet categorized into one of 6 intent

categories• Promotion• Appreciation• Information Sharing• Criticism• Enquiry• Complaint

• Experimented with 3 types of crowd• Known, expert workers

• Workers from AMT with no special qualification or training

• Experienced workers from AMT who had taken our short training for qualification

Experiment Results

Agent Type % Accuracy Time (days) Cost ($)

Public (3 votes) 57.2 1 15

Public (5 votes) 78 1 25

Expert Workers (3 votes)

91.8 1 37.5

Public Qualified (3 votes)

80.4 7 15

HP Autonomy IDOL 67.2 < 1 1.25

Performance of different Agent Types

Results for 250 tweets dataset

Simulation Results

Case 1 Action (Action 1): change 𝜆 in favor of machines in order to increase ρ(𝑡)

Case 2 Action (Action 2): subject to meeting 𝐵∗, increase 𝑘ℎ = # of assignments per HIT to improve accuracy by getting more answers

Case 3 Action (Action 3): subject to 𝐵∗, change to mobile expert workers to improve both 𝐴(𝑡) and

ρ(𝑡)

Simulated mobile expert workers from expert crowd data - sans the wait time for the tasks to be picked up (random wait time added to offset pickup time of each task)

Assumption: mobile workers start working on their tasks within next polling instance 𝜏𝑝𝑜𝑙𝑙 = 30 𝑚𝑖𝑛𝑠

Used Subset of Corrective Actions

Simulation Results

X-axis represents duration of 2 days

Blue dashed and solid lines represent 𝐴∗ =80% and 𝐴 𝑡 respectively

Red dashed and solid lines represent 𝐵∗ =$20 and 𝑐𝑜𝑠𝑡(𝑡) respectively

Using data from 250 tweets, consistently achieved

Accuracy ≥ 𝟖𝟒% in 1.5 days

Budget spent ≤ $𝟏𝟗. 𝟓9

Best results: 85.2% accuracy in 1.5 days for $19.59

Simulation Results

Time when action-2 taken is critical

Similar results seen for other corrective actions

Conclusion

Proposed new human-machine hybrid computing platform for unstructured data analytics

Dynamic microtask execution management to actively attempt satisfying 𝐴∗, 𝐵∗, 𝑇∗ SLO specifications

Introduced new System Control Points of HM-Ratio 𝜆 and MCR 𝜌

Simulation results show DMTEC algorithm outperforms both purely human and purely

machine computing agents for a given 𝐴∗, 𝐵∗, 𝑇∗

Thank You