Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe:...

33
Pricing Crowdsourcing-based Software Development Tasks Insitute of Software, Chinese Academy of Sciences Ke Mao (ICSE'13-NIER) dreamict[at]gmail.com Feb. 11th, 2013 @ CREST, UCL

Transcript of Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe:...

Page 1: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Pricing Crowdsourcing-based Software Development Tasks

Insitute of Software, Chinese Academy of Sciences

Ke Mao

(ICSE'13-NIER)

dreamict[at]gmail.com

Feb. 11th, 2013@ CREST, UCL

Page 2: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

2222

Author

http://www.linkedin.com/in/kemao

Page 3: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Overview

� Background� Crowdsourcing: Micro task VS. Complex task� The TopCoder Platform

� Motivation� New Phenomenon� The Pricing Issue

� Methodology

� Experiments & Insights

� Conclusion

3333

Page 4: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Overview

� Background� Crowdsourcing: Micro task VS. Complex task� The TopCoder Platform

� Motivation� New Phenomenon� The Pricing Issue

� Methodology

� Experiments & Insights

� Conclusion

� Background� Crowdsourcing: Micro task VS. Complex task� The TopCoder Platform

4444

Page 5: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

A Recent News...

� Typical work day of one star developer:� 09:00 a.m. – Arrive and surf Reddit, watch cat videos� 11:30 a.m. – Take lunch� 01:00 p.m. – Ebay time� 02:00 p.m. – Facebook updates – LinkedIn� 04:30 p.m. – End of day update e-mail to management� 05:00 p.m. – Go home

5555

Page 6: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Introduction to Crowdsourcing

� A proper way...� Labor of the Internet� Low cost� Suprising deliverable

� Wisdom of the CrowdCrowdCrowdCrowd

6666

Page 7: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

What is Crowdsourcing ?

� "Crowdsourcing" defined by Jeff Howe:� The act of a company or institution taking a function

once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call.

� Crowdsourcing VS. Outsourcing:�The crucial prerequisite is the use of the open call open call open call open call

formatformatformatformat and the large networklarge networklarge networklarge network of potential laborers.

7777

Page 8: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Micro Task VS. Complex Task8888

Page 9: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Micro Task VS. Complex Task

Credit: http://sandfishdesign.co.uk, © 2012, Crowdsourcing, LLC

9999

Page 10: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

� The world's largest competitive community for crowdsourced software development

� The TopCoder Community is 425,993 strong

� Membership�China� India�U.S.

What is TopCoder ?10101010

Page 11: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

What is TopCoder ?

� What kinds of projects can I do with TopCoder?What kinds of projects can I do with TopCoder?What kinds of projects can I do with TopCoder?What kinds of projects can I do with TopCoder?�Mobile Applications�Analytics and Optimization�Scientific Algorithm Development�Online Communities�Open Platforms�Digital Media�Business Systems�…

11111111

Page 12: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

How Does It Work ?12121212

Credit: www.topcoder.com, © 2007, TopCoder, Inc

Page 13: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Overview

� Background� Crowdsourcing: Micro task VS. Complex task� The TopCoder Platform

� Motivation� New Phenomenon� The Pricing Issue

� Methodology

� Experiments & Insights

� Conclusion

� Motivation� New Phenomenon� The Pricing Issue

13131313

Page 14: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Motivation - New Phenomenon

� New Paradigm�Crowdsourced development�1. Open call format�2. Large networked potential labor

Fig.1 Illustration of crowdsourcing-based software development process.

2. Global Labor

1. Competition

14141414

Page 15: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Motivation - New Phenomenon

� New Phenomenon In SE activity

�2 examples that challenge traditional law

�Parkinson's Law

�COCOMO Model

15151515

Page 16: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Motivation - New Phenomenon

� Parkinson's Law

Fig.2 Correlation between the time allocated and the actual time consumed

(“Work expands so as to fill the time available for its completion. ” )

16161616

Page 17: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Motivation - New Phenomenon

� Basic COCOMO Model

Fig.3 The effort estimated by COCOMO model, compared to the actual effort.

( EFFORT = a * SIZE b )

17171717

Page 18: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Motivation - The Pricing Issue

� Inappropriate price often lead to low capital efficiency and task starvation

� How to build empirical pricing models?How to build empirical pricing models?How to build empirical pricing models?How to build empirical pricing models?

18181818

Fig.4 Active Component Development Contests on TopCoder.com

Page 19: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Overview

� Background� Crowdsourcing: Micro task VS. Complex task� The TopCoder Platform

� Motivation� New Phenomenon� The Pricing Issue

� Methodology

� Experiments & Insights

� Conclusion

� Methodology

19191919

Page 20: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Methodology

� Price Drivers

Development Type

TABLE I. DESCRIPTIVE STATISTICS AND REGRESSION COEFFICIENTS OF PROPOSED FACTORS

Quality of Input

Input Complexity

Previous Phase Decision

20202020

Page 21: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Methodology

� Predictive Models�Multiple Linear Regression Model:

�8 other Machine Learning & Statistical models

3 Decision Tree 3 Decision Tree 3 Decision Tree 3 Decision Tree based learnersbased learnersbased learnersbased learners

2 Instance 2 Instance 2 Instance 2 Instance based learnersbased learnersbased learnersbased learners

1 Neural Net1 Neural Net1 Neural Net1 Neural Net1 Support 1 Support 1 Support 1 Support Vector Vector Vector Vector Machine Machine Machine Machine

1 Logistic 1 Logistic 1 Logistic 1 Logistic RegressionRegressionRegressionRegressionC5.0, CART,

QUESTKNN-1,KNN-k∈[3, 7]

21212121

Page 22: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Overview

� Background� Crowdsourcing: Micro task VS. Complex task� The TopCoder Platform

� Motivation� New Phenomenon� The Pricing Issue

� Methodology

� Experiments & Insights

� Conclusion

� Experiments & Insights

22222222

Page 23: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Experiments

� Aim:�To answer the following RQs.

� RQs:�Baseline Comparison�How much better?

�Performance Assessment�Which is the best?

�Actionable Insights�What guidance can we offer?

23232323

Page 24: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Experiments

� Dataset�Sep 29th 2003 to Sep 2nd 2012�2,895 design and 3,015 development tasks�490 successful sw dev projects from TopCoder

� Validation method

�LOOCV

24242424

Page 25: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

� Performance Measures:

Experiments25252525

Page 26: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Experimental Results

� Answer to RQ1:� Outperformed by all 9 predictive models, according

to Pred(30) measure

26262626

Fig.5 Performance of pricing models learned by each approach

Page 27: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Experimental Results

� Answer to RQ2:� Decision tree based learners� C5.0, QUEST, CART

27272727

Fig.5 Performance of pricing models learned by each approach

Page 28: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Insights

� Answer to RQ3:�Significance Anlysis

TABLE I. DESCRIPTIVE STATISTICS AND REGRESSION COEFFICIENTS OF PROPOSED FACTORS

28282828

Page 29: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Insights

� Answer to RQ3:�Rules of Thumb�ISUP => $70↓

�COMP(4 pages) => $30↑�SEQU(4 diagrams) => $30↑�SIZE(1 KSLOC) => $30↑

�May not alway be right�But "Why am I bucking the trend?"

29292929

Page 30: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Overview

� Background� Crowdsourcing: Micro task VS. Complex task� The TopCoder Platform

� Motivation� New Phenomenon� The Pricing Issue

� Methodology

� Experiments & Insights

� Conclusion� Conclusion

30303030

Page 31: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Conclusion

� Analyzed 5,910 sw dev tasks on TopCoder

� Proposed 16 price drivers

� Assessed 12 empirical pricing models

� Useful prediction quality is achievable (Pred(30)>0.8)

� Actionable advice can be extracted from our models to assist the developers on TopCoder

31313131

Page 32: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

Future Work

� Quality & Risk Factors

� Price / Quality Trade off� Assessing task complexity via UML design

� Muti-objective Optimization� Price / Quality / Risk

32323232

Page 33: Pricing Crowdsourcing-based - UCL · What is Crowdsourcing ? "Crowdsourcing" defined by Jeff Howe: The act of a company or institution taking a function once performed by employees

谢谢!

Thanks !