Peer Evaluation Through Crowdsourcing

Vinod Kumar OraM.Tech Project

Under the guidance of

Prof. Deepak B. Phatakand

Prof. Ganesh Ramakrishnan

Computer Science & EngineeringIndian Institute of Technology, Bombay

October 28, 2014

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 1 / 21

1 Introduction

2 Types of EvaluationProcedure for Peer Evaluation System

3 Available Tool’s For Peer Evaluation

4 Problems in Peer Evaluation

5 Problem Scope

6 Proposed SolutionFinding interest of the Participants for EvaluationGroup Formation in Peer EvaluationAllocation Strategies of Projects for Evaluation, Based on TheirInterestIdentification of Biasing in EvaluationCalculating Final Grade

7 Conclusion and Future WorkVinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 2 / 21

Introduction

CrowdsourcingCrowdsourcing is the process of obtaining needed services, ideas, orcontent by soliciting contributions from a large group of people.

Crowdsourcing users can be classified in two categories:

1 Requester

2 Workers

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 2 / 21

Peer Evaluation

Peer evaluation is the evaluation of work by one or more people of similarcompetence.

Advantage

1 Motivation towards performing better

2 Better understanding of same topic while evaluating others

Disadvantage

1 Biasing, which is over-marking to friends and under-marking to others

2 A notion of ignorance towards feedback and comment

3 Disbelief on accuracy of system

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 3 / 21

Types of Evaluation

1 Expert Evaluation

2 Self Evaluation

3 Auto Evaluation

4 Peer Evaluation

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 4 / 21

Procedure for Peer Evaluation System

1 Group Formation

2 Peer Evaluation

3 Identification of Biasing(if any)

4 Calculate Final Grade

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 5 / 21

Available Tool’s For Peer Evaluation-I

PeEv(Peer Evaluation System)

1 Used for Project Evaluation

2 Evaluation is done based on some fixed set of criteria

3 Student marks their group members on each criteria on a certain scale

4 programmed in MYSQL

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 6 / 21

Available Tool’s For Peer Evaluation-II

SPARK(Self and Peer Assessment Resource Kit)

1 Works same as PeEv

2 The motivation behind that is students get equal group marks forunequal contribution

3 Additionally provide an option to negotiate the criteria with thestudents

4 Student can apply for re-evaluation, if he is not satisfied

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 7 / 21

Problems in Peer Evaluation

Reliability

Fairness

Predictive validity

Controlling biasing factor

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 8 / 21

Related Issue

Diversity in Intelligence

Time Consuming

Lack of motivation

Group formation

Validation of group formation

Verification and validation of evaluation system

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 9 / 21

Problem Scope

Problem ScopeDiversity in intelligence and domain knowledge, among participants,creates problem of ensuring fairness in evaluation. Fairness here refers tothe ability of participants to evaluates their peers assignments correctly

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 10 / 21

Proposed Solution

Proposed Solution

The problem of ensuring fairness in peer evaluation system may beresolved by allotting the works to participants based on their interest.

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 11 / 21

Finding interest of the Participants for Evaluation

To find interest of student, we can simply float a form which containdifferent sub-topics of Quiz

Student can give priority in numerical form say between 1 to 10.

Higher Number indicate, Higher interest/priority

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 12 / 21

Group Formation in Peer Evaluation

If There are total n students and m sub-topics, then in each group,there will be n/m students.

If n is not proper divisor of m, then n-(n/m)*m random students willbe available for manual allocation which may be done by instructor orthey may be available for next quiz.

Within the sub-group instructor can decide manually that how manystudents will evaluate to a single student.

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 13 / 21

Projects Allocation Strategies

For each question we will find a student who has given higher priorityor interest and allocate them to those sub-group

We will parform this step, for all sub-part alternately, to maintainquality of each sub-part

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 14 / 21

Identification of Biasing in Evaluation-I

To Identify biasing, instructor may set a biasing limit.

If suppose 10 students are evaluating to a student, some studentsmay give biased marks.

So we are going to find percentage(%) difference between max andmin marks, which are given by all 10 students.

If Percentage(%) difference is higher than biasing limit then we willre-schedule evaluation.

We may notedown the names of student who gave biased marks toimprove future evaluation.

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 15 / 21

Identification of Biasing in Evaluation-II

In following table we are assuming that biasing limit is 20%.

TotalMarks

Maxmarks

MinMarks

Difference (%)Diff Biased

10 8 3 5 50(%) YES10 9 7 2 20(%) NO10 10 6 4 40(%) YES20 16 13 3 15(%) NO100 75 60 15 15(%) NO50 42 30 12 24(%) YES

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 16 / 21

Calculating Final Grade

One simple solution is to take average of all marks given byparticipants. Then this average will become final grade for thatparicular student.

Since we can identified biasing in marks, so we need not to worryabout any descripancy in marks given by participants and can takedirctly average of all marks.

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 17 / 21

Conclusion

Large number of participants in MOOC

Traditional approach of evaluation is not suitable

Use peer evaluation

Challenge: Diversity in intelligence and knowledge among peers

Control over evaluation system

Minimize instructor workload in evaluation

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 18 / 21

Future Work

Implement strategies, for finding priority of students

Implement strategies, for controlling biasing factor in evaluation

Perform experiments with large number of participants

Implement this system with edx platform and deploy it

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 19 / 21

Refrences

Christina M Cestone, Ruth E Levine, and Derek R Lane. Peerassessment and evaluation in team-based learning. New Directions forTeaching and Learning, 2008(116):67-68, 2008

Hans-Dieter Daniel, Sandra Mittag, and Lutz Bornmann. Thepotential and problems of peer evaluation in higher education andresearch. Quality assessment for higher education in Europe, pages71-82

FJRC Dochy, Mien Segers, and Dominique Sluijsmans. The use ofself, peer and co-assessment in higher education: A review. Studies inHigher education 24(3):331-350, 1999

Sunny SJ Lin, Eric Zhi-Feng Liu, and Shyan-Ming Yuan. Web-basedpeer assessment: feedback for students with various thinking- styles.Journal of Computer Assisted Learning, 17(4):420-432, 2001

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 20 / 21

Thanks

Vinod Kumar Ora M.Tech Project Under the guidance of Prof. Deepak B. Phatak and Prof. Ganesh Ramakrishnan Computer Science & Engineering Indian Institute of Technology, BombayPeer Evaluation Through Crowdsourcing October 28, 2014 21 / 21