Online Student Peer Reviews

William J. Wolfe

Professor of Computer Science

California State University Channel Islands

william.wolfe@csuci.edu

Online Peer Reviews: Outline

• Pros and cons

• Implementation Considerations

• Students Websites

• Course Website

• Peer Review Process

• Results

Peer Reviews – Pro

• Students learn from each other.• Students get lots of feedback.• Students develop skills as evaluators.• Students learn to appreciate evaluation criteria.• Students see how they compare to their peers.• Students see the class from teacher’s perspective.• Students get to know each other.• Teacher plays role of supervisor

(A much better use of the teacher’s skills/knowledge).

Peer Reviews – Con

• Students don’t know the subject.• Students are not skilled evaluators.• Students can not, or will not, do that much work.• Students will copy (cheat)!• Keeping track of the reviews is very difficult.• Student privacy.

Considerations

• What type of assignment?• How many reviews does each student do?• How many reviews does each student get?• Who reviews whom?• Does the reviewer have to be “qualified”?• Will students grade fairly and accurately?• Anonymous reviews?• Grades based on peer reviews?• Grade the peer reviews?• Opportunity to revise based on reviews?• Peer review of the peer reviews?

Peer Review System

Student Website

• Students use their own website.

• Students must have basic web page skills.

• Student must have access to a web server.

• Student cannot (easily) hide their identity.

  COMP 449   Human Computer Interaction

  John Doe

Weekly Assignments COMP 449 Assignment #1

COMP 449 Assignment #2

COMP 449 Assignment #3

COMP 449 Assignment #4

COMP 449 Assignment #5

COMP 449 Assignment #6

COMP 449 Assignment #7

COMP 449 Assignment #8

COMP 449 Assignment #9

COMP 449 Assignment #10

COMP 449 Assignment #11

COMP 449 Assignment #12

COMP 449 Assignment #13

COMP 449 Assignment #14

Course Website

• Instructor sets up a course website

• Web pages

• Database

• Server-side scripts

• Let’s try it out:

Peer Review Process

• Student logs on the course website

• Sees list of links to other students pages

• Accesses student assignments

• Enters “score + comment” for each review

Entering a Peer Review

Grading Criteria (Rubric)

 ”Looks pretty good”

Sample Peer Review

Perfunctory Reviews

perfunctory \pur-FUNGK-tuh-ree\ --adjective : Done merely to carry out a duty; performed mechanically; done in a careless and superficial manner; characterized by indifference

You should have requirements that detail the concepts in section 4.2. Although you had some very good points (i.e. the database should look up student's degree requirements; view should list courses, etc...) almost all your requirements can be more detailed. Go through section 4.2 (each of the sections) and think of what the program would need to do to effective run. Some good examples of what requirements are necessary are on others' websites, however I'll give some to you now:1.Is there a timeline requirement?2.Is there a requirement on how much(or how little) this will cost?3.Is there security requirements?4.Is there user view requirements?These(and many other questions) are what you should answer in your requirements definition document. Good luck on Assignment #3.

Sample Peer Review

Let’s try it out:

http://faculty.csuci.edu/william.wolfe/ucd/online/

Average Review Score Comp 449 Spring 2005

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

score

Stu

den

t R

anki

ng

# Missing Homworks Comp 449 Spring 2005

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Student Ranking

Sco

re

Average Score Given Comp 449 Spring 2005

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Student Ranking

Sco

re

# Reviews Received Comp 449 Spring 2005

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Student Ranking

Sco

re

# Reviews Given Comp 449 Spring 2005

0

20

40

60

80

100

120

140

160

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Student Ranking

Sco

re

Length of Comments Given Comp 449 Spring 2005

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Student Ranking

Sco

re

Length of Comments Received Comp 449 Spring 2005

0

2000

4000

6000

8000

10000

12000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Student Ranking

Sco

re

Logon ID Comp 449 Spring 2005

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Student Ranking

ID #

Comp 449 Spring 2005 Assignment 1 Student ID:6

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

student

sc

ore You Gave

Class Average

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10

You Gave

Class Avg

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10

You Gave

Class Avg

Summary

Stimulated class activity.

Some passionate participation.

The “audience effect”: brought up all performance levels.

Very accurate evaluations (as a whole).

Immediate access to examples of good and poor work.

Addressed late, incomplete, and sloppy work.

Needed access to web servers and web page skills.

1. Online Student Peer Reviews, Proceedings of ACM SIGITE Annual Conference, Salt Lake City Utah, Oct. 28-30, 2004.

2. Student Peer Reviews in an Upper-Division Mathematics Class, exchanges THE ONLINE JOURNAL OF TEACHING AND LEARNING IN

THE CSU, (From the Classroom), September, 2003.

3. Course Web Site:

http://compsci.csuci.edu/wwolfe/ucd/online

Password: GUEST

4. William.Wolfe@csuci.edu

References

Acknowledgements

Carol Holder (Director of Faculty Development CSUCI)

Paul Rivera (Economics, CSUCI)

Harley Baker (Psychology, CSUCI)

Bob Bleicher (Education, CSUCI)

Ivona Grzegorzcyk (Mathematics, CSUCI)

Todd Gibson (Colorado Institute of Technology)

Michael Cook (Forstmann Leff).

Peer Reviews – How?

• Student Web Pages: – Students post homework solutions on their

own web page.

• Course Web Site:– Set up course web site to manage all the peer

review activity. Keep track of: • Links to student web pages, • Peer Reviews:

– Scores,– Comments.

• Anonymous reviews.

The Course Web Site

List of Student Links

Student Web Pages

Peer Reviews Received

Average Peer Review Score

Scoring Comparison

Software Engineering (CSC 4508):

34 students

Theory:

1 Assignment: 1,122 reviews.

15 Assignments: 16,830 reviews.

Fact:

1 Assignment: 300 – 400 reviews.

15 Assignments: 5,212 reviews.

Number of Reviews

Average Review Score

0

1

2

3

4

5

6

7

8

9

10

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

Students

Score

Software Engineering (CSC4508) Fall 2002

Software Engineering (CSC 4508) Fall 2002

Average Review Score Given

0

1

2

3

4

5

6

7

8

9

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

Student

sco

re

Number of Reviews Received(CSC 4508 Fall 2002)

0255075

100125150175200225250275300325350375400

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

Student Ranking

Number of Reviews Given(CSC 4508 Fall 2002)

0255075

100125150175200225250275300325350375400

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

Student Ranking

Avg Deviation(CSC 4508 Fall 2002)

00.10.20.30.40.50.60.70.80.9

11.11.21.31.41.51.61.71.81.9

2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

Student Ranking

Avg Delta(CSC 4508 Fall 2002)

-0.1

-0.05

0

0.05

0.1

0.15

0.2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

Student Ranking

Distribution of Scores

0

200

400

600

800

1000

1200

1400

1600

1 2 3 4 5 6 7 8 9 10

Score

Cou

nt

Software Engineering (CSC 4508) Fall 2002

Logon Sequence(CSC 4508 Fall 2002)

0

5

10

15

20

25

30

35

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

Students

Average Review Score Received

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Student Ranking

sco

re

Real Analysis (Math 351) Spring 2003