Evaluating the Effectiveness of Feedback in SQL-Tutor

Antonija Mitrovic, Brent Martin

Intelligent Computer Tutoring GroupComputer Science Department

University of Canterbury

Christchurch, New Zealand

Overview

• SQL-Tutor

• Feedback types

• Evaluation study

• Results

Architecture of the standalone version of SQL-Tutor

SQLT-Web: a Web-enabled SQL tutor

• http://ictg.cosc.canterbury.ac.nz:8000/sql-tutor/

• Interface developed in CL HTTP-server

• Used in COSC courses since 1999

• Open for outside users on March 29, 2000

Architecture of SQLT-Web

Feedback types

• Positive/negative• Error flag• Hint• All errors• Partial solution• Complete solution

Hypothesis:

• Positive/negative and Error flag too general

• Low-level feedback contraproductive

(partial and complete solution)

• Constraint-based feedback most effective

(hint and all errors)

Evaluation study

• COSC313, May 1999 (Web)

• Single 2-hour session

• 6 lectures + 8 hours of labs

• 33 students

Three groups:

• Detailed (complete and partial solution)

• General (hint and all errors)

• Limited (pos/neg and error flag)

Results 1

Group Solved (%) NoAttempts Time/attempt

Detailed 87.07 2.21 65.26s

General 83.49 2.16 47.80s

Limited 84.10 2.24 78.05s

Difference in times is significant

Results 2Feedback Const Succ Fail Learn Corr (%)

Pos/neg 436 254 636 29% 78.0

Error flag 116 98 126 44% 81.8

Hint 43 33 43 43% 74.4

All errors 64 72 81 47% 80.0

Partial 26 22 10 69% 91.6

Full sol 18 6 16 27% 44.2

Initial learning rate

• All errors 0.44

• Error Flag 0.40

• Pos/Neg 0.29

• Hint 0.26

• Partial 0.15

• Full solution 0.13

Conclusions

• General feedback is most effective

• Detailed feedback is detrimental to learning

• Feedback level needs to be adapted