Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using...
Transcript of Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using...
![Page 1: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/1.jpg)
Automatic feedback for student mistakes using examples
Björn Hartmann
Gustavo Soares
Loris D’Antoni
1
![Page 2: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/2.jpg)
Personalized Education
• CCC CAPE 2016 Workshop Vision:“Personalization can meet the demands of heterogeneous backgrounds and different learning styles, and ensure engagement and retention.”
• Enabled by online learning platforms (MOOCs and on-campus blended classes) that allow tailoring problem selection and feedback generation to individual students’ needs.
2
![Page 3: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/3.jpg)
Example: AutomataTutor
3
Draw the DFA accepting the language:
{ s | ‘ab’ appears in s exactly 2 times }
Student’s Solution:
Feedback: Your DFA accepts the language
{ s | ‘ab’ appears in s at least 2 times }
(D’Antoni et al, TOCHI 2015; deployed at 20 universities)
![Page 4: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/4.jpg)
Generating Problem-Specific Hints
4
Student’s attempt
Correct solution
Feedback
GeneratorHint
Additional
Problem-Specific
Information
Challenge: How to include information about errors that are specific to the problem, but general enough to capture many different student instances of the error?
![Page 5: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/5.jpg)
Problem-Specific Hints for Code
• Autograder for Introductory Programming(Singh, Gulwani, Solar-Lezama): Instructors must manually write down error model.
• Our approach:Assume many students are working on the same problem set. Learn hints from student data.
5
![Page 6: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/6.jpg)
Setting: CS61a – Intro CS at UC Berkeley
• ~1500 students/semester
• Autograding infrastructure already in place: students submit problems to grading server (http://okpy.org), which runs a test suite, logs code and status, and returns feedback to student
• >200GB of plain-text python source code from last few semesters
• Next: DS8 Intro to Data Science –needs more instrumentation
6
![Page 7: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/7.jpg)
Approach
• Focus on code that runs but fails test suite.
• Capture (incorrect correct) source pairs from students (cf. Hartmann et al., CHI 2010)
• Extract a reusable transformation script from each source pair using synthesis.
• If script succeeds to fix new program, turn template into conceptual hint.
7
![Page 8: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/8.jpg)
8
def product(n, term):i = 1product = 1while i <= n:
product *= ii += 1
return product
def product(n, term)i = 1product = 1while i <= n:
product *= term(i)i += 1
return product
Program synthesis
Input-output example specification
𝑥 → term 𝑥Hint
generation
When should you call the passed-in
function “term” in your while loop?
Write a function product(n, term) that returns
term(1) * ... * term(n)
![Page 9: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/9.jpg)
9
DSL
syntax
DSL
semantics
Synthesis
AlgorithmsRanking
Transformation script synthesis
Input-output
ExampleInput-output
Example
A technique for learningtransformations using examples
Top Ranked
scripts
![Page 10: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/10.jpg)
10
DSL
syntax
DSL
semantics
Synthesis
AlgorithmsRanking
Transformation script synthesis
Input-output
ExampleInput-output
ExampleTop Ranked
scripts
Microsoft PROSE (FlashMeta) as the framework for program synthesis
[Polozov and Gulwani, OOPSLA’15]
A technique for learningtransformations using examples
![Page 11: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/11.jpg)
11
DSL example
i
+=
p
call
+=
p
term Arg
i
![Page 12: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/12.jpg)
12
DSL example
i
+=
p
call
+=
p
term Arg
i
edit = Insert(node, ConstNode(CallExpression, Children( LeafConstNode(NameExpression-term), SingleChild( ConstNode(Arg, SingleChild(ReferenceNode(node, NameExpression(“i”))))))), 1)
![Page 13: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/13.jpg)
13
DSL example
i
+=
p
call
+=
p
term Arg
i
edit = Insert(node, ConstNode(CallExpression, Children( LeafConstNode(NameExpression-term), SingleChild( ConstNode(Arg, SingleChild(ReferenceNode(node, NameExpression(“i”))))))), 1)
![Page 14: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/14.jpg)
14
DSL example
i
+=
p
call
+=
p
term Arg
i
edit = Insert(node, ConstNode(CallExpression, Children( LeafConstNode(NameExpression-term), SingleChild( ConstNode(Arg, SingleChild(ReferenceNode(node, NameExpression(“i”))))))), 1)
selectedNodes = Selected(λx => Match(x, AugmentedAssignStatement {NameExpression NameExpression}), InOrderSort(ast))
![Page 15: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/15.jpg)
15
DSL example
i
+=
p
call
+=
p
term Arg
i
EditMap(λnode => edit, selectedNodes) Where
edit = Insert(node, ConstNode(CallExpression, Children( LeafConstNode(NameExpression-term), SingleChild( ConstNode(Arg, SingleChild(ReferenceNode(node, NameExpression(“i”))))))), 1)
selectedNodes = Selected(λx => Match(x, AugmentedAssignStatement {NameExpression NameExpression}), InOrderSort(ast))
![Page 16: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/16.jpg)
16
@start IEnumerable<PythonAst> transformation := Apply(ast, patch);Patch patch := Patch(editList) | ConcatPatch(editList, patch);IEnumerable<Edit> editList := EditMap(edit, selectednodes) =
Map(λnode : PythonNode => edit, selectednodes);IEnumerable<PythonNode> selectednodes := Selected(match, nodes) =
Filter(λx : PythonNode => match,nodes); bool match := Match(x, template);Edit edit := Update(node, n) | Insert(node, n, k) | Delete(node, r);Node r := ReferenceNode(node, template); Node n := LeafConstNode(info) | ConstNode(info, children) | r;IEnumerable<Node> children := SingleChild(n) | Children(n, children);IEnumerable<PythonNode> nodes := InOrderSort(ast);
@input PythonNode ast;@feature[Score=TemplateScore] PythonNode template;@feature[Score=InfoScore] NodeInfo info; @feature[Score=KScore] int k;
DSL syntax
![Page 17: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/17.jpg)
Program synthesis
17
Example
InputOutput
p += ip += term(i)
Apply(patch) Insert(term(i)), 0, 1)
Insert(term(i)), 0, 1) Insert(node, n) term(i)
i LeafConstNode(type, value) NameExpression
ReferenceNode(value, template)i
i
+=
p
i
![Page 18: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/18.jpg)
18
Ranking
• Prefer ReferenceNode to ConstNode
• Prefer abstract templates to concrete templates
![Page 19: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/19.jpg)
Preliminary Evaluation
19
Can our technique generate fixes for student submissions
based on previous data?
CS 61a
![Page 20: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/20.jpg)
20
Leave-one-out
cross validation
Question Mistakes
Product 215
Repeated 108
• Mistake is a pair of submissions of a single student containing her correct submission and her last incorrect submission
• Mistakes were taken from a sample of submissions clustered by test cases
![Page 21: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/21.jpg)
Our technique fixed 81% of the submissions
21
Question Mistakes Fixed mistakes Scripts
Product 215 190 (88%) 8
Repeated 108 73 (68%) 10
![Page 22: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/22.jpg)
22
Examples
Insert(b𝑖𝑛𝑎𝑟𝑦𝐸𝑥𝑝, term(𝑥), 1)Insert(𝒂𝐬𝐬𝐢𝐠𝐧𝐒𝐭𝐦𝐭, 𝐭𝐞𝐫𝐦(𝒙), 1)Insert(𝑟𝑒𝑡𝑢𝑟𝑛𝑆𝑡𝑚𝑡, 𝑡𝑒𝑟𝑚(𝑥), 1)Update(𝑙𝑖𝑡𝑒𝑟𝑎𝑙, n), Insert(𝑟𝑒𝑡𝑢𝑟𝑛𝑆𝑡𝑚𝑡, term(𝑛), 1)Update(𝑛𝑎𝑚𝑒𝐸𝑥𝑝, Identity)Update(𝑙𝑖𝑡𝑒𝑟𝑎𝑙, Identity)Insert(if (𝒏 == 𝟎) return identity, ifStmt, 0)Update(𝑒𝑥𝑝𝑆𝑡𝑚𝑡, return 𝑒𝑥𝑝𝑆𝑡𝑚𝑡)Insert(return 𝒏𝒂𝒎𝒆𝑬𝒙𝒑, 𝑺𝒖𝒊𝒕𝒆𝑺𝒕𝒎𝒕, 0)
Update(𝒍𝒊𝒕𝒆𝒓𝒂𝒍, 𝟏)Update(𝒍𝒊𝒕𝒆𝒓𝒂𝒍, 𝟎)Delete(𝑎𝑟𝑔, 𝑛 − 1), Insert(𝑛, 𝑎𝑟𝑔, 0)Insert(𝑎𝑟𝑔, 𝑛 + 1)Update(𝑏𝑖𝑛𝑎𝑟𝑦𝐸𝑥𝑝,<)Update(𝑛𝑎𝑚𝑒𝐸𝑥𝑝, j),Update(𝑛𝑎𝑚𝑒𝐸𝑥𝑝, j), Update(𝑙𝑖𝑡𝑒𝑟𝑎𝑙, 0)
General
Question Specific
![Page 23: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/23.jpg)
Next steps
23
![Page 24: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/24.jpg)
Data-centric hints
24
1. def product(n, term):2. i = 13. product = 14. while i <= n:5. product *= i6. i += 17. return product
1. def product(n, term):2. i = 13. product = 14. while i <= n:5. product *= term(i)6. i += 17. return product
Actual
Expected
![Page 25: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/25.jpg)
Further studies
First deployment in CS61a summer or fall 2016
User study: controlled experiment of different hints
Comparison to Autograder
25
![Page 26: Automatic feedback for student mistakes examples · Automatic feedback for student mistakes using examples Björn Hartmann Gustavo Soares Loris D’Antoni 1. Personalized Education](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb992b9cc3e340cfe19b991/html5/thumbnails/26.jpg)
Automatic feedback for student mistakes using examples
Björn Hartmann
Gustavo Soares
Loris D’Antoni
26