Constraint Based Synthesis for Beginners
description
Transcript of Constraint Based Synthesis for Beginners
Constraint Based Synthesis for Beginners
PSY 2012
Armando Solar-Lezama
CAP View of Synthesis
Synthesis Methodology Code
A different approach
Synthesis Methodology
CodeDomain Specific Tool
EXHIBIT A: SYNTHESIS OF SQLWith Alvin Cheung and Sam Madden
Motivation•It turns out SQL is challenging to learn
Who would have thought?
•Frameworks simplify program/DB interface
You can access the DB without using SQL Which can lead to some interesting code...
Examples: Explicit Selectpublic Set<Project> getUnfinishedProjects() { Set<Project> unfinishedP = new HashSet<Project>(); List<Project> projects = this.projectDao.getAllProjects();
for (Project project : projects) { if (!(project.getIsFinished())) { unfinishedP.add(project); } } return unfinishedP; }
Get list of projects from the DB
Select unfinished projects
SELECT * FROM Projects WHERE isFinished=FALSE
Examples: Explicit Selectpublic List<WilosUser> getRoleUser(){ List<WilosUser> listUser = new ArrayList<WilosUser>(); List<WilosUser> user= this.userDao.getUsers(); List<Role> role = this.roleDao.getRoles(); for(int i = 0; i < user.size(); i ++){ for(int a = 0; a < role.size(); a++){ if(user.get(i).getRole_id(). equals(role.get(a).getRole_id())){ WilosUser userok = user.get(i); listUser.add(userok); } } return listUser;}
Find users in the Roles list and add them to the output
Start with users and
roles
SELECT u FROM users u, roles r WHERE u.roleId == r.id
Why is this so bad?•These can be performance bottlenecks
Where performance matters, people write SQL
•More controversial arguments can be made
Is this a synthesis problem?•It is
SynthesizerImperative code with loop nests
Equivalent Formula in Relational Algebra
Domain knowledge
about relational algebra
Framing the problemList getUsersWithRoles () { List users = getUsersFromDB(); List roles = getRolesFromDB(); List results = []; int i = j = 0; while (i < users.size()) { while (j < roles.size()) { if (users[i].roleId == roles[j].id) results.add(users[i]); } }return results;}
Precondition: truePostcondition:
Framing the problem•We want to synthesize a post-condition!
•Challenges Defining the language Solving the synthesis problem Generating code
Defining the language•Requirements
Should make reasoning about the program easy
Should have expressiveness comparable to SQL
Language•Just like relational algebra...•...but with lists rather than sets
The program is written in terms of lists, not sets
• •
InvariantsList getUsersWithRoles () { List users = getUsersFromDB(); List roles = getRolesFromDB(); List results = []; int i = j = 0; while (i < users.size()) { while (j < roles.size()) { if (users[i].roleId == roles[j].id) results.add(users[i]); } }return results;}
Verification•Easy if you know the invariants
Z3 can do it in seconds
•All you need are a few axioms:
:...
Synthesis•Synthesis works best in a finite domain
•Invariant generation can be framed as a sketch
1) Model operations in terms of their source code
2) Construct the verification condition leaving invariant and post condition as unknowns
3) Create a sketch for unknowns4) Solve!
Results so far•We have tried this with a dozen loop nests from real open source projects
•All solve in less than 7 minutes Slow, but not very optimized.
EXHIBIT B: AUTOMATED GRADINGWith Rishabh Singh and Sumit Gulwani
The real software problem•The Software Quality problem is a symptom
•The real problem:The demand for software in our society far
exceeds the supply of people skilled enough to produce it
Three pronged attack
Make programmers more productive Make programming more accessible Reduce the cost of training the next generation
Grading Programming Assignments•Test-cases based grading
No precise correctness correlation No student tailored feedback
•Manual grading by TAs Error-prone Time consuming Expensive
•Manual grading will not scale to 100K students
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
22
Buggy Program for Array Reverse
6:28::50 AM
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length-1; i < a.Length-1; i--){
b[count] = a[i];count++;
}return b;
} }
23
Buggy Program for Array Reverse
6:32::01 AM
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length-1; i < a.Length-1; i--){
b[count] = a[i];count++;
}return b;
} }
24
Buggy Program for Array Reverse
6:32::32 AM
No change! Sign of Frustation?
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i <= a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
25
Buggy Program for Array Reverse
6:33::19 AM
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){ Console.Writeline(i);
b[count] = a[i];count++;
}return b;
} }
26
Buggy Program for Array Reverse
6:33::55 AM
Same as initial attempt except Console.Writeline!
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){ Console.Writeline(i);
b[count] = a[i];count++;
}return b;
} }
27
Buggy Program for Array Reverse
6:34::06 AM
No change! Sign of Frustation?
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i <= a.Length; i--){ Console.Writeline(i);
b[count] = a[i];count++;
}return b;
} }
28
Buggy Program for Array Reverse
6:34::56 AM
The student has tried this before!
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
29
Buggy Program for Array Reverse
6:36::24 AM
Same as initial attempt!
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length-1; i < a.Length-1; i--){
b[count] = a[i];count++;
}return b;
} }
30
Buggy Program for Array Reverse
6:37::39 AM
The student has tried this before!
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i > 0; i--){
b[count] = a[i];count++;
}return b;
} }
31
Buggy Program for Array Reverse
6:38::11 AM
Almost correct! (a[i-1] instead of a[i] in loop body)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i >= 0; i--){
b[count] = a[i];count++;
}return b;
} }
32
Buggy Program for Array Reverse
6:38::44 AM
Student going in wrong direction!
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
33
Buggy Program for Array Reverse
6:39::33 AM
Back to bigger error!
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
34
Buggy Program for Array Reverse
6:39::45 AM
No change! Frustation!
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
35
Buggy Program for Array Reverse
6:40::27 AM
No change! More Frustation!!
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
36
Buggy Program for Array Reverse
6:40::57 AM
No change! Too Frustated now!!! Gives up.
AutoGrader•Automate grading
Find semantic errors Feedback to fix them
Students make similar mistakes
Is this a synthesis problem?•It is
SynthesizerBuggy implementation
Corrections for student program
Error ModelReference Solution
Array Reverse
i = 1
i <= a.Length
Challenge 1: Different Algorithms
Challenge 2: Scalability
1010 different possible candidate corrections
Our Approach•Use data of previous student solutions
•Correction rules based on observed errors
•Create a set of candidate solutions using rules and find closest correct solution.
Aren’t rewrite systems hard?•Can teachers really write rewrite rules?
•Angelic non-determinism helps Ambiguities and redundancies no longer matter
Results: Problems Fixed
F1 F2 F3 F4 F50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Problems fixed
Array ReversePalindromeMaxFactorialisIncreasingSort
Error Models
Frac
tion
of
Prob
lem
s fix
ed
Results: Performance
F1 F2 F3 F4 F50
5
10
15
20
25
30
35
Performance with Error Models
reversepalindromemaxisIncreasingfactorialsort
Error Models
Run
ning
Tim
e (i
n s)
Results: Generalization
palindrome array max isIncreasing sort0
10
20
30
40
50
60
70
80
Generalization of Error models
Specific Error modelArray reverse error model
Num
ber
of fi
xes
Feedback for Tutoring System
OK for Grading,
But not ideal for teaching
Broad research agenda ahead•Transformative for students in under-funded schools
Reduce the resources required to support quality instruction Enable “true” distance education for programming courses
•Same technology could be used for automatic tutoring Identify errors stemming from deep misconceptions (e.g. not understanding difference in values vs. references)
Synthesize small examples that make misconceptions explicit
48
Sketch Tutorial on Saturday
http://bit.ly/sketch2012