Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2....
Transcript of Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2....
![Page 1: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/1.jpg)
Automatic Programming:How Far Can Machines Go?
Hila Peleg
Technion
![Page 2: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/2.jpg)
![Page 3: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/3.jpg)
Laziness
![Page 4: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/4.jpg)
Laziness Impatience
![Page 5: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/5.jpg)
Laziness Impatience Hubris
![Page 6: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/6.jpg)
Automatic Programming
Clean up my spreadsheet!Right away, boss!
![Page 7: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/7.jpg)
Program synthesis to the rescue
![Page 8: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/8.jpg)
Program Synthesis
⇒ ⇒
![Page 9: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/9.jpg)
What we really want
⇒ ⇒
This is what I want
![Page 10: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/10.jpg)
What if we have automatic programming?
• Hyper-intelligent program generation for your every need
![Page 11: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/11.jpg)
What if we have automatic programming?
• Hyper-intelligent program generation for your every need
• Self-aware, self-augmenting AI
![Page 12: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/12.jpg)
What if we have automatic programming?
• Hyper-intelligent program generation for your every need
• Self-aware, self-augmenting AI
• The singularity
![Page 13: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/13.jpg)
What if we have automatic programming?
• Hyper-intelligent program generation for your every need
• Self-aware, self-augmenting AI
• The singularity
• ???
![Page 14: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/14.jpg)
What if we have automatic programming?
• Hyper-intelligent program generation for your every need
• Self-aware, self-augmenting AI
• The singularity
• ???
• THE ROBOT APOCALYPSE
![Page 15: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/15.jpg)
The Robot Apocalypse
![Page 16: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/16.jpg)
But it’s ok!
![Page 17: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/17.jpg)
But it’s ok!
Understand users Build a program
![Page 18: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/18.jpg)
But it’s ok!
Generalizing partial intent is hard
Understand users Build a program
![Page 19: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/19.jpg)
But it’s ok!
Generalizing partial intent is hard
Solving HALT is hard*
Understand users Build a program
![Page 20: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/20.jpg)
![Page 21: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/21.jpg)
User intent is hard
Clean up my spreadsheet!Right away, boss!
![Page 22: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/22.jpg)
User intent is hard
Clean up my spreadsheet!Right away, boss!
What’s “clean”?Is this right?What am I doing with my life?
![Page 23: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/23.jpg)
Intent via examples
![Page 24: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/24.jpg)
Intent via examples
![Page 25: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/25.jpg)
Intent via examples
![Page 26: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/26.jpg)
Intent via examples
![Page 27: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/27.jpg)
Query knowledge about (some kind of) code
I want to apply foo() torequests to my http server
![Page 28: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/28.jpg)
Query knowledge about (some kind of) code
I want to apply foo() torequests to my http server
val routes : Route = ???
val bindingFuture = Http().
bindAndHandle(routes,
"localhost",
8080)
![Page 29: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/29.jpg)
Commit Strip said it best…
http://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/
![Page 30: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/30.jpg)
Commit Strip said it best…
http://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/
![Page 31: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/31.jpg)
Commit Strip said it best…
http://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/
![Page 32: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/32.jpg)
Commit Strip said it best…
http://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/
![Page 33: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/33.jpg)
Building a program is also hard*
Get me a program that takes a program and an input and tells me if that program stops on that input.
![Page 34: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/34.jpg)
Building a program is also hard*
Get me a program that takes a program and an input and tells me if that program stops on that input.
(i.e., the halting problem)
![Page 35: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/35.jpg)
Adjusting our expectations
Here’s a grammar of 20 functions and 10 constants, get me a program that I’m certain is in this space.
![Page 36: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/36.jpg)
Adjusting our expectations
Here’s a grammar of 20 functions and 10 constants, get me a program that I’m certain is in this space.
![Page 37: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/37.jpg)
Adjusting our expectations
Here’s a grammar of 20 functions and 10 constants, get me a program that I’m certain is in this space.
![Page 38: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/38.jpg)
What does this all mean?
![Page 39: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/39.jpg)
What does this all mean?
• Generally, “find me a program that—” cannot be solved
![Page 40: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/40.jpg)
What does this all mean?
• Generally, “find me a program that—” cannot be solved
• Still, we’re not giving up
![Page 41: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/41.jpg)
What does this all mean?
• Generally, “find me a program that—” cannot be solved
• Still, we’re not giving up
• Realistic expectations for realistic program synthesis
![Page 42: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/42.jpg)
Realistic expectations for realistic synthesis
We still want
• Partial specifications
• To not have to know
everything
• A result!
But we’ll have to live without
• Checking every possible
program
• Fully automatic solution
• Single-step solution
![Page 43: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/43.jpg)
Realistic expectations for realistic synthesis
We still want
• Partial specifications
• To not have to know
everything
• A result!
But we’ll have to live without
• Checking every possible
program
• Fully automatic solution
• Single-step solution
Synthesis Engine Interaction Model
![Page 44: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/44.jpg)
The synthesis engine
• Predicts code for intent
• Draws its understanding from language syntax• and/or crowd wisdom
• and/or semantic specifications
• Reduce the number of programs seen
![Page 45: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/45.jpg)
Reducing equivalent programs
• We’ve seen x+y, so we don’t want y+x
![Page 46: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/46.jpg)
Reducing equivalent programs
• We’ve seen x+y, so we don’t want y+x
• But how do we know they’re the same?
![Page 47: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/47.jpg)
Reducing equivalent programs
• We’ve seen x+y, so we don’t want y+x
• But how do we know they’re the same?
1. Heuristics
![Page 48: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/48.jpg)
Reducing equivalent programs
• We’ve seen x+y, so we don’t want y+x
• But how do we know they’re the same?
1. Heuristics
2. Solvers (e.g., Z3)
![Page 49: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/49.jpg)
Reducing equivalent programs
• We’ve seen x+y, so we don’t want y+x
• But how do we know they’re the same?
1. Heuristics
2. Solvers (e.g., Z3)
3. Observational Equivalence
![Page 50: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/50.jpg)
Reducing equivalent programs
• We’ve seen x+y, so we don’t want y+x
• But how do we know they’re the same?
1. Heuristics
2. Solvers (e.g., Z3)
3. Observational Equivalence
Equivalence:
𝑝1 ≡ 𝑝2 i.f.f. for every possible input 𝑖 ever, 𝑝1 𝑖 = 𝑝2 (𝑖)
![Page 51: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/51.jpg)
Reducing equivalent programs
• We’ve seen x+y, so we don’t want y+x
• But how do we know they’re the same?
1. Heuristics
2. Solvers (e.g., Z3)
3. Observational Equivalence
Observational equivalence:
𝑝1 ≡𝑂𝐸 𝑝2 i.f.f. for every input 𝑖 the user cares about, 𝑝1 𝑖 = 𝑝2 (𝑖)
Equivalence:
𝑝1 ≡ 𝑝2 i.f.f. for every possible input 𝑖 ever, 𝑝1 𝑖 = 𝑝2 (𝑖)
![Page 52: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/52.jpg)
Trying a different strategy altogether
• Use knowledge bases instead of the language grammar
• They no longer contain every program (neither limited grammars)
• Searchable via graph algorithms or probability equations
![Page 53: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/53.jpg)
The Interaction Model
• Aimed at programmers• Specify intent
• Express yourself
• Think like a programmer
![Page 54: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/54.jpg)
Specifying (and re-specifying) intent
Task: find the median of a list
User: examples!
1. [1,2,3]→2
2. [7,8,7,3]→7
Synthesis engine:
input[input.length/2]
User:
![Page 55: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/55.jpg)
Specifying (and re-specifying) intent
Task: find the median of a list
User: examples!
1. [1,2,3]→2
2. [7,8,7,3]→7
Synthesis engine:
input[input.length/2]
User:
It managed to find a single formula, let’s
make a counterexample
![Page 56: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/56.jpg)
Specifying (and re-specifying) intent
Task: find the median of a list
User: examples!
1. [1,2,3]→2
2. [7,8,7,3]→7
Synthesis engine:
input[input.length/2]
User:
It managed to find a single formula, let’s
make a counterexample
Principle #1:Cost of communicating intent +
consuming result << cost of manually performing the task
![Page 57: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/57.jpg)
Programming Not Only by Example
input[input.length/2]
[]
input /
. 2
input length
• A programmer can talk at the level of the program
• Read debug info
• Reason about subtrees or sequences of methods
• Even rewrite the program
• But also give examples, if those happen to be easier
![Page 58: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/58.jpg)
Programming Not Only by Example
input[input.length/2]
[]
input /
. 2
input length
• A programmer can talk at the level of the program
• Read debug info
• Reason about subtrees or sequences of methods
• Even rewrite the program
• But also give examples, if those happen to be easier
Retain
Exclude
![Page 59: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/59.jpg)
Programming Not Only by Example
input[input.length/2]
[]
input /
. 2
input length
• A programmer can talk at the level of the program
• Read debug info
• Reason about subtrees or sequences of methods
• Even rewrite the program
• But also give examples, if those happen to be easier
Principle #2:Let developers be developers
Retain
Exclude
![Page 60: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/60.jpg)
When models “out-think” the programmer
def counts(l : List[String]) : Map[String,Int]=
l.
![Page 61: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/61.jpg)
When models “out-think” the programmer
def counts(l : List[String]) : Map[String,Int]=
l.
groupBy(
map(
fold(
![Page 62: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/62.jpg)
When models “out-think” the programmer
def counts(l : List[String]) : Map[String,Int]=
l.groupBy(
identity)
x => x.length)
x => x[0])
![Page 63: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/63.jpg)
When models “out-think” the programmer
def counts(l : List[String]) : Map[String,Int]=
l.groupBy(identity).
map(x => x._1 -> x._2
x => x.length)
x => x[0])
![Page 64: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/64.jpg)
When models “out-think” the programmer
def counts(l : List[String]) : Map[String,Int]=
l.groupBy(identity).
map(x => x._1 -> x._2.
filter(y => y.startsWith(
x._2.length
x._2[0])
![Page 65: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/65.jpg)
When models “out-think” the programmer
def counts(l : List[String]) : Map[String,Int]=
l.groupBy(identity).
map(x => x._1 -> x._2.
filter(y => y.startsWith(
![Page 66: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/66.jpg)
When models “out-think” the programmer
def counts(l : List[String]) : Map[String,Int]=
l.groupBy(identity).
map(x => x._1 -> x._2.
filter(y => y.startsWith(
![Page 67: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/67.jpg)
Keep understanding what’s going on
??
![Page 68: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/68.jpg)
Keep understanding what’s going on
??
![Page 69: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/69.jpg)
Keep understanding what’s going on
?? Principle #3:Results must be explainable
![Page 70: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/70.jpg)
Programmers aren’t as good as they think
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
histogram no. lines with text most frequent word
% c
orr
ect
answ
er
PBE Only new operations All operations
![Page 71: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/71.jpg)
Read-Eval-Synth Loops
![Page 72: Automatic Programming: How Far Can Machines Go?hpeleg/yow-noanim.pdf · 2019. 11. 11. · 2. Solvers (e.g., Z3) 3. Observational Equivalence Observational equivalence: 𝑝1≡𝑂𝐸𝑝2](https://reader033.fdocuments.us/reader033/viewer/2022060912/60a6e5ffadff4306967caebd/html5/thumbnails/72.jpg)
What if we have automatic programming?
• Hyper-intelligent program generation for your every need
• Self-aware, self-augmenting AI
• The singularity
• ???
• THE ROBOT APOCALYPSE