CENTRAL VALLEY POSTHARVEST NEWSLETTER - UC Davis: Welcome to UC Davis
SmartSynth: Synthesizing Smartphone Automation Scripts from Natural Language Vu Le (UC Davis) Sumit...
-
Upload
ambrose-palmer -
Category
Documents
-
view
223 -
download
0
Transcript of SmartSynth: Synthesizing Smartphone Automation Scripts from Natural Language Vu Le (UC Davis) Sumit...
SmartSynth: Synthesizing Smartphone Automation
Scripts from Natural Language
Vu Le (UC Davis)Sumit Gulwani (MSR Redmond)
Zhendong Su (UC Davis)
motivation
programming
App Inventor Tasker
natural languageWhen I receive a new SMS, if the phone is connected to my car’s bluetooth, it reads the message content and replies the sender “I’m driving.”
scriptwhen (number, content) := MessageReceived()
if (IsConnectedToBTDevice(Car_BT) then
Speak(content);
SendMessage(number, "I'm driving");
SmartScriptScript P ::= I E T C M
Parameter I ::= input (i1, …, in) | ε
Event E ::= (r1, …, rn) := when Event() | ε
Conversions T ::= F1; …; Fn;
Condition C ::= if (Π1 ∧ … ∧ Πn) then | ε
Body M ::= Stmt1; … ; Stmtn;
Conversion F ::= x := Convert(a)
Predicate Π ::= Predicate (a1, …, an)
Statement Stmt ::= S | foreach x in a do S1; … ; Sn; od
Atomic Stmt. S ::= A | F
Action A ::= (r1, …, rn) := Action (a1, …, an)
Argument a ::= x | i | r | l
approach
Script = Components + Relations
approach
Script = Components + Relations
approach
Script = Components + Relations
approach
Script = Components + Relations
definitiono Component
• API: MessageReceived
• Entity• Literal: “I’m driving”
• API return value: MessageReceived.TextO
o Relation = <Entity, API parameter>• E.g., < “I’m driving”, SendMessage.TextI
>
component discovery
Description Component
When I receive a new SMS
MessageReceived
if the phone is connected to
IsConnectedToBTDevice
my car’s bluetooth Car_BT
reads Speak
the message content MessageReceived.TextO
replies SendMessage
the sender MessageReceived.NumberO
“I’m driving” "I'm driving"
mapping featureso Regular expressionso Bag of wordso Phrase lengtho Punctuationo Parse tree (Stanford NLP parser)
relation discoveryo Rule-based relation detection
• Relative locations of APIs and entities
Return Value or Literal API Parameter
Car_BT IsConnectedToBTDevice.TextI
MessageReceived.TextO Speak.TextI
MessageReceived.NumberO
SendMessage.NumberI
“I’m driving” SendMessage.TextI
script generationwhen (number, content) := MessageReceived()
if (IsConnectedToBTDevice(Car_BT) then
Speak(content);
SendMessage(number, "I'm driving");
ambiguityDescription Possible Component
Mappings
When I receive a new SMS
MessageReceived
if the phone is connected to
IsConnectedToBTDevice
IsConnectedToWifiNetwork
IsConnectedToDataService
my car’s bluetooth Car_BT
reads Speak
the message content MessageReceived.TextO
repliesSendMessage
SendEmail
“I’m driving” "I'm driving"
incompletenesswhen (number, content) := MessageReceived()
if (IsConnectedToBTDevice(Car_BT) then
Speak(content);
SendMessage(number, "I'm driving");
insight
NLP + Program Synthesis
ambiguity{ MessagedReceived, IsConnectedToBTDevice, Car_BT,
Speak, MessageReceived.TextO, (SendMessage OR
SendEmail), "I'm driving“ }
Ranking
Script 1 (SendMessage)
SendMessage
SendEmails
Script 1
Script 2
Synthesizer
incompletenesso Search for most likely missing
relations
q & ao Distinguishing multiple choice
questions• Question: API parameter
• Answers: Equally-likely related entitiesWhat do you want the phone to speak?A. The received message contentB. “I’m driving”
architecture
Component Discovery
ScriptDiscovery
Natural LanguageDescription
MappingComponents
IntendedScript
Natural LanguageQ&A (Optional)
1
2
3
5
User
Feedback onDescription (Optional)
6
Feedback onComp. Mapping
4
evaluationo 50 scripts collected from user forumso User study: give NL descriptions from
scripts• 11 students participated
• Each student was given 25 problems
• 640 correct descriptions (725 total)
component mapping
[1] [1] [2] [1] [2] [3] [1][2][3][4] SmartSynth0
10
20
30
40
50
60
70
80
90
100
Series1[1] Regex + Bags-of-Words [2] Phrase length[3] Punctuation [4] Parse tree
relation detection
1 2 3 4 7 80
1
2
3
4
5
# Relations
# D
etec
ted
Rela
tions
completing ratio
0 1 2 3 4 7 80
50
100
150
200
250
Detected by NLP Engine Completed by Synthesis Engine
# Relations
# D
escr
iptio
ns
synthesis time
2 3 4 5 6 7 8 10 11 120
1
2
3
4
5
6
7
Parsing time Total time
# Components
Tim
e (s
)
related worko General purpose programming using
natural languageo Natural language interfaceso Specification extraction from natural
language
thank you
Script = Components + Relations
NLP
Program Synthesis
Identify
Detect
Resolve
Complete
v.s. Siri
Conversation
Single phone API
Composition of APIs
Siri
SmartSynth
v.s. Tasker
TaskerSmartSynth
Time
Prod
uctiv
ity