Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence...
Transcript of Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence...
![Page 1: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/1.jpg)
Natural Language Dialogues withSequence-To-Sequence Learning
Dirk von Grünigen
Deep Learning Day 2017, 22nd September 2017
![Page 2: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/2.jpg)
2 Deep Learning Day 2017Dirk von Grünigen
What Kind Of Chatbots Are There?
Sources: https://chatbotslife.com/ultimate-guide-to-leveraging-nlp-machine-learning-for-you-chatbot-531ff2dd870c
![Page 3: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/3.jpg)
3 Deep Learning Day 2017Dirk von Grünigen
Rule-Based: TheMovieMan
![Page 4: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/4.jpg)
4 Deep Learning Day 2017Dirk von Grünigen
Smart-Machine: Reddit Movie Chatbot
![Page 5: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/5.jpg)
5 Deep Learning Day 2017Dirk von Grünigen
General-AI: Neural Conversational Model
Source: Oriol Vinyals & Quoc Le (2015), Neural Conversational Model
![Page 6: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/6.jpg)
6 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Architecture
Source: http://suriyadeepan.github.io/2016-12-31-practical-seq2seq/
![Page 7: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/7.jpg)
7 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Training
Timespan of ~4 weeks
Sources: http://colah.github.io/posts/2015-09-Visual-Information/
q = [0.1, 0.08, … , 0.001, 0.01]
p = [0, 0, 0, 0, 0, 1, 0 … , 0, 0]
![Page 8: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/8.jpg)
8 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Evaluation
The president meets the press in Illinois Donald Trump welcomes the media in Chicago
Sentence #1: Sentence #2:
BLEU, METEOR,ROUGE,Cross-Entropy
Evaluation
Result?
![Page 9: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/9.jpg)
9 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Evaluation
The president meets the press in Illinois
Sentence #1: Sentence #2:
BLEU, METEOR,ROUGE,Cross-Entropy
Evaluation
Donald Trump welcomes the media in Chicago
![Page 10: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/10.jpg)
10 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Evaluation
Sources: http://sebastianruder.com/word-embeddings-1/, https://www.tensorflow.org/tutorials/word2vec
![Page 11: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/11.jpg)
11 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Language Diversity
Source: https://blogemis.com/2015/09/26/zipfs-law-and-the-math-of-reason/
![Page 12: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/12.jpg)
12 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Language Diversity
Source: Li et. al (2016), A Diversity-Promoting Objective Function for Neural Conversation Models
![Page 13: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/13.jpg)
13 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Language Diversity
Source: Li et. al (2016), A Diversity-Promoting Objective Function for Neural Conversation Models
![Page 14: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/14.jpg)
14 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Language Diversity
Source: Li et. al (2016), A Diversity-Promoting Objective Function for Neural Conversation Models
![Page 15: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/15.jpg)
15 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Language Diversity
Sources: https://research.googleblog.com/2016/05/chat-smarter-with-allo.html
t1 t2 t3
![Page 16: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/16.jpg)
16 Deep Learning Day 2017Dirk von Grünigen
Seq2Seq: Language Diversity
Sources: https://research.googleblog.com/2016/05/chat-smarter-with-allo.html
t1 t2 t3
![Page 17: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/17.jpg)
17 Deep Learning Day 2017Dirk von Grünigen
Conclusion
• Chatbots are already working for specific tasks in closed domains• Mostly rule-based, generative will become more popular in the
next years
• Seq2Seq works, but results are still not „production-ready“
• Still a lot of challenges to overcome:• Evaluation, Language Models, Context, Personality, ...
• Very active research area with a lot of open questions!
![Page 18: Natural Language Dialogues with Seq2Seq · Natural Language Dialogues with Sequence-To-Sequence Learning Dirk von Grünigen Deep Learning Day 2017, 22nd September 2017](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5654b7a7d9c40ab73cf65/html5/thumbnails/18.jpg)
18 Deep Learning Day 2017Dirk von Grünigen
Thanks for the attention! J