Theory and Practice of Logic Programming Redundant Sudoku rules
Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules...
Transcript of Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules...
![Page 2: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/2.jpg)
Distributional semantics• Target word = “stars”
![Page 3: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/3.jpg)
Distributional semantics• Collect the contextual words for “stars”
![Page 4: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/4.jpg)
Distributional semantics• Distributional word representation • Distributional hypothesis: words that are used and occur in the same
contexts tend to purport similar meanings
• Implementations with distributed word embedding models • Word2vec • GloVec
![Page 5: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/5.jpg)
Distributional semantics
![Page 6: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/6.jpg)
A huge success of deep contextualized models• EMLo
• Char-level word encoding + 2 BiLSTM layers • Pretrained language models + can be fine-tuned according to specific tasks
• Transformer • Pairwise interaction, called self-attention (with positional embedding) • Multi-head mechanism • Deep architecture (using six layers)
• BERT • Built on top of Transformer • Bidirectional context • Masked LM + Next sentence prediction • Very deep
• BERT-Base: 12-layer, 768-hidden, 12-head • BERT-Large: 24-layer, 1024-hidden, 16-head
![Page 7: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/7.jpg)
Basic motivations
• Deeper architecture and larger context scope • Pretrained language models that can be fine-tuned
![Page 8: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/8.jpg)
Connecting the dots• Deep architecture
• NLPers always want to use very deep neural networks as CVers
• Contextualized models • Distributional semantics
• Word-by-word attention • Reasoning about entailment with neural attention
• Self-match • R-Net: Machine Reading Comprehension with Self-Matching
![Page 9: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/9.jpg)
The context scope can be even larger • Document-level information • Document Context Neural Machine Translation with Memory Network
![Page 10: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/10.jpg)
The context scope can be even larger • Document-level information • Improving the Transformer Translation Model with Document-Level
Context
![Page 11: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/11.jpg)
Structural knowledge • Triplets • Form: (h, r, t) • Examples:
• <YAO Ming, birthPlace, Shanghai> • <YAO Ming, gender, male>
• Embedding methods • E.g., TransE
![Page 12: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/12.jpg)
Structural knowledge • Knowledge base (or KB like) information • Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension
with External Commonsense Knowledge
![Page 13: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/13.jpg)
Structural knowledge • Linguistic information • Type-Aware Question Answering over Knowledge Base with Attention-
Based Tree-Structured Neural Networks
![Page 14: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/14.jpg)
Structural knowledge • Logic rules • Harnessing Deep Neural Networks with Logic Rules
First-order logic rules on two tasks: sentiment classification and NER
![Page 15: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/15.jpg)
Structural knowledge • Demographic attributes • Mining Product Adopter Information from Online Reviews for Improving
Product Recommendation
![Page 16: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/16.jpg)
Structural knowledge • Demographic attributes • Mining Product Adopter Information from Online Reviews for Improving
Product Recommendation
• Based on the analysis of 13.9 million JD reviews, about 10.8% reviews contain at least an adopter mention • We can even infer the information about the buyer
• Marital status • Age range
![Page 17: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/17.jpg)
Structural knowledge • Demographic attributes • Adversarial Removal of Demographic Attributes from Text Data
![Page 18: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/18.jpg)
Knowledge utilization• Point #1: • Enriching information for the NLP tasks • The widely used procedure
• Knowledge retrieval ! Knowledge contextualization ! Knowledge utilization
Knowledge-powered conversational agents
![Page 19: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/19.jpg)
Knowledge utilization• Point #1: • Enriching information for the NLP tasks • The widely used procedure
• Knowledge retrieval ! Knowledge contextualization ! Knowledge utilization
Commonsense Knowledge Aware Conversation Generation with Graph Attention
![Page 20: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/20.jpg)
Knowledge utilization• Point #1: • Enriching information for the NLP tasks • Challenging problems:
• How to identify and find suitable knowledge resources to use • How to learn knowledge representations that are useful for some specific tasks
![Page 21: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/21.jpg)
Knowledge utilization• Point #2: • Making models more explainable
• e-SNLI: Natural Language Inference with Natural Language Explanations
![Page 22: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/22.jpg)
Knowledge utilization• Point #2: • Making models more explainable
• Improving Sequential Recommendation with Knowledge-Enhanced Memory Networks
![Page 23: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/23.jpg)
Knowledge utilization• Point #2: • Making models more explainable
• Challenging problems • How to define explainability • How to balance explainability and effectiveness
![Page 24: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/24.jpg)
Knowledge utilization• Point #3: • Knowledge can guide the model design
• Sentence Encoding with Tree-constrained Relation Networks
![Page 25: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/25.jpg)
Knowledge utilization• Point #3: • Knowledge can guide the model design
• Taxonomy-Aware Multi-Hop Reasoning Networks for Sequential Recommendation
![Page 26: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/26.jpg)
Knowledge utilization• Point #3: • Knowledge can guide the model design
• Challenging problems: • Given some kind of knowledge, what is the suitable model to integrate it? • Given some existing models, how to adapt it to fully utilize knowledge information?
![Page 27: Knowledge-driven semantic understandinghainanumeeting.net/YSSNLP2019/file/1.pdf · • Logic rules • Harnessing Deep Neural Networks with Logic Rules First-order logic rules on](https://reader035.fdocuments.us/reader035/viewer/2022071023/5fd87da75fca1a09f1324223/html5/thumbnails/27.jpg)
Conclusion