Semi-supervised Learning for Neural Machine...
Transcript of Semi-supervised Learning for Neural Machine...
![Page 1: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/1.jpg)
Semi-supervised Learning for Neural Machine Translation
Yong Cheng
joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu
1
![Page 2: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/2.jpg)
Machine Translation
2
Automated translation using computer software
![Page 3: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/3.jpg)
Machine Translation
3
Rule-based Machine Translation 1970s
Example-based Machine Translation 1984
Statical Machine Translation (SMT) 1993
Neural Machine Translation NMT 2014
Trends: learning to translate from DATA
![Page 4: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/4.jpg)
Machine Translation
4
Parallel Corpora
Monolingual Corpora
Parallel corpora are usually limited in
quantity quality coverage& &
![Page 5: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/5.jpg)
Monolingual Corpora Used in SMT and NMT
N-gram language model in SMT Koehn et al., [2007]
Monolingual corpora as decipherment Ravi and Knight [2011]
Integrate a neural language model into NMT. Gulccehre et al. [2015]
Additional pseudo parallel corpus. Sennrich et al. [2016]
5
![Page 6: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/6.jpg)
Supervised Training
Parallel Corpus
Objective
6
![Page 7: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/7.jpg)
Unsupervised Training
Monolingual Corpus
7
![Page 8: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/8.jpg)
cc Our Approach — Autoencoders
8
bushi yu shalong juxing le huitan x
![Page 9: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/9.jpg)
cc Our Approach — Autoencoders
9
bushi yu shalong juxing le huitan xP(y | x;
!θ )
![Page 10: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/10.jpg)
cc Our Approach — Autoencoders
10
bushi yu shalong juxing le huitan x
Bush held a talk with sharon y
P(y | x;!θ )
latent
![Page 11: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/11.jpg)
cc Our Approach — Autoencoders
11
bushi yu shalong juxing le huitan x
Bush held a talk with sharon y
P(y | x;!θ )
P(x | y;!θ )
latent
![Page 12: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/12.jpg)
cc Our Approach — Autoencoders
12
bushi yu shalong juxing le huitan x
Bush held a talk with sharon
′xbushi yu shalong juxing le huitan
y
P(y | x;!θ )
P(x | y;!θ )
latent
![Page 13: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/13.jpg)
cc Our Approach — Autoencoders
13
source autoencoder target autoencoder
![Page 14: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/14.jpg)
Unsupervised Training (Autoencoders)
Monolingual Corpus
14
target autoencoder
![Page 15: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/15.jpg)
Semi-supervised Training
15
Training Objective
![Page 16: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/16.jpg)
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
16
![Page 17: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/17.jpg)
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
17
![Page 18: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/18.jpg)
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
18
![Page 19: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/19.jpg)
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
19
![Page 20: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/20.jpg)
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
20
![Page 21: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/21.jpg)
Translation Results
Compared with Sennrich et al. [2015a]
21
![Page 22: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/22.jpg)
Example Translation of Monolingual Corpus
22
![Page 23: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/23.jpg)
ConclusionMonolingual corpora is an important resource for neural machine translation.
We have proposed a semi-supervised approach to training bidirectional neural machine translation models for exploiting monolingual corpora.
As our method is sensitive to the OOVs present in monolingual corpora, we plan to integrate Jean et al. (2015)’s technique on using very large vocabulary into our approach.
23
![Page 24: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/24.jpg)
Thank You !
24
![Page 25: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/25.jpg)
Effect of Sample Size
ZH-EN EN-ZH
25
![Page 26: Semi-supervised Learning for Neural Machine Translationpeople.iiis.tsinghua.edu.cn/~weixu/Krvdro9c/acl... · Monolingual corpora is an important resource for neural machine translation.](https://reader033.fdocuments.us/reader033/viewer/2022060504/5f1d7ad48e61ec19aa6d1189/html5/thumbnails/26.jpg)
Effect of OOV ratio
ZH-EN EN-ZH
26