[DL輪読会]Generating Wikipedia by Summarizing Long Sequences
-
Upload
deep-learning-jp -
Category
Technology
-
view
42 -
download
1
Transcript of [DL輪読会]Generating Wikipedia by Summarizing Long Sequences
![Page 1: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/1.jpg)
1
DEEP LEARNING JP[DL Papers]
http://deeplearning.jp/
Generating Wikipedia by Summarizing Long Sequences(ICLR 2018)
Toru Fujino, scalab, UTokyo
![Page 2: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/2.jpg)
��
• . /0•s ( -:: 8 2 I
• : p 2>2 > goo.gl/wSuuS9 k
•1 2ILCG B iI rd ie
• a Ird i I R l• G ItW )• B W DC noI2>> > : g
![Page 3: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/3.jpg)
������
• , ) 1• ,
•• : ,•
•• :•
•• , , ( )• , ,
![Page 4: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/4.jpg)
1
• .G DC L N•
• 51 3 2 : 0 6 4 1 (
•
• R• // 1 2 : /1 1:1 4 1 ) •
••
1) Rush et al. “A Neural Attention Model for Sentence Summarization”, EMNLP 2015
2) Nallapati et al. “Abstractive Text Summarization using Sequence-to-Sequence RNNs and Beyond”, CoNLL 2016
![Page 5: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/5.jpg)
• (,,, ),( )
•e /• S
•/ • 2 A / /
2) Nallapati et al. “Abstractive Text Summarization using Sequence-to-Sequence RNNs and Beyond”, CoNLL 2016
2)
![Page 6: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/6.jpg)
�������
• 2. 1/
• 2
![Page 7: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/7.jpg)
• 1 W Ra
•• 2 G• 00 . c
•• 1 d
��
https://en.wikipedia.org/wiki/Deep_learning
![Page 9: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/9.jpg)
• ,
��1
��2
��3
��4
���� �����
![Page 10: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/10.jpg)
••• -•
![Page 11: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/11.jpg)
• 3 43 4 Y p• CD 4 M i• ac c 43 , 4 a• d ac c n nY
r Ly
• ot ldA CN m m3 43 e
• 4 , 3 43 m u• ) 24 : 42 3 43 m
• s e3) A. Vaswani et al. “Attention is All You Need”, NIPS 2017
4) N. Shazzer et al. “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, ICLR 2017
![Page 12: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/12.jpg)
• . .
• 3 .) • .) A • ( 3 .) : :
3) A. Vaswani et al. “Attention is All You Need”, NIPS 2017
![Page 13: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/13.jpg)
• ( 2• E !" = [!%", !'", … , !)"" ] • E !+ = [!%+ , !'+ , … , !)++ ] • ) E
•A : D5) M.-T. Luong et al. “Effective Approaches to Attention-based Neural Machine translation”, EMNLP 2015
5)
![Page 14: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/14.jpg)
• ( - E• : D• : D• ) : D
• E : A
![Page 15: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/15.jpg)
• • A , -
•
![Page 16: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/16.jpg)
• K K V 25 /6 ) ,
• ( A 6
• 6
![Page 17: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/17.jpg)
• L K 3 ASA
• V = • . • 11,/1 /
•
![Page 18: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/18.jpg)
) (
• )
•/(
4) N. Shazzer et al. “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, ICLR 2017
4)
![Page 19: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/19.jpg)
• 9 L• 2 / 02 2 / 02 2 /
• 9• M - - - 9 =• -1 5
![Page 20: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/20.jpg)
•����������� ���
![Page 21: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/21.jpg)
• - - • - • -
• :
![Page 22: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/22.jpg)
• ����������������� ���
![Page 23: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/23.jpg)
![Page 24: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/24.jpg)
)& (
• ( ()
![Page 25: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/25.jpg)
![Page 26: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/26.jpg)
•�������• ���� ��������
![Page 27: [DL輪読会]Generating Wikipedia by Summarizing Long Sequences](https://reader036.fdocuments.us/reader036/viewer/2022081802/5aaa85d17f8b9af9198b4677/html5/thumbnails/27.jpg)
������
•M• / / -,/ p k lsr im e• W lsr f :• n s k• a a > - - /
• - / y• -2 2 - -, - / Wy
•• ot C L c A• L M d L C• lsr : L