Towards BetterUDParsingyjliu.net/cv/res/2018-10-12-conll.compressed.pdf2018/10/12 · Towards...
Transcript of Towards BetterUDParsingyjliu.net/cv/res/2018-10-12-conll.compressed.pdf2018/10/12 · Towards...
Towards Better UD Parsing:Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation
Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, Ting Liu
Research Center for Social Computing and Information Retrieval Harbin Institute of Technology
Overview of Our Techniques
• Rank 1st according to LAS
• Baseline model: Dozat et al.(2017)
• Winning strategies:• ELMo: +0.84
• Ensemble: +0.55
• Treebank Concat.: +0.42(estimated on Dev set.)
Our Extension to Dozat et al. (2017)
Form
Sum
Embed
Token W2V Char ELMo
!"#$% = ∑()*+ ℎ%,(
(/0)
Two Extensions on AllenNLP ELMo
• Supporting Unicode range
• Training with sample softmax• use a window of 8192 surrounding words as negative samples
• More stable training and better performance
• One language takes 3 days on Nvida P100
Other Techniques that Contributes
• Improved POS tagging:• Ranked 2nd in the UPOS evaluation (1st on the big treebanks)
• Biaffine tagger + ELMo
• Improved tokenization:• Ranked 2nd in the Tokenization-F1 evaluation
• BiLSTM sequence labeling + unigram character ELMo
• Wins in zh_gsd (+6.6), ja_gsd (+4.1), vi_vtb (+7.2), ja_morden (+9.7)
Effects of ELMo:Improvements on Gold Segmentation
95
95.5
96
96.5
97
97.5
98
UPOS
Baseline +ELMo
83
83.5
84
84.5
85
85.5
86
86.5
87
LAS
Baseline +ELMo
0.91.1
Effects of ELMo:OOV Rate against Improvements
Effects of ELMo:
ELMo’s Effects on IV and OOV
85
87
89
91
93
95
97
99
IV OOV
Basline +ELMo
76
78
80
82
84
86
88
90
IV OOV
Basline +ELMo
Effects of ELMo:
Better OOV Abstraction
Conclusion
• We made several improvements including incorporating deep contextualized word embeddings, parser ensemble, and treebank concatenation.
• Analysis shows that ELMo mainly improves the OOVperformance via learning better abstraction.
• We release the pre-trained ELMo at: https://github.com/HIT-SCIR/ELMoForManyLangs