Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processingand the 9th International Joint Conference on Natural Language Processing, pages 1061–1070,Hong Kong, China, November 3–7, 2019. c©2019 Association for Computational Linguistics

1061

Tree Transformer: Integrating Tree Structures into Self-Attention

Yau-Shian Wang Hung-Yi Lee Yun-Nung ChenNational Taiwan University, Taipei, Taiwan

king6101@gmail.com hungyilee@ntu.edu.tw y.v.chen@ieee.org

Abstract

Pre-training Transformer from large-scale rawtexts and fine-tuning on the desired task haveachieved state-of-the-art results on diverseNLP tasks. However, it is unclear what thelearned attention captures. The attention com-puted by attention heads seems not to matchhuman intuitions about hierarchical structures.This paper proposes Tree Transformer, whichadds an extra constraint to attention heads ofthe bidirectional Transformer encoder in orderto encourage the attention heads to follow treestructures. The tree structures can be automat-ically induced from raw texts by our proposed“Constituent Attention” module, which is sim-ply implemented by self-attention betweentwo adjacent words. With the same trainingprocedure identical to BERT, the experimentsdemonstrate the effectiveness of Tree Trans-former in terms of inducing tree structures,better language modeling, and further learningmore explainable attention scores1.

1 Introduction

Human languages exhibit a rich hierarchical struc-ture which is currently not exploited nor mir-rored by the self-attention mechanism that isthe core of the now popular Transformer archi-tecture. Prior work that integrated hierarchi-cal structure into neural networks either used re-cursive neural networks (Tree-RNNs) (C.Gollerand A.Kuchler, 1996; Socher et al., 2011; Taiet al., 2015) or simultaneously generated a syn-tax tree and language in RNN (Dyer et al.,2016), which have shown beneficial for manydownstream tasks (Aharoni and Goldberg, 2017;Eriguchi et al., 2017; Strubell et al., 2018; Zare-moodi and Haffari, 2018). Considering the re-quirement of the annotated parse trees and the

1The source code is publicly available at https://github.com/yaushian/Tree-Transformer.

costly annotation effort, most prior work relied onthe supervised syntactic parser. However, a su-pervised parser may be unavailable when the lan-guage is low-resourced or the target data has dif-ferent distribution from the source domain.

Therefore, the task of learning latent tree struc-tures without human-annotated data, called gram-mar induction (Carroll and Charniak, 1992; Kleinand Manning, 2002; Smith and Eisner, 2005), hasbecome an important problem and attractd moreattention from researchers recently. Prior workmainly focused on inducing tree structures fromrecurrent neural networks (Shen et al., 2018a,b) orrecursive neural networks (Yogatama et al., 2017;Drozdov et al., 2019), while integrating tree struc-tures into Transformer remains an unexplored di-rection.

Pre-training Transformer from large-scale rawtexts successfully learns high-quality languagerepresentations. By further fine-tuning pre-trainedTransformer on desired tasks, wide range of NLPtasks obtain the state-of-the-art results (Radfordet al., 2019; Devlin et al., 2018; Dong et al.,2019). However, what pre-trained Transformerself-attention heads capture remains unknown. Al-though an attention can be easily explained by ob-serving how words attend to each other, only somedistinct patterns such as attending previous wordsor named entities can be found informative (Vig,2019). The attention matrices do not match ourintuitions about hierarchical structures.

In order to make the attention learned byTransformer more interpretable and allow Trans-former to comprehend language hierarchically, wepropose Tree Transformer, which integrates treestructures into bidirectional Transformer encoder.At each layer, words are constrained to attend toother words in the same constituents. This con-straint has been proven to be effective in priorwork (Wu et al., 2018). Different from the prior

1062

work that required a supervised parser, in TreeTransformer, the constituency tree structures is au-tomatically induced from raw texts by our pro-posed “Constituent Attention” module, which issimply implemented by self-attention. Motivatedby Tree-RNNs, which compose each phrase andthe sentence representation from its constituentsub-phrases, Tree Transformer gradually attachesseveral smaller constituents into larger ones fromlower layers to higher layers.

The contributions of this paper are 3-fold:

• Our proposed Tree Transformer is easy toimplement, which simply inserts an addi-tional “Constituent Attention” module im-plemented by self-attention to the originalTransformer encoder, and achieves good per-formance on the unsupervised parsing task.

• As the induced tree structures guide wordsto compose the meaning of longer phraseshierarchically, Tree Transformer improvesthe perplexity on masked language modelingcompared to the original Transformer.

• The behavior of attention heads learnedby Tree Transformer expresses better inter-pretability, because they are constrained tofollow the induced tree structures. By visu-alizing the self-attention matrices, our modelprovides the information that better matchsthe human intuition about hierarchical struc-tures than the original Transformer.

2 Related Work

This section reviews the recent progress aboutgrammar induction. Grammar induction is the taskof inducing latent tree structures from raw textswithout human-annotated data. The models forgrammar induction are usually trained on othertarget tasks such as language modeling. To obtainbetter performance on the target tasks, the mod-els have to induce reasonable tree structures andutilize the induced tree structures to guide text en-coding in a hierarchical order. One prior attemptformulated this problem as a reinforcement learn-ing (RL) problem (Yogatama et al., 2017), wherethe unsupervised parser is an actor in RL and theparsing operations are regarded as its actions. Theactor manages to maximize total rewards, whichare the performance of downstream tasks.

PRPN (Shen et al., 2018a) and On-LSTM (Shenet al., 2018b) induce tree structures by introducing

a bias to recurrent neural networks. PRPN pro-poses a parsing network to compute the syntacticdistance of all word pairs, and a reading networkutilizes the syntactic structure to attend relevantmemories. On-LSTM allows hidden neurons tolearn long-term or short-term information by theproposed new gating mechanism and new activa-tion function. In URNNG (Kim et al., 2019b), theyapplied amortized variational inference between arecurrent neural network grammar (RNNG) (Dyeret al., 2016) decoder and a tree structures inferencenetwork, which encourages the decoder to gen-erate reasonable tree structures. DIORA (Droz-dov et al., 2019) proposed using inside-outside dy-namic programming to compose latent represen-tations from all possible binary trees. The repre-sentations of inside and outside passes from samesentences are optimized to be close to each other.Compound PCFG (Kim et al., 2019a) achievesgrammar induction by maximizing the marginallikelihood of the sentences which are generated bya probabilistic context-free grammar (PCFG) in acorpus.

3 Tree Transformer

Given a sentence as input, Tree Transformer in-duces a tree structure. A 3-layer Tree Trans-former is illustrated in Figure 1(A). The build-ing blocks of Tree Transformer is shown in Fig-ure 1(B), which is the same as those used inbidirectional Transformer encoder, except the pro-posed Constituent Attention module. The blocksin Figure 1(A) are constituents induced from theinput sentence. The red arrows indicate the self-attention. The words in different constituents areconstrained to not attend to each other. In the 0-th layer, some neighboring words are merged intoconstituents; for example, given the sentence “thecute dog is wagging its tail”, the tree Transformerautomatically determines that “cute” and “dog”form a constituent, while “its” and “tail” also formone. The two neighboring constituents may mergetogether in the next layer, so the sizes of con-stituents gradually grow from layer to layer. In thetop layer, the layer 2, all words are grouped intothe same constituent. Because all words are intothe same constituent, the attention heads freely at-tend to any other words, in this layer, Tree Trans-former behaves the same as the typical Trans-former encoder. Tree Transformer can be trainedin an end-to-end fashion by using “masked LM”,

1063

Figure 1: (A) A 3-layer Tree Transformer, where the blocks are constituents induced from the input sentence. Thetwo neighboring constituents may merge together in the next layer, so the sizes of constituents gradually growfrom layer to layer. The red arrows indicate the self-attention. (B) The building blocks of Tree Transformer. (C)Constituent prior C for the layer 1.

which is one of the unsupervised training task usedfor BERT training.

Whether two words belonging to the same con-stituent is determined by “Constituent Prior” thatguides the self-attention. Constituent Prior is de-tailed in Section 4, which is computed by theproposed Constituent Attention module in Sec-tion 5. By using BERT masked language model astraining, latent tree structures emerge from Con-stituent Prior and unsupervised parsing is therebyachieved. The method for extracting the con-stituency parse trees from Tree Transformer is de-scribed in Section 6.

4 Constituent Prior

In each layer of Transformer, there are a query ma-trix Q consisting of query vectors with dimensiondk and a key matrix K consisting of key vectorswith dimension dk. The attention probability ma-trix is denoted as E, which is an N by N matrix,where N is the number of words in an input sen-tence. Ei,j is the probability that the position iattends to the position j. The Scaled Dot-ProductAttention computes the E as:

E = softmax(QKT

d), (1)

where the dot-product is scaled by 1/d. In Trans-former, the scaling factor d is set to be

√dk.

In Tree Transformer, the E is not only deter-mined by the query matrix Q and key matrix K,but also guided by Constituent Prior C generat-ing from Constituent Attention module. Same asE, the constituent prior C is also a N by N ma-trix, where Ci,j is the probability that word wi and

word wj belong to the same constituency. Thismatrix is symmetric that Ci,j is same as Cj,i. Eachlayer has its own Constituent Prior C. An exam-ple of Constituent Prior C is illustrated in Fig-ure 1 (C), which indicates that in layer 1, “the cutedog” and “is wagging its tail” are two constituents.

To make each position not attend to the positionin different constituents, Tree Transformer con-strains the attention probability matrix E by con-stituent prior C as below,

E = C � softmax(QKT

d), (2)

where � is the element-wise multiplication.Therefore, if Ci,j has small value, it indicatesthat the positions i and j belong to different con-stituents, where the attention weightEi,j would besmall. As Transformer uses multi-head attentionwith h different heads, there are h different querymatrices Q and key matrices K at each position,but here in the same layer, all attention heads inmulti-head attention share the same C. The multi-head attention module produces the output of di-mension dmodel = h× dk.

5 Constituent Attention

The proposed Constituent Attention module isto generate the constituent prior C. Instead ofdirectly generating C, we decompose the prob-lem into estimating the breakpoints between con-stituents, or the probability that two adjacentwords belong to the same constituent. In eachlayer, the Constituent Attention module generatesa sequence a = {a1, ..., ai, ..., aN}, where ai isthe probability that the word wi and its neighbor

1064

Figure 2: The example illustration about how neigh-boring attention works.

word wi+1 are in the same constituent. The smallvalue of ai implies that there is a breakpoint be-tween wi and wi+1, so the constituent prior C isobtained from the sequence a as follows. Ci,j isthe multiplication of all ai6k<j between word wi

and word wj :

Ci,j =

j−1∏k=i

ak. (3)

In (3), we choose to use multiplication instead ofsummation, because if one of ai6k<j between twowords wi and wj is small, the value of Ci,j withmultiplication also becomes small. In implemen-tation, to avoid probability vanishing, we use log-sum instead of directly multiplying all a:

Ci,j = e∑j−1

k=i log(ak). (4)

The sequence a is obtained based on the follow-ing two mechanisms: Neighboring Attention andHierarchical Constraint.

5.1 Neighboring AttentionWe compute the score si,i+1 indicating that wi

links to wi+1 by scaled dot-product attention:

si,i+1 =qi · ki+1

d, (5)

where qi is a link query vector of wi with dmodel

dimensions, and ki+1 is a link key vector of wi+1

with dmodel dimensions. We use qi · ki+1 to rep-resent the tendency that wi and wi+1 belong to thesame constituent. Here, we set the scaling factord to be dmodel

2 . The query and key vectors in (5)are different from (1). They are computed by thesame network architecture, but with different setsof network parameters.

For each word, we constrain it to either link toits right neighbor or left neighbor as illustrated inFigure 2. This constraint is implemented by ap-plying a softmax function to two attention links ofwi:

pi,i+1, pi,i−1 = softmax(si,i+1, si,i−1), (6)

where pi,i+1 is the probability that wi attends towi+1, and (pi,i+1+pi,i−1) = 1. We find that with-out the constraint of the softmax operation in (6)the model prefers to link all words together andassign all words to the same constituency. That is,giving both si,i+1 and si,i−1 large values, so the at-tention head freely attends to any position withoutrestriction of constituent prior, which is the sameas the original Transformer. Therefore, the soft-max function is to constraint the attention to besparse.

As pi,i+1 and pi+1,i may have different values,we average its two attention links:

ai =√pi,i+1 × pi+1,i. (7)

The ai links two adjacent words only if two wordsattend to each other. ai is used in the next subsec-tion to obtain ai.

5.2 Hierarchical ConstraintAs mentioned in Section 3, constituents in thelower layer merge into larger one in the higherlayer. That is, once two words belong to the sameconstituent in the lower layer, they would still be-long to the same constituent in the higher layer. Toapply the hierarchical constraint to the tree Trans-former, we restrict alk to be always larger than al−1

k

for the layer l and word index k. Hence, at thelayer l, the link probability alk is set as:

alk = al−1k + (1− al−1

k )alk, (8)

where al−1k is the link probability from the previ-

ous layer l− 1, and alk is obtained from Neighbor-ing Attention (Section 5.1) of the current layer l.Finally, at the layer l, we apply (4) for computingC l from al. Initially, different words are regardedas different constituents, and thus we initialize a−1

k

as zero.

6 Unsupervised Parsing from TreeTransformer

After training, the neighbor link probability a canbe used for unsupervised parsing. The small valueof a suggests this link be the breakpoint of twoconstituents. By top-down greedy parsing (Shenet al., 2018a), which recursively splits the sentenceinto two constituents with minimum a, a parse treecan be formed.

However, because each layer has a set of al, wehave to decide to use which layer for parsing. In-stead of using a from a specific layer for parsing

1065

Algorithm 1 Unsupervised Parsing with MultipleLayers

1: a← link probabilities2: m← minimum layer id . Discard the a

from layers below minimum layer3: thres← 0.8 . Threshold of breakpoint4: procedure BUILDTREE(l, s, e) . l: layer

index, s: start index, e: end index5: if e− s < 2 then. The constituent cannot

be split6: return (s, e)

7: span← als≤i<e

8: b← argmin(span) . Get breakpoint9: last← max(l − 1,m) . Get index of last

layer10: if alb > thres then11: if l = m then12: return (s, e)

13: return BuildTree(last, s, e)14: tree1← BuildTree(last, s, b)15: tree2← BuildTree(last, b+ 1, e)16: return (tree1, tree2) . Return tree

(Shen et al., 2018b), we propose a new parsing al-gorithm, which utilizes a from all layers for un-supervised parsing. As mentioned in Section 5.2,the values of a are strictly increasing, which indi-cates that a directly learns the hierarchical struc-tures from layer to layer. Algorithm 1 details howwe utilize hierarchical information of a for unsu-pervised parsing.

The unsupervised parsing starts from the toplayer, and recursively moves down to the last layerafter finding a breakpoint until reaching the bot-tom layerm. The bottom layerm is a hyperparam-eter needed to be tuned, and is usually set to 2 or3. We discard a from layers below m, because wefind the lowest few layers do not learn good rep-resentations (Liu et al., 2019) and thus the parsingresults are poor (Shen et al., 2018b). All values ofa on top few layers are very close to 1, suggestingthat those are not good breakpoints. Therefore, weset a threshold for deciding a breakpoint, where aminimum a will be viewed as a valid breakpointonly if its value is below the threshold. As we findthat our model is not very sensitive to the thresholdvalue, we set it to be 0.8 for all experiments.

7 Experiments

In order to evaluate the performance of our pro-posed model, we conduct the experiments detailedbelow.

7.1 Model Architecture

Our model is built upon a bidirectional Trans-former encoder. The implementation of our Trans-former encoder is identical to the original Trans-former encoder. For all experiments, we set thehidden size dmodel of Constituent Attention andTransformer as 512, the number of self-attentionheads h as 8, the feed-forward size as 2048 andthe dropout rate as 0.1. We analyze and discussthe sensitivity of the number of layers, denoted asL, in the following experiments.

7.2 Grammar Induction

In this section, we evaluate the performance of ourmodel on unsupervised constituency parsing. Ourmodel is trained on WSJ training set and WSJ-all(i.e. including testing and validation sets) by us-ing BERT Masked LM (Devlin et al., 2018) as un-supervised training task. We use WordPiece (Wuet al., 2016) tokenizer from BERT to tokenizewords with a 16k token vocabulary. Our best re-sult is optimized by adam with a learning rate of0.0001, β1 = 0.9 and β2 = 0.98. Following theevaluation settings of prior work (Htut et al., 2018;Shen et al., 2018b)2, we evaluate F1 scores of ourmodel on WSJ-test and WSJ-10 of Penn Treebank(PTB) (Marcus et al., 1993). The WSJ-10 has7422 sentences from whole PTB with sentencelength restricted to 10 after punctuation removal,while WSJ-test has 2416 sentences from the PTBtesting set with unrestricted sentence length.

The results on WSJ-test are in Table 1. Wemainly compare our model to PRPN (Shen et al.,2018a), On-lstm (Shen et al., 2018b) and Com-pound PCFG(C-PCFG) (Kim et al., 2019a), inwhich the evaluation settings and the training dataare identical to our model. DIORA (Drozdovet al., 2019) and URNNG (Kim et al., 2019b) use arelative larger training data and the evaluation set-tings are slightly different from our model. Ourmodel performs much better than trivial trees (i.e.right and left-branching trees) and random trees,which suggests that our proposed model success-fully learns meaningful trees. We also find that

2https://github.com/yikangshen/Ordered-Neurons

1066

Model Data F1median F1max

PRPN WSJ-train 35.0 42.8On-lstm WSJ-train 47.7 49.4C-PCFG WSJ-train 55.2 60.1Tree-T,L=12 WSJ-train 48.4 50.2Tree-T,L=10 WSJ-train 49.5 51.1Tree-T,L=8 WSJ-train 48.3 49.6Tree-T,L=6 WSJ-train 47.4 48.8DIORA NLI 55.7 56.2URNNG Billion - 52.4Tree-T,L=10 WSJ-all 50.5 52.0Random - 21.6 21.8LB - 9.0 9.0RB - 39.8 39.8

Table 1: The F1 scores on WSJ-test. Tree Trans-former is abbreviated as Tree-T, and L is the num-ber of layers(blocks). DIORA is trained on multi-NLIdataset (Williams et al., 2018). URNNG is trained onthe subset of one billion words (Chelba et al., 2013)with 1M training data. LB and RB are the left andright-brancing baselines.

Figure 3: A parse tree induced by Tree Transformer.As shown in the figure, because we set a threshold inAlgorithm 1, the leaf nodes are not strictly binary.

increasing the layer number results in better per-formance, because it allows the Tree Transformerto model deeper trees. However, the performancestops growing when the depth is above 10. Thewords in the layers above the certain layer areall grouped into the same constituent, and there-fore increasing the layer number will no longerhelp model discover useful tree structures. In Ta-ble 2, we report the results on WSJ-10. Some ofthe baselines including CCM (Klein and Manning,2002), DMV+CCM (Klein and Manning, 2005)and UML-DOP (Bod, 2006) are not directly com-parable to our model, because they are trained us-ing POS tags our model does not consider.

In addition, we further investigate what kindsof trees are induced by our model. FollowingURNNG, we evaluate the performance of con-

Model Data F1median F1max

PRPN WSJ-train 70.5 71.3On-lstm WSJ-train 65.1 66.8C-PCFG WSJ-train 70.5 -Tree-T,L=10 WSJ-train 66.2 67.9DIORA NLI 67.7 68.5Tree-T,L=10 WSJ-all 66.2 68.0CCM WSJ-10 - 71.9DMV+CCM WSJ-10 - 77.6UML-DOP WSJ-10 - 82.9Random - 31.9 32.6LB - 19.6 19.6RB - 56.6 56.6

Table 2: The F1 scores on WSJ-10. Tree Transformeris abbreviated as Tree-T, and L is the number of layers(blocks).

Label Tree-T URNNG PRPNNP 67.6 39.5 63.9VP 38.5 76.6 27.3PP 52.3 55.8 55.1ADJP 24.7 33.9 42.5SBAR 36.4 74.8 28.9ADVP 55.1 50.4 45.1

Table 3: Recall of constituents by labels. The re-sults of URNNG and PRPN are taken from Kim et al.(2019b).

stituents by its label in Table 3. The trees in-duced by different methods are quite different.Our model is inclined to discover noun phrases(NP) and adverb phrases (ADVP), but not easyto discover verb phrases (VP) or adjective phrases(ADJP). We show an induced parse tree in Fig-ure 3 and more induced parse trees can be foundin Appendix.

7.3 Analysis of Induced Structures

In this section, we study whether Tree Trans-former learns hierarchical structures from layers tolayers. First, we analyze the influence of the hy-perparameter minimum layer m in Algorithm. 1given the model trained on WSJ-all in Table 1.As illustrated in Figure 4(a), setting m to be 3yields the best performance. Prior work discov-ered that the representations from the lower lay-ers of Transformer are not informative (Liu et al.,2019). Therefore, using syntactic structures fromlower layers decreases the quality of parse trees.On the other hand, most syntactic information is

1067

(a) F1 in terms of different minimum layers. (b) F1 of parsing via a specific layer.

Figure 4: Performance of unsupervised parsing.

missing when a from top few layers are close to 1,so too large m also decreases the performance.

To further analyze which layer contains richerinformation of syntactic structures, we evaluatethe performance on obtaining parse trees from aspecific layer. We use al from the layer l forparsing with the top-down greedy parsing algo-rithm (Shen et al., 2018a). As shown in Fig-ure 4(b), using a3 from the layer 3 for parsingyields the best F1 score, which is 49.07. The re-sult is consistent to the best value of m. However,compared to our best result (52.0) obtained by Al-gorithm 1, the F1-score decreases by 3 (52.0 →49.07). This demonstrates the effectiveness ofTree Transformer in terms of learning hierarchi-cal structures. The higher layers indeed capturethe higher-level syntactic structures such as clausepatterns.

7.4 Interpretable Self-Attention

This section discusses whether the attention headsin Tree Transformer learn hierarchical structures.Considering that the most straightforward way ofinterpreting what attention heads learn is to visu-alize the attention scores, we plot the heat maps ofConstituent Attention prior C from each layer inFigure 5.

In the heat map of constituent prior from firstlayer (Figure 5(a)), as the size of constituent issmall, the words only attend to its adjacent words.We can observe that the model captures some sub-phrase structures, such as the noun phrase “deltaair line” or “american airlines unit”. In Fig-ure 5(b)-(d), the constituents attach to each otherand become larger. In the layer 6, the words from“involved” to last word “lines” form a high-level

Model L Params PerplexityTransformer 8 40M 48.8Transformer 10 46M 48.5Transformer 10-B 67M 49.2Transformer 12 52M 48.1Tree-T 8 44M 46.1Tree-T 10 51M 45.7Tree-T 12 58M 45.6

Table 4: The perplexity of masked words. Params isthe number of parameters. We denote the number oflayers as L. In Transformer L = 10−B, the increasedhidden size results in more parameters.

adjective phrase (ADJP). In the layer 9, all wordsare grouped into a large constituent except thefirst word “but”. By visualizing the heat mapsof the constituent prior from each layer, we caneasily know what types of syntactic structures arelearned in each layer. The parse tree of this exam-ple can be found in Figure 3 of Appendix. We alsovisualize the heat maps of self-attention from theoriginal Transformer layers and one from the TreeTransformer layers in Appendix A. As the self-attention heads from our model are constrainedby the constituent prior, compared to the originalTransformer, we can discover hierarchical struc-tures more easily.

Those attention heat maps demonstrate that: (1)the size of constituents gradually grows from layerto layer, and (2) at each layer, the attention headstend to attend to other words within constituentsposited in that layer. Those two evidences supportthe success of the proposed Tree Transformer interms of learning tree-like structures.

1068

(a) The layer 0. (b) The layer 3.

(c) The layer 6. (d) The layer 9.

Figure 5: The constituent prior heat maps.

7.5 Masked Language Modeling

To investigate the capability of Tree Transformerin terms of capturing abstract concepts and syn-tactic knowledge, we evaluate the performance onlanguage modeling. As our model is a bidirec-tional encoder, in which the model can see itssubsequent words, we cannot evaluate the lan-guage model in a left-to-right manner. We evaluatethe performance on masked language modelingby measuring the perplexity on masked words3.To perform the inference without randomness, foreach sentence in the testing set, we mask all wordsin the sentence, but not at once. In each maskedtesting data, only one word is replaced with a“[MASK]” token. Therefore, each sentence cre-ates the number of testing samples equal to its

3The perplexity of masked words is e−

∑log(p)

nmask , where pis the probability of correct masked word to be predicted andnmask is the total number of masked words.

length.In Table 4, the models are trained on WSJ-train

with BERT masked LM and evaluated on WSJ-test. All hyperparameters except the number oflayers in Tree Transformer and Transformer areset to be the same and optimized by the same opti-mizer. We use adam as our optimizer with learn-ing rate of 0.0001, β1 = 0.9 and β2 = 0.999. Ourproposed Constituent Attention module increasesabout 10% hyperparameters to the original Trans-former encoder and the computational speed is 1.2times slower. The results with best performanceon validation set are reported. Compared to theoriginal Transformer, Tree Transformer achievesbetter performance on masked language modeling.As the performance gain is possibly due to moreparameters, we adjust the number of layers or in-crease the number of hidden layers in TransformerL = 10 − B. Even with fewer parameters thanTransformer, Tree Transformer still performs bet-

1069

ter.The performance gain is because the induced

tree structures guide the self-attention processeslanguage in a more straightforward and human-like manner, and thus the knowledge can be bet-ter generalized from training data to testing data.Also, Tree Transformer acquires positional infor-mation not only from positional encoding but alsofrom the induced tree structures, where the wordsattend to other words from near to distant (lowerlayers to higher layers)4.

7.6 Limitations and Discussion

It is worth mentioning that we have tried to initial-ize our Transformer model with pre-trained BERT,and then fine-tuning on WSJ-train. However, inthis setting, even when the training loss becomeslower than the loss of training from scratch, theparsing result is still far from our best results. Thissuggests that the attention heads in pre-trainedBERT learn quite different structures from thetree-like structures in Tree Transformer. In ad-dition, with a well-trained Transformer, it is notnecessary for the Constituency Attention moduleto induce reasonable tree structures, because thetraining loss decreases anyway.

8 Conclusion

This paper proposes Tree Transformer, a first at-tempt of integrating tree structures into Trans-former by constraining the attention heads to at-tend within constituents. The tree structures areautomatically induced from the raw texts by ourproposed Constituent Attention module, whichattaches the constituents to each other by self-attention. The performance on unsupervised pars-ing demonstrates the effectiveness of our model interms of inducing tree structures coherent to hu-man expert annotations. We believe that incorpo-rating tree structures into Transformer is an im-portant and worth exploring direction, because itallows Transformer to learn more interpretable at-tention heads and achieve better language model-ing. The interpretable attention can better explainhow the model processes the natural language andguide the future improvement.

4We do not remove the positional encoding from theTransformer and we find that without positional encoding thequality of induced parse trees drops.

Acknowledgements

We would like to thank reviewers for their insight-ful comments. This work was financially sup-ported from the Young Scholar Fellowship Pro-gram by Ministry of Science and Technology(MOST) in Taiwan, under Grant 108-2636-E002-003.

ReferencesRoee Aharoni and Yoav Goldberg. 2017. Towards

string-to-tree neural machine translation. In Pro-ceedings of the 55th Annual Meeting of the Associa-tion for Computational Linguistics (Volume 2: ShortPapers).

Rens Bod. 2006. An all-subtrees approach to unsuper-vised parsing. In Proceedings of the 21st Interna-tional Conference on Computational Linguistics andthe 44th Annual Meeting of the Association for Com-putational Linguistics.

Glenn Carroll and Eugene Charniak. 1992. Two exper-iments on learning probabilistic dependency gram-mars from corpora. In WORKING NOTES OF THEWORKSHOP STATISTICALLY-BASED NLP TECH-NIQUES.

C.Goller and A.Kuchler. 1996. Learning task-dependent distributed representations by backpropa-gation through structure. In Proceedings of Interna-tional Conference on Neural Networks (ICNN’96).

Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge,Thorsten Brants, Phillipp Koehn, and Tony Robin-son. 2013. One billion word benchmark for measur-ing progress in statistical language modeling. arXivpreprint arXiv:1312.3005.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018. Bert: Pre-training of deepbidirectional transformers for language understand-ing. arXiv preprint arXiv:1810.04805.

Li Dong, Nan Yang, Wenhui Wang, Furu Wei,Xiaodong Liu, Yu Wang, Jianfeng Gao, MingZhou, and Hsiao-Wuen Hon. 2019. Unifiedlanguage model pre-training for natural languageunderstanding and generation. arXiv preprintarXiv:1905.03197.

Andrew Drozdov, Pat Verga, Mohit Yadav, MohitIyyer, and Andrew McCallum. 2019. Unsupervisedlatent tree induction with deep inside-outside recur-sive autoencoders. In North American Associationfor Computational Linguistics.

Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros,and Noah A. Smith. 2016. Recurrent neural networkgrammars. In Proc. of NAACL.

1070

Akiko Eriguchi, Yoshimasa Tsuruoka, and KyunghyunCho. 2017. Learning to parse and translate improvesneural machine translation. In Proceedings of the55th Annual Meeting of the Association for Compu-tational Linguistics (Volume 2: Short Papers).

Phu Mon Htut, Kyunghyun Cho, and Samuel Bowman.2018. Grammar induction with neural languagemodels: An unusual replication. In Proceedings ofthe 2018 EMNLP Workshop BlackboxNLP: Analyz-ing and Interpreting Neural Networks for NLP. As-sociation for Computational Linguistics.

Yoon Kim, Chris Dyer, and Alexander Rush. 2019a.Compound probabilistic context-free grammars forgrammar induction. In Proceedings of the 57th An-nual Meeting of the Association for ComputationalLinguistics, pages 2369–2385.

Yoon Kim, Alexander M. Rush, Lei Yu, AdhigunaKuncoro, Chris Dyer, and Gabor Melis. 2019b.Unsupervised recurrent neural network grammars.arXiv preprint arXiv:1904.03746.

Dan Klein and Christopher D. Manning. 2002. Agenerative constituent-context model for improvedgrammar induction. In Proceedings of 40th AnnualMeeting of the Association for Computational Lin-guistics.

Dan Klein and Christopher D. Manning. 2005. Nat-ural language grammar induction with a generativeconstituent-context model. Pattern Recogn.

Nelson F. Liu, Matt Gardner, Yonatan Belinkov,Matthew E. Peters, and Noah A. Smith. 2019. Lin-guistic knowledge and transferability of contextualrepresentations. arXiv preprint arXiv:1903.08855.

Mitchell P. Marcus, Mary Ann Marcinkiewicz, andBeatrice Santorini. 1993. Building a large annotatedcorpus of english: The penn treebank. Comput. Lin-guist.

Alec Radford, Jeff Wu, Rewon Child, David Luan,Dario Amodei, and Ilya Sutskever. 2019. Languagemodels are unsupervised multitask learners.

Yikang Shen, Zhouhan Lin, Chin wei Huang, andAaron Courville. 2018a. Neural language modelingby jointly learning syntax and lexicon. In Interna-tional Conference on Learning Representations.

Yikang Shen, Shawn Tan, Alessandro Sordoni, andAaron Courville. 2018b. Ordered neurons: Integrat-ing tree structures into recurrent neural networks.arXiv preprint arXiv:1810.09536.

Noah A. Smith and Jason Eisner. 2005. Guiding un-supervised grammar induction using contrastive es-timation. In In Proc. of IJCAI Workshop on Gram-matical Inference Applications.

Richard Socher, Cliff Chiung-Yu Lin, Andrew Y. Ng,and Christopher D. Manning. 2011. Parsing natu-ral scenes and natural language with recursive neu-ral networks. In Proceedings of the 28th Interna-tional Conference on International Conference onMachine Learning.

Emma Strubell, Patrick Verga, Daniel Andor,David I Weiss, and Andrew McCallum. 2018.Linguistically-informed self-attention for semanticrole labeling. In EMNLP.

Kai Sheng Tai, Richard Socher, and Christopher D.Manning. 2015. Improved semantic representationsfrom tree-structured long short-term memory net-works. arXiv preprint arXiv:1503.00075.

Jesse Vig. 2019. Visualizing attention in transformer-based language representation models. arXivpreprint arXiv:1904.02679.

Adina Williams, Nikita Nangia, and Samuel Bowman.2018. A broad-coverage challenge corpus for sen-tence understanding through inference. In Proceed-ings of the 2018 Conference of the North AmericanChapter of the Association for Computational Lin-guistics: Human Language Technologies, Volume 1(Long Papers).

Wei Wu, Houfeng Wang, Tianyu Liu, and ShumingMa. 2018. Phrase-level self-attention networks foruniversal sentence encoding. In Proceedings of the2018 Conference on Empirical Methods in NaturalLanguage Processing, pages 3729–3738.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V.Le, Mohammad Norouzi, Wolfgang Macherey,Maxim Krikun, Yuan Cao, Qin Gao, KlausMacherey, and et al. 2016. Google’s neural ma-chine translation system: Bridging the gap betweenhuman and machine translation. arXiv preprintarXiv:1609.08144.

Dani Yogatama, Phil Blunsom, Chris Dyer, EdwardGrefenstette, and Wang Ling. 2017. Learning tocompose words into sentences with reinforcementlearning. In International Conference on LearningRepresentations.

Poorya Zaremoodi and Gholamreza Haffari. 2018. In-corporating syntactic uncertainty in neural machinetranslation with a forest-to-sequence model. In Pro-ceedings of the 27th International Conference onComputational Linguistics.