Learning Neural Templates for Text Generation · hand-engineered rules (Angeli et...

Learning Neural Templates for Text Generation

Sam Wiseman Stuart M. Shieber Alexander M. RushSchool of Engineering and Applied Sciences

Harvard UniversityCambridge, MA, USA

{swiseman,shieber,srush}@seas.harvard.edu

Abstract

While neural, encoder-decoder models havehad significant empirical success in text gener-ation, there remain several unaddressed prob-lems with this style of generation. Encoder-decoder models are largely (a) uninterpretable,and (b) difficult to control in terms of theirphrasing or content. This work proposes aneural generation system using a hidden semi-markov model (HSMM) decoder, which learnslatent, discrete templates jointly with learningto generate. We show that this model learnsuseful templates, and that these templatesmake generation both more interpretable andcontrollable. Furthermore, we show that thisapproach scales to real data sets and achievesstrong performance nearing that of encoder-decoder text generation models.

1 Introduction

With the continued success of encoder-decodermodels for machine translation and related tasks,there has been great interest in extending thesemethods to build general-purpose, data-driven nat-ural language generation (NLG) systems (Meiet al., 2016; Dusek and Jurcıcek, 2016; Lebretet al., 2016; Chisholm et al., 2017; Wiseman et al.,2017). These encoder-decoder models (Sutskeveret al., 2014; Cho et al., 2014; Bahdanau et al.,2015) use a neural encoder model to represent asource knowledge base, and a decoder model toemit a textual description word-by-word, condi-tioned on the source encoding. This style of gen-eration contrasts with the more traditional divisionof labor in NLG, which famously emphasizes ad-dressing the two questions of “what to say” and“how to say it” separately, and which leads to sys-tems with explicit content selection, macro- andmicro-planning, and surface realization compo-nents (Reiter and Dale, 1997; Jurafsky and Martin,2014).

Source Entity: Cottotype[coffee shop], rating[3 out of 5],food[English], area[city centre],price[moderate], near[The Portland Arms]

System Generation:Cotto is a coffee shop serving English foodin the moderate price range. It is locatednear The Portland Arms. Its customer rating is3 out of 5.

Neural Template:

| The

...|

is ais an

is an expensive...

| |providingservingoffering

...

|

|food

cuisinefoods...

|in thewith a

and has a...

| |price range

price bracketpricing...

| . |It’sIt is

The place is...

|located in thelocated near

near...

| | . |Its customer rating is

Their customer rating isCustomers have rated it

...

| | .

Figure 1: An example template-like generation from the E2EGeneration dataset (Novikova et al., 2017). Knowledge basex (top) contains 6 records, and y (middle) is a system gen-eration; records are shown as type[value]. An inducedneural template (bottom) is learned by the system and em-ployed in generating y. Each cell represents a segment inthe learned segmentation, and “blanks” show where slots arefilled through copy attention during generation.

Encoder-decoder generation systems appear tohave increased the fluency of NLG outputs, whilereducing the manual effort required. However,due to the black-box nature of generic encoder-decoder models, these systems have also largelysacrificed two important desiderata that are oftenfound in more traditional systems, namely (a) in-terpretable outputs that (b) can be easily controlledin terms of form and content.

This work considers building interpretable andcontrollable neural generation systems, and pro-poses a specific first step: a new data-driven gen-eration model for learning discrete, template-like

arX

iv:1

808.

1012

2v2

[cs

.CL

] 1

3 Se

p 20

18

structures for conditional text generation. Thecore system uses a novel, neural hidden semi-markov model (HSMM) decoder, which providesa principled approach to template-like text gener-ation. We further describe efficient methods fortraining this model in an entirely data-driven wayby backpropagation through inference. Generat-ing with the template-like structures induced bythe neural HSMM allows for the explicit repre-sentation of what the system intends to say (in theform of a learned template) and how it is attempt-ing to say it (in the form of an instantiated tem-plate).

We show that we can achieve performance com-petitive with other neural NLG approaches, whilemaking progress satisfying the above two desider-ata. Concretely, our experiments indicate that wecan induce explicit templates (as shown in Figure1) while achieving competitive automatic scores,and that we can control and interpret our gener-ations by manipulating these templates. Finally,while our experiments focus on the data-to-textregime, we believe the proposed methodology rep-resents a compelling approach to learning discrete,latent-variable representations of conditional text.

2 Related Work

A core task of NLG is to generate textual descrip-tions of knowledge base records. A common ap-proach is to use hand-engineered templates (Ku-kich, 1983; McKeown, 1992; McRoy et al., 2000),but there has also been interest in creating tem-plates in an automated manner. For instance,many authors induce templates by clustering sen-tences and then abstracting templated fields withhand-engineered rules (Angeli et al., 2010; Kon-dadadi et al., 2013; Howald et al., 2013), or with apipeline of other automatic approaches (Wang andCardie, 2013).

There has also been work in incorporating prob-abilistic notions of templates into generation mod-els (Liang et al., 2009; Konstas and Lapata, 2013),which is similar to our approach. However, theseapproaches have always been conjoined with dis-criminative classifiers or rerankers in order to ac-tually accomplish the generation (Angeli et al.,2010; Konstas and Lapata, 2013). In addition,these models explicitly model knowledge basefield selection, whereas the model we present isfundamentally an end-to-end model over genera-tion segments.

Recently, a new paradigm has emerged aroundneural text generation systems based on machinetranslation (Sutskever et al., 2014; Cho et al.,2014; Bahdanau et al., 2015). Most of thiswork has used unconstrained black-box encoder-decoder approaches. There has been some workon discrete variables in this context, including ex-tracting representations (Shen et al., 2018), incor-porating discrete latent variables in text model-ing (Yang et al., 2018), and using non-HSMM seg-mental models for machine translation or summa-rization (Yu et al., 2016; Wang et al., 2017; Huanget al., 2018). Dai et al. (2017) develop an approx-imate inference scheme for a neural HSMM usingRNNs for continuous emissions; in contrast wemaximize the exact log-marginal, and use RNNsto parameterize a discrete emission distribution.Finally, there has also been much recent interest insegmental RNN models for non-generative tasksin NLP (Tang et al., 2016; Kong et al., 2016; Luet al., 2016).

The neural text generation community has alsorecently been interested in “controllable” text gen-eration (Hu et al., 2017), where various aspectsof the text (often sentiment) are manipulated ortransferred (Shen et al., 2017; Zhao et al., 2018; Liet al., 2018). In contrast, here we focus on control-ling either the content of a generation or the way itis expressed by manipulating the (latent) templateused in realizing the generation.

3 Overview: Data-Driven NLG

Our focus is on generating a textual descriptionof a knowledge base or meaning representation.Following standard notation (Liang et al., 2009;Wiseman et al., 2017), let x= {r1 . . . rJ} be acollection of records. A record is made up ofa type (r.t), an entity (r.e), and a value (r.m).For example, a knowledge base of restaurantsmight have a record with r.t = Cuisine, r.e =Denny’s, and r.m = American. The aim isto generate an adequate and fluent text descriptiony1:T = y1, . . . , yT of x. Concretely, we considerthe E2E Dataset (Novikova et al., 2017) and theWikiBio Dataset (Lebret et al., 2016). We showan example E2E knowledge base x in the top ofFigure 1. The top of Figure 2 shows an exam-ple knowledge base x from the WikiBio dataset,where it is paired with a reference text y= y1:T atthe bottom.

The dominant approach in neural NLG has been

sopoulos, 2007; Turner et al., 2010). Generation isdivided into modular, yet highly interdependent, de-cisions: (1) content planning defines which parts ofthe input fields or meaning representations shouldbe selected; (2) sentence planning determines whichselected fields are to be dealt with in each outputsentence; and (3) surface realization generates thosesentences.

Data-driven approaches have been proposed toautomatically learn the individual modules. One ap-proach first aligns records and sentences and thenlearns a content selection model (Duboue and McK-eown, 2002; Barzilay and Lapata, 2005). Hierar-chical hidden semi-Markov generative models havealso been used to first determine which facts to dis-cuss and then to generate words from the predi-cates and arguments of the chosen facts (Liang et al.,2009). Sentence planning has been formulated as asupervised set partitioning problem over facts whereeach partition corresponds to a sentence (Barzilayand Lapata, 2006). End-to-end approaches havecombined sentence planning and surface realiza-tion by using explicitly aligned sentence/meaningpairs as training data (Ratnaparkhi, 2002; Wong andMooney, 2007; Belz, 2008; Lu and Ng, 2011). Morerecently, content selection and surface realizationhave been combined (Angeli et al., 2010; Kim andMooney, 2010; Konstas and Lapata, 2013).

At the intersection of rule-based and statisti-cal methods, hybrid systems aim at leveraging hu-man contributed rules and corpus statistics (Langk-ilde and Knight, 1998; Soricut and Marcu, 2006;Mairesse and Walker, 2011).

Our approach is inspired by the recent success ofneural language models for image captioning (Kiroset al., 2014; Karpathy and Fei-Fei, 2015; Vinyals etal., 2015; Fang et al., 2015; Xu et al., 2015), ma-chine translation (Devlin et al., 2014; Bahdanau etal., 2015; Luong et al., 2015), and modeling conver-sations and dialogues (Shang et al., 2015; Wen et al.,2015; Yao et al., 2015).

Our model is most similar to Mei et al. (2016)who use an encoder-decoder style neural networkmodel to tackle the WEATHERGOV and ROBOCUPtasks. Their architecture relies on LSTM units andan attention mechanism which reduces scalabilitycompared to our simpler design.

Figure 1: Wikipedia infobox of Frederick Parker-Rhodes. Theintroduction of his article reads: “Frederick Parker-Rhodes (21March 1914 – 21 November 1987) was an English linguist,plant pathologist, computer scientist, mathematician, mystic,and mycologist.”.

3 Language Modeling for ConstrainedSentence generation

Conditional language models are a popular choiceto generate sentences. We introduce a table-conditioned language model for constraining textgeneration to include elements from fact tables.

3.1 Language modelGiven a sentence s = w1, . . . , wT with T wordsfrom vocabularyW , a language model estimates:

P (s) =

T∏

t=1

P (wt|w1, . . . , wt−1) . (1)

Let ct = wt−(n−1), . . . , wt−1 be the sequence ofn − 1 context words preceding wt. An n-gram lan-guage model makes an order n Markov assumption,

P (s) ≈T∏

t=1

P (wt|ct) . (2)

3.2 Language model conditioned on tablesA table is a set of field/value pairs, where values aresequences of words. We therefore propose languagemodels that are conditioned on these pairs.

Local conditioning refers to the informationfrom the table that is applied to the description of thewords which have already generated, i.e. the previ-ous words that constitute the context of the language

2

Frederick Parker-Rhodes (21 March 1914 - 21 November

1987) was an English linguist, plant pathologist, computer

scientist, mathematician, mystic, and mycologist.

Figure 2: An example from the WikiBio dataset (Lebretet al., 2016), with a database x (top) for Frederick Parker-Rhodes and corresponding reference generation y (bottom).

to use an encoder network over x and then a condi-tional decoder network to generate y, training thewhole system in an end-to-end manner. To gener-ate a description for a given example, a black-boxnetwork (such as an RNN) is used to produce a dis-tribution over the next word, from which a choiceis made and fed back into the system. The entiredistribution is driven by the internal states of theneural network.

While effective, relying on a neural decodermakes it difficult to understand what aspects ofx are correlated with a particular system output.This leads to problems both in controlling fine-grained aspects of the generation process and ininterpreting model mistakes.

As an example of why controllability is im-portant, consider the records in Figure 1. Giventhese inputs an end-user might want to generatean output meeting specific constraints, such as notmentioning any information relating to customerrating. Under a standard encoder-decoder stylemodel, one could filter out this information eitherfrom the encoder or decoder, but in practice thiswould lead to unexpected changes in output thatmight propagate through the whole system.

As an example of the difficulty of interpret-ing mistakes, consider the following actual gen-eration from an encoder-decoder style system for

the records in Figure 2: ”frederick parker-rhodes(21 november 1914 - 2 march 1987) was an en-glish mycology and plant pathology, mathematicsat the university of uk.” In addition to not beingfluent, it is unclear what the end of this sentenceis even attempting to convey: it may be attempt-ing to convey a fact not actually in the knowledgebase (e.g., where Parker-Rhodes studied), or per-haps it is simply failing to fluently realize infor-mation that is in the knowledge base (e.g., Parker-Rhodes’s country of residence).

Traditional NLG systems (Kukich, 1983; McK-eown, 1992; Belz, 2008; Gatt and Reiter, 2009), incontrast, largely avoid these problems. Since theytypically employ an explicit planning component,which decides which knowledge base records tofocus on, and a surface realization component,which realizes the chosen records, the intent of thesystem is always explicit, and it may be modifiedto meet constraints.

The goal of this work is to propose an approachto neural NLG that addresses these issues in a prin-cipled way. We target this goal by proposing anew model that generates with template-like ob-jects induced by a neural HSMM (see Figure 1).Templates are useful here because they representa fixed plan for the generation’s content, and be-cause they make it clear what part of the genera-tion is associated with which record in the knowl-edge base.

4 Background: Semi-Markov Models

What does it mean to learn a template? It is nat-ural to think of a template as a sequence of typedtext-segments, perhaps with some segments actingas the template’s “backbone” (Wang and Cardie,2013), and the remaining segments filled in fromthe knowledge base.

A natural probabilistic model conforming withthis intuition is the hidden semi-markov model(HSMM) (Gales and Young, 1993; Ostendorfet al., 1996), which models latent segmentationsin an output sequence. Informally, an HSMM ismuch like an HMM, except emissions may lastmultiple time-steps, and multi-step emissions neednot be independent of each other conditioned onthe state.

We briefly review HSMMs following Murphy(2002). Assume we have a sequence of ob-served tokens y1 . . . yT and a discrete, latent statezt ∈{1, . . . ,K} for each timestep. We addition-

ally use two per-timestep variables to model multi-step segments: a length variable lt ∈{1, . . . , L}specifying the length of the current segment, and adeterministic binary variable ft indicating whethera segment finishes at time t. We will consider inparticular conditional HSMMs, which conditionon a source x, essentially giving us an HSMM de-coder.

An HSMM specifies a joint distribution on theobservations and latent segmentations. Letting θdenote all the parameters of the model, and usingthe variables introduced above, we can write thecorresponding joint-likelihood as follows

p(y, z, l, f |x; θ) =T−1∏t=0

p(zt+1, lt+1 | zt, lt, x)ft

×T∏t=1

p(yt−lt+1:t | zt, lt, x)ft ,

where we take z0 to be a distinguished start-state, and the deterministic ft variables are usedfor excluding non-segment log probabilities. Wefurther assume p(zt+1, lt+1 | zt, lt, x) factors asp(zt+1 | zt, x) × p(lt+1 | zt+1). Thus, the likeli-hood is given by the product of the probabilitiesof each discrete state transition made, the proba-bility of the length of each segment given its dis-crete state, and the probability of the observationsin each segment, given its state and length.

5 A Neural HSMM Decoder

We use a novel, neural parameterization of anHSMM to specify the probabilities in the likeli-hood above. This full model, sketched out in Fig-ure 3, allows us to incorporate the modeling com-ponents, such as LSTMs and attention, that makeneural text generation effective, while maintainingthe HSMM structure.

5.1 Parameterization

Since our model must condition on x, let rj ∈Rd

represent a real embedding of record rj ∈x, andlet xa ∈Rd represent a real embedding of the en-tire knowledge base x, obtained by max-poolingcoordinate-wise over all the rj . It is also usefulto have a representation of just the unique typesof records that appear in x, and so we also definexu ∈Rd to be the sum of the embeddings of theunique types appearing in x, plus a bias vector andfollowed by a ReLU nonlinearity.

x

z1

RNN

y1 y2 y3 y4

RNN

z4T

Figure 3: HSMM factor graph (under a known segmenta-tion) to illustrate parameters. Here we assume z1 is in the“red” state (out of K possibilities), and transitions to the“blue” state after emitting three words. The transition model,shown as T , is a function of the two states and the neural en-coded source x. The emission model is a function of a “red”RNN model (with copy attention over x) that generates words1, 2 and 3. After transitioning, the next word y4 is generatedby the “blue” RNN, but independently of the previous words.

Transition Distribution The transition distribu-tion p(zt+1 | zt, x) may be viewed as aK ×K ma-trix of probabilities, where each row sums to 1. Wedefine this matrix to be

p(zt+1 | zt, x) ∝ AB +C(xu)D(xu),

where A∈RK×m1 , B ∈Rm1×K are state embed-dings, and where C : Rd → RK×m2 and D :Rd → Rm2×K are parameterized non-linear func-tions of xu. We apply a row-wise softmax to theresulting matrix to obtain the desired probabilities.

Length Distribution We simply fix all lengthprobabilities p(lt+1 | zt+1) to be uniform up to amaximum length L.1

Emission Distribution The emission modelmodels the generation of a text segment condi-tioned on a latent state and source information,and so requires a richer parameterization. Inspiredby the models used for neural NLG, we base thismodel on an RNN decoder, and write a segment’sprobability as a product over token-level probabil-ities,

p(yt−lt+1:t | zt= k, lt= l, x) =

lt∏i=1

p(yt−lt+i | yt−lt+1:t−lt+i−1, zt= k, x)

× p(</seg> | yt−lt+1:t, zt= k, x)× 1{lt = l},

1We experimented with parameterizing the length distri-bution, but found that it led to inferior performance. Forcingthe length probabilities to be uniform encourages the modelto cluster together functionally similar emissions of differ-ent lengths, while parameterizing them can lead to states thatspecialize to specific emission lengths.

where </seg> is an end of segment token. TheRNN decoder uses attention and copy-attentionover the embedded records rj , and is conditionedon zt= k by concatenating an embedding corre-sponding to the k’th latent state to the RNN’s in-put; the RNN is also conditioned on the entire xby initializing its hidden state with xa.

More concretely, let hki−1 ∈Rd be the state of

an RNN conditioned on x and zt= k (as above)run over the sequence yt−lt+1:t−lt+i−1. We let themodel attend over records rj using hk

i−1 (in thestyle of Luong et al. (2015)), producing a contextvector cki−1. We may then obtain scores vi−1 foreach word in the output vocabulary,

vi−1=W tanh(gk1 ◦ [hk

i−1, cki−1]),

with parameters gk1 ∈R2d and W ∈RV×2d. Note

that there is a gk1 vector for each of K discrete

states. To additionally implement a kind of slotfilling, we allow emissions to be directly copiedfrom the value portion of the records rj using copyattention (Gulcehre et al., 2016; Gu et al., 2016;Yang et al., 2016). Define copy scores,

ρj = rTj tanh(gk2 ◦ hk

i−1),

where gk2 ∈Rd. We then normalize the output-

vocabulary and copy scores together, to arrive at

vi−1=softmax([vi−1, ρ1, . . . , ρJ ]),

and thus

p(yt−lt+i=w | yt−lt+1:t−lt+i−1, zt= k, x) =

vi−1,w +∑

j:rj .m=w

vi−1,V+j .

An Autoregressive Variant The model as spec-ified assumes segments are independent condi-tioned on the associated latent state and x. Whilethis assumption still allows for reasonable perfor-mance, we can tractably allow interdependencebetween tokens (but not segments) by having eachnext-token distribution depend on all the previ-ously generated tokens, giving us an autoregres-sive HSMM. For this model, we will in fact usep(yt−lt+i=w | y1:t−lt+i−1, zt= k, x) in definingour emission model, which is easily implementedby using an additional RNN run over all the pre-ceding tokens. We will report scores for bothnon-autoregressive and autoregressive HSMM de-coders below.

5.2 Learning

The model requires fitting a large set of neu-ral network parameters. Since we assume z, l,and f are unobserved, we marginalize over thesevariables to maximize the log marginal-likelihoodof the observed tokens y given x. The HSMMmarginal-likelihood calculation can be carried outefficiently with a dynamic program analogous toeither the forward- or backward-algorithm famil-iar from HMMs (Rabiner, 1989).

It is actually more convenient to use thebackward-algorithm formulation when usingRNNs to parameterize the emission distributions,and we briefly review the backward recurrenceshere, again following Murphy (2002). We have:

βt(j) = p(yt+1:T | zt= j, ft=1, x)

=K∑k=1

β∗t (k) p(zt+1= k | zt = j)

β∗t (k) = p(yt+1:T | zt+1 = k, ft = 1, x)

=L∑l=1

[βt+l(k) p(lt+1= l | zt+1= k)

p(yt+1:t+l | zt+1= k, lt+1= l)],

with base case βT (j)= 1. We can nowobtain the marginal probability of y asp(y |x)=

∑Kk=1 β

∗0(k) p(z1= k), where we

have used the fact that f0 must be 1, and wetherefore train to maximize the log-marginallikelihood of the observed y:

ln p(y |x; θ) = ln

K∑k=1

β∗0(k) p(z1= k). (1)

Since the quantities in (1) are obtained from adynamic program, which is itself differentiable,we may simply maximize with respect to the pa-rameters θ by back-propagating through the dy-namic program; this is easily accomplished withautomatic differentiation packages, and we usepytorch (Paszke et al., 2017) in all experiments.

5.3 Extracting Templates and Generating

After training, we could simply condition on a newdatabase and generate with beam search, as is stan-dard with encoder-decoder models. However, thestructured approach we have developed allows usto generate in a more template-like way, giving usmore interpretable and controllable generations.

[The Golden Palace]55 [is a]59 [coffee shop]12[providing]3 [Indian]50 [food]1 [in the]17 [£20-25]26 [price range]16 [.]2 [It is]8 [located inthe]25 [riverside]40 [.]53 [Its customer rating is]19[high]23 [.]2

Figure 4: A sample Viterbi segmentation of a training text;subscripted numbers indicate the corresponding latent state.From this we can extract a template with S=17 segments;compare with the template used at the bottom of Figure 1.

First, note that given a database x and refer-ence generation y we can obtain the MAP assign-ment to the variables z, l, and f with a dynamicprogram similar to the Viterbi algorithm familiarfrom HMMs. These assignments will give us atyped segmentation of y, and we show an exampleViterbi segmentation of some training text in Fig-ure 4. Computing MAP segmentations allows usto associate text-segments (i.e., phrases) with thediscrete labels zt that frequently generate them.These MAP segmentations can be used in an ex-ploratory way, as a sort of dimensionality reduc-tion of the generations in the corpus. More im-portantly for us, however, they can also be used toguide generation.

In particular, since each MAP segmentation im-plies a sequence of hidden states z, we may runa template extraction step, where we collect themost common “templates” (i.e., sequences of hid-den states) seen in the training data. Each “tem-plate” z(i) consists of a sequence of latent states,with z(i)= z

(i)1 , . . . z

(i)S representing the S distinct

segments in the i’th extracted template (recall thatwe will technically have a zt for each time-step,and so z(i) is obtained by collapsing adjacent zt’swith the same value); see Figure 4 for an exampletemplate (with S=17) that can be extracted fromthe E2E corpus. The bottom of Figure 1 shows avisualization of this extracted template, where dis-crete states are replaced by the phrases they fre-quently generate in the training data.

With our templates z(i) in hand, we can thenrestrict the model to using (one of) them duringgeneration. In particular, given a new input x, wemay generate by computing

y(i) = argmaxy′

p(y′, z(i) |x), (2)

which gives us a generation y(i) for each extractedtemplate z(i). For example, the generation in Fig-ure 1 is obtained by maximizing (2) with x set tothe database in Figure 1 and z(i) set to the template

extracted in Figure 4. In practice, the argmax in(2) will be intractable to calculate exactly due tothe use of RNNs in defining the emission distribu-tion, and so we approximate it with a constrainedbeam search. This beam search looks very similarto that typically used with RNN decoders, exceptthe search occurs only over a segment, for a par-ticular latent state k.

5.4 Discussion

Returning to the discussion of controllability andinterpretability, we note that with the proposedmodel (a) it is possible to explicitly force the gen-eration to use a chosen template z(i), which is it-self automatically learned from training data, and(b) that every segment in the generated y(i) istyped by its corresponding latent variable. We ex-plore these issues empirically in Section 7.1.

We also note that these properties may be use-ful for other text applications, and that they offeran additional perspective on how to approach la-tent variable modeling for text. Whereas there hasbeen much recent interest in learning continuouslatent variable representations for text (see Sec-tion 2), it has been somewhat unclear what the la-tent variables to be learned are intended to capture.On the other hand, the latent, template-like struc-tures we induce here represent a plausible, proba-bilistic latent variable story, and allow for a morecontrollable method of generation.

Finally, we highlight one significant possible is-sue with this model – the assumption that seg-ments are independent of each other given the cor-responding latent variable and x. Here we notethat the fact that we are allowed to condition on xis quite powerful. Indeed, a clever encoder couldcapture much of the necessary interdependencebetween the segments to be generated (e.g., thecorrect determiner for an upcoming noun phrase)in its encoding, allowing the segments themselvesto be decoded more or less independently, given x.

6 Data and Methods

Our experiments apply the approach outlinedabove to two recent, data-driven NLG tasks.

6.1 Datasets

Experiments use the E2E (Novikova et al., 2017)and WikiBio (Lebret et al., 2016) datasets, ex-amples of which are shown in Figures 1 and 2,respectively. The former dataset, used for the

2018 E2E-Gen Shared Task, contains approxi-mately 50K total examples, and uses 945 distinctword types, and the latter dataset contains approx-imately 500K examples and uses approximately400K word types. Because our emission modeluses a word-level copy mechanism, any recordwith a phrase consisting of n words as its value isreplaced with n positional records having a singleword value, following the preprocessing of Lebretet al. (2016). For example, “type[coffee shop]”in Figure 1 becomes “type-1[coffee]” and “type-2[shop].”

For both datasets we compare with publishedencoder-decoder models, as well as with directtemplate-style baselines. The E2E task is eval-uated in terms of BLEU (Papineni et al., 2002),NIST (Belz and Reiter, 2006), ROUGE (Lin,2004), CIDEr (Vedantam et al., 2015), and ME-TEOR (Banerjee and Lavie, 2005).2 The bench-mark system for the task is an encoder-decoderstyle system followed by a reranker, proposed byDusek and Jurcıcek (2016). We compare to thisbaseline, as well as to a simple but competitivenon-parametric template-like baseline (“SUB” intables), which selects a training sentence withrecords that maximally overlap (without includingextraneous records) the unseen set of records wewish to generate from; ties are broken at random.Then, word-spans in the chosen training sentenceare aligned with records by string-match, and re-placed with the corresponding fields of the new setof records.3

The WikiBio dataset is evaluated in terms ofBLEU, NIST, and ROUGE, and we compare withthe systems and baselines implemented by Lebretet al. (2016), which include two neural, encoder-decoder style models, as well as a Kneser-Ney,templated baseline.

6.2 Model and Training Details

We first emphasize two additional methodologicaldetails important for obtaining good performance.

Constraining Learning We were able to learnmore plausible segmentations of y by constrainingthe model to respect word spans yt+1:t+l that ap-pear in some record rj ∈x. We accomplish this bygiving zero probability (within the backward re-

2We use the official E2E NLG Challenge scoring scripts athttps://github.com/tuetschek/e2e-metrics.

3For categorical records, like “familyFriendly”, whichcannot easily be aligned with a phrase, we simply select onlycandidate training sentences with the same categorical value.

currences in Section 5) to any segmentation thatsplits up a sequence yt+1:t+l that appears in somerj , or that includes yt+1:t+l as a subsequence ofanother sequence. Thus, we maximize (1) subjectto these hard constraints.

Increasing the Number of Hidden StatesWhile a larger K allows for a more expressive la-tent model, computing K emission distributionsover the vocabulary can be prohibitively expen-sive. We therefore tie the emission distribution be-tween multiple states, while allowing them to havea different transition distributions.

We give additional architectural details of ourmodel in the Supplemental Material; here we notethat we use an MLP to embed rj ∈Rd, and a 1-layer LSTM (Hochreiter and Schmidhuber, 1997)in defining our emission distributions. In order toreduce the amount of memory used, we restrict ouroutput vocabulary (and thus the height of the ma-trix W in Section 5) to only contain words in ythat are not present in x; any word in y present in xis assumed to be copied. In the case where a wordyt appears in a record rj (and could therefore havebeen copied), the input to the LSTM at time t+1 iscomputed using information from rj ; if there aremultiple rj from which yt could have been copied,the computed representations are simply averaged.

For all experiments, we set d=300 and L=4.At generation time, we select the 100 most com-mon templates z(i), perform beam search with abeam of size 5, and select the generation with thehighest overall joint probability.

For our E2E experiments, our best non-autoregressive model has 55 “base” states, dupli-cated 5 times, for a total of K =275 states, andour best autoregressive model uses K =60 states,without any duplication. For our WikiBio exper-iments, both our best non-autoregressive and au-toregressive models uses 45 base states duplicated3 times, for a total of K =135 states. In all cases,K was chosen based on BLEU performance onheld-out validation data. Code implementing ourmodels is available at https://github.com/harvardnlp/neural-template-gen.

7 Results

Our results on automatic metrics are shown inTables 1 and 2. In general, we find that thetemplated baselines underperform neural models,whereas our proposed model is fairly competi-tive with neural models, and sometimes even out-

https://github.com/tuetschek/e2e-metrics

https://github.com/harvardnlp/neural-template-gen

https://github.com/harvardnlp/neural-template-gen

BLEU NIST ROUGE CIDEr METEOR

Validation

D&J 69.25 8.48 72.57 2.40 47.03SUB 43.71 6.72 55.35 1.41 37.87NTemp 64.53 7.66 68.60 1.82 42.46NTemp+AR 67.07 7.98 69.50 2.29 43.07

Test

D&J 65.93 8.59 68.50 2.23 44.83SUB 43.78 6.88 54.64 1.39 37.35NTemp 55.17 7.14 65.70 1.70 41.91NTemp+AR 59.80 7.56 65.01 1.95 38.75

Table 1: Comparison of the system of Dusek and Jurcıcek(2016), which forms the baseline for the E2E challenge, anon-parametric, substitution-based baseline (see text), andour HSMM models (denoted “NTemp” and “NTemp+AR”for the non-autoregressive and autoregressive versions, resp.)on the validation and test portions of the E2E dataset.“ROUGE” is ROUGE-L. Models are evaluated using the of-ficial E2E NLG Challenge scoring scripts.

BLEU NIST ROUGE-4

Template KN † 19.8 5.19 10.7NNLM (field) † 33.4 7.52 23.9NNLM (field & word) † 34.7 7.98 25.8NTemp 34.2 7.94 35.9NTemp+AR 34.8 7.59 38.6

Seq2seq (Liu et al., 2018) 43.65 - 40.32

Table 2: Top: comparison of the two best neural systems ofLebret et al. (2016), their templated baseline, and our HSMMmodels (denoted “NTemp” and “NTemp+AR” for the non-autoregressive and autoregressive versions, resp.) on the testportion of the WikiBio dataset. Models marked with a † arefrom Lebret et al. (2016), and following their methodologywe use ROUGE-4. Bottom: state-of-the-art seq2seq-style re-sults from Liu et al. (2018).

performs them. On the E2E data, for example,we see in Table 1 that the SUB baseline, despitehaving fairly impressive performance for a non-parametric model, fares the worst. The neuralHSMM models are largely competitive with theencoder-decoder system on the validation data, de-spite offering the benefits of interpretability andcontrollability; however, the gap increases on test.

Table 2 evaluates our system’s performance onthe test portion of the WikiBio dataset, compar-ing with the systems and baselines implementedby Lebret et al. (2016). Again for this dataset wesee that their templated Kneser-Ney model under-performs on the automatic metrics, and that neu-ral models improve on these results. Here theHSMMs are competitive with the best model ofLebret et al. (2016), and even outperform it onROUGE. We emphasize, however, that recent, so-phisticated approaches to encoder-decoder style

Travellers Rest Beefeater

name[Travellers Rest Beefeater], customerRating[3 out of 5],area[riverside], near[Raja Indian Cuisine]

1. [Travellers Rest Beefeater]55 [is a]59 [3 star]43[restaurant]11 [located near]25 [Raja Indian Cuisine]40 [.]53

2. [Near]31 [riverside]29 [,]44 [Travellers Rest Beefeater]55[serves]3 [3 star]50 [food]1 [.]2

3. [Travellers Rest Beefeater]55 [is a]59 [restaurant]12[providing]3 [riverside]50 [food]1 [and has a]17[3 out of 5]26 [customer rating]16 [.]2 [It is]8 [near]25[Raja Indian Cuisine]40 [.]53

4. [Travellers Rest Beefeater]55 [is a]59 [place to eat]12[located near]25 [Raja Indian Cuisine]40 [.]53

5. [Travellers Rest Beefeater]55 [is a]59 [3 out of 5]5[rated]32 [riverside]43 [restaurant]11 [near]25[Raja Indian Cuisine]40 [.]53

Table 3: Impact of varying the template z(i) for a single xfrom the E2E validation data; generations are annotated withthe segmentations of the chosen z(i). Results were obtainedusing the NTemp+AR model from Table 1.

database-to-text generation have since surpassedthe results of Lebret et al. (2016) and our own,and we show the recent seq2seq style results of Liuet al. (2018), who use a somewhat larger model, atthe bottom of Table 2.

7.1 Qualitative EvaluationWe now qualitatively demonstrate that our gener-ations are controllable and interpretable.

Controllable Diversity One of the powerful as-pects of the proposed approach to generation isthat we can manipulate the template z(i) whileleaving the database x constant, which allows foreasily controlling aspects of the generation. In Ta-ble 3 we show the generations produced by ourmodel for five different neural template sequencesz(i), while fixing x. There, the segments in eachgeneration are annotated with the latent states de-termined by the corresponding z(i). We see thatthese templates can be used to affect the word-ordering, as well as which fields are mentioned inthe generated text. Moreover, because the discretestates align with particular fields (see below), it isgenerally simple to automatically infer to whichfields particular latent states correspond, allowingusers to choose which template best meets their re-quirements. We emphasize that this level of con-trollability is much harder to obtain for encoder-decoder models, since, at best, a large amount ofsampling would be required to avoid generatingaround a particular mode in the conditional distri-bution, and even then it would be difficult to con-trol the sort of generations obtained.

kenny warren

name: kenny warren, birth date: 1 april 1946, birth name: kenneth warren deutscher, birth place: brooklyn, new york,occupation: ventriloquist, comedian, author, notable work: book - the revival of ventriloquism in america

1. [kenneth warren deutscher]132 [ ( ]75 [born]89 [april 1, 1946]101 [ ) ]67 [is an american]82 [author]20 [and]1[ventriloquist and comedian]69 [.]88

2. [kenneth warren deutscher]132 [ ( ]75 [born]89 [april 1, 1946]101 [ ) ]67 [is an american]82 [author]20[best known for his]95 [the revival of ventriloquism]96 [.]88

3. [kenneth warren]16 [“kenny” warren]117 [ ( ]75 [born]89 [april 1, 1946]101 [ ) ]67 [is an american]127[ventriloquist, comedian]28 [.]133

4. [kenneth warren]16 [“kenny” warren]117 [ ( ]75 [born]89 [april 1, 1946]101 [ ) ]67 [is a]104 [new york]98 [author]20 [.]1335. [kenneth warren deutscher]42 [is an american]82 [ventriloquist, comedian]118 [based in]15 [brooklyn, new york]84 [.]88

Table 4: Impact of varying the template z(i) for a single x from the WikiBio validation data; generations are annotated withthe segmentations of the chosen z(i). Results were obtained using the NTemp model from Table 2.

Interpretable States Discrete states also pro-vide a method for interpreting the generations pro-duced by the system, since each segment is explic-itly typed by the current hidden state of the model.Table 4 shows the impact of varying the templatez(i) for a single x from the WikiBio dataset. Whilethere is in general surprisingly little stylistic varia-tion in the WikiBio data itself, there is variation inthe information discussed, and the templates cap-ture this. Moreover, we see that particular discretestates correspond in a consistent way to particularpieces of information, allowing us to align stateswith particular field types. For instance, birthnames have the same hidden state (132), as donames (117), nationalities (82), birth dates (101),and occupations (20).

To demonstrate empirically that the learnedstates indeed align with field types, we calculatethe average purity of the discrete states learned forboth datasets in Table 5. In particular, for eachdiscrete state for which the majority of its gen-erated words appear in some rj , the purity of astate’s record type alignment is calculated as thepercentage of the state’s words that come fromthe most frequent record type the state represents.This calculation was carried out over training ex-amples that belonged to one of the top 100 mostfrequent templates. Table 5 indicates that discretestates learned on the E2E data are quite pure. Dis-crete states learned on the WikiBio data are lesspure, though still rather impressive given that thereare approximately 1700 record types representedin the WikiBio data, and we limit the number ofstates to 135. Unsurprisingly, adding autoregres-siveness to the model decreases purity on bothdatasets, since the model may rely on the autore-gressive RNN for typing, in addition to the state’sidentity.

NTemp NTemp+AR

E2E 89.2 (17.4) 85.4 (18.6)WikiBio 43.2 (19.7) 39.9 (17.9)

Table 5: Empirical analysis of the average purity of dis-crete states learned on the E2E and WikiBio datasets, for theNTemp and NTemp+AR models. Average purities are givenas percents, and standard deviations follow in parentheses.See the text for full description of this calculation.

8 Conclusion and Future Work

We have developed a neural, template-like gen-eration model based on an HSMM decoder,which can be learned tractably by backpropagat-ing through a dynamic program. The method al-lows us to extract template-like latent objects ina principled way in the form of state sequences,and then generate with them. This approach scalesto large-scale text datasets and is nearly competi-tive with encoder-decoder models. More impor-tantly, this approach allows for controlling thediversity of generation and for producing inter-pretable states during generation. We view thiswork both as the first step towards learning dis-crete latent variable template models for more dif-ficult generation tasks, as well as a different per-spective on learning latent variable text models ingeneral. Future work will examine encouragingthe model to learn maximally different (or mini-mal) templates, which our objective does not ex-plicitly encourage, templates of larger textual phe-nomena, such as paragraphs and documents, andhierarchical templates.

Acknowledgments

SW gratefully acknowledges the support of aSiebel Scholars award. AMR gratefully acknowl-edges the support of NSF CCF-1704834, Intel Re-search, and Amazon AWS Research grants.

ReferencesGabor Angeli, Percy Liang, and Dan Klein. 2010. A

simple domain-independent probabilistic approachto generation. In Proceedings of the 2010 Confer-ence on Empirical Methods in Natural LanguageProcessing, pages 502–512. Association for Com-putational Linguistics.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2015. Neural machine translation by jointlylearning to align and translate. In ICLR.

Satanjeev Banerjee and Alon Lavie. 2005. Meteor: Anautomatic metric for mt evaluation with improvedcorrelation with human judgments. In Proceedingsof the acl workshop on intrinsic and extrinsic evalu-ation measures for machine translation and/or sum-marization, pages 65–72.

Anja Belz. 2008. Automatic generation of weatherforecast texts using comprehensive probabilisticgeneration-space models. Natural Language Engi-neering, 14(04):431–455.

Anja Belz and Ehud Reiter. 2006. Comparing auto-matic and human evaluation of nlg systems. In 11thConference of the European Chapter of the Associa-tion for Computational Linguistics.

Andrew Chisholm, Will Radford, and Ben Hachey.2017. Learning to generate one-sentence biogra-phies from wikidata. CoRR, abs/1702.06235.

KyungHyun Cho, Bart van Merrienboer, Dzmitry Bah-danau, and Yoshua Bengio. 2014. On the propertiesof neural machine translation: Encoder-decoder ap-proaches. Eighth Workshop on Syntax, Semanticsand Structure in Statistical Translation.

Hanjun Dai, Bo Dai, Yan-Ming Zhang, Shuang Li,and Le Song. 2017. Recurrent hidden semi-markovmodel. In International Conference on LearningRepresentations.

Ondrej Dusek and Filip Jurcıcek. 2016. Sequence-to-sequence generation for spoken dialogue via deepsyntax trees and strings. In The 54th Annual Meet-ing of the Association for Computational Linguis-tics, page 45.

Mark JF Gales and Steve J Young. 1993. The theoryof segmental hidden Markov models. University ofCambridge, Department of Engineering.

Albert Gatt and Ehud Reiter. 2009. Simplenlg: A re-alisation engine for practical applications. In Pro-ceedings of the 12th European Workshop on NaturalLanguage Generation, pages 90–93. Association forComputational Linguistics.

Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K.Li. 2016. Incorporating copying mechanism insequence-to-sequence learning. In ACL.

Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati,Bowen Zhou, and Yoshua Bengio. 2016. Pointingthe unknown words. In ACL.

Sepp Hochreiter and Jurgen Schmidhuber. 1997. Longshort-term memory. Neural Comput., 9:1735–1780.

Blake Howald, Ravikumar Kondadadi, and FrankSchilder. 2013. Domain adaptable semantic clus-tering in statistical nlg. In Proceedings of the 10thInternational Conference on Computational Seman-tics (IWCS 2013)–Long Papers, pages 143–154.

Zhiting Hu, Zichao Yang, Xiaodan Liang, RuslanSalakhutdinov, and Eric P Xing. 2017. Toward con-trolled generation of text. In International Confer-ence on Machine Learning, pages 1587–1596.

Po-Sen Huang, Chong Wang, Sitao Huang, DengyongZhou, and Li Deng. 2018. Towards neural phrase-based machine translation. In International Confer-ence on Learning Representations.

Dan Jurafsky and James H Martin. 2014. Speech andlanguage processing. Pearson London.

Ravi Kondadadi, Blake Howald, and Frank Schilder.2013. A statistical nlg framework for aggregatedplanning and realization. In Proceedings of the 51stAnnual Meeting of the Association for Computa-tional Linguistics (Volume 1: Long Papers), vol-ume 1, pages 1406–1415.

Lingpeng Kong, Chris Dyer, and Noah A Smith. 2016.Segmental recurrent neural networks. In Interna-tional Conference on Learning Representations.

Ioannis Konstas and Mirella Lapata. 2013. A globalmodel for concept-to-text generation. J. Artif. Intell.Res.(JAIR), 48:305–346.

Karen Kukich. 1983. Design of a knowledge-based re-port generator. In ACL, pages 145–150.

Remi Lebret, David Grangier, and Michael Auli. 2016.Neural text generation from structured data withapplication to the biography domain. In EMNLP,pages 1203–1213.

J. Li, R. Jia, H. He, and P. Liang. 2018. Delete, retrieve,generate: A simple approach to sentiment and styletransfer. In North American Association for Compu-tational Linguistics (NAACL).

Percy Liang, Michael I Jordan, and Dan Klein. 2009.Learning semantic correspondences with less super-vision. In ACL, pages 91–99. Association for Com-putational Linguistics.

Chin-Yew Lin. 2004. Rouge: A package for auto-matic evaluation of summaries. Text SummarizationBranches Out.

Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang,and Zhifang Sui. 2018. Table-to-text generation bystructure-aware seq2seq learning. In Proceedings ofthe Thirty-Second AAAI Conference on Artificial In-telligence.

Liang Lu, Lingpeng Kong, Chris Dyer, Noah A Smith,and Steve Renals. 2016. Segmental recurrent neuralnetworks for end-to-end speech recognition. Inter-speech 2016, pages 385–389.

Thang Luong, Hieu Pham, and Christopher D. Man-ning. 2015. Effective approaches to attention-basedneural machine translation. In Proceedings of the2015 Conference on Empirical Methods in NaturalLanguage Processing, EMNLP 2015, pages 1412–1421.

Kathleen McKeown. 1992. Text generation - using dis-course strategies and focus constraints to generatenatural language text. Studies in natural languageprocessing. Cambridge University Press.

Susan W McRoy, Songsak Channarukul, and Syed SAli. 2000. Yag: A template-based generator for real-time systems. In Proceedings of the first interna-tional conference on Natural language generation-Volume 14, pages 264–267. Association for Compu-tational Linguistics.

Hongyuan Mei, Mohit Bansal, and Matthew R. Walter.2016. What to talk about and how? selective gener-ation using lstms with coarse-to-fine alignment. InNAACL HLT, pages 720–730.

Kevin P Murphy. 2002. Hidden semi-markov models(hsmms). unpublished notes, 2.

Vinod Nair and Geoffrey E Hinton. 2010. Rectifiedlinear units improve restricted boltzmann machines.In Proceedings of the 27th international conferenceon machine learning (ICML-10), pages 807–814.

Jekaterina Novikova, Ondrej Dusek, and Verena Rieser.2017. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th AnnualMeeting of the Special Interest Group on Discourseand Dialogue, Saarbrucken, Germany.

Mari Ostendorf, Vassilios V Digalakis, and Owen AKimball. 1996. From hmm’s to segment models:A unified view of stochastic modeling for speechrecognition. IEEE Transactions on speech and au-dio processing, 4(5):360–378.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval-uation of machine translation. In Proceedings ofthe 40th annual meeting on association for compu-tational linguistics, pages 311–318. Association forComputational Linguistics.

Adam Paszke, Sam Gross, Soumith Chintala, Gre-gory Chanan, Edward Yang, Zachary DeVito, Zem-ing Lin, Alban Desmaison, Luca Antiga, and AdamLerer. 2017. Automatic differentiation in pytorch.NIPS 2017 Autodiff Workshop.

Lawrence R Rabiner. 1989. A tutorial on hiddenmarkov models and selected applications in speechrecognition. Proceedings of the IEEE, 77(2):257–286.

Ehud Reiter and Robert Dale. 1997. Building appliednatural language generation systems. Natural Lan-guage Engineering, 3(1):57–87.

Tianxiao Shen, Tao Lei, Regina Barzilay, and TommiJaakkola. 2017. Style transfer from non-parallel textby cross-alignment. In Advances in Neural Informa-tion Processing Systems, pages 6833–6844.

Yikang Shen, Zhouhan Lin, Chin wei Huang, andAaron Courville. 2018. Neural language modelingby jointly learning syntax and lexicon. In Interna-tional Conference on Learning Representations.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky,Ilya Sutskever, and Ruslan Salakhutdinov. 2014.Dropout: A simple way to prevent neural networksfrom overfitting. The Journal of Machine LearningResearch, 15(1):1929–1958.

Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014.Sequence to sequence learning with neural net-works. In Advances in Neural Information Process-ing Systems (NIPS), pages 3104–3112.

Hao Tang, Weiran Wang, Kevin Gimpel, and KarenLivescu. 2016. End-to-end training approachesfor discriminative segmental models. In SpokenLanguage Technology Workshop (SLT), 2016 IEEE,pages 496–502. IEEE.

Ke M Tran, Yonatan Bisk, Ashish Vaswani, DanielMarcu, and Kevin Knight. 2016. Unsupervised neu-ral hidden markov models. In Proceedings of theWorkshop on Structured Prediction for NLP, pages63–71.

Ramakrishna Vedantam, C Lawrence Zitnick, and DeviParikh. 2015. Cider: Consensus-based image de-scription evaluation. In Proceedings of the IEEEconference on computer vision and pattern recog-nition, pages 4566–4575.

Chong Wang, Yining Wang, Po-Sen Huang, Abdel-rahman Mohamed, Dengyong Zhou, and Li Deng.2017. Sequence modeling via segmentations. In In-ternational Conference on Machine Learning, pages3674–3683.

Lu Wang and Claire Cardie. 2013. Domain-independent abstract generation for focused meetingsummarization. In Proceedings of the 51st AnnualMeeting of the Association for Computational Lin-guistics (Volume 1: Long Papers), volume 1, pages1395–1405.

Sam Wiseman, Stuart Shieber, and Alexander Rush.2017. Challenges in data-to-document generation.In Proceedings of the 2017 Conference on Empiri-cal Methods in Natural Language Processing, pages2253–2263.

Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, andWilliam W. Cohen. 2018. Breaking the softmaxbottleneck: A high-rank RNN language model. InInternational Conference on Learning Representa-tions.

Zichao Yang, Phil Blunsom, Chris Dyer, and WangLing. 2016. Reference-aware language models.CoRR, abs/1611.01628.

Lei Yu, Jan Buys, and Phil Blunsom. 2016. Online seg-ment to segment neural transduction. In Proceed-ings of the 2016 Conference on Empirical Methodsin Natural Language Processing, pages 1307–1316.

Junbo Jake Zhao, Yoon Kim, Kelly Zhang, Alexan-der M. Rush, and Yann LeCun. 2018. Adversari-ally regularized autoencoders. In Proceedings of the35th International Conference on Machine Learn-ing, ICML 2018, pages 5897–5906.

A Supplemental Material

A.1 Additional Model and Training DetailsComputing rj A record rj is represented byembedding a feature for its type, its position, andits word value in Rd, and applying an MLP withReLU nonlinearity (Nair and Hinton, 2010) toform rj ∈Rd, similar to Yang et al. (2016) andWiseman et al. (2017).

LSTM Details The initial cell and hidden-state values for the decoder LSTM are givenby Q1xa and tanh(Q2xa), respectively, whereQ1,Q2 ∈Rd×d.

When a word yt appears in a record rj , the inputto the LSTM at time t + 1 is computed using anMLP with ReLU nonlinearity over the concatena-tion of the embeddings for rj’s record type, wordvalue, position, and a feature for whether it is thefinal position for the type. If there are multiple rjfrom which yt could have been copied, the com-puted representations are averaged. At test time,we use the MAP rj to compute the input, even ifthere are multiple matches. For yt which could nothave been copied, the input to the LSTM at timet+1 is computed using the same MLP over yt andthree dummy features.

For the autoregressive HSMM, an additional 1-layer LSTM with d hidden units is used. We ex-perimented with having the autoregressive HSMMconsume either tokens y1:t in predicting yt+1, orthe average embedding of the field types corre-sponding to copied tokens in y1:t. The formerworked slightly better for the WikiBio dataset(where field types are more ambiguous), while thelatter worked slightly better for the E2E dataset.

Transition Distribution The functionC(xu), which produces hidden state em-beddings conditional on the source, is de-fined as C(xu)=U2(ReLU(U1xu)), where

U1 ∈Rm3×d and U2 ∈RK×m2×m3 ; D(x) is de-fined analogously. For all experiments, m1=64,m2=32, and m3=64.

Optimization We train with SGD, using a learn-ing rate of 0.5 and decaying by 0.5 each epochafter the first epoch in which validation log-likelihood fails to increase. When using an au-toregressive HSMM, the additional LSTM is op-timized only after the learning rate has been de-cayed. We regularize with Dropout (Srivastavaet al., 2014).

A.2 Additional Learned TemplatesIn Tables 6 and 7 we show visualizations of addi-tional templates learned on the E2E and WikiBiodata, respectively, by both the non-autoregressiveand autoregressive HSMM models presented inthe paper. For each model, we select a set of fivedissimilar templates in an iterative way by greed-ily selecting the next template (out of the 200 mostfrequent) that has the highest percentage of statesthat do not appear in the previously selected tem-plates; ties are broken randomly. Individual stateswithin a template are visualized using the threemost common segments they generate.

Learning Neural Templates for Text Generation · hand-engineered rules (Angeli et...

Documents

Transcript of Learning Neural Templates for Text Generation · hand-engineered rules (Angeli et...