Query-Based Single Document Summarization Using an Ensemble NoisyAuto-Encoder

Mahmood Yousefi Azar, Kairit Sirts, Diego Molla Aliod and Len HameyDepartment of Computing

Macquarie University, Australiamahmood.yousefiazar@students.mq.edu.au,

{kairit.sirts, diego.molla-aliod, len.hamey}@mq.edu.au

Abstract

In this paper we use a deep auto-encoderfor extractive query-based summarization.We experiment with different input repre-sentations in order to overcome the prob-lems stemming from sparse inputs charac-teristic to linguistic data. In particular, wepropose constructing a local vocabulary foreach document and adding a small randomnoise to the input. Also, we propose us-ing inputs with added noise in an Ensem-ble Noisy Auto-Encoder (ENAE) that com-bines the top ranked sentences from mul-tiple runs on the same input with differentadded noise. We test our model on a pub-licly available email dataset that is specifi-cally designed for text summarization. Weshow that although an auto-encoder can bea quite effective summarizer, adding noiseto the input and running a noisy ensemblecan make improvements.

1 Introduction

Recently, deep neural networks have gained pop-ularity in a wide variety of applications, in partic-ular, they have been successfully applied to vari-ous natural language processing (NLP) tasks (Col-lobert et al., 2011; Srivastava and Salakhutdinov,2012). In this paper we apply a deep neural net-work to query-based extractive summarization task.Our model uses a deep auto-encoder (AE) (Hintonand Salakhutdinov, 2006) to learn the latent repre-sentations for both the query and the sentences inthe document and then uses a ranking function tochoose certain number of sentences to compose thesummary.

Typically, automatic text summarization systemsuse sparse input representations such as tf-idf. How-ever, sparse inputs can be problematic in neuralnetwork training and they may make the training

slow. We propose two techniques for reducingsparsity. First, we compose for each document alocal vocabulary which is then used to constructthe input representations for sentences in that doc-ument. Second, we add small random noise to theinputs. This technique is similar to the denoisingauto-encoders (Vincent et al., 2008). However, thedenoising AE adds noise to the inputs only duringtraining, while during test time we also add noiseto input.

An additional advantage of adding noise dur-ing testing is that we can use the same input withdifferent added noise in an ensemble. Typically,an ensemble learner needs to learn several differ-ent models. However, the Ensemble Noisy Auto-Encoder (ENAE) proposed in this paper only needsto train one model and the ensemble is created fromapplying the model to the same input several times,each time with different added noise.

Text summarization can play an important role indifferent application domains. For instance, whenperforming a search in the mailbox according to akeyword, the user could be shown short summariesof the relevant emails. This is especially attractivewhen using a smart phone with a small screen. Wealso evaluate our model on a publicly availableemail dataset (Loza et al., 2014). In addition tosummaries, this corpus has also been annotatedwith keyword phrases. In our experiments we useboth the email subjects and annotated keywords asqueries.

The contributions of the paper are the following:

1. We introduce an unsupervised approach forextractive summarization using AEs. Al-though AEs have been previously applied tosummarization task as a word filter (Liu et al.,2012), to the best of our knowledge we are thefirst to use the representations learned by theAE directly for sentence ranking.

2. We add small Gaussian noise to the sparseinput representations both during training and

Mahmood Yousefi Azar, Kairit Sirts, Len Hamey and Diego Molla Aliod. 2015. Query-Based Single DocumentSummarization Using an Ensemble Noisy Auto-Encoder . In Proceedings of Australasian Language TechnologyAssociation Workshop, pages 2−10.

testing. To the best of our knowledge, nois-ing the inputs during test time is novel in theapplication of AEs.

3. We introduce the Ensemble Noisy Auto-Encoder (ENAE) in which the model is trainedonce and used multiple times on the same in-put, each time with different added noise.

Our experiments show that although a deep AEcan be a quite effective summarizer, adding stochas-tic noise to the input and running an ensemble onthe same input with different added noise can makeimprovements.

We start by giving the background in section 2.The method is explained in section 3. Section 4describes the input representations. The Ensem-ble Noisy Auto-Encoder is introduced in section 5.The experimental setup is detailed in section 6. Sec-tion 7 discusses the results and the last section 8concludes the paper.

2 Background

Automatic summarization can be categorized intotwo distinct classes: abstractive and extractive. Anabstractive summarizer re-generates the extractedcontent (Radev and McKeown, 1998; Harabagiuand Lacatusu, 2002; Liu et al., 2015). Extractivesummarizer, on the other hand, chooses sentencesfrom the original text to be included in the summaryusing a suitable ranking function (Luhn, 1958; De-nil et al., 2014b). Extractive summarization hasbeen more popular due to its relative simplicitycompared to the abstractive summarization and thisis also the approach taken in this paper.

Both extractive and abstractive summarizers canbe designed to perform query-based summariza-tion. A query-based summarizer aims to retrieveand summarize a document or a set of documentssatisfying a request for information expressed bya user’s query (Daume III and Marcu, 2006; Tanget al., 2009; Zhong et al., 2015), which greatly fa-cilitates obtaining the required information fromlarge volumes of structured and unstructured data.Indeed, this is the task that the most popular searchengines (e.g. Google) are performing when theypresent the search results, including snippets of textthat are related to the query.

There has been some previous work on usingdeep neural networks for automatic text summa-rization. The most similar to our work is the modelby Zhong et al. (2015) that also uses a deep AEfor extractive summarization. However, they use

the learned representations for filtering out relevantwords for each document which are then used toconstruct a ranking function over sentences, whilewe use the learned representations directly in theranking function. Denil et al. (2014a) propose asupervised model based on a convolutional neuralnetwork to extract relevant sentences from docu-ments. Cao et al. (2015) use a recursive neuralnetwork for text summarization. However, alsotheir model is supervised and uses hand-craftedword features as inputs while we use an AE forunsupervised learning.

The method of adding noise to the input pro-posed in this paper is very similar to the denoisingauto-encoders (Vincent et al., 2008). In a denoisingAE, the input is corrupted and the network triesto undo the effect of the corruption. The intuitionis that this rectification can occur if the networklearns to capture the dependencies between the in-puts. The algorithm adds small noise to the inputbut the reconstructed output is still the same asuncorrupted input, while our model attempts toreconstruct the noisy input. While denoising AEonly uses noisy inputs only in the training phase,we use the input representations with added noiseboth during training and also later when we use thetrained model as a summarizer.

Previously, also re-sampling based methods havebeen proposed to solve the problem of sparsity forAE (Genest et al., 2011).

3 The Method Description

An AE (Figure 1) is a feed-forward network thatlearns to reconstruct the input x. It first encodes theinput x by using a set of recognition weights intoa latent feature representation C(x) and then de-codes this representation back into an approximateinput x using a set of generative weights.

While most neural-network-based summariza-tion methods are supervised, using an AE providesan unsupervised learning scheme. The general pro-cedure goes as follows:

1. Train the AE on all sentences and queries inthe corpus.

2. Use the trained AE to get the latent represen-tations (a.k.a codes or features) for each queryand each sentence in the document;

3. Rank the sentences using their latent represen-tations to choose the query-related sentencesto be included into the summary.

3

Figure 1: The structure of an AE for dimension-ality reduction. x and x denote the input and re-constructed inputs respectively. hi are the hiddenlayers and wi are the weights. Features/codes C(x)are in this scheme the output of the hidden layerh4.

The AE is trained in two phases: pre-trainingand fine-tuning. Pre-training performs a greedylayer-wise unsupervised learning. The obtainedweights are then used as initial weights in the fine-tuning phase, which will train all the network layerstogether using back-propagation. The next subsec-tions will describe all procedures in more detail.

3.1 Pre-training Phase

In the pre-training phase, we used restricted Boltz-mann machine (RBM) (Hinton et al., 2006). AnRBM (Figure 2) is an undirected graphical modelwith two layers where the units in one layer areobserved and in the other layer are hidden. It hassymmetric weighted connections between hiddenand visible units and no connections between theunits of the same layer. In our model, the firstlayer RBM between the input and the first hiddenrepresentation is Gaussian-Bernoulli and the otherRBMs are Bernoulli-Bernoulli.

The energy function of a Bernoulli-BernoulliRBM, i.e. where both observed and hidden unitsare binary, is bilinear (Hopfield, 1982):

Figure 2: The structure of the restricted Boltzmannmachine (RBM) as an undirected graphical model:x denotes the visible nodes and h are the hiddennodes.

E(x,h; θ) =−∑i∈V

bixi −∑j∈H

ajhj

−∑i,j

xihjwij ,(1)

where V and H are the sets of visible and hiddenunits respectively, x and h are the input and hid-den configurations respectively, wij is the weightbetween the visible unit xi and the hidden unit hj ,and bi and aj are their biases. θ = {W,a,b}denotes the set of all network parameters.

The joint distribution over both the observed andthe hidden units has the following equation:

p(x,h; θ) =exp (−E(x,h; θ))

Z, (2)

where Z =∑

x′,h′ exp(−E(x′,h′; θ)

)is the par-

tition function that normalizes the distribution.The marginal probability of a visible vector is:

p(x; θ) =

∑h exp (−E(x,h; θ))

Z(3)

The conditional probabilities for a Bernoulli-Bernoulli RBM are:

p(hj = 1|x; θ) =exp (

∑iwijxi + aj)

1 + exp (∑

iwijxi + aj)

= sigm(∑i

wijxi + aj)(4)

p(xi = 1|h; θ) =exp (

∑j wijhj + bi)

1 + exp (∑

j wijhj + bi)

= sigm(∑j

wijhj + bi)(5)

4

When the visible units have real values and thehidden units are binary, e.g. the RBM is Gaussian-Bernoulli, the energy function becomes:

E(x,h; θ) =∑i∈V

(xi − bi)2

2σ2i−∑j∈H

ajhj

−∑i,j

xiσihjwij ,

(6)

where σi is the standard deviation of the ith visibleunit. With unit-variance the conditional probabili-ties are:

p(hj = 1|x; θ) =exp (

∑iwijxi + aj)

1 + exp (∑

iwijxi + aj)

= sigm(∑i

wijxi + aj)(7)

p(xi|h; θ) =1√2π

exp

(−

(x− bi −∑

j wijhj)2

2

)

= N

∑j

wijhj + bi, 1

(8)

To estimate the parameters of the network, max-imum likelihood estimation (equivalent to mini-mizing the negative log-likelihood) can be applied.Taking the derivative of the negative log-probabilityof the inputs with respect to the weights leads toa learning algorithm where the update rule for theweights of a RBM is given by:

∆wij = ε(〈xi, hj〉data − 〈xi, hj〉model), (9)

where ε is the learning rate, angle brackets denotethe expectations and 〈xi, hj〉data is the so-calledpositive phase contribution and 〈xi, hj〉model is theso-called negative phase contribution. In particular,the positive phase is trying to decrease the energyof the observation and the negative phase increasesthe energy defined by the model. We use k-stepcontrastive divergence (Hinton, 2002) to approx-imate the expectation defined by the model. Weonly run one step of the Gibbs sampler, which pro-vides low computational complexity and is enoughto get a good approximation.

The RBM blocks can be stacked to form thetopology of the desired AE. During pre-training

Figure 3: Several generative RBM models stackedon top of each other.

the AE is trained greedily layer-wise using individ-ual RBMs, where the output of one trained RBMis used as input for the next upper layer RBM (Fig-ure 3).

3.2 Fine-tuning PhaseIn this phase, the weights obtained from the pre-training are used to initialise the deep AE. For thatpurpose, the individual RBMs are stacked on topof each other and unrolled, i.e. the recognition andgeneration weights are tied.

Ngiam et al. (2011) evaluated different types ofoptimization algorithm included stochastic gradientdescent (SGD) and Conjugate gradient (CG). It hasbeen observed that mini-batch CG with line searchcan simplify and speed up different types of AEscompared to SGD. In this phase, the weights of theentire network are fine-tuned with CG algorithmusing back-propagation. The cost function to beminimised is the cross-entropy error between thegiven and reconstructed inputs.

3.3 Sentence RankingExtractive text summarization is also known assentence ranking. Once the AE model has beentrained, it can be used to extract the latent represen-tations for each sentence in each document and foreach query. We assume that the AE will place thesentences with similar semantic meaning close toeach other in the latent space and thus, we can usethose representations to rank the sentences accord-

5

Figure 4: The Ensemble Noisy Auto-Encoder.

ing to their relevance to the query. We use cosinesimilarity to create the ranking ordering betweensentences.

4 Input Representations

The most common input representation used ininformations retrieval and text summarization sys-tems is tf-idf (Wu et al., 2008), which representseach word in the document using its term frequencytf in the document, as well as over all documents(idf ). In the context of text summarization the tf-idfrepresentations are constructed for each sentence.This means that the input vectors are very sparse be-cause each sentence only contains a small numberof words.

To address the sparsity, we propose computingthe tf representations using local vocabularies. Weconstruct the vocabulary for each document sepa-rately from the most frequent terms occurring inthat document. We use the same number of wordsin the vocabulary for each document.

This local representation is less sparse comparedto the tf-idf because the dimensions in the inputnow correspond to words that all occur in the cur-rent document. Due to the local vocabularies theAE input dimensions now correspond to differentwords in different documents. As a consequence,the AE positions the sentences of different docu-ments into different semantic subspaces. However,this behaviour causes no adverse effects becauseour system extracts each summary based on a sin-gle document only.

In order to reduce the sparsity even more, weadd small Gaussian noise to the input. The idea isthat when the noise is small, the information in thenoisy inputs is essentially the same as in the inputvectors without noise.

5 Ensemble Noisy Auto-Encoder

After ranking, a number of sentences must be se-lected to be included into the summary. A straight-forward selection strategy adopted in most extrac-tive summarization systems is just to use the topranked sentences. However, we propose a morecomplex selection strategy that exploits the noisyinput representations introduced in the previous sec-tion. By adding random noise to the input we canrepeat the experiment several times using the sameinput but with different noise. Each of those ex-periments potentially produces a slightly differentranking, which can be aggregated into an ensemble.

In particular, after running the sentence rankingprocedure multiple times, each time with differ-ent noise in the input, we use a voting scheme foraggregating the ranks. In this way we obtain thefinal ranking which is then used for the sentenceselection. The voting scheme counts, how manytimes each sentence appears in all different rank-ings in the n top positions, where n is a predefinedparameter. Currently, we use the simple countingand do not take into account the exact position ofthe sentence in each of the top rankings. Based onthose counts we produce another ranking over onlythose sentences that appeared in the top rankingsof the ensemble runs. Finally, we just select the topsentences according to the final ranking to producethe summary.

A detailed schematic of the full model is pre-sented in Figure 4. The main difference betweenthe proposed approach and the commonly usedensemble methods lies in the number of trainedmodels. Whereas during ensemble learning severaldifferent models are trained, our proposed approachonly needs to train a single model and the ensembleis created by applying it to a single input repeatedly,each time perturbing it with different noise.

6

6 Experimental Setup

We perform experiments on a general-purpose sum-marization and keyword extraction dataset (SKE)(Loza et al., 2014) that has been annotated withboth extractive and abstractive summaries, and ad-ditionally also with keyword phrases. It consistsof 349 emails from which 319 have been selectedfrom the Enron email corpus and 30 emails wereprovided by the volunteers. The corpus containsboth single emails and email threads that all havebeen manually annotated by two different annota-tors.

We conduct two different experiments on theSKE corpus. First, we generate summaries basedon the subject of each email. As some emails in thecorpus have empty subjects we could perform thisexperiment only on the subset of 289 emails thathave non-empty subjects. Secondly, we generatesummaries using the annotated keyword phrasesas queries. As all emails in the corpus have beenannotated with keyword phrases, this experimentwas performed on the whole dataset. The annotatedextractive summaries contain 5 sentences and thuswe also generate 5 sentence summaries.

ROUGE (Lin, 2004) is the fully automatic metriccommonly used to evaluate the text summarizationresults. In particular, ROUGE-2 recall has beenshown to correlate most highly with human evalua-tor judgements (Dang and Owczarzak, 2008). Weused 10-fold cross-validation, set the confidenceinterval to 95% and used the jackknifing procedurefor multi-annotation evaluation (Lin, 2004).

Our deep AE implementation is based on G. Hin-ton’s software, which is publicly available.1 Weused mini-batch gradient descent learning in bothpre-training and fine-tuning phases. The batch sizewas 100 data items during pre-training and 1000data items during fine-tuning phase. During pre-training we trained a 140-40-30-10 network withRBMs and in fine-tuning phase we trained a 140-40-30-10-30-40-140 network as the AE. Here, 140is the size of the first hidden layer and 10 is the sizeof the sentence representation layer, which is usedin the ranking function.

As a pre-processing step, we stem the docu-ments with the Porter stemmer and remove thestop words.2

1http://www.cs.toronto.edu/˜hinton/MatlabForSciencePaper.html

2Stop word list obtained from http://xpo6.com/list-of-english-stop-words

Model Subject Phrases

tf-idf V=1000 0.2312 0.4845tf-idf V=5% 0.1838 0.4217tf-idf V=2% 0.1435 0.3166tf-idf V=60 0.1068 0.2224

AE (tf-idf V=2%) 0.3580 0.4795AE (tf-idf V=60) 0.3913 0.4220

L-AE 0.4948 0.5657L-NAE 0.4664 0.5179L-ENAE 0.5031 0.5370

Table 1: ROUGE-2 recall for both subject-orientedand key-phrase-oriented summarization. The uppersection of the table shows tf-idf baselines with var-ious vocabulary sizes. The middle section showsAE with tf-idf as input representations. The bottomsection shows the AE with input representationsconstructed using local vocabularies (L-AE), L-AEwith noisy inputs (L-NAE) and the Ensemble NoisyAE (L-ENAE).

We use tf-idf 3 as the baseline. After preprocess-ing, the SKE corpus contains 6423 unique termsand we constructed tf-idf vectors based on the 1000,320 (5% of the whole vocabulary), 128 (2% of thewhole vocabulary), and 60 most frequently occur-ring terms. V = 60 is the size of the tf representa-tion used in our AE model. 4

We apply the AE model to several different inputrepresentations: tf-idf, tf constructed using localvocabularies as explained in Section 4 (L-AE), tfusing local vocabularies with added Gaussian noise(L-NAE) and in the noisy ensemble (L-ENAE).

7 Results and Discussion

Table 1 shows the ROUGE-2 scores of the tf-idfbaselines and the AE model with various inputrepresentations. The columns show the scores ofthe summaries generated using the subjects andkeyword phrases as queries respectively.

The main thing to note is that AE performs inmost cases much better than the tf-idf baseline, es-pecially when using subjects as queries. The onlyscenario where the tf-idf can compete with the AE

3We use the inverse frequency as the idf term.4We did not train the AE-s with larger vocabularies be-

cause this would have required changing the network structureas during the preliminary experiments we noticed that thenetwork did not improve much when using inputs larger thanthe first hidden layer.

7

is with the vocabulary of size 1000 and when usingkeyword phrases as queries. This is because themanually annotated keyword phrases do in manycases contain the same words as extracted summarysentences, especially because the queries and sum-maries where annotated by the same annotators.However, when the vocabulary size is the same asused in the AE, tf-idf scores are much lower in bothexperimental settings.

The second thing to notice is that although AEwith tf-idf input performs better than plain tf-idf,it is still quite a bit worse than AE using tf repre-sentations derived from the local vocabularies. Webelieve it is so because the deep AE can extract ad-ditional information from the tf-idf representations,but the AE learning is more effective when usingless sparse inputs, provided by the tf representa-tions constructed from local vocabularies.

Although we hoped that the reduced sparsitystemming from the added noise will improve theresults even more, the experiments show that this isnot the case—AE without noise performs in bothsettings better than the noisy AE. However, whencombining the rankings of the noisy ensemble, theresults are much better than a single noisy AE andmay even improve over the simple AE. This is thecase when extracting summaries based on the sub-ject. The subjects of the emails are less informativethan the annotated keyword phrases. Perhaps thisexplains why the ENAE was able to make use ofthe noisy inputs to gain small improvements overthe AE without any noise in this scenario.

There is considerable difference between the re-sults when using the email subjects or keywordphrases as queries with keyword phrases leading tobetter summaries. This is to be expected becausethe keyword phrases have been carefully extractedby the annotators. The keyword phrases give thehighest positive contribution to the td-idf baselineswith largest vocabularies, which clearly benefitsfrom the fact that the annotated sentences containthe extracted keyword phrases. The ENAE showsthe smallest difference between the subject-basedand keyword-based summaries. We believe it isbecause the ENAE is able to make better use of thewhole latent semantic space of the document to ex-tract the relevant sentences to the query, regardlessof whether the query contains the exact relevantterms or not.

Figure 5 illustrates the ROUGE-2 recall of thebest baseline and the AE models with both tf-idf

Figure 5: ROUGE-2 recall for summaries contain-ing different number of sentences using the key-word phrases as queries.

and tf input representations using keyword phrasesas queries and varying the length of the generatedsummaries. In this experiment, each summary wasevaluated against the annotated summary of thesame length. As is expected, the results improvewhen the length of the summary increases. Whilethe AE model’s results improve almost linearlyover the 5 sentences, tf-idf gains less from increas-ing the summary length from 4 to 5 sentences. Thescores are almost the same for the tf-idf and theAE with tf representation with 1 and 2 sentences.Starting from 3 sentences, the AE performs clearlybetter.

To get a better feel what kind of summaries theENAE system is generating we present the resultsof a sample email thread (ECT020). This typicalemail thread contains 4 emails and 13 lines. Thesummaries extracted by the ENAE system usingboth subjects and keyword phrases are given inFigure 6. The annotated summaries consist of sen-tences [03, 04, 10, 11, 05] and [03, 04, 11, 06, 05]for the first and the second annotator respectively.

Both generated summaries contain the sentences03, 04 and 06. These were also the sentences cho-sen by the annotators (03 and 04 by the first anno-tation and all three of them by the second). Thesentence 11 present in the subject-based summarywas also chosen by both annotators, while sentence10 in keyword-based summary was also annotatedby the first annotator. The only sentences that werenot chosen by the annotators are 08 in the subject-based summary and 12 in the keyword-based sum-mary. Both annotators had also chosen sentence 05,which is not present in the automatically generatedsummaries. However, this is the sentence that bothannotators gave the last priority in their rankings.

In general the order of the sentences generatedby the system and chosen by the annotators is the

8

a) ENAE summary based on subject03 Diamond-san, As I wrote in the past, Nissho

Iwai’s LNG related department has beentransferred into a new joint venture companybetween Nissho and Sumitomo Corp. as ofOctober 1, 2001, namely, ”LNG Japan Corp.”.

08 We are internally discussing when we start ourofficial meeting.

04 In this connection, we would like to concludeanother NDA with LNG Japan Corp, as perattached.

06 Also, please advise us how we should treatNissho’s NDA in these circumstances.

11 They need to change the counterparty name dueto a joint venture.

b) ENAE summary based on keyword phrases10 Please approve or make changes to their new

NDA.03 Diamond-san, As I wrote in the past, Nissho

Iwai’s LNG related department has beentransferred into a new joint venture companybetween Nissho and Sumitomo Corp. as ofOctober 1, 2001, namely, ”LNG Japan Corp.”.

04 In this connection, we would like to concludeanother NDA with LNG Japan Corp, as perattached.

12 I wanted to let you know this was coming in assoon as Mark approves the changes.

06 Also, please advise us how we should treatNissho’s NDA in these circumstances.

Figure 6: Examples of subject-based (left) and keyword-based (right) summaries extrated by the EnsembleNoisy AE.

a) First annotatorLNG Japan Corp. is a new joint venture betweenNissho and Sumitomo Corp. Given this situationa new NDA is needed and sent for signature toDaniel Diamond. Daniel forward the NDA toMark for revision.

b) Second annotatorAn Enron employee is informed by an employeeof Nissho Iwai that the Nissho Iwai’s LNG relateddepartment has been transferred into a new jointventure company, namely, ’LNG Japan Corp.’. Asa result, there is a need to change the counterpartyname in the new NDA. The new change has to beapproved and then applied to the new NDA withLNG Japan Corporation

Figure 7: The abstractive summaries created by theannotators for the example email.

same in both example summaries. The only excep-tion is sentence 10, which is ranked as top in thesummary generated based on the keyword phrasesbut chosen as third after the sentences 03 and 04by the first annotator.

Looking at the annotated abstractive summarieswe found that the sentence 12 chosen by thekeyword-based summarizer is not a fault extrac-tion. Although neither of the annotators chose thissentence for the extractive summary, the informa-tion conveyed in this sentence can be found in both

annotated abstractive summaries (Figure 7).

8 Conclusion

In this paper we used a deep auto-encoder (AE) forquery-based extractive summarization. We testedour method on a publicly available email datasetand showed that the auto-encoder-based modelsperform much better than the tf-idf baseline. Weproposed using local vocabularies to construct in-put representations and showed that this improvesover the commonly used tf-idf, even when the lat-ter is used as input to an AE. We proposed addingsmall stochastic noise to the input representationsto reduce sparsity and showed that constructing anensemble by running the AE on the same inputmultiple times, each time with different noise, canimprove the results over the deterministic AE.

In future, we plan to compare the proposed sys-tem with the denoising auto-encoder, as well asexperiment with different network structures andvocabulary sizes. Also, we intend to test our En-semble Noisy Auto-Encoder on various differentdatasets to explore the accuracy and stability of themethod more thoroughly.

References

Ziqiang Cao, Furu Wei, Li Dong, Sujian Li, and MingZhou. 2015. Ranking with recursive neural net-works and its application to multi-document summa-

9

rization. In Twenty-Ninth AAAI Conference on Arti-ficial Intelligence.

Ronan Collobert, Jason Weston, Lon Bottou, MichaelKarlen, Koray Kavukcuoglu, and Pavel Kuksa.2011. Natural language processing (almost) fromscratch. The Journal of Machine Learning Research,12:2493–2537.

Hoa Trang Dang and Karolina Owczarzak. 2008.Overview of the tac 2008 update summarization task.In Proceedings of text analysis conference, pages 1–16.

Hal Daume III and Daniel Marcu. 2006. Bayesianquery-focused summarization. In Proceedings ofACL06, pages 305–312. Association for Computa-tional Linguistics.

Misha Denil, Alban Demiraj, and Nando de Freitas.2014a. Extraction of salient sentences from labelleddocuments. arXiv preprint arXiv:1412.6815.

Misha Denil, Alban Demiraj, Nal Kalchbrenner, PhilBlunsom, and Nando de Freitas. 2014b. Modelling,visualising and summarising documents with a sin-gle convolutional neural network. arXiv preprintarXiv:1406.3830.

Pierre-Etienne Genest, Fabrizio Gotti, and Yoshua Ben-gio. 2011. Deep learning for automatic summaryscoring. In Proceedings of the Workshop on Auto-matic Text Summarization, pages 17–28.

Sanda M Harabagiu and Finley Lacatusu. 2002. Gen-erating single and multi-document summaries withgistexter. In Document Understanding Conferences.

Geoffrey E Hinton and Ruslan R Salakhutdinov. 2006.Reducing the dimensionality of data with neural net-works. Science, 313(5786):504–507.

Geoffrey E Hinton, Simon Osindero, and Yee-WhyeTeh. 2006. A fast learning algorithm for deep be-lief nets. Neural computation, 18(7):1527–1554.

Geoffrey E Hinton. 2002. Training products of expertsby minimizing contrastive divergence. Neural com-putation, 14(8):1771–1800.

John J Hopfield. 1982. Neural networks and physi-cal systems with emergent collective computationalabilities. Proceedings of the national academy ofsciences, 79(8):2554–2558.

Chin-Yew Lin. 2004. Rouge: A package for auto-matic evaluation of summaries. In Text summariza-tion branches out: Proceedings of the ACL-04 work-shop, volume 8.

Yan Liu, Sheng-hua Zhong, and Wenjie Li. 2012.Query-oriented multi-document summarization viaunsupervised deep learning. In Twenty-Sixth AAAIConference on Artificial Intelligence.

Fei Liu, Jeffrey Flanigan, Sam Thomson, NormanSadeh, and Noah A Smith. 2015. Toward abstrac-tive summarization using semantic representations.

Vanessa Loza, Shibamouli Lahiri, Rada Mihalcea, andPo-Hsiang Lai. 2014. Building a dataset for summa-rization and keyword extraction from emails. Pro-ceedings of the Ninth International Conference onLanguage Resources and Evaluation (LREC’14).

H. P. Luhn. 1958. The automatic creation of literatureabstracts. IBM Journal of Research Development,2(2):159.

Jiquan Ngiam, Adam Coates, Ahbik Lahiri, BobbyProchnow, Quoc V Le, and Andrew Y Ng. 2011.On optimization methods for deep learning. In Pro-ceedings of the 28th International Conference onMachine Learning (ICML-11), pages 265–272.

Dragomir R Radev and Kathleen R McKeown. 1998.Generating natural language summaries from mul-tiple on-line sources. Computational Linguistics,24(3):470–500.

Nitish Srivastava and Ruslan R Salakhutdinov. 2012.Multimodal learning with deep boltzmann machines.In Advances in neural information processing sys-tems, pages 2222–2230.

Jie Tang, Limin Yao, and Dewei Chen. 2009. Multi-topic based query-oriented summarization. In SDM,volume 9, pages 1147–1158. SIAM.

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, andPierre-Antoine Manzagol. 2008. Extracting andcomposing robust features with denoising autoen-coders. In Proceedings of the 25th internationalconference on Machine learning, pages 1096–1103.ACM.

Ho Chung Wu, Robert Wing Pong Luk, Kam Fai Wong,and Kui Lam Kwok. 2008. Interpreting tf-idf termweights as making relevance decisions. ACM Trans-actions on Information Systems (TOIS), 26(3):13.

Sheng-hua Zhong, Yan Liu, Bin Li, and Jing Long.2015. Query-oriented unsupervised multi-documentsummarization via deep learning model. Expert Sys-tems with Applications, 42(21):8146–8155.

10