CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann,...

SUMMARIZATION

CS276BWeb Search and Mining

Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M.

Venkataramani, R. Altman, L. Hirschman, and D. Radev)

Automated Text summarization Tutorial — COLING/ACL’98

3

an exciting challenge...

...put a book on the scanner, turn the dial to ‘2 pages’, and read the result...

...download 1000 documents from the web, send them to the summarizer, and select the best ones by reading the summaries of the clusters...

...forward the Japanese email to the summarizer, select ‘1 par’, and skim the translated summary.

4

Headline news — informing

5

TV-GUIDES — decision making

6

Abstracts of papers — time saving

7

Graphical maps — orienting

8

Textual Directions — planning

9

Cliff notes — Laziness support

10

Real systems — Money making

11

Questions

What kinds of summaries do people want? What are summarizing, abstracting,

gisting,...?

How sophisticated must summ. systems be? Are statistical techniques sufficient? Or do we need symbolic techniques and

deep understanding as well?

What milestones would mark quantum leaps in summarization theory and practice? How do we measure summarization

quality?

What is a Summary?

Informative summary Purpose: replace original document Example: executive summary

Indicative summary Purpose: support decision: do I want to

read original document yes/no? Example: Headline, scientific abstract

Summarization

• Information overload problem• Increasing need for IR and automated

text summarization systems• Summarization: Process of distilling the

most salient information from a source/sources for a particular user and task

Summary : “a reductive transformation of source text to summary text through content condensation by selection and/or generalization on what is important in the source.” (Sparck Jones,1999)

Steps for Summarization

Transform text into an internal representation. Detect important text units. Generate summary

In extracts ▪ no generation▪ information ordering, ▪ anaphora resolution (or avoiding anaphoric structures)

In abstracts ▪ text generation. ▪ Sentence fusion, ▪ paraphrasing, ▪ Natural Language Generation.

16

Summarizing factors Input (Sparck Jones

2007) subject type: domain genre: newspaper articles, editorials, letters, reports... form: regular text structure; free-form source size: single doc; multiple docs (few; many)

Purpose situation: embedded in larger system (MT, IR) or not? audience: focused or general usage: IR, sorting, skimming...

Output completeness: include all aspects, or focus on some? format: paragraph, table, etc. style: informative, indicative, aggregative, critical...

17

ExamplesExercise: summarize the following

texts for the following readers:

text1: Coup Attempt

text2: childrens’ story

reader1: your friend, who knows nothing about South Africa.

reader2: someone who lives in South Africa and knows the political position.

reader3: your 4-year-old niece.reader4: the Library of Congress.

Summarization Techniques

Surface level: Shallow features Term frequency statistics, position in text,

presence of text from the title, cue words/phrases: e.g. “in summary”, “important”

Entity level: Model text entities and their relationship Vocabulary overlap, distance between text

units, co-occurence, syntactic structure, coreference

Discourse level: Model global structure of text Document outlines, narrative stucture

Hybrid

19

‘Genres’ of Summary? Indicative vs. informative

...used for quick categorization vs. content processing.

Extract vs. abstract...lists fragments of text vs. re-phrases content

coherently.

Generic vs. query-oriented...provides author’s view vs. reflects user’s interest.

Background vs. just-the-news...assumes reader’s prior knowledge is poor vs. up-

to-date.

Single-document vs. multi-document source...based on one text vs. fuses together many texts.

20

A Summarization Machine

EXTRACTS

ABSTRACTS

?

MULTIDOCS

Extract Abstract

Indicative

Generic

Background

Query-oriented

Just the news

10%

50%

100%

Very BriefBrief

Long

Headline

Informative

DOC QUERY

CASE FRAMESTEMPLATESCORE CONCEPTSCORE EVENTSRELATIONSHIPSCLAUSE FRAGMENTSINDEX TERMS

21

The Modules of the Summarization Machine

EXTRACTION

INTERPRETATION

EXTRACTS

ABSTRACTS

?

CASE FRAMESTEMPLATESCORE CONCEPTSCORE EVENTSRELATIONSHIPSCLAUSE FRAGMENTSINDEX TERMS

MULTIDOC

EXTRACTS

GENERATION

FILTERING

DOCEXTRACTS

Why Automatic Summarization? Algorithm for reading in many

domains is:1) read summary2)decide whether relevant or not3) if relevant: read whole document

Summary is gate-keeper for large number of documents.

Information overload Often the summary is all that is read.

Human-generated summaries are expensive.

Summary Length (Reuters)

Goldstein et al. 1999

Summarization Algorithms Keyword summaries

Display most significant keywords Easy to do Hard to read, poor representation of content

Sentence extraction Extract key sentences Medium hard Summaries often don’t read well Good representation of content

Natural language understanding / generation Build knowledge representation of text Generate sentences summarizing content Hard to do well

Something between the last two methods?

Sentence Extraction

Represent each sentence as a feature vector

Compute score based on features Select n highest-ranking sentences Present in order in which they occur in text. Postprocessing to make summary more

readable/concise Eliminate redundant sentences Anaphors/pronouns

▪ A woman walks. She smokes. Delete subordinate clauses, parentheticals

Sentence Extraction: Example

Sigir95 paper on summarization by Kupiec, Pedersen, Chen

Trainable sentence extraction

Proposed algorithm is applied to its own description (the paper)

Sentence Extraction: Example

Feature Representation

Fixed-phrase feature Certain phrases indicate summary, e.g. “in summary”

Paragraph feature Paragraph initial/final more likely to be important.

Thematic word feature Repetition is an indicator of importance

Uppercase word feature Uppercase often indicates named entities. (Taylor)

Sentence length cut-off Summary sentence should be > 5 words.

Feature Representation (cont.) Sentence length cut-off

Summary sentences have a minimum length. Fixed-phrase feature

True for sentences with indicator phrase▪ “in summary”, “in conclusion” etc.

Paragraph feature Paragraph initial/medial/final

Thematic word feature Do any of the most frequent content words

occur? Uppercase word feature

Is uppercase thematic word introduced?

Training

Hand-label sentences in training set (good/bad summary sentences)

Train classifier to distinguish good/bad summary sentences

Model used: Naïve Bayes

Can rank sentences according to score and show top n to user.

Evaluation

Compare extracted sentences with sentences in abstracts

Evaluation of features

Baseline (choose first n sentences): 24% Overall performance (42-44%) not very good. However, there is more than one good summary.

Multi-Document (MD) Summarization

Summarize more than one document Harder but benefit is large (can’t scan 100s

of docs) To do well, need to adopt more specific

strategy depending on document set. Other components needed for a production

system, e.g., manual post-editing. DUC: government sponsored bake-off

200 or 400 word summaries Longer → easier

Types of MD Summaries

Single event/person tracked over a long time period Elizabeth Taylor’s bout with pneumonia Give extra weight to character/event May need to include outcome (dates!)

Multiple events of a similar nature Marathon runners and races More broad brush, ignore dates

An issue with related events Gun control Identify key concepts and select sentences

accordingly

Determine MD Summary Type

First, determine which type of summary to generate

Compute all pairwise similarities Very dissimilar articles → multi-event

(marathon) Mostly similar articles

Is most frequent concept named entity? Yes → single event/person (Taylor) No → issue with related events (gun

control)

MultiGen Architecture (Columbia)

Generation

Ordering according to date Intersection

Find concepts that occur repeatedly in a time chunk

Sentence generator

Processing

Selection of good summary sentences

Elimination of redundant sentences Replace anaphors/pronouns with

noun phrases they refer to Need coreference resolution

Delete non-central parts of sentences

Newsblaster (Columbia)

Query-Specific Summarization

So far, we’ve look at generic summaries.

A generic summary makes no assumption about the reader’s interests.

Query-specific summaries are specialized for a single information need, the query.

Summarization is much easier if we have a description of what the user wants.

Genre

Some genres are easy to summarize Newswire stories Inverted pyramid structure The first n sentences are often the best

summary of length n Some genres are hard to summarize

Long documents (novels, the bible) Scientific articles?

Trainable summarizers are genre-specific.

Discussion

Correct parsing of document format is critical. Need to know headings, sequence, etc.

Limits of current technology Some good summaries require natural

language understanding Example: President Bush’s nominees for

ambassadorships▪ Contributors to Bush’s campaign▪ Veteran diplomats▪ Others

44

.Summarization techniques

45

Computational Approach

Top-Down: I know what I want!

User needs: only certain types of info

System needs: particular criteria of interest, used to focus search

Bottom-Up: I’m dead curious:

what’s in the text?

User needs: anything that’s important

System needs: generic importance metrics, used to rate content

46

Review of Methods

Text location: title, position

Cue phrases Word frequencies Internal text cohesion:

word co-occurrences local salience co-reference of names,

objects lexical similarity semantic rep/graph

centrality Discourse structure

centrality

Information extraction templates

Query-driven extraction: query expansion lists co-reference with query

names lexical similarity to

query

Bottom-up methods Top-down methods

47

Query-Driven vs. Text-Driven Focus Top-down: Query-driven focus

Criteria of interest encoded as search specs.

System uses specs to filter or analyze text portions.

Examples: templates with slots with semantic characteristics; termlists of important terms.

Bottom-up: Text-driven focus Generic importance metrics encoded as

strategies. System applies strategies over rep of whole

text. Examples: degree of connectedness in

semantic graphs; frequency of occurrence of tokens.

48

Bottom-Up, using Info. Retrieval IR task: Given a query, find the relevant

document(s) from a large set of documents. Summ-IR task: Given a query, find the

relevant passage(s) from a set of passages (i.e., from one or more documents).

• Questions: 1. IR techniques work on large

volumes of data; can they scale down accurately enough?

2. IR works on words; do abstracts require abstract representations?

xx xxx xxxx x xx xxxx xxx xx xxx xx xxxxx xxxx xx xxx xx x xxx xx xx xxx x xxx xx xxx x xx x xxxx xxxx xxxx xxxx xxxxxx xx xx xxxx x xxxxx x xx xx xxxxx x x xxxxx xxxxxx xxxxxx x xxxxxxxx xx x xxxxxxxxxx xx xx xxxxx xxx xx xxx xxxx xxx xxxx xx xxxxx xxxxx xx xxx xxxxxx xxx

49

Top-Down, using Info. Extraction

IE task: Given a template and a text, find all the information relevant to each slot of the template and fill it in.

Summ-IE task: Given a query, select the best template, fill it in, and generate the contents.• Questions:1. IE works only for very particular

templates; can it scale up?

2. What about information that doesn’t fit into any template—is this a generic limitation of IE?

xx xxx xxxx x xx xxxx xxx xx xxx xx xxxxx xxxx xx xxx xx x xxx xx xx xxx x xxx xx xxx x xx x xxxx xxxx xxxx xxxx xxxx xxxxxx xx xx xxxx x xxxxx x xx xx xxxxx x x xxxxx xxxxxx xxxxxx x xxxxxxxx xx x xxxxxxxxxx xx xx xxxxx xxx xx x xxxx xxxx xxx xxxx xx xxxxx xxxxx xx xxx xxxxxx xxx

Xxxxx: xxxx Xxx: xxxx Xxx: xx xxx Xx: xxxxx xXxx: xx xxx Xx: x xxx xx Xx: xxx x Xxx: xx Xxx: x

50

NLP/IE:• Approach: try to ‘understand’

text—re-represent content using ‘deeper’ notation; then manipulate that.

• Need: rules for text analysis and manipulation, at all levels.

• Strengths: higher quality; supports abstracting.

• Weaknesses: speed; still needs to scale up to robust open-domain summarization.

IR/Statistics:• Approach: operate at lexical

level—use word frequency, collocation counts, etc.

• Need: large amounts of text.

• Strengths: robust; good for query-oriented summaries.

• Weaknesses: lower quality; inability to manipulate information at abstract levels.

Paradigms: NLP/IE vs.

ir/statistics

51

Toward the Final Answer...

Problem: What if neither IR-like nor IE-like methods work?

Solution: semantic analysis of the text

(NLP), using adequate knowledge bases

that support inference (AI).

Mrs. Coolidge: “What did the preacher preach about?”

Coolidge: “Sin.”Mrs. Coolidge: “What did he

say?”Coolidge: “He’s against it.”

– sometimes counting and templates are insufficient,

– and then you need to do inference to understand.

Word counting

Inference

52

The Optimal Solution...

Combine strengths of both paradigms…

...use IE/NLP when you have suitable template(s),

...use IR when you don’t…

…but how exactly to do it?

53

Overview of Extraction Methods Word frequencies throughout the text Position in the text

lead method; optimal position policy title/heading method

Cue phrases in sentences Cohesion: links among words

word co-occurrence coreference lexical chains

Discourse structure of the text Information Extraction: parsing and

analysis

54

Word-frequency-based method (1)

Claim: Important sentences contain words that occur “somewhat” frequently.

Method: Increase sentence score for each frequent word.

Evaluation: Straightforward approach empirically shown to be mostly detrimental in summarization systems.

words

Wordfrequency

The resolving power of words

(Luhn, 59)

Word-frequency-based method

Computes measures of significance

Words: Stemmingdiffer,difference bag of words

Sentences: concentration of high-

score words Cutoff values

established in experiments

SIGNIFICANT WORDS

ALL WORDS

* * * * 1 2 3 4 5 6 7

SENTENCE

SCORE = 42/7 2.3

56

POSition-based method (1)Claim: Important sentences occur

at the beginning (and/or end) of texts.

Lead method: just take first sentence(s)!

Experiments: In 85% of 200 individual paragraphs

the topic sentences occurred in initial position and in 7% in final position (Baxendale, 58).

Only 13% of the paragraphs of contemporary writers start with topic sentences (Donlan, 80).

POSition-based method

Cue method: stigma words (“hardly”, “impossible”) bonus words (“significant”)

Key method: similar to Luhn

Title method: title + headings

Location method: sentences under headings sentences near beginning or end of

document and/or paragraphs (also [Baxendale 58])

58

Optimum Position Policy (OPP) Claim: Important sentences are located at

positions that are genre-dependent; these positions can be determined automatically through training (Lin and Hovy, 97). Corpus: 13000 newspaper articles (ZIFF corpus). Step 1: For each article, determine overlap

between sentences and the index terms for the article.

Step 2: Determine a partial ordering over the locations where sentences containing important words occur: Optimal Position Policy (OPP)

OPP for ZIFF corpus:

(T) > (P2,S1) > (P3,S1) > (P2,S2) > {(P4,S1),(P5,S1),(P3,S2)} >…

(T=title; P=paragraph; S=sentence)OPP for Wall Street Journal: (T)>(P1,S1)>...

59

Title-Based Method (1)

Claim: Words in titles and headings are positively relevant to summarization.

Shown to be statistically valid at 99% level of significance (Edmundson, 68).

Empirically shown to be useful in summarization systems.

60

Cue-Phrase method (1)

Claim 1: Important sentences contain ‘bonus phrases’, such as significantly, In this paper we show, and In conclusion, while non-important sentences contain ‘stigma phrases’ such as hardly and impossible.

Claim 2: These phrases can be detected automatically (Kupiec et al. 95; Teufel and Moens 97).

Method: Add to sentence score if it contains a bonus phrase, penalize if it contains a stigma phrase.

61

Cohesion-based methodsClaim: Important

sentences/paragraphs are the highest connected entities in more or less elaborate semantic structures.

Classes of approaches word co-occurrences; local salience and grammatical relations; co-reference; lexical similarity (WordNet, lexical

chains); combinations of the above.

62

Cohesion: WORD co-occurrence (1)

Apply IR methods at the document level: texts are collections of paragraphs (Salton et al., 94; Mitra et al., 97; Buckley and Cardie, 97): Use a traditional, IR-based, word similarity

measure to determine for each paragraph Pi the set Si of paragraphs that Pi is related to.

Method: determine relatedness score Si for each

paragraph, extract paragraphs with largest Si scores.

P1P2

P3

P4

P5P6

P7

P8

P9

63

Word co-occurrence method (2) Cornell’s Smart-based approach

expand original query compare expanded query against paragraphs select top three paragraphs (max 25% of

original) that are most similar to the original query (SUMMAC,98): 71.9% F-score for relevance judgment

CGI/CMU approach maximize query-relevance while minimizing

redundancy with previous information.(SUMMAC,98): 73.4% F-score for relevance judgment

In the context of query-based summarization

64

Cohesion: Local salience Method

Assumes that important phrasal expressions are given by a combination of grammatical, syntactic, and contextual parameters (Boguraev and Kennedy, 97):

CNTX: 50 iff the expression is in the current discourse segmentSUBJ: 80 iff the expression is a subjectEXST: 70 iff the expression is an existential constructionACC: 50 iff the expression is a direct objectHEAD: 80 iff the expression is not contained in another phraseARG: 50 iff the expression is not contained in an adjunct

65

Cohesion: Lexical chains method (1)

But Mr. Kenny’s move speeded up work on a machine which uses micro-computers to control the rate at which an anaesthetic is pumpedinto the blood of patients undergoing surgery. Such machines are nothing new. But Mr. Kenny’s device uses two personal-computers to achievemuch closer monitoring of the pump feeding the anaesthetic into the patient. Extensive testing of the equipment has sufficiently impressedthe authorities which regulate medical equipment in Britain, and, so far,four other countries, to make this the first such machine to be licensedfor commercial sale to hospitals.

Based on (Morris and Hirst, 91)

66

Lexical chains-based method (2)

Assumes that important sentences are those that are ‘traversed’ by strong chains (Barzilay and Elhadad, 97). Strength(C) = length(C) -

#DistinctOccurrences(C) For each chain, choose the first sentence

that is traversed by the chain and that uses a representative set of concepts from that chain.

LC algorithm Lead-based algorithm[Jing et al., 98] corpus Recall Prec Recall Prec

10% cutoff 67% 61% 82.9% 63.4%

20% cutoff 64% 47% 70.9% 46.9%

67

Cohesion: Coreference method

Build co-reference chains (noun/event identity, part-whole relations) between query and document - In the context of query-based

summarization title and document sentences within document

Important sentences are those traversed by a large number of chains: a preference is imposed on chains (query > title > doc)

Evaluation: 67% F-score for relevance (SUMMAC, 98). (Baldwin and Morton, 98)

68

Cohesion: Connectedness method (1)

Map texts into graphs: The nodes of the graph are the words of

the text. Arcs represent adjacency, grammatical,

co-reference, and lexical similarity-based relations.

Associate importance scores to words (and sentences) by applying the tf.idf metric.

Assume that important words/sentences are those with the highest scores.

(Mani and Bloedorn, 97)

69

Connectedness method (2)

When a query is given, by applying a spreading-activation algorithms, weights can be adjusted; as a results, one can obtain query-sensitive summaries.

Evaluation (Mani and Bloedorn, 97): IR categorization task: close to full-

document categorization results.[Marcu,97] corpus TF-IDF method Spreading activation

10% cutoff F-score 25.2% 32.4%

20% cutoff F-score 35.8% 45.4%

In the context of query-based summarization

70

Claim: The multi-sentence coherence structure of a text can be constructed, and the ‘centrality’ of the textual units in this structure reflects their importance.

Tree-like representation of texts in the style of Rhetorical Structure Theory (Mann and Thompson,88).

Use the discourse representation in order to determine the most important textual units. Attempts: (Ono et al., 94) for Japanese. (Marcu, 97) for English.

Discourse-based method

71

Rhetorical parsing (Marcu,97)

[With its distant orbit {– 50 percent farther from the sun than Earth –} and slim atmospheric blanket,1] [Mars experiences frigid weather conditions.2] [Surface temperatures typically average about –60 degrees Celsius (–76 degrees Fahrenheit) at the equator and can dip to –123 degrees C near the poles.3] [Only the midday sun at tropical latitudes is warm enough to thaw ice on occasion,4] [but any liquid water formed that way would evaporate almost instantly5] [because of the low atmospheric pressure.6]

[Although the atmosphere holds a small amount of water, and water-ice clouds sometimes develop,7] [most Martian weather involves blowing dust or carbon dioxide.8] [Each winter, for example, a blizzard of frozen carbon dioxide rages over one pole, and a few meters of this dry-ice snow accumulate as previously frozen carbon dioxide evaporates from the opposite polar cap.9] [Yet even on the summer pole, {where the sun remains in the sky all day long,} temperatures never warm enough to melt frozen water.10]

72

Rhetorical parsing (2)

Use discourse markers to hypothesize rhetorical relations rhet_rel(CONTRAST, 4, 5) rhet_rel(CONTRAT, 4, 6) rhet_rel(EXAMPLE, 9, [7,8]) rhet_rel(EXAMPLE, 10, [7,8])

Use semantic similarity to hypothesize rhetorical relations if similar(u1,u2) then

rhet_rel(ELABORATION, u2, u1) rhet_rel(BACKGROUND, u1,u2)else

rhet_rel(JOIN, u1, u2)

rhet_rel(JOIN, 3, [1,2]) rhet_rel(ELABORATION, [4,6], [1,2]) Use the hypotheses in order to derive a valid discourse

representation of the original text.

73

Rhetorical parsing (3)

5Evidence

Cause

5 6

4

4 5Contrast

3

3Elaboration

1 2

2BackgroundJustification

2Elaboration

7 8

8Concession

9 10

10Antithesis

8Example

2Elaboration

Summarization = selection of the most important units

2 > 8 > 3, 10 > 1, 4, 5, 7, 9 > 6

Kupiec et al. 95

Extracts of roughly 20% of original text

Feature set: sentence length

▪ |S| > 5 fixed phrases

▪ 26 manually chosen paragraph

▪ sentence position in paragraph

thematic words▪ binary: whether sentence is included in the

set of highest scoring sentences uppercase words

▪ not common acronyms Corpus:

▪ 188 document + summary pairs from scientific journals

Kupiec et al. 95

Uses Bayesian classifier:

• Assuming statistical independence:

k

j j

k

j j

kFP

SsPSsFPFFFSsP

1

121

)(

)()|(),...,|(

),(

)()|,...,(),...,|(

,...21

2121

k

kk FFFP

SsPSsFFFPFFFSsP

Centroid (Radev, 2004)

Centroids consist of words which are central not only to one article in a cluster, but to all the articles.

Hypothesize that sentences that contain the words from the centroid are more indicative of the topic of the cluster.

A centroid is a pseudo-document which consists of words which have Count*IDF scores above a predefined threshold in the documents that constitute the cluster. Count: average number of occurrences of a

word across the entire cluster IDF: computed from a large corpus.

Centroid example

77

78

Topic Signature Method (Hovy and Lin, 98)Claim: Can approximate script

identification at lexical level, using automatically acquired ‘word families’.

Idea: Create topic signatures: each concept is defined by frequency distribution of its related words (concepts):

TS = {topic, signature } = {topic, (t1,w1) (t2,w2) ...}

restaurant -visit waiter + menu + food + eat...

(inverse of query expansion in IR.)

Signature term extraction

likelihood ratio (Dunning 1993) Hypothesis testing method:

79

80

Information extraction Method (1)

Idea: content selection using templates Predefine a template, whose slots specify what is of

interest. Use a canonical IE system to extract from a (set of)

document(s) the relevant information; fill the template. Generate the content of the template as the summary.

Previous IE work: FRUMP (DeJong, 78): ‘sketchy scripts’ of terrorism,

natural disasters, political visits... (Mauldin, 91): templates for conceptual IR. (Rau and Jacobs, 91): templates for business. (McKeown and Radev, 95): templates for news.

81

Information Extraction method (2)

Example template:

MESSAGE:ID TSL-COL-0001SECSOURCE:SOURCE ReutersSECSOURCE:DATE 26 Feb 93

Early afternoonINCIDENT:DATE 26 Feb 93INCIDENT:LOCATION World Trade CenterINCIDENT:TYPE BombingHUM TGT:NUMBER AT LEAST 5

82

Finally: Combining the Evidence

Problem: which extraction methods to believe?

Answer: assume they are independent, and combine their evidence: merge individual sentence scores.

Studies: (Kupiec et al., 95; Aone et al., 97, Teufel and

Moens, 97): Bayes’ Rule. (Mani and Bloedorn,98): SCDF, C4.5, inductive

learning. (Lin and Hovy, 98b): C4.5. (Marcu, 98): rhetorical parsing tuning.

83

And Now, an Example...

84

Example System: SUMMARISTThree stages: (Hovy and Lin, 98)

1. Topic Identification Modules: Positional Importance, Cue Phrases (under construction), Word Counts, Discourse Structure (under construction), ...

2. Topic Interpretation Modules: Concept Counting /Wavefront, Concept Signatures (being extended)

3. Summary Generation Modules (not yet built): Keywords, Template Gen, Sent. Planner & Realizer

SUMMARY = TOPIC ID + INTERPRETATION + GENERATION

86

Internal Format: Word-by-WordFormer <pno=1 sno=1 pos=JJ cwd=1 mph=- frq=1 tfidf=0.000

sig=-,-|-,-|-,- cue=0,- opp=-,->

hostage <pno=1 sno=1 pos=NN cwd=0 mph=- frq=6 tfidf=8.613 sig=1,12.169|33,1.370|2,5.791 cue=0,- opp=2.445,0.898>

John-Paul <pno=1 sno=1 pos=NNP cwd=0 mph=- frq=1 tfidf=0.000 sig=-,-|-,-|-,- cue=0,- opp=0.898,0.898>

Kauffmann <pno=1 sno=1 pos=NNP cwd=0 mph=- frq=6 tfidf=0.000 sig=-,-|-,-|-,- cue=0,- opp=4.578,0.898>

on <pno=1 sno=1 pos=IN cwd=1 mph=- frq=4 tfidf=0.000 sig=-,-|-,-|-,- cue=0,- opp=-,->

Monday <pno=1 sno=1 pos=NNP cwd=0 mph=- frq=3 tfidf=0.000 sig=-,-|-,-|-,- cue=0,- opp=2.076,0.898>

urged <pno=1 sno=1 pos=VBD cwd=0 mph=urge frq=1 tfidf=0.000 sig=-,-|-,-|274,0.492 cue=0,- opp=0.898,0.898>

87

Example Output, with Keywords

OPP tf.idfsignature

<QNUM>138</QNUM><DOCNO>AP890417-0167</DOCNO><TITLE>Former Hostage Accuses Britain of Weakness </TITLE><TEXT>Former hostage John-Paul Kauffmann on Monday urged Britain to follow the example set by France and West Germany and negotiate the release of its citizens held captive in Lebanon .Kauffmann said Britain `` has abandoned '' John McCarthy , 32 , a television reporter abducted on his way to Beirut...Keywords:western moslem iranian middle kill march east syrian free anderson group palestinian </TEXT></DOC>

88

Summarization exercise

Write a one-sentence summary for each of the following texts.

89

Flu stopperA new compound is set for human testing (Times)

Running nose. Raging fever. Aching joints. Splitting headache. Are there any poor souls suffering from the flu this winter who haven’t longed for a pill to make it all go away? Relief may be in sight. Researchers at Gilead Sciences, a pharmaceutical company in Foster City, California, reported last week in the Journal of the American Chemical Society that they have discovered a compound that can stop the influenza virus from spreading in animals. Tests on humans are set for later this year.The new compound takes a novel approach to the familiar flu virus. It targets an enzyme,

called neuraminidase, that the virus needs in order to scatter copies of itself throughout thebody. This enzyme acts like a pair of molecular scissors that slices through the protectivemucous linings of the nose and throat. After the virus infects the cells of the respiratorysystem and begins replicating, neuraminidase cuts the newly formed copies free to invadeother cells. By blocking this enzyme, the new compound, dubbed GS 4104, prevents theinfection from spreading.

90

Plant matters How do you regulate an herb? (Scientific American)

If Harlan Page Hubbard were alive, he might be the president of a dietary supplementscompany. In the late 19th century Hubbard sold Lydia E. Pinkham’s Vegetable Compoundfor kidney and sexual problems. The renowned huckster is remembered each year by nationalconsumer and health organizations who confer a “Hubbard” – a statuette clutching a freshlemon – for the “most misleading, unfair and irresponsible advertising of the past 12 months.”

Appropriately enough, one of this year’s winners was a product that Hubbard might havepeddled alongside his Lydia Pinkham elixir. Ginkay, an extract of the herb gingko, received its lemon for advertising and labelling claims that someone ingesting the product will havea better memory. Whereas some studies have shown that gingko improves mental functioningin people with dementia, none has proved that it serves as brain tonic for healthy.

91

Table of contents

1. Motivation.2. Genres and types of summaries.3. Approaches and paradigms.4. Summarization methods (& exercise).

Topic Extraction. Interpretation. Generation.

5. Evaluating summaries.6. The future.

92

From extract to abstract: topic interpretation or concept fusion.

Experiment (Marcu, 98): Got 10 newspaper texts, with human abstracts. Asked 14 judges to extract corresponding

clauses from texts, to cover the same content. Compared word lengths of extracts to

abstracts: extract_length 2.76 abstract_length !!

xx xxx xxxx x xx xxxx xxx xx xxx xx xxxxx xxxx xx xxx xx x xxx xx xx xxx x xxx xx xxx x xx x xxxx xxxx xxxx xxxx xxxx xxxxxx xx xx xxxx x xxxxx x xx xx xxxxx x x xxxxx xxxxxx xxxxxx x xxxxxxxx xx x xxxxxxxxxx xx xx xxxxx xxx xx x xxxx xxxx xxx xxxx xx

Topic Interpretationxxx xx xxx xxxx xxxxx x xxxx x xx xxxxxx xxx xxxx xx x xxxxxx xxxx x xxx x xxxxx xx xxxxx x x xxxxxxxxx xx x xxxxxxxxxx xx xx xxxxx xxx xxxxx xx xxxx x xxxxxxx xxxxx x

93

Some Types of Interpretation Concept generalization:

Sue ate apples, pears, and bananas Sue ate fruit

Meronymy replacement:Both wheels, the pedals, saddle, chain…

the bike Script identification: (Schank and

Abelson, 77)

He sat down, read the menu, ordered, ate, paid, and left He ate at the restaurant

Metonymy:A spokesperson for the US Government

announced that… Washington announced that...

94

General Aspects of Interpretation Interpretation occurs at the conceptual

level...…words alone are polysemous (bat animal and

sports instrument) and combine for meaning (alleged murderer murderer).

For interpretation, you need world knowledge...…the fusion inferences are not in the text!

Little work so far: (Lin, 95; McKeown and Radev, 95; Reimer and Hahn, 97; Hovy and Lin, 98).

95

Template-based operations

Claim: Using IE systems, can aggregate templates by detecting interrelationships.

1. Detect relationships (contradictions, changes of perspective, additions, refinements, agreements, trends, etc.).

2. Modify, delete, aggregate templates using rules (McKeown and Radev, 95):

Given two templates,if (the location of the incident is the same and

the time of the first report is before the time of the second report and

the report sources are different and at least one slot differs in value)then combine the templates using a contradiction

operator.

96

Claim: Can perform concept generalization, using WordNet (Lin, 95).

Find most appropriate summarizing concept:

Concept Generalization: Wavefront

Cash register

Mainframe

Dell Mac IBM

Computer

Calculator

18

65

20

5

0

2

20

PC

1. Count word occurrences in text; score WN concs

2. Propagate scores upward3. R Max{scores} / scores

4. Move downward until no obvious child: R<Rt

5. Output that concept

97

Wavefront Evaluation 200 BusinessWeek articles about

computers: typical length 750 words (1 page). human abstracts, typical length 150

words (1 par). several parameters; many variations

tried. Rt = 0.67; StartDepth = 6; Length =

20%:

Conclusion: need more elaborate taxonomy.

Random Wavefront

Precision 20.30% 33.80%Recall 15.70% 32.00%

98

Inferences in terminological Logic ‘Condensation’ operators (Reimer and Hahn,

97).

1. Parse text, incrementally build a terminological rep.

2. Apply condensation operators to determine the salient concepts, relationships, and properties for each paragraph (employ frequency counting and other heuristics on concepts and relations, not on words).

3. Build a hierarchy of topic descriptions out of salient constructs.

Conclusion: No evaluation.

99

Topic Signatures (1)Claim: Can approximate script

identification at lexical level, using automatically acquired ‘word families’ (Hovy and Lin, 98).

Idea: Create topic signatures: each concept is defined by frequency distribution of its related words (concepts):

signature = {head (c1,f1) (c2,f2) ...}

restaurant waiter + menu + food + eat...

(inverse of query expansion in IR.)

100

Example SignaturesRANKaerospace banking environment telecommunication

1 contract bank epa at&t2 air_force thrift waste network3 aircraft banking environmental fcc4 navy loan water cbs5 army mr. ozone6 space deposit state bell7 missile board incinerator long-distance8 equipment fslic agency telephone9 mcdonnell fed clean telecommunication

10 northrop institution landfill mci11 nasa federal hazardous mr.12 pentagon fdic acid_rain doctrine13 defense volcker standard service14 receive henkel federal news15 boeing banker lake turner16 shuttle khoo garbage station17 airbus asset pollution nbc18 douglas brunei city sprint19 thiokol citicorp law communication20 plane billion site broadcasting21 engine regulator air broadcast22 million national_bankprotection programming23 aerospace greenspan violation television24 corp. financial management abc25 unit vatican reagan rate

101

Topic Signatures (2)Experiment: created 30 signatures

from 30,000 Wall Street Journal texts, 30 categories: Used tf.idf to determine uniqueness in

category. Collected most frequent 300 words per

term.Evaluation: classified 2204 new

texts: Created document signature and

matched against all topic signatures; selected best match.

Results: Precision 69.31%; Recall 75.66% 90%+ for top 1/3 of categories; rest

lower, because less clearly delineated (overlapping signatures).

102

Table of contents

1. Motivation.2. Genres and types of summaries.3. Approaches and paradigms.4. Summarization methods (& exercise).

Topic Extraction. Interpretation. Generation.

5. Evaluating summaries.6. The future.

103

NL Generation for Summaries Level 1: no separate generation

Produce extracts, verbatim from input text.

Level 2: simple sentences Assemble portions of extracted clauses

together.

Level 3: full NLG 1. Sentence Planner: plan sentence content,

sentence length, theme, order of constituents, words chosen... (Hovy and Wanner, 96)

2. Surface Realizer: linearize input grammatically (Elhadad, 92; Knight and Hatzivassiloglou, 95).

104

Full Generation Example

Challenge: Pack content densely!

Example (McKeown and Radev, 95): Traverse templates and assign values to

‘realization switches’ that control local choices such as tense and voice.

Map modified templates into a representation of Functional Descriptions (input representation to Columbia’s NL generation system FUF).

FUF maps Functional Descriptions into English.

105

Generation Example (McKeown and Radev, 95)

NICOSIA, Cyprus (AP) – Two bombs exploded near government ministries in Baghdad, but there was no immediate word of any casualties, Iraqi dissidents reported Friday. There was no independentconfirmation of the claims by the Iraqi National Congress. Iraq’sstate-controlled media have not mentioned any bombings.

Multiple sources and disagreement

Explicit mentioning of “no information”.

106

Table of contents

1. Motivation.2. Genres and types of

summaries.3. Approaches and paradigms.4. Summarization methods (&

exercise).5. Evaluating summaries.6. The future.

107

How can You Evaluate a Summary? When you already have a

summary… ...then you can compare a new one to it:1. choose a granularity (clause; sentence;

paragraph),2. create a similarity measure for that granularity

(word overlap; multi-word overlap, perfect match),

3. measure the similarity of each unit in the new to the most similar unit(s) in the gold standard,

4. measure Recall and Precision.

e.g., (Kupiec et al., 95).

……………..…. but when you don’t?

108

Toward a Theory of EvaluationTwo Measures:

Measuring length: Number of letters? words?

Measuring information: Shannon Game: quantify information

content. Question Game: test reader’s

understanding. Classification Game: compare

classifiability.

Compression Ratio: CR = (length S) / (length T)

Retention Ratio: RR = (info in S) / (info in T)

109

Compare Length and InformationCase 1: just adding

info; no special leverage from summary.

Case 2: ‘fuser’ concept(s) at knee add a lot of information.

Case 3: ‘fuser’ concepts become progressively weaker.

RR

CR

RR

CR

RR

CR

110

Small Evaluation Experiment (Hovy, 98)

Can you recreate what’s in the original? the Shannon Game [Shannon 1947–50]. but often only some of it is really

important. Measure info retention (number of

keystrokes): 3 groups of subjects, each must recreate

text:▪ group 1 sees original text before starting. ▪ group 2 sees summary of original text before

starting. ▪ group 3 sees nothing before starting.

Results (# of keystrokes; two different paragraphs):

Group 1 Group 2 Group 3approx. 10 approx. 150 approx. 1100

111

Q&A Evaluation

Can you focus on the important stuff? The Q&A Game—can be tailored to your interests!

Measure core info. capture by Q&A game: Some people (questioners) see text, must create

questions about most important content. Other people (answerers) see:

1. nothing—but must try to answer questions (baseline),2. then: summary, must answer same questions,3. then: full text, must answer same questions again.

Information retention: % answers correct.

112

How can You Evaluate a Summary?

When you generate a summary… ..…:1. the gold standard summaries (human),2. choose a granularity (clause; sentence;

paragraph),3. create a similarity measure for that

granularity (word overlap; multi-word overlap, perfect match),

4. measure the similarity of each unit in the new to the most similar unit(s) in the gold standard,

5. measure Recall and Precision.

e.g., (Kupiec et al., 95).

Human summaries

Two persons A and B extract sentences from documents as summaries

Their agreement? Kappa value

▪ ≥0.75 Excellent, 0.40 to 0.75 Fair to good, < 0.40 as Poor

▪ 0 to 0.20 Slight, >0.20 to 0.40 Fair, >0.40 to 0.60 Moderate, >0.60 to 0.80 Substantial, >0.80 Almost perfect

113

Kappa

P(A): the observed agreement among raters

P(E): the expectation probability of chance agreement

)(1

)()(

EP

EPAP

B

Yes No

AYes 20 5

No 10 15

115

P(A) = (20 + 15) / 50 = 0.70P(E) = 0.5*0.6+ 0.5* 0.4 = 0.3 + 0.2 = 0.5

)(1

)()(

EP

EPAP

Evaluation Manual

Linguistic Quality(readability)▪ Grammaticality▪ Non-redundancy▪ Referential clarity▪ Focus▪ Structure

Five-point scale (1 very poor, 5 very good)

Pyramid: SCU

Automatic Rouge

▪ ROUGE 2▪ ROUGE SU4

4

3

21

– Responsiveness(content)

– BE(basic element)

Pyramid(1)

The pyramid method is designed to address the observation: summaries from different humans always have partly overlapping content.

The pyramid method includes a manual annotation method to represent Summary Content Units (SCUs) and to quantify the proportion of model summaries that express this content.

All SCUs have a weight representing the number of models they occur in, thus from 1 to maxn, where maxn is the total number of models There are very few SCUs expressed in all models (i.e.,

weight=maxn), and increasingly many SCUs at each lower weight, with the most SCUs at weight=1.

1

2

34

SCU example

118

Pyramid(2)

The approach involves two phases of manual annotation: pyramid construction annotation against the pyramid to

determine which SCUs in the pyramid have been expressed in the peer summary.

The total weight is

ROUGE basics

Rouge(Recall-Oriented Understudy for Gisting Evaluation) Recall-oriented, within-sentence word overlap with model(s)

Models - no theoretical limit to number compared system output to 4 models compared manual summaries to 3 models

Using n-gram Correlate reasonably with human coverage judgements Not address summary discourse characteristics, and suffer

from lack of text cohesion or coherence ROUGE v1.2.1 measures

ROUGE-1,2,3,4: N-gram matching where N = 1,2,3,4 ROUGE-LCS: Longest common substring

ROUGE: Recall-Oriented Understudy for Gisting Evaluation

Rouge – Ngram co-occurrence metrics measuring content overlap

Counts of n-gram overlaps between candidate and model summaries

Total n-grams in summary model

ROUGE

Where Nn represents the set of all n-grams and i is one member from Nn. Xn(i) is the number of times the n-gram i occurred in the summary and Mn(i,j) is the number of times the n-gram i ocurred in the j-th model reference(human) summary. There are totally h human summaries.

1

1

min( ( ), ( , ))( )

( , )n

n

h

n nj i Nn h

nj i N

X i M i jR X

M i j

Example

peer: A B C D E F G A B H1: A B G A D E C D (7) H2: A C E F G A D (6)

How to compute the ROUGE-2 score?

124

125

Table of contents

1. Motivation.2. Genres and types of

summaries.3. Approaches and paradigms.4. Summarization methods .5. Evaluating summaries.6. The future.

In the last decade there has been a surge of interest in automatic summarizing. … There has been some progress, but there is much to do.

Karen Sparck Jones

Karen Sparck Jones, automatic summarising: the state of the art, Information Processing & Management, 43(6): 1449--1481, 2007.

127

The Future (1) — There’s much to do!

Data preparation: Collect large sets of texts with abstracts, all

genres. Build large corpora of <Text, Abstract, Extract>

tuples. Investigate relationships between extracts and

abstracts (using <Extract, Abstract> tuples). Topic Identification:

Develop new identification methods (discourse, etc.).

Develop heuristics for method combination (train heuristics on <Text, Extract> tuples).

128

The Future (2)

Concept Interpretation (Fusion): Investigate types of fusion (semantic, evaluative…). Create large collections of fusion knowledge/rules. Study incorporation of User’s knowledge in

interpretation. Generation:

Develop sentence generation rules (using <Extract, Abstract> pairs).

Evaluation: Develop better automatic evaluation metrics.

129

SUMMAC Q&A EvaluationProcedure (SUMMAC,

98):1. Testers create questions for

each category.2. Systems create summaries,

not knowing questions.3. Humans answer questions

from originals and from summaries.

4. Testers measure answer Recall: how many questions can be answered correctly from the summary?(many other measures as well)

• Results:Large variation by topic,

even within systems...

Normalized Answer Recall

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3

Topic number

130

Task Evaluation: Text Classification

Can you perform some task faster? example: the Classification Game. measures: time and effectiveness.

TIPSTER/SUMMAC evaluation: February, 1998 (SUMMAC, 98). Two tests: 1. Categorization 2. Ad Hoc (query-sensitive) 2 summaries per system: fixed-length

(10%), best. 16 systems (universities, companies; 3

intern’l).

CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann,...

Documents

Transcript of CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann,...