Speech Summarization

Julia Hirschberg (thanks to Sameer Maskey for some slides)

CS4706

Summarization Distillation

• ‘…the process of distilling the most important information from a source (or sources) to produce an abridged version for a particular user (or users) and task (or tasks) [Mani and Maybury, 1999]

• Why summarize? Too much data!

Types of Summarization

• Indicative– Describes the document and its contents

• Informative– ‘Replaces’ the document

• Extractive– Concatenate pieces of existing document

• Generative– Creates a new document

• Document compression

Extraction Training w/ manual Summaries

Sentence ExtractionSimilarity Measures

Concept LevelExtract concepts units

Generate Words/Phrases

Use of Structured Data

SOME SUMMARIZATIONTECHNIQUES BASED

ON TEXT (LEXICAL FEATURES)

[Salton, et al., 1995]

[McKeown, et al., 2001]

[Hovy & Lin, 1999]

[Witbrock & Mittal, 1999]

[Maybury, 1995]

Sentence Extraction/Similarity measures [Salton, et al. 1995]

• Extract sentences by their similarity to a topic sentence and their dissimilarity to sentences already in summary (Maximal Marginal Relativity)

• Similarity measures – Cosine Measure – Vocabulary Overlap– Topic word overlap– Content Signatures Overlap

Concept/content level extraction [Hovy & Lin, 1999]

• Present key-words as summary• Builds concept signatures by finding relevant

words in 30,000 WSJ documents, each categorized into different topics

• Phrase concatenation of relevant concepts/content

• Sentence planning for generation

Feature-based statistical models [Kupiec, et al., 1995]

• Create manual summaries• Extract features• Train statistical model using various ML techniques• Use the trained model to score each sentence in the test

data• Extract N highest-scoring sentences

• Where S is summary given k features Fj and P(Fj) & P(Fj|s of S) can be computed by counting occurrences

k

jj

k

jj

k

FP

SsPSsFP

FFFSsP

1

1,21

)(

)()|(

)...,|(

Structured Database [Maybury, 1995]

• Summarize text represented in structured form: database, templates– E.g. generation of a medical history from a

database of medical ‘events’

• Link analysis (semantic relations within the structure)

• Domain dependent importance of events

events all of # Total

event E of soccurrence of # Eoffrequency Relative

Comparing Speech and Text Summarization

• Alike– Identifying important

information– Some lexical,

discourse features– Extraction or

generation or compression

• Different– Speech Signal– Prosodic features– NLP tools?– Segments –

sentences?– Generation?– Errors– Data size

Text vs. Speech Summarization (NEWS)Speech Signal

Speech Channels- phone, remote satellite, station

Transcripts- ASR, Close Captioned

Many Speakers- speaking styles

Prosodic Features-pitch, energy, duration

Structure-Anchor, Reporter Interaction

Commercials, Weather Report

Transcript- Manual

Some Lexical Features

Story presentationstyle

Error-free Text

Lexical Features

Segmentation-sentences

NLP tools

Speech Summarization Today

• Mostly extractive:– Words, sentences, content units

• Some compression methods • Generation-based summarization difficult

– Text or synthesized speech?

• SENT27 a trial that pits the cattle industry against tv talk show host oprah winfrey is under way in amarillo , texas.

• SENT28 jury selection began in the defamation lawsuit began this morning .• SENT29 winfrey and a vegetarian activist are being sued over an exchange on her April

16, 1996 show .• SENT30 texas cattle producers claim the activists suggested americans could get mad

cow disease from eating beef .• SENT31 and winfrey quipped , this has stopped me cold from eating another burger • SENT32 the plaintiffs say that hurt beef prices and they sued under a law banning false

and disparaging statements about agricultural products • SENT33 what oprah has done is extremely smart and there's nothing wrong with it she

has moved her show to amarillo texas , for a while • SENT34 people are lined up , trying to get tickets to her show so i'm not sure this hurts

oprah .• SENT35 incidentally oprah tried to move it out of amarillo . she's failed and now she has

brought her show to amarillo .• SENT36 the key is , can the jurors be fair • SENT37 when they're questioned by both sides, by the judge , they will be asked, can

you be fair to both sides • SENT38 if they say , there's your jury panel • SENT39 oprah winfrey's lawyers had tried to move the case from amarillo , saying they

couldn't get an impartial jury • SENT40 however, the judge moved against them in that matter … story summary

Generation or Extraction?

Word scoringwith dependency structure

Sentence extraction withsimilarity measures

Classification

User access information

Weighted finite statetransducers

Removing disfluencies

SPEECH SUMMARIZATIONTECHNIQUES

[Christensen et al., 2004]

[Hori C. et al., 1999, 2002] , [Hori T. et al., 2003]

[Koumpis & Renals, 2004]

[He et al., 1999]

[Hori T. et al., 2003]

[Zechner, 2001]

Content/Context sentence level extraction for speech summary [Christensen et al., 2004]

Find sentences similar to the lead topic sentences Use position features to find the relevant nearby

sentences after detecting a topic sentence where Sim is a similarity measure between two

sentences or a sentence and a document (D) and E is the set of sentences already in the summary

Choose a new sentence which is most like D and most different from E

)},({maxarg 1/

^

iEDs

k ssSimsSi

)},({maxarg/

^

iEDs

k sDSimsSi

Weighted finite state transducers for speech summarization

[Hori T. et al., 2003] • Summarization includes speech recognition, paraphrasing,

sentence compaction integrated into single Weighted Finite State Transducer

• Decoder can use all knowledge sources in one-pass strategy• Speech recognition using WFST

– Where H is state network of triphone HMMs, C is triphone connection rules, L is pronunciation and G is trigram language model

• Paraphrasing can be looked at as a kind of machine translation with translation probability P(W|T) where W is source language and T is the target language

• If S is the WFST representing translation rules and D is the language model of the target language speech summarization can be looked at as the following composition

GLCHR

DSGLCHZ

H C L G S DSpeech recognizer Translator

Speech Translator

User Access Identifies What to Include [He et al., 1999]

• Summarize lectures or shows by extracting parts that have been viewed the longest

• Needs multiple users of the same show, meeting or lecture for training

• E.g. To summarize lectures compute the time spent on each slide

• Summarizer based on user access logs did as well as summarizers that used linguistic and acoustic features – Average score of 4.5 on a scale of 1 to 8 for the

summarizer (subjective evaluation)•

Word level extraction by scoring/classifying words [Hori C. et al., 1999, 2002]

Score each word in the sentence and extract a set of words to form a sentence whose total score is the product/sum of the scores of each word

Example: Word Significance score (topic words) Linguistic Score (bigram probability) Confidence Score (from ASR) Word Concatenation Score (dependency structure

grammar)

Where M is the number of words to be extracted, and I C T are weighting factors for balancing among L, I, C, and T r

)()()()...|({)( ,11

1 mmTmcm

M

mImm vvTrvCvIvvLVS

Segmentation Using Discourse Cues [Maybury, 1998]

Discourse Cue-Based Story Segmentation Discourse Cues in CNN

Start and end of broadcast Anchor/Reporter handoff, Reporter/Anchor handoff Cataphoric Segment (“still ahead …”)

Time Enhanced Finite State Machine representing discourse states such as anchor segment, reporter segment, advertisement

Other features: named entities, part of speech, discourse shifts “>>” speaker change, “>>>” subject changeSource Precision Recall

ABC 90 94

CNN 95 75

Jim Lehrer Show 77 52

CU: Summarization without Words: Does importance of ‘what’ is said correlates with ‘how’ it

is said?

• Hypothesis: “Speakers change their amplitude, pitch, speaking rate to signify importance of words, phrases, sentences.”– If so, then the prediction labels for sentences predicted

using acoustic features (A) should correlate with labels predicted using lexical features (L)

– In fact, this seems to be true (corr .74 between precitions of A and L

Is It Possible to Build ‘good’ Automatic Speech Summarization Without Any Transcripts?

Feature Set F-Measure ROUGE-avg

L+S+A+D 0.54 0.80

L 0.49 0.70

S+A 0.49 0.68

A 0.47 0.63

Baseline 0.43 0.50

• Just using A+S without any lexical features we get 6% higher F-measure and 18% higher ROUGE-avg than the baseline

Evaluation using ROUGE

• F-measure too strict– Predicted summary sentences must match

summary sentences exactly– What if content is similar but not identical?

• ROUGE(s)…

ROUGE metric

• Recall-Oriented Understudy for Gisting Evaluation (ROUGE)• ROUGE-N (where N=1,2,3,4 grams)• ROUGE-L (longest common subsequence)• ROUGE-S (skip bigram)• ROUGE-SU (skip bigram counting unigrams as well)

• Does ROUGE solve the problem?

Next Class

• Emotional speech• HW 4 assigned