COMS 6998-06 Network TheoryWeek 7

Dragomir R. RadevWednesdays, 6:10-8 PM

325 Pupin TerraceFall 2010

(17) Lexical networks

The large-scale structure of semantic networks:statistical analyses and a model of semantic growthM. Steyvers, J. B. Tenenbaum (2005)Cognitive Science, 29(1)

Free word associations

Semantic network

Meredith yesterday apples

bought

green

Dependency network

Dependency network

Random network

Language networks

• A special case of networks where nodes are words or documents and edges link semantically related nodes

• Other examples:– Words used in dictionary definitions– Names of people mentioned in the same story– Words that translate to the same word

Comparing the dependency graph to a random (Poisson) graph

Random Actual

n 5563 5584

M 14440 14472

Diameter 21 13

Asp 8.788 4.01

W/S cc 0.00062 0.092

n/a 2.2

Some real networks

• From Newman 2002:Network n Mean degree z Cc Cc for random graph

Internet (AS level) 6,374 3.8 0.24 0.00060

WWW (sites) 153,127 35.2 0.11 0.00023

Power grid 4,941 2.7 0.080 0.00054

Biology collaborations 1,520,251 15.5 0.081 0.000010

Mathematics collaborations 253,339 3.9 0.15 0.000015

Film actor collaborations 449,913 113.4 0.20 0.00025

Company directors 7,673 14.4 0.59 0.0019

Word co-occurrence 460,902 70.1 0.44 0.00015

Neural network 282 14.0 0.28 0.049

Metabolic network 315 28.3 0.59 0.090

Food web 134 8.7 0.22 0.065

Properties of lexical networks

• Entries in a thesaurus[Motter et al. 2002]

• c/c0 = 260 (n=30,244; m=60)

• Co-occurrence networks [Dorogovtsev and Mendes 2001, Sole and Ferrer i Cancho 2001]

• c/c0 = 1,000 (n=400,000)

• Mental lexicon [Vitevitch 2005]• c/c0 = 278 (n=19,340)

letter

actor

character nature

universe

world

syntactic dependency degree distribution(loglog scale)

0

1

2

3

4

5

6

7

8

0 1 2 3 4 5 6 7 8

Sigman and Cecchi 2002

• Meronymy– Component (branch-tree)– Member (tree-forest)– Stuff (glass-bottle)

• Polysemy– E.g., board

• Hypernymy/hyponymy– E.g., oak-tree

Masucci and Rodgers 2006

• n=8,992• m=117,687 • alpha = 1.1

It was a bright cold day in April, and the clocks were striking thirteen. Winston Smith, his chin nuzzled into hisbreast in an effort to escape the vile wind, slipped quickly through the glass doors of Victory Mansions, though not quickly enough to prevent a swirl of gritty dust from entering along with him.

The hallway smelt of boiled cabbage and old rag mats. At one end of it a coloured poster, too large for indoor display, had been tacked to the wall. It depicted simply an enormous face, more than a metre wide: the face of a man of about forty-five, with a heavy black moustache and ruggedly handsome features. Winston made for the stairs. It was no use trying the lift. Even at the best of times it was seldom working, and at present the electric current was cut off during daylight hours. It was part of the economy drive in preparation for Hate Week. The flat was seven flights up, and Winston, who was thirty-nine and had a varicose ulcer above his right ankle, went slowly, resting several times on the way. On each landing, opposite the lift-shaft, the poster with the enormous face gazed from the wall. It was one of those pictures which are so contrived that the eyes follow you about when you move. BIG BROTHER IS WATCHING YOU, the caption beneath it ran.

Masucci and Rodgers (cont’d)

Solé et al. 2004

Ferrer i Cancho et al. 2003

Based on [Mehler 2007]

Lexical centrality

• In social networks, some people are called mavens/centers due to their popularity (or gregariousness)

• What is the analog to a highly connected person in a language network?

• In general, how can network structure be used to identify content in a language network?

Red Sox Win the World Series, More Titles Might FollowBack in the Red: World Series crown returns to Boston as Rockies hit the canvasFans celebrate Red Sox winRockies Vanish In Thin AirPolice Arrest Dozens After Red Sox World Series WinRed Sox 4, Rockies 3 Boston Sweeps World Series AgainVictory walk leads to dynasty talkRed Sox Win Baseball's World Series Title by Sweeping RockiesBoston enjoys sweep smell of successSox sweep Rockies to win SeriesWorld Series: Red Sox sweep RockiesRed Sox cruise to World Series titleRed Sox Are World ChampsBoston sweep Colorado to win World SeriesRed Sox Sweep Colorado in World SeriesRed Sox Take World SeriesHow sweep it is! Red Sox breeze to second World Series titleBoston owners dedicate triumph to Red Sox Nation

How many ways to say it?

Red Sox Win the World Series, More Titles Might FollowBack in the Red: World Series crown returns to Boston as Rockies hit the canvasFans celebrate Red Sox winRockies Vanish In Thin AirPolice Arrest Dozens After Red Sox World Series WinRed Sox 4, Rockies 3 Boston Sweeps World Series AgainVictory walk leads to dynasty talkRed Sox Win Baseball's World Series Title by Sweeping RockiesBoston enjoys sweep smell of successSox sweep Rockies to win SeriesWorld Series: Red Sox sweep RockiesRed Sox cruise to World Series titleRed Sox Are World ChampsBoston sweep Colorado to win World SeriesRed Sox Sweep Colorado in World SeriesRed Sox Take World SeriesHow sweep it is! Red Sox breeze to second World Series titleBoston owners dedicate triumph to Red Sox NationRed Sox win World SeriesShort wait for bosox this timeWorld Series: Red Sox complete sweep of RockiesIt's Leap YearBoston Red Sox blank Rockies to clinch World SeriesRed Sox Sweep Rockies 4-3 In Game 4Red Sox claim World Series titleBoston Red Sox win World Series, lose imageSox sweep Rockies for 2nd title in 4 seasonsRed Sox Complete Sweep Of Rockies For World Series VictoryRed sox wrap up world series routRockies feel the pain, but not the shameRed Sox, tarnished for so long, now baseball's gold standardRed sox take titleBoston Celebrates World Series WinBad news AL, Red Sox built to lastBoston sweeps it Heartbreak is history for Red SoxFans Celebrate Red Sox World Series WinFrom cursed to charmed: Red Sox sweep World SeriesRed Sox Sweep Rockies To Win World SeriesRed Sox make Boston jump for joy, Series ChampsCrowds fill streets after Red Sox winPapelbon, Timlin savoring Series winRed Sox scale the RockiesEven Sox fan losing passion for winningRookies respond in first crack at the big timeRed Sox Get 2nd World Series Sweep In 4 YearsRed Sox looking like a dynastyBoston Red Sox are America's teamRed Sox Win 2007 World SeriesBelieving Pays Off! Sox World Series ChampsRed Sox go from cursed to blessed

Boston lowers the broomWe Are the Champions: Red Sox 4, Rockies 3Sweep and red sox for everybodyTwo titles four years apart impossible to compareRed Sox cash inThe Boston Red Sox swept the Colorado Rockies to win the World SeriesRed Sox "do little things" to win second title in four yearsWorld Series victory for Red SoxBoston reigns supremeRockies' heads held high despite lossRockies just failed to executeBoston Fans Fill Streets To Celebrate Series SweepIt's easy to embrace these Red SoxYoung stars lead Red Sox to World Series titleThey spend money in Boston, but they winBoston fans celebrate World Series win; 37 arrests reportedSoxcess started upstairsBoston sweep is completeRed Sox sweep 2007 World Series in DenverRookies rise to occasion!Sox sweep World Series againPoor pitching, poorer hitting doom RockiesRed Sox top Rockies and sweep to second World Series championship ...Sweeping off to BostonAnother in a Series of sweepsPutting an end to baseball as we've known itRed Sox Sweep Rockies to Capture Second Title in Four YearsRed Sox accomplished the expected, unbelievableRockies Find Being Good Isnt EnoughBoston sweep-walks to titleRed Sox play party crashers in DenverUnhappy ending for Colorado - MLBSox on, Rocks off in sweeping winTimlin gets to ring up another oneRed Sox complete World Series sweepWild celebrations in Boston after World Series winRed Sox seal sweep of RockiesRed Sox Win World SeriesSox are kings of diamondRockies: Sweep, sweep, sweptRed Sox sweep World SeriesMonsters of Beantown: Red Sox win SeriesRed Sox claim World Series gloryHow sweep it isRed Sox: Dynasty in the makingRed Sox sweep upstart RockiesBoston Sweeps Colorado To Win World SeriesRed Sox sweep Rockies, take World SeriesWhat curse? Red Sox win Series again

• Red Sox• Sweep• World Series• Rockies• Celebrate• Boston• Colorado• Dynasty• 2007• Four years• Second time• 4:3 score• Easy• Expectations• No curse• Young players• Timlin• not (baseball)

List of topics

LexRank – Centrality in Text Graphs

Vertices

Units of text (sentences or documents)

Edges

Pairwise similarity between text

(tf-idf cosine or language model)

LexRank – Centrality in Text Graphs

Intuition

LexRank score is propagated through

edges

Central vertices are those that are similar

to other central vertices

LexRank – Centrality in Text Graphs

Recurrence Relation

sCan guarantee solution by allowing “jump” probability

d/N.

0.5

0.3

0.80.2

0.1

0.3

0.9

0.2 0.4

0.01718 Red Sox Win Baseball's World Series Title by Sweeping Rockies 0.01712 Red Sox Sweep Rockies To Win World Series 0.01647 World Series: Red Sox sweep Rockies 0.01630 Red Sox sweep Rockies, take World Series 0.01608 Red Sox 4, Rockies 3 Boston Sweeps World Series Again 0.01597 World Series: Red Sox complete sweep of Rockies 0.01584 Red Sox sweep World Series 0.01579 Red Sox Sweep Colorado in World Series 0.01573 Red Sox Complete Sweep Of Rockies For World Series Victory 0.01531 Red Sox complete World Series sweep ... 0.01057 Boston Red Sox blank Rockies to clinch World Series 0.01052 Red Sox: Dynasty in the making 0.01037 Sox sweep Rockies for 2nd title in 4 seasons 0.01034 Police Arrest Dozens After Red Sox World Series Win 0.01027 Rookies respond in first crack at the big time 0.01018 Rockies: Sweep, sweep, swept 0.01016 Sweeping off to Boston 0.01013 Rookies rise to occasion! 0.01012 Fans celebrate Red Sox win 0.01010 Short wait for bosox this time ... 0.00441 Sox are kings of diamond 0.00414 Rockies just failed to execute 0.00408 Rockies Find Being Good Isnt Enough 0.00407 Rockies' heads held high despite loss 0.00391 Boston lowers the broom 0.00390 Rockies Vanish In Thin Air 0.00390 Poor pitching, poorer hitting doom Rockies 0.00390 Rockies feel the pain, but not the shame 0.00375 Two titles four years apart impossible to compare 0.00362 Boston reigns supreme

0.01718 Red Sox Win Baseball's World Series Title by Sweeping Rockies 0.01712 Red Sox Sweep Rockies To Win World Series 0.01647 World Series: Red Sox sweep Rockies 0.01630 Red Sox sweep Rockies, take World Series 0.01608 Red Sox 4, Rockies 3 Boston Sweeps World Series Again 0.01597 World Series: Red Sox complete sweep of Rockies 0.01584 Red Sox sweep World Series 0.01579 Red Sox Sweep Colorado in World Series 0.01573 Red Sox Complete Sweep Of Rockies For World Series Victory 0.01531 Red Sox complete World Series sweep ... 0.01057 Boston Red Sox blank Rockies to clinch World Series 0.01052 Red Sox: Dynasty in the making 0.01037 Sox sweep Rockies for 2nd title in 4 seasons 0.01034 Police Arrest Dozens After Red Sox World Series Win 0.01027 Rookies respond in first crack at the big time 0.01018 Rockies: Sweep, sweep, swept 0.01016 Sweeping off to Boston 0.01013 Rookies rise to occasion! 0.01012 Fans celebrate Red Sox win 0.01010 Short wait for bosox this time ... 0.00441 Sox are kings of diamond 0.00414 Rockies just failed to execute 0.00408 Rockies Find Being Good Isnt Enough 0.00407 Rockies' heads held high despite loss 0.00391 Boston lowers the broom 0.00390 Rockies Vanish In Thin Air 0.00390 Poor pitching, poorer hitting doom Rockies 0.00390 Rockies feel the pain, but not the shame 0.00375 Two titles four years apart impossible to compare 0.00362 Boston reigns supreme

Stationary solutions

• The fundamental Ergodic Theorem for Markov chains [Grimmett and Stirzaker 1989] says that the Markov chain with kernel E has a stationary distribution p under three conditions:– E is stochastic

– E is irreducible

– E is aperiodic

• To make these conditions true:– All rows of E add up to 1 (and no value is negative)

– Make sure that E is strongly connected

– Make sure that E is not bipartite

• Example: PageRank [Brin and Page 1998]: use “teleportation”

LexRank (Cosine centrality)

1 2 3 4 5 6 7 8 9 10 11

1 1.00 0.45 0.02 0.17 0.03 0.22 0.03 0.28 0.06 0.06 0.00

2 0.45 1.00 0.16 0.27 0.03 0.19 0.03 0.21 0.03 0.15 0.00

3 0.02 0.16 1.00 0.03 0.00 0.01 0.03 0.04 0.00 0.01 0.00

4 0.17 0.27 0.03 1.00 0.01 0.16 0.28 0.17 0.00 0.09 0.01

5 0.03 0.03 0.00 0.01 1.00 0.29 0.05 0.15 0.20 0.04 0.18

6 0.22 0.19 0.01 0.16 0.29 1.00 0.05 0.29 0.04 0.20 0.03

7 0.03 0.03 0.03 0.28 0.05 0.05 1.00 0.06 0.00 0.00 0.01

8 0.28 0.21 0.04 0.17 0.15 0.29 0.06 1.00 0.25 0.20 0.17

9 0.06 0.03 0.00 0.00 0.20 0.04 0.00 0.25 1.00 0.26 0.38

10 0.06 0.15 0.01 0.09 0.04 0.20 0.00 0.20 0.26 1.00 0.12

11 0.00 0.00 0.00 0.01 0.18 0.03 0.01 0.17 0.38 0.12 1.00

d4s1

d1s1

d3s2

d3s1

d2s3

d2s1

d2s2

d5s2d5s3

d5s1

d3s3

Lexical centrality (t=0.3)

d4s1

d1s1

d3s2

d3s1

d2s3

d2s1

d2s2

d5s2d5s3

d5s1

d3s3

Lexical centrality (t=0.2)

d4s1

d1s1

d3s2

d3s1

d2s3d3s3

d2s1

d2s2

d5s2d5s3

d5s1

Lexical centrality (t=0.1)

Sentences vote for the most central sentence!Need to worry about diversity reranking.

d4s1

d3s2

d2s1

DUC 2004 results

Peer code

Task ROUGE-1 ROUGE-2 ROUGE-3 ROUGE-4

141 3 5 2 1 1

142 3 5 1 1 1

143 4 1 2 1 1

144 4 3 1 1 1

145 4 1 2 2 2

Content-based evaluation (ROUGE)

Latent networks

• When there is no explicit network, we can create one.

• Latent networks – generating functions for networks.

• Complex system with one parameter (akin to temperature).

• With a change in temperature, latent links become active (or inactive).

• Examples: social network in a small town; network of business connections.

• Looking for phase transitions.

ap158

t cutoff

degree

counts