Introduction Spam in Society Email Spam IM Spam Text Spam Blog Spamming Spam Blogs.
(with an application of Web Spam detection) CS315-Web Search and Mining Power Laws and...
-
Upload
agatha-evans -
Category
Documents
-
view
216 -
download
0
Transcript of (with an application of Web Spam detection) CS315-Web Search and Mining Power Laws and...
(with an application of Web Spam detection)
CS315-Web Search and Mining
Power Laws and Rich-Get-Richer Phenomena
What do these have in common?
The grades of students in a class. The weights of apples.
The high temperatures in Boston on July 4th. The heights of Dutch men. The speed of cars on I-90.
These measurements are well-characterized by the average and the standard deviation.
Most instances are typical.Seeing an outlier is very surprising.
City populations
1. New York 8,310,2122. Los Angeles 3,834,340 3. Chicago 2,836,6584. Houston 2,208,180 5. Phoenix 1,552,2596. Philadelphia 1,449,634 7. San Antonio 1,328,9848. San Diego 1,266,731 9. Dallas 1,266,372 10.San Jose 939,899
City populations
1. New York 8,310,2122. Los Angeles 3,834,340 3. Chicago 2,836,658
21. Boston, MA 625,087
248. Cambridge, MA 106,038
25,375. Lost Springs, WY 1
A few cities with high population
Many cities with low population
City populations
Cities ordered on population range
Word Frequencies
Power Law: The number of cities with population > k is proportional to k-c.
“fraction of items”
“popularity = k”
Power Law: Fraction f(k) of items with popularity k is proportional to k-c.
f(k) k-c
log [f(k)] log [k-c]
log [f(k)] -c log [k]
y -c x
A power law is a straight line on a log-log plot.
Number of Web page in-links (Broder+)
Examples (some better than others)
frequency of words protein-interaction degree distributionInternet (AS) degree distributionseverity of inter-state warsseverity of terrorist attacksfrequency of bird sightingssize of blackoutsbook salespopulation of US citiessize of religionsnumber of citationspapers authoredpopularity of surnamesnumber of web hitsnumber of web links, with cut-offnumber of phone callssize of email address booknumber of species per genus
What is going on?
Nature seems to create bell curves(range around an average)
Human activity seems to create power laws(popularity skewing)
Network Science: Scale-Free Property 2012
“seems to”
How can we use this to… fight spam?
The main idea behind “Spam, Damn Spam and Statistics”Spammers manufacture pages and links to fool search enginesIn this process, they will overdo itTheir actions would likely fall outside the normal human activity
Let’s look for outliers in the power laws!
Web page out-degreesThere are 158,290 pages with out-degree 1301, while according to the overall trend only 1,700 such pages are expected.
Web page in-degreesThere are 369,457 pages have the in-degree of 1001, while according to the trend only 2,000 such pages are expected
Length of the URL’s host
The 100 longest hostnames reveal that 80 of them belong to adult site and 11 refer to the financial and credit related sites
Number of host name resolutions to a single IP
There are 100,000’s host names mapped to a single IP, The record-breaking IP is referred by 8,967,154 host names
Clusters of similar pages (shingling)
The blue group is mainly spam. 15 of 20 largest clusters have 2,080,112 spam pages
The red group has duplicated content, not spam).
Spammers are studious!
Why does data exhibit power laws?
imitation Power law
Can imitation explain the size of the Web parts?
Constructing a model of the Web
1. Pages are created in order, named 1, 2, …, N2. When created, page j links to a page randomly:
1. With probability p, picking a page i uniformly at random from pages 1, …, j-1
2. With probability (1-p), pick page i uniformly at random and link to the page that i links too imitation
randomness
This is the well-studied “preferential attachment” model
of Web generation
The rich get richer
2 b) With prob. (1-p), pick page i uniformly at random and link to the page that i links too
1/43/4
The rich get richer
2 b) With prob. (1-p), pick page i uniformly at random and link to the page that i links too
Equivalently,2 b) With prob. (1-p), pick a page
proportional to its in-degree and link to it
Information cascades and the rich
Information cascade = some people get a little bit richer by chance
and then rich-get-richer dynamics = the random rich people
get a lot richer very fast
Is popularity predictable?
Why is Harry Potter popular?
If we could re-play history, would we still read Harry Potter en masse,
or would it be some other book?
(But then, why JK Rowling had troublespublishing it at first?)
Is popularity… random?
Why “hits” in cultural markets are much more successful than average (and yet so hard to predict)?Can we study it with an experiment?“Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market”14,000 participants randomly assigned to “social influence” and “independent” conditionschose between 48 songs by unknown bandsin 8+1 parallel worlds
Subject
See what othersdownloaded
No information
World 1
World 8
World 0
Music download site – 8+1 worlds
1. “Let’s go driving,” Barzin
2. “Silence is sexy,” Einsturzende Neubauten
3. “Go it alone,” Noonday Underground
10.“Picadilly Lilly,” Tiger Lillies
1. “Let’s go driving,” Barzin
2. “Silence is sexy,” Einsturzende Neubauten
3. “Go it alone,” Noonday Underground
10.“Picadilly Lilly,” Tiger Lillies
18
3
47
2
The best songs never went to the bottom, the worse never became popular. But their order changed a lot.