CS 345A Data Mining Lecture 1 Introduction to Web Mining.
-
date post
19-Dec-2015 -
Category
Documents
-
view
221 -
download
2
Transcript of CS 345A Data Mining Lecture 1 Introduction to Web Mining.
![Page 1: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/1.jpg)
CS 345AData MiningLecture 1
Introduction to Web Mining
![Page 2: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/2.jpg)
What is Web Mining?
Discovering useful information from the World-Wide Web and its usage patterns
![Page 3: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/3.jpg)
Web Mining v. Data Mining
Structure (or lack of it) Textual information and linkage structure
Scale Data generated per day is comparable to
largest conventional data warehouses Speed
Often need to react to evolving usage patterns in real-time (e.g., merchandising)
![Page 4: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/4.jpg)
Web Mining topics
Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 5: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/5.jpg)
Web Mining topics
Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 6: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/6.jpg)
Size of the Web
Number of pages Technically, infinite Much duplication (30-40%) Best estimate of “unique” static HTML
pages comes from search engine claims Google = 8 billion(?), Yahoo = 20 billion
Number of web sites Netcraft survey says 72 million sites
(http://news.netcraft.com/archives/web_server_survey.html)
![Page 7: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/7.jpg)
Netcraft survey
http://news.netcraft.com/archives/web_server_survey.html
![Page 8: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/8.jpg)
The web as a graph
Pages = nodes, hyperlinks = edges Ignore content Directed graph
High linkage 8-10 links/page on average Power-law degree distribution
![Page 9: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/9.jpg)
Structure of Web graph
Let’s take a closer look at structure Broder et al (2000) studied a crawl of
200M pages and other smaller crawls Bow-tie structure
Not a “small world”
![Page 10: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/10.jpg)
Bow-tie Structure
Source: Broder et al, 2000
![Page 11: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/11.jpg)
What can the graph tell us?
Distinguish “important” pages from unimportant ones Page rank
Discover communities of related pages Hubs and Authorities
Detect web spam Trust rank
![Page 12: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/12.jpg)
Web Mining topics
Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 13: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/13.jpg)
Power-law degree distribution
Source: Broder et al, 2000
![Page 14: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/14.jpg)
Power-laws galore
Structure In-degrees Out-degrees Number of pages per site
Usage patterns Number of visitors Popularity
![Page 15: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/15.jpg)
The Long Tail
Source: Chris Anderson (2004)
![Page 16: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/16.jpg)
The Long Tail
Shelf space is a scarce commodity for traditional retailers Also: TV networks, movie theaters,…
The web enables near-zero-cost dissemination of information about products Action moves from Hits to Niches
![Page 17: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/17.jpg)
The Long Tail
More choice necessitates better filters Recommendation engines (e.g., Amazon) How Into Thin Air made Touching the
Void a bestseller In fact, page rank can be seen as a
long tail filter Tapping into the Wisdom of Crowds
![Page 18: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/18.jpg)
Web Mining topics
Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 19: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/19.jpg)
Extracting Structured Data
http://www.simplyhired.com
![Page 20: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/20.jpg)
Extracting structured data
http://www.fatlens.com
![Page 21: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/21.jpg)
Web Mining topics
Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 22: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/22.jpg)
Searching the Web
Content aggregatorsThe Web Content consumers
![Page 23: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/23.jpg)
Ads vs. search results
![Page 24: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/24.jpg)
Ads vs. search results
Search advertising is the revenue model Multi-billion-dollar industry Advertisers pay for clicks on their ads
Interesting problems What ads to show for a search? If I’m an advertiser, which search terms
should I bid on and how much to bid?
![Page 25: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/25.jpg)
Sidebar: What’s in a name?
Geico sued Google, contending that it owned the trademark “Geico” Thus, ads for the keyword geico couldn’t
be sold to others Court Ruling: search engines can sell
keywords including trademarks No court ruling yet: whether the ad
itself can use the trademarked word(s)
![Page 26: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/26.jpg)
Web Mining topics
Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 27: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/27.jpg)
Systems architecture
Memory
Disk
CPU
Machine Learning, Statistics
“Classical” Data Mining
![Page 28: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/28.jpg)
Very Large-Scale Data Mining
Mem
Disk
CPU
Mem
Disk
CPU
Mem
Disk
CPU…
Cluster of commodity nodes
![Page 29: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/29.jpg)
Systems Issues
Web data sets can be very large Tens to hundreds of terabytes
Cannot mine on a single server! Need large farms of servers
How to organize hardware/software to mine multi-terabye data sets Without breaking the bank!
![Page 30: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/30.jpg)
Web Mining topics
Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 31: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/31.jpg)
Web Mining Project
Lots of interesting project ideas If you can’t think of one please come
discuss with us Data and Infrastructure
Webbase data (older Stanford web crawl) Recent web crawl and server courtesy of
Kosmix
![Page 32: CS 345A Data Mining Lecture 1 Introduction to Web Mining.](https://reader035.fdocuments.us/reader035/viewer/2022062313/56649d395503460f94a125d4/html5/thumbnails/32.jpg)
The World-Wide Web
Our modern-day Library of Alexandria
The Web