A Web Crawler Design for Data Min-ing

Mike ThelwallUniversity of Wolverhampton, Wolverhampton, UKJournal of Information Science 2001

27 April 2011Presentation @ IDB Lab Seminar

Presented by Jee-bum Park

2

Outline Introduction Architecture Implementation System Testing Conclusion

3

Introduction

- Motive

The importance of the web has guaranteed academic interest in it, not only for affiliated technologies, but also for its content

4

Introduction

- Motive

Information scientists and others wish to perform data mining on large numbers of web pages

They will require the services of a web crawler,– To extract patterns from the web– To extract meaning from the link structure of the web

The necessity of an effective paradigm for a web min-ing crawler

5

Introduction

- Web Crawler

A web crawler, robot or spider

A program that is capable of iteratively and automat-ically,– Downloading web pages– Extracting URLs from their HTML– Fetching them

6

Introduction

- Web Crawler: Workflow

/• index.html• login.php

/images/

• logo.gif• menu.jpg• bg.png

/board/

• index.php• index.php?

id=2• Index.php?

id=3

/board/files/

• a.jpg• b.txt• c.zip

http://idb.s-nu.ac.kr/

Web Crawler

7

Introduction

- Web Crawler: Architecture

8

Introduction

- Web Crawler: Roles

A sophisticated web crawler may also perform,– Identifying pages judged relevant to the crawl– Rejecting pages as duplicates of ones previously visited– Supporting the action of search engines

For example, constructing the searchable index

9

Introduction

- Web Crawler: Issue

In the normal course of operation,a simple crawler will spend most of its time awaiting data– Requesting a web page– Receiving a web page

For this reason, crawlers are normally multi-threaded If the crawling task requires more complex process-

ing,the speed of the crawler will be reduced

A distributed approach for crawlers is needed

10

Introduction

- Distributed Systems

Using idle computers connected to the internet– To gain extra processing power– To distribute processing power

For personal site-specific crawlers, a single personal computer solution may be fast enough

An alternative is a distributed model– A central control unit– Many crawlers operating on individual personal computers

11

Outline Introduction Architecture Implementation System Testing Conclusion

12

Architecture The crawler/analyzer units The control unit

Four constraints1. Almost all processing should be conducted on idle com-

puters2. The distributed architecture should not increase network

traffic3. The system must be able to operate through a firewall4. The components must be easy to install and remove

13

Architecture

Control unit

Crawleridb.snu.ac.kr

Crawlerbrahma.snu.ac.kr

Crawlersugang.snu.ac.kr

Crawleretl.snu.ac.kr

Crawlermy.snu.ac.kr

Crawlersiva.snu.ac.kr

14

Architecture

- The Crawler/Analyzer Units

The program– Crawl a site or set of sites– Analyze the pages– Report its results

It can execute on the type of computers on whichthere will be spare time, normally personal comput-ers

15

Architecture

- The Crawler/Analyzer Units: Data Management

Accessing permanent storage space to save the web pages– Linking to a database– Using the normal file storage system

Pages must be saved on each host computer,in order to minimize network traffic

If the system is capable of handling enough data,a large-scale server-based database can be used

It must provide a facility for the user to delete all saved data

16

Architecture

- The Crawler/Analyzer Units: Interface

Immediate stop

Clear all data from the computer

17

Architecture

- The Control Unit

The control unit will live on a web server

When a crawler unit requests a job or sends some data,It will be triggered

It will need to store the commands– The owner wishes to be executed– Indicating status

Completed In progress Unallocated

18

Architecture

Control unit

Crawleridb.snu.ac.kr

Crawlerbrahma.snu.ac.kr

Crawlersugang.snu.ac.kr

Crawleretl.snu.ac.kr

Crawlermy.snu.ac.kr

Crawlersiva.snu.ac.kr

19

Outline Introduction Architecture Implementation System Testing Conclusion

20

Implementation

- The Crawler/Analyzer Units

The architecture was employed to create a system for analyzing the link structure of university web sites

21

Implementation

- The Crawler/Analyzer Units

Previous system– Running a single crawler/analyzer program

Issues– Not run quickly enough– Individually set up and run on a number of computers– Inefficient in terms of both human time and processor use!

New system– The existing stand-alone crawler was used as the basis– Communication and easy installation features added– Buttons to instantly close the program and remove any

saved data– Processed by compressor for easy distribution

22

Implementation

- The Crawler/Analyzer Units

Choice of the types of checking for duplicate pages– No page checking– HTML page checking– Weak HTML page checking

Comparing methods– Comparing each page against all of the others

Naive

– Various numbers were calculated from the text of each page For example, the length of the page, MD5 or SHA-1 hash, etc.

23

Implementation

- The Control Unit

Entirely new!

It was given a reporting facility– Statistics– To deliver a summary of crawlers

24

Outline Introduction Architecture Implementation System Testing Conclusion

25

System Testing In June and July of 2000

A set of sites or web pages to download An analysis to perform on the downloaded sites

26

System Testing

- Result

The total number of crawler units– Peaked at just over 100 with three rooms of computers

9112 tasks completed by the system Over 100,000 pages downloaded

Each crawler used approximately 1 GB of hard disk space

The system had become a virtual computer with over100 GB of disk space and over 100 processors

27

System Testing

- Limitations

The system was not able to run fully automatically

The problem was randomly generated web pages– For example, a huge set of web pages containing usage sta-

tistics for electronic equipment with one page per device per day

The solution was– To manually check the root cause of the problem– To add their URLs to a banned list operated by the control

unit

There is the alternative of designing a heuristic to avoid problems– For example, a maximum crawl depth

28

Outline Introduction Architecture Implementation System Testing Conclusion

29

Conclusion The distributed architecture has shown itself

– Capable of crawling a large collection of web sites– By using idle processing power and disk space

The testing of the system has shown that– It cannot operate fully automatically– Without an effective heuristic for identifying duplicate pages

30

Conclusion The architecture is particularly suited to situations

– Where a task can be decomposed into a collection of crawl-ing based tasks

It would be unsuitable if– The crawls had to cross-reference each other– The data mining had to be performed in an integrated way

The architecture is an effective way to use idle com-puting resources in order to perform large-scale web data mining tasks

Thank You!Any Questions or Comments?