T.Sharon-A.Frank1

Internet Resources Discovery (IRD)

Intelligent IRD

T.Sharon-A.Frank2

Motivation for Intelligence

“We are drowning in information but starved of knowledge “ John Naisbit

T.Sharon-A.Frank3

Content• Classical IRD characteristics and the

Information food chain

• Agents - Softbots family

• Meta SE - Metacrawler

• Homepage finder - Ahoy!

• ILA – Internet Learning Agent

• Shopbot – Jango et al. See Oren Ezioni’s Web site at:

http://www.cs.washington.edu/research/projects/WebWare1/www/softbots/softbots.html

T.Sharon-A.Frank4

No Time for Intelligence!

Classical IRD Characteristics

• Massive memory and network resources required.

• Amortized over millions of queries per day.

• Minimal cycles devoted to each individual.

• No memory of previous requests.

• Least common denominator service.

T.Sharon-A.Frank5

Classical Information Food Chain

T.Sharon-A.Frank6

Intelligent Information Food Chain

T.Sharon-A.Frank7

Definition: Softbots

• Softbots are intelligent agents that use software tools and services on a person’s behalf.

• Make intensive use of artificial intelligence (AI) techniques: planning, scheduling, learning, etc.

T.Sharon-A.Frank8

Softbot Family Tree

Rodney

Sims

InfoManifold

Occam

Simon MetaCrawler

Ahoy!

ILA

ShopBot

BargainFinder

T.Sharon-A.Frank9

General problems to be solved

• Discovery– How to find new information sources (IS) ?

• Extraction– What to send and how to parse the response ?

• Translation– How to interpret the response in terms of internal

concepts ?

• Evaluation– How to evaluate the quality of IS ?

T.Sharon-A.Frank10

Metacrawler

Ahoy!

Discovery, Evaluation:

Extraction:

Translation:

Main Focus of the Robots

ILA

T.Sharon-A.Frank11

Meta Search Engine

MetaCrawler

Yahoo Web Crawler Open Text Lycos InfoSeek Inktomi Galaxy Excite

T.Sharon-A.Frank12

Search Service - Motivation

1. The number and variety of Search services.2. Each service provides an incomplete snapshot of Web.3. Users are forced to try and retry their queries across

different indices.4. Each service has its own interface.5. Irrelevant, outdated or unavailable responses.6. There is no time for intelligence.7. Each query is independent.8. No individual customization.9. The result is not homogenized.

T.Sharon-A.Frank13

The Web Community Demands

• Robustness– A working system, accessible 24 hours a day.

• Speed– Transmitting useful information within seconds.

• Added Value– Any increase in sophistication had better yield a

tangible benefit to users.

T.Sharon-A.Frank14

Premises of MetaCrawler

• No single search is sufficient.

• Problem in expressing the query.

• Low quality references can be detected.

T.Sharon-A.Frank15

MetaCrawler

T.Sharon-A.Frank16

MetaCrawler is a Meta-Service

• It doesn’t use a database of its own.

• It uses other external search services that provide the information necessary to fulfill user queries.

T.Sharon-A.Frank17

MetaCrawler Advantages

• It access multiple databases and provides large number of higher quality references.

• It does not depend upon the implementation or existence of any specific search service.

• It access the search services simultaneously.

• Users need not remember the address, interfaces, … of each search service.

T.Sharon-A.Frank18

How It Works?

• It currently accesses a few services: InfoSeek, Lycos, WebCrawler, Yahoo, etc.

• It submits a query to every search service it knows in parallel.

• It collates the results by merging all hits returned.

• It has a sorting and verify option.

• It presents a results page consisting of a list of references.

T.Sharon-A.Frank19

Meta-Search

• http://www.metacrawler.com

T.Sharon-A.Frank20

Meta Search Results