Federated Searching

In a Nutshell

21st Century Literacies18 November 2005

Federated Searching

Rex KrajewskiReference Services Librarian

Simmons Collegeweb.simmons.edu/~krajewsk/library/federatedsearching.html

What is Federated Searching?

Process of searching multiple sources simultaneously

More Specifically

Conducted using federated search

engines

Federated Searching AKA

parallel search meta search broadcast search one-search cross searching cross-database searching distributed searching single search

Well-known Models

Dialog allows user to search many databases simultaneously (think: Dialindex One Search Categories)

Metasearch Engines—like Dogpile, Clusty, Mamma, and Metacrawler—allow users to search multiple search engines’ top results with a single search

Federated Searching

The term “federated searching” came from the Open Archives Initiative’s Protocol for Metadata Harvesting (OIA-PMH)

Single server harvests metadata for records from the holdings databases of “federated” databases. The resulting centralized data is searchable.

What’s in a name?

NISO says “metasearching” Vendors prefer “federated

search engines” because metasearching use by meta-search engines like Dogpile, Mamma, and Ask Jeeves.

Federated vs. Meta

Meta search engines aggregate material that has already been searched and is freely available to anyone on the Web.

Federated search engines run searches at the time they are queried and can include proprietary material and other Internet content not spidered by search engines

Why Federated Searching?

Libraries offer hundreds of databases to search

Allows libraries and users alike to manage the hundreds of database search tools.

Digital Reference: Why?

97% of surveyed adult internet users expect to find the information they need on government, health, commerce, and news on the internet.*

*Counting on the Internet: Most expect to find key information online, most find the information they seek,many now turn to the Internet first, Pew Internet& American Life Project,

29 September 2002, Date accessed: 15 November 2005.

Digital Reference: Why?

“Nearly three-quarters (73%) of college students say they use the Internet more than the library, while only 9% said they use the library more than the Internet for information searching.”*

*The Internet Goes to College: How Students are Living in the Future with Today's Technology, Pew Internet& American Life Project,15 September 2002, Date accessed: 03 November 2005.

Digital Reference: Why?

“71% of students report using the Internet at their primary source for their last major project, and they also report accessing online study aids like Sparknotes or CliffNotes.”*

*The Internet and Education, Pew Internet& American Life Project,01 September, 2001, Date accessed: 21 October, 2005

Why Federated Searching

In the age of Google, users expect the world of knowledge available quickly and easily at their fingertips…they expect the same kind of one-stop searching to be available in the library

How It Plays Out

Searcher enters a single search into the federated search engine

women and sports

What does the user see?

A single research entry point

A familiar interface A consistent search syntax

How It Plays Out

Federated search engine translates query into syntax of multiple resource

wom#n and sport*

women and sports

(woman or women) and sport!

women and sports

How do they do it?

Federated search engines translate single search query into syntax of multiple databases:

Z39.50 XML Gateways HTTP Protocol “Others”

How It Plays Out

Individual resources execute search based on the query supplied by the federated search engine wom#n and sport*

(woman or women) and sport!

women and sports

What does the user see?

A friendly message indicating a search is in process

Federated Search Engines

Search multiple databases: E-Journals Abstracting and indexing databases E-Books Web Online catalog(s) Any other searchable online source

How It Plays Out

Federated search engine aggregates results

Results

ResultsResults

Results

Results

Results

Results

+

What does the user see?

Combined results of the search

How It Plays Out

Searcher receives a single list of all the results from all the resources searched by the federated search engine

ResultsResults

What does the user see?

A single, combined list of results:

Value added in results

Federated search engines deliver results from multiple databases in a single list:

Standardized format De-duped Connect to fulltext using link

resolvers Relevancy ranked

What are they selling?

The technology behind federated searching is straight-forward enough to prompt one industry insider to describe it as a “commodity.”

The Major Players

The major players attempt to distinguish themselves by adding value to the basic technology:

Maintaining linksUpdating translators to remain

compatible with search interfacesResults delivery: de-duping, ranking,

sorting, fulltext linking, etc.

Ex Libris MetaLib

http://www.exlibris-usa.com/metalib.htm

Customer List

WebFeat’s Prism

http://www.webfeat.org/products/prism.htm

Customer List

Fretwell-Downing Zportal

http://www.fdusa.com/products/zportal.html

Customer List

Endeavor ENCompass

http://encompass.endinfosys.com/

Customer List

More Major Players

Sirsi Rooms

TDNet TES

MuseGlobal

What’s not to love?

One-stop searching No danger of missing a possible

source of information Users do not have to figure out

where to start…just search them all Those expensive databases won’t

be missed by searchers who could use them

What’s not to love?

The whole process of research—even for scholarly, technical, and professional information—

has been Googlized!

There’s a catch, right?

While the search may be quick and broad, it is

neither precise nor deep

Not for Power Searching

The searching syntax among databases vary:

Truncation, Boolean searching, phrase searching, and proximity searching may be lost

Use of limits is limited Searchable fields may be eliminated—

controlled vocabularies lose their punch

Even keyword searching tough. MS = Microsoft or multiple sclerosis

Clusty Advanced Search

Dogpile Advanced Search

Dogpile Advanced Search vs. Google Advanced Search

May not be all they claim

True de-duping is virtually impossible

Too many variables for reliable relevancy ranking

Sorting—a single basket for apples and oranges?

Still Has Much to Improve

Access and verification—especially “off-site” users

Not all federated search engines can search all sources—not everyone is using the Z39.50 or XML protocol

Expensive and labor intensive

Sample searching

•BPL’s Big Dig

•Duke’s Metasearch

•BC’s MetaQuest

BPL’s Big Dig

Powered by WebFeat http://www.bpl.org/electronic/index.htm

BC’s Metaquest

Powered by ExLibris MetaLib v 2 http://metaquest.bc.edu

Duke’s Metasearch

Powered by ExLibris MetaLib v 3 http://metasearch.library.duke.edu

And, What About…?

Library OPACS/ILS Integration Google Scholar Amazon A9