Evaluation: Preliminary Results from the Server Side

Frank A. SettleElizabeth Blackmer

Thomas Whaley

The Alsos Digital Library for Nuclear IssuesWashington & Lee University

Lexington, Va

User

ServerServer log data

Who? What?How? Why?

Evaluation from the Server Side

Preliminary Observations from the Server Side

• Robots, crawlers, and other automated searches produce over 50% of the visits. While they inflate the number of visitors, they do produce human visits indirectly.

• Search engines (Google and Yahoo) provide the majority of referrals.

• Sites that have embedded, directed links to specific bibliographical information provide over 10% of the referrals. • Referrals from the NSDL portal as well as other sites that include Alsos in their list constitute a small percentage of referrals. However, users may be using a search engine to follow up on an Alsos link they saw in a list on another site.

Month Total visitors

(including robots, etc.)

Visitors from top referrers

per month

Prints

per month

Estimated human visitors per day

Visitors

from NSDL portal

Per month

September 2002

1696 na 410 29 na

March 2003

2619 na 364 44 na

May 2003 2008 na 312 33 na

June 2003 1455 na 260 23 na

August

2003

2489 914 505 43 5

September

2003

4810 1474 1500 89 18

Overall Usage - September 2002 – September 2003

Top Referral Sites – September 2003

Visitors •www.google.com 687• search.yahoo.com 289• chemcases.org 227• search.msn.com 138• www.google.ca 54• aolsearch.aol.com 39• www.atomicarchive.com 38

Embedded Link from Atomic Archive to Alsos

Preliminary Observations from the Server Side

• Printing (hardcopy or files) indicates real engagement of the user. However there are also serious users who do not print so this criteria for estimating engaged users is low.

• Almost all of the visitors listed as referrals from other sites are human users, not robots, crawlers, etc.

• Based on a typical day (Tuesday, September 30, 2003) a large number of users (~72%) who are referred directly to an annotation by another site, print the annotation and exit Alsos immediately.

• User access to the six general browse areas (Issues, Warfare, etc.) indicates an equal distribution of interests. This validates the selection of these areas and shows the broad interests of users.

No 1 6 36 43

72%

Yes 4 6

7 17

28%

Total 5

8%

12

20%

43

72%

60

Did user move beyond printing the annotation? Homepage Search Annotation

Entry Point

Engaged Users’ BehaviorTuesday, Sept. 30, 2003

Distribution of Browse Topics for September 2003(human visitors)

Home Page

Browse Issues Area

Results for Nuclear Waste

Complete Citation with Annotation

Processed Information from Server

• Tabular session displays – summary information for each session

• Individual session display – tracks the user through a session

• Suffix statistics – provides information on types and also locations of international users

• Hit counts for individual references – skewed by robots and multiple hits during a single session

Processed Server Log Data

Problem – hits are skewed by robots, crawlers, etc.

Types of Users

• Robots, crawlers, etc.

• Humans who scan topics, site information, etc., but don’t access specific annotations

• Single reference users referred to Alsos from a search engine or website to a specific annotation. User gets information and exits. Often prints.

• Human browsers who go to a browse area, obtain a list of references, and then examine specific annotations. May or may not print.

• Human searchers who enter keywords(s), title, or creator to trigger a search. Most users in this category do not use the advanced search feature.

Single reference user from Google

Searcher enters keyword “Iran”

Browser in People Area

Casual Visitor – Scans Information

Summary

• Massive amounts of raw server log data must be transformed into smaller, organized sets of information.

• This information can provide useful information on who is using the collection and how they use it.

• This information is useful in improving the utility of the collection and the effectiveness of the user interface.

• This information is critical in seeking funds to sustain and expand the collection