Evaluating Web Server Log Analysis Tools David Strom [email protected] SD’98 2/13/98.

Evaluating Web Server Log Analysis Tools

David Strom

[email protected]

SD’98 2/13/98

SD'98 (c) David Strom, Inc. 2

Summary

• Examine different log files

• What you can and can’t learn from your logs

• Pros and cons of various tools


Different types of log files

• Access

• Error

• Referral

• Other


Access logs

• Domain name

• Date, time

• Server command processed and result

• URL of visitor

• Bytes transmitted


Sample access log data

• rm258.fav.usu.edu [31/May/1995:09:03:23 +0600] "GET /NEI.html HTTP/1.0" 302 396

• rm258.fav.usu.edu [31/May/1995:09:03:28 +0600] "GET /xculture/nei/nei.html HTTP/1.0" 200 2114

• rm258.fav.usu.edu [31/May/1995:09:03:30 +0600] "GET /gifs/sedlbutton.gif HTTP/1.0" 200 1336

• 129.71.83.161 [31/May/1995:09:20:32 +0600] "GET /RELs.html HTTP/1.0" 304 0

• Leslie-Francis.tenet.edu [31/May/1995:09:36:06 +0600] "GET / HTTP/1.0" 200 1867

• ls973.ulib.albany.edu [31/May/1995:09:40:52 +0600] "GET /viii1.html HTTP/1.0" 404 244


Errors reported in your logs

• Clients that time out (or leave in frustration!)

• Scripts that don’t produce any output

• Server bugs

• User authentication or configuration problems


Sample error log data

• [Thu May 30 07:25:32 1996] send timed out for bamberg.sedl.org

• [Thu May 30 07:57:41 1996] send timed out for kenya.sedl.org

• [Thu May 30 08:23:11 1996] send timed out for ppp092.kyoto-inet.or.jp

• [Thu May 30 09:15:52 1996] access to /usr/local/www/htdocs/scimath/compass/vol03 failed for 170.211.67.51, reason: File does not exist

• [Thu May 30 09:57:56 1996] send timed out for dd10-048.compuserve.com

• [Thu May 30 10:47:25 1996] read timed out for ncia110b.ncia.net


Referral logs

• Who links to your site?

• Who downloads your pages?


Sample referral log data

• http://www.isisnet.com/ ->/change/welcome.html• http://www.ipl.org/ref/RR/EDU/Research-rr.html

->/welcome.html• http://www.tenet.edu/snp/main.html

->/policy/networks/toc.html• http://www.tenet.edu/new/main.html

->/policy/networks/toc.html• http://guide-p.infoseek.com/NS/Titles?qt=teacher+training -

>/resources/SCIMAST/announcement.html• http://www.tenet.edu/new/main.html

->/policy/networks/toc.html• http://www.tenet.edu/new/main.html

->/policy/networks/toc.html• http://www.nwrel.org/national/regional-labs.html

->/welcome.html


Common log format

• Output by most standard servers

• Needed by most third-party log analyzers• hoohoo.ncsa.uiuc.edu/docs/setup/httpd/Overview.html


Extended/custom log formats

• Log whatever you wish in whatever order you wish

• Useful if you will read them regularly!

• But can’t work with the analyzers

• Now in IIS v4, NSCP v3, others.


What you can learn from your log files

• Hits per day

• Domain origins

• The path people take in and around your web

• Problem areas


HITS

• (How Idiots Track Success)

• Nobody uses this word anymore

• Doesn’t really measure individual users, just access

• Catching servers and proxies mess up these statistics


Domain origins

• Where users are coming from -- sometimes

• Just because they are from ibm.net doesn’t mean they work at IBM!

• Forgotten accounts, friends and family using the account

• Hacked user names

• Proxies don’t help here either


The path people take in and around your web

• Search engines help sometimes

• Which search site was the most popular front door

• Who links to you and why

• Is there a pattern or a random walk?


Problem areas to deal with

• Broken links (locally)

• Broken outbound links

• Time outs (sunspots?)


What you can’t learn from your logs

• Who are these people, anyway?– No specific user names– Is it a bot or a real human?

• How long did they view a page?– Most people don’t spend much time on your

web– Where did they go visit next?


What technologies are available?

• Built-in analyzer tools

• Sites that capture user info

• Secure sites with registration

• Build your own from perl

• Third-party tools


Built-in tools

• WebSite, website.ora.com

• IIS with Site Server, www.microsoft.com/iis

• Netscape servers, www.netscape.com

• Easy to use but limited


WebSite Professional v2

• Win NT, 95

• Best web server for learning about logs, best docs

• QuickStats module for instant analysis:– single report but nice set of information– shows today, last two days requests and unique

hosts– IP addresses of visitors, average requests/hour


IIS Site Server

• NT Server v4 w/SP3 only

• Lots of preconfigured reports

• Two versions, Express and Full (customized reports)

• backoffice.microsoft.com/products/siteserver/express/


Netscape v3 web servers

• Various NT, Unix versions

• Reports for a few variables but nothing too extensive

• Best to use a third-party tool here


Sites that capture user info

• WebCounter, www.digits.com -- third-party hit counter

• Someone else does the programming and debugging

• But beyond your control


Secure sites with registration

• You know your users

• But many won’t register, or forget their passwords

• Requires scripting, database integration, more maintenance


Build your own from perl

• Needs some in-house support

• Works best with Unix-based webs

• Examples:– refstats,

members.aol.com/htmlguru/refstats.html– surfreport, bienlogic.com/SurfReport/


Third-party tools

• WebTracker, www.CQMInc.com/webtrack

• WebTrends, www.webtrends.com

• net.Genesis, www.netgen.com

• MarketWave, www.marketwave.com

• IIS Assistant, www.go-iis.com


Third-party tools (con’t)

• Can make very pretty reports

• Customizable

• Make sure they support your particular log format

• Not that expensive, mostly run on Windows

Evaluating Web Server Log Analysis Tools David Strom [email protected] SD’98 2/13/98.

Documents

Transcript of Evaluating Web Server Log Analysis Tools David Strom [email protected] SD’98 2/13/98.