server tools

Site server tools can be utilised to build, host, track and monitor transactions on a business site.

• There are a wide range of possibilities but today we are looking at Web server logs.

• Server based statistics analysis tools utilise server logs for their information

Built in web server logging tools

• Web servers such as Apache and MS IIS provide built-in site activity logging functionality to provide information such as:

• Who has accessed the website?• When was it accessed?• What was accessed?• This information can make it possible to identify

bottlenecks, security breaches, content popularity and facilitate troubleshooting.

File format of the web log files

• The W3C specification for extended log file format, describes a number of log fields that can be used for web server activity logging. The specification calls for an ASCII file format and time is logged as UTC (Greenwich Mean Time).

• Although different servers have different log formats they all look very similar

Information in Server LogsWeb server logs are plain text (ASCII) files, independent of server platform.

There are some differences between server software, but traditionally there are four types of server logs:

1. Transfer (access) log 2. Error log 3. Referrer log 4. Agent log The first two types of log files are standard. The referrer and agent logs may

or may not be “turned on” at the server or may be added to the transfer log file to create an “extended” log file format. Each HTTP protocol transaction, whether completed or not, is recorded in the logs, and some transactions are recorded in more that one log. For example, most (but not all) HTTP errors are recorded in the transfer log and the error log.

Transfer (access) log

• The line below is an example of a single line in a common transfer log. This typically displays as one long line of ASCII text, separated by tabs and spaces (useful for importing it into a spreadsheet program or other analytical program).1Cust216.tnt1.tpg.com.au - -[08/May/2007:12:13:03 -0700] GET /gen/meeting/ssi/next/HTTP/1.0 200 9887 http://www.north.tafe.edu/ Mozilla/7.01-C-MACOS8 (Macintosh; I; PPC) GET /gen/meeting/ssi/next/ - HTTP/1.0

Lets look at each section of this entry.

Remotehost Address or IP 1Cust216.tnt1.tpg.com.au

• This is the address of the computer making the HTTP request. The server records the IP and then, if configured, will lookup the Domain Name Server (DNS). However, with all the dynamically assigned IP addresses these days, you don’t learn as much as you’d expect from the domain name. In this case the visitor seems to be a customer of an ISP.

RFC931 (or identification) -

• Rarely used, the field was designed to identify the requestor. If this information is not recorded, a hyphen (-) holds the column in the log.(The remote logname of the user)

Authuser-

• List the authenticated user, if required for access. This authentication is sent via clear text, so it is not really intended for security. This field is usually filled by a hyphen (-).

Time Stamp[08/May/2007:12:13:03 -0700]

• The date, time, and offset from Greenwich Mean Time (GMT x 100) are recorded for each hit. The date and time format is: DD/Mon/YYYY HH:MM:SS. The example above shows that the transaction was recorded at 12:13 pm on May 8, 2007 at a location 7 hours behind GMT.

requestGET /gen/meeting/ssi/next/index.html HTTP/1.0

• The request line exactly as it came from the client

• GET is the standard request for a document or program. POST tells the server that data is following. HEAD is used by link checking programs, not browsers, and downloads just the information in the HEAD tag information. The specific level of HTTP protocol is also recorded.

status 200

• The resulting status code. "200" is success. If the requested URL didn't exist, this is where the dreaded "404" would have shown up in the log.

• Response codes fall into 4 groups• Success (200 series) • Redirect (300 series) • Failure (400 series) • Server Error (500 series) • A status code of 200 means the transaction was successful. Common 300-series

codes are 302, for a redirect from http://www.mydomain.com to http://www.mydomain.com/, and 304 for a conditional GET. This occurs when the server checks if the version of the file or graphic already in cache is still the current version and directs the browser to use the cached version. The most common failure codes are 401 (failed authentication), 403 (forbidden request to a restricted subdirectory), and the dreaded 404 (file not found) messages. Sever errors are red flags for the server administrator.

example server response codes

• 200- Request successful• 206 - Partial request successful (not complete)

302 - URL has been redirected to another document 400 - Bad request was made by the client 401 - Authorization is required for this document 403 - Access to this document is forbidden 404 - Document not found 500 - Server internal error 501 - Application method (either GET or POST) is not implemented 503 - Server is out of resources

Transfer Volume9887

• For GET HTTP transactions, the last field is the number of bytes transferred. For other commands this field will be a hyphen (-) or a zero (0).

• The transfer volume statistic marks the end of the common log file. The remaining fields make up the referrer and agent logs, added to the common log format to create the “extended” log file format. Lets look at these fields.

Referer URLhttp://www.north.tafe.edu/

• The referrer URL indicates the page where the visitor was located when making the next request. The actual request is shown in the last field of the entry

• GET /gen/meeting/ssi/next/ - HTTP/1.0 and is duplicated from the HTTP Request, the fifth field in this log.

• If you were looking at just the referrer log, not integrated into the transfer log, it would be made up of just two fields. The left field is the starting URL and the right field is where the reader went from the URL. Transfers within your site would also show in the transfer log. For example, movement from one page to another within a web site might show in the referrer log as:

• http://www. north.tafe.edu/ -> /gen/meeting/ssi/next/ The visitor went from the top-level page to another deeper level page through a link on the page.

User AgentMozilla/7.01-C-MACOS8 (Macintosh; I; PPC)

• The user agent is information about the browser, version, and operating system of the reader. The general format is:

• Browser name/version (operating system) The confusion comes from the word “Mozilla,” which is the original code name for Netscape. Now almost all browsers compatible with Netscape use the Mozilla code.

Exercises• 1. Interpreting Transaction Log File Data• Take a look at the following excerpt from a real extended-format web server log file

and answer the questions on the next slide• Transaction #1• dejh.ipm.ac.ir - - [08/May/2007:00:47:07 -0700] "GET /spires/form/hepfnal.html

HTTP/1.0" 200 3529 "http://www-spires.slac.stanford.edu/spires/forms.html" "Mozilla/4.05 [en] (Win2003; I)" GET /spires/form/hepfnal.html - "HTTP/1.0"

• Transaction #2• 202.41.102.153 - - [08/May/2007:02:11:25 -0700] "POST /cgi-bin/form-mail.pl

HTTP/1.1" 200 649 "http://www.slac.stanford.edu/spires/find/hepnames/wwwupd?ID=RCV&NODE=PBI.ERNET.IN" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)" POST /cgi-bin/form-mail.pl - "HTTP/1.1"

• Transaction #3• oeias1-p2.telepac.pt - - [08/May/2007:03:16:08 -0700] "GET

/BFROOT/Images/BABAR2.gif HTTP/1.1" 404 360 "http://www.slac.stanford.edu/BFROOT/old-www/Physics/Workshops/wkshp_home.html" "Mozilla/4.0 (compatible; MSIE 4.01; WinXP Pro)" GET /BFROOT/Images/BABAR2.gif - "HTTP/1.1"

Exercise Questions

1. How many visitors are reflected in these web transactions? Explain.

2. On what date did these transactions take place? 3. How many minutes passed between the first and last entry?

Were all three transactions successful? If not, explain. 4. Which transaction requested the largest file? 5. What size was the file? 6. What browsers are being used to access these pages? 7. What platforms are being used to access these pages? 8. Can you determine the path the visitor from 202.41.102.153

took through the website?

A few things to note :

• Each line in the web log files represents a single "hit" on a file on the web server

• A web server "hit" is not the same as a web page "hit". For example in a log file if there a user requests a page which consists of 3 gifs, 1 jpg then in the log file there will be 4 “hits” or records

• If this page also included a graphic which is being retrieved from a different site then it doesn't actually show up in this site's log.