Reporter Administrators Guide 9.x.b

7/21/2019 Reporter Administrators Guide 9.x.b

1/134

Blue CoatSystemsReporter 9.x

Administrators Guide

Reporter Versions 9.4.x


2/134

Blue Coat Reporter 9.x Administrators Guide

ii

Contact Information

Americas:Blue Coat Systems Inc.420 North Mary AveSunnyvale, CA 94085-4121

Rest of the World:Blue Coat Systems International SARL3a Route des Arsenaux1700 Fribourg, Switzerland

http://www.bluecoat.com/contact/customer-support

http://www.bluecoat.com

For concerns or feedback about the documentation:[email protected]
http://www.bluecoat.com/mailto:[email protected]:[email protected]://www.bluecoat.com/


3/134

iii

Copyright 1999-2013 Blue Coat Systems, Inc. All rights reserved worldwide. No part of this document may be reproduced by any means

nor modified, decompiled, disassembled, published or distributed, in whole or in part, or translated to any electronic medium or othermeans without thewritten consent of Blue Coat Systems, Inc. All right, title and interest in andto theSoftware anddocumentation areandshall remain the exclusive property of Blue Coat Systems, Inc. and its licensors. ProxyAV, ProxyOne, CacheOS, SGOS, SG,Spyware Interceptor, Scope, ProxyRA Connector, ProxyRA Manager, Remote Access and MACH5 are trademarks of BlueCoat Systems, Inc. and CacheFlow, Blue Coat, Accelerating The Internet, ProxySG, WinProxy, PacketShaper, PacketShaperXpress, PolicyCenter, PacketWise, AccessNow, Ositis, Powering Internet Management, The Ultimate Internet SharingSolution, Cerberian, Permeo, Permeo Technologies, Inc., and the Cerberian and Permeo logos are registered trademarks of BlueCoat Systems, Inc. All other trademarks contained in this document and in the Software are the property of their respective owners.

BLUE COAT SYSTEMS, INC. AND BLUE COAT SYSTEMS INTERNATIONAL SARL (COLLECTIVELY BLUE COAT) DISCLAIM ALLWARRANTIES, CONDITIONS OR OTHER TERMS, EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, ON SOFTWARE ANDDOCUMENTATION FURNISHED HEREUNDER INCLUDING WITHOUT LIMITATION THE WARRANTIES OF DESIGN,MERCHANTABILITYOR FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL BLUECOAT,ITS SUPPLIERS OR ITS LICENSORS BE LIABLE FOR ANY DAMAGES, WHETHER ARISING IN TORT, CONTRACT OR ANY OTHERLEGAL THEORY EVEN IF BLUE COAT SYSTEMS, INC. HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Americas: Rest of the World:Blue Coat Systems, Inc. Blue Coat Systems International SARL

420 N. Mary Ave. 3a Route des Arsenaux

Sunnyvale, CA 94085 1700 Fribourg, Switzerland

Document Number:Document Revision: Reporter 9.4.1: 11/2012


4/134


iv


5/134

iii

Contents

Chapter 1: Preface

About This Document ........................................................................................................................7Document Conventions .....................................................................................................................7Notes and Warnings...........................................................................................................................7Navigating This Document ............................................................................................................... 8

Chapter 2: Reporter Concepts

Chapter Contents ................................................................................................................................ 9Reporter Overview ............................................................................................................................. 9About the Reporter Architecture......................................................................................................9

About the Page View Combiner.....................................................................................................10Deployment Overview.....................................................................................................................12

Standard FTP DeploymentOne Server................................................................................12Standard FTP DeploymentTwo Servers..............................................................................14Direct ProxySG Streaming Deployment ................................................................................. 15Download Access Log Data from the Blue Coat Cloud........................................................16

About Optimizing Log Processing Configurations .....................................................................17About Access Log Naming Conventions................................................................................17About Chronological Ordering ................................................................................................19About Known Conditions for Efficiency/In-efficiency........................................................19About Database Purging ........................................................................................................... 20

About the Default Browse Time Calculations .............................................................................. 20The Page View Criteria Used for Browse Time .....................................................................20Examples......................................................................................................................................21

Report Field/Log Field Names.......................................................................................................21Log Field Best Practices ............................................................................................................. 22Main Logs .................................................................................................................................... 23

Reports/Log Field Matrix................................................................................................................24Main Log Field Matrix...............................................................................................................26Web Application Reports..........................................................................................................28Video Usage Reports..................................................................................................................28

Chapter 3: Administrative Tasks

How Do I...? .......................................................................................................................................31

Section A: Reporter Administration Tasks

Linux Root User Installation Procedure ........................................................................................ 32Uninstalling Reporter.......................................................................................................................33
http://-/?-http://-/?-


6/134

Blue Coat Reporter 9.x Administrator Guide

iv

About the Reporter Improvement Program.................................................................................33Securing the Reporter Web Server Transport Protocol...............................................................34

Default Certificate ...................................................................................................................... 34

Selected Certificate.....................................................................................................................35Connecting Reporter to E-mail Servers.........................................................................................38Creating ProxySG Policy That Backs Up Access Log Files.........................................................38Processing Log Files With Encoded Spaces in User Names.......................................................43Process Access Logs From the Blue Coat Cloud Service ............................................................ 44

Prerequisites................................................................................................................................44ThreatPulse Configuration .......................................................................................................44Reporter Configuration.............................................................................................................45Create a Database or Assign Log Source................................................................................47

Section B: Reporter Performance Best Practices

Basic Best Practices ........................................................................................................................... 48Reporter Server...........................................................................................................................48Log Processing............................................................................................................................48Managing Data ........................................................................................................................... 49Reporter System Maintenance..................................................................................................49

About Log File Names ..................................................................................................................... 49About UNC Paths.............................................................................................................................50

Section C: Advanced Filtering Tasks

Filtering Based on Custom Text File Contents ............................................................................. 51Filtering Based on IP Addresses with CIDR Notations .............................................................. 51

Section D: TroubleshootingHow do I...?........................................................................................................................................53About Compression Modes in a Direct ProxySG Connection Deployment............................ 53Configuring Reporter to Send Alerts.............................................................................................54Uploading System Diagnostics to Blue Coat................................................................................55Reviewing Reporter Event Logs.....................................................................................................55Troubleshooting HTTPS Configuration on Linux ....................................................................... 57

Chapter 4: Managing User Access to Reporter

About Users.......................................................................................................................................59

Section A: Planning the Role-Based Access

Recommended Database Fields for Roles.....................................................................................60Determining Role Access.................................................................................................................62

Section B: Configuring Reporter Role-Based Access

Section C: Authenticating Users With LDAP

About LDAP and Reporter..............................................................................................................64About LDAP Nested Group Support ............................................................................................ 65


7/134

Contents

v

LDAP Procedure...............................................................................................................................66Use Case: Role-based Access for Managers Viewing Direct Report Data................................67

Section D: Auditing Reporter Users

Chapter 5: Web API

About the Web API ..........................................................................................................................71Additional Support...........................................................................................................................71About Web API Endpoints..............................................................................................................71

Security Requirements...............................................................................................................72Downloading Reports ...............................................................................................................72Parameter Syntax .......................................................................................................................72

Common Parameters........................................................................................................................72Parameter: username.................................................................................................................72Parameter: password ................................................................................................................. 73Parameter: reportId....................................................................................................................73Parameter: responseFormat......................................................................................................73

End Point: /api/create.....................................................................................................................74Parameter: database...................................................................................................................75Parameter: role............................................................................................................................75Parameter: label..........................................................................................................................75Parameter: format.......................................................................................................................75Parameter: summarizeBy..........................................................................................................76Parameter: columns ................................................................................................................... 76Parameter: rows..........................................................................................................................76

Parameter: sort............................................................................................................................76Parameter: action........................................................................................................................77Parameter: filterN.......................................................................................................................77Parameter: graphType...............................................................................................................77Parameter: graphColumns........................................................................................................77Parameter: dateRelativeUnit ....................................................................................................78Parameter: dateStart ..................................................................................................................78Parameter: dateEnd....................................................................................................................78Parameter: showLast .................................................................................................................78

End Point: /api/status.....................................................................................................................79

End Point: /api/cancel .................................................................................................................... 79End Point: /api/download ............................................................................................................. 79End Point: /api/listDatabases........................................................................................................79End Point: /api/listFields ............................................................................................................... 79Debugging .........................................................................................................................................80Relative Dates....................................................................................................................................81Trend Reports....................................................................................................................................81
http://-/?-http://-/?-


8/134

Blue Coat Reporter 9.x Administrator Guide

vi


9/134

7

Chapter 1: Preface

About This DocumentAudience: Network Administrators (Security)

Abstract: This document describes various Reporter components, operationalconcepts, and how to view and manage reports, and provides proceduresassociated with tuning and troubleshooting Reporter performance.

This document assumes you have read and performed the tasks in the Blue CoatReporter 9.4 Initial Configuration Guide; that is, Reporter is installed on adedicated server and one or more ProxySG appliances are forwarding accesslogs by way of FTPthe exceptions are alternate deployment scenarios andupload methods, which are described in this document.

Document ConventionsThis document adheres to the following typographical and document designprinciples.

Notes and Warnings

The following is provided for your information and to caution you againstactions that can result in data loss or personal injury:

Table 11 Document Conventions

Conventions Definition

Italics The first use of a new or Blue Coat-proprietary term.

Courier font Screen output. For example, command line text, file names, andBlue Coat Content Policy Language (CPL).

Courier Italics A command line variable that is to be substituted with a literal

name or value pertaining to the appropriate facet of yournetwork system.

Courier Boldface A Blue Coat literal to be entered as shown.

Arial Boldface Screen elements in the Management Console.

{ } One of the parameters enclosed within the braces must besupplied

[ ] An optional parameter or parameters.

| Either the parameter before or after the pipe character can ormust be selected, but not both.

Note: Information to which you should pay attention.


10/134

Blue Coat Reporter 9.x. Administrators Guide

8

Related Documentation

The following documents are available on the Blue Coat download site:

Blue Coat Reporter 9.4 Release NotesLinked to from the Reporter softwaredownload page.

Blue Coat Reporter 9.4 Initial Configuration Guide

Blue Coat Reporter Online Help System (available through the Reporteruser interface)

These document PDFs are available at:https://bto.bluecoat.com/documentation/pubs/Reporter

The intended reference path is:

Release Notes > Initial Configuration Guide > Help System > AdministratorsGuide

For any Reporter documentation issues, send e-mail to:[email protected].

Navigating This Document

List of Chapters and task descriptions. Chapter 2: "Reporter Concepts"on page 9Describes various Reporter

operations.

Chapter 3: "Administrative Tasks"on page 31Provides various commonadministrative procedures and lists some basic Reporter best practices.

Chapter 4: "Managing User Access to Reporter"on page 59Describeshow to plan and configure Reporter role-based access and externalauthentication (LDAP).

Chapter 5: "Web API"on page 71Describes scripting report creation andgeneration and supported HTTP endpoints.

Important: Critical information that is not related to equipment damage orpersonal injury (for example, data loss).

WARNING! Usedonlyto inform you of danger of personal injury or physical

damage to equipment. An example is a warning against electrostatic discharge(ESD) when installing equipment.


11/134

9


This chapter discusses various components of Blue Coat Reporter.

Chapter Contents

This chapter contains the following sections:

"Reporter Overview"on page 9

"About the Reporter Architecture"on page 9

"About the Page View Combiner"on page 10

"Deployment Overview"on page 12

"About Optimizing Log Processing Configurations"on page 17

"About the Default Browse Time Calculations"on page 20 "Report Field/Log Field Names"on page 22

"Reports/Log Field Matrix"on page 24

Reporter Overview

Blue Coat Reporter is a key component in the Blue Coat Secure Web Gatewaysolution. Reporter generates and displays reports based on Web traffic accesslog data that is sent from one or more gateway ProxySG appliances. Analyzingreports gives insight regarding the integrity of the network and user Web

browsing habits and policy compliance. This allows you to:

Identify possible security threats (such as malware/spyware)

View user activity by user, group, URLs, or other aspect

View blocked Web traffic (such as categories and URLs)

Identify which users consume how much network bandwidth from Webuse

About the Reporter Architecture

The Reporter application accomplishes major tasks:

Processes raw log data received from ProxySG appliances and populatesdatabases.

Manages the databases and generates reports.

Manages the Reporter server functions.

Log processing itself involves the following components:

Log Reader: Reads access log data into memory on the Reporter server.


12/134


10

Page View Combiner (PVC): This sub-component of the log reader attemptsto provide more realistic user browsing statistics by combining the initialrequest and its secondary referral requests as one page count. For detailedinformation about the PVC, see"About the Page View Combiner"on page10.

Log Processor: Populates the databases with the log data.

Figure 21 Access log to database process.

About the Page View Combiner

The Page View Combiner (PVC) is called during Blue Coat Reporter log

processing. The PVC combines multiple HTTP requests that are associated witha single Web page into a single log line. When a user browses to a Web page,most often that page triggers requests for more content, either from the sameWeb server or another server (for example, a media server that stores video orimage content). Rather than regard each of these as separate requests, the PVCcombines all of the bytes into the original request.


13/134


11

Figure 22 PVC concept diagram.

The goals of the PVC are to:

Reduce the number of database entries from the original log file, whichimproves report generation performance.

More closely represent user browsing activity, as each object (requested bythe first page from content servers) is not counted as a separate entry.

It is possible that a Web request that would normally be combined to representone page view might be split into two page views. This occurs when, as a resultof internal processing, the log sources are halted or restarted or the request isrecorded across two log files.

LEGEND

A: Enterprise users

B: Gateway ProxySG appliance

C: Example.com Server Farm: C-1: main server; C-2: ad farm; C-3: media serverD: Reporter server (and log source staging)

E: Reporter Administrator User

PROCESS FLOW

1: An enterprise user initiates a Web request for a news story at: www.example.com.

2: example.comsends further requests to internal servers for advertisement and video content and

returns four data objects:

example.com/main.html

i.example.com/ads/sponsor1.gif

example.com/news/story1.html

example.com/news/video1.asf

3: The ProxySG appliance adds access log entries for requested content pages.4: The ProxySG appliance forwards the requested content to the user (successful policy check).

5: At a scheduled time, the ProxySG appliance sends (FTP) the log files to the Reporter server.

6: The PVC combines the log lines into one page view and saves that in the database. The Reporter user

generates and views a report that contains one page view entry for the original request to

www.example.com.

Internet


14/134


12

If this occurs, no data is lost, but the database contains two page views.Continuing with the example in the previous illustration:

8:40:20 cnn.com/html

8:40:20 i.cnn.com/ads/sponsor1.gif

[------end of log file------------]

[----beginning of new log file----]

8:40:21 cnn.com/news/story1.html8:40:21 cnn.com/news/video1.asf

The first two entries are shown as one page view; the second two as anotherwithin the database. However, they represent a single page view requested by auser.

Requirements

The PVC requires the following fields in the logs:

cs-referer

sc-status

rs(Content-Type)

The Blue Coat-recommended log formats contain these fields (see"ReportField/Log Field Names"on page 22).

If these log fields are not present, no page-view combining occurs, and reportdata represents each and every Web request.

Deployment Overview

This section describes the Blue Coat-recommended deployment of Reporter.

Standard FTP DeploymentOne Server

In this deployment, gateway ProxySG appliances use FTP to send access logfiles to the same server on which Reporter is installed. For this deployment,Blue Coat strongly recommends staging the logs and installing the Reporterapplication on different physical hard disk drives.

Important: HTTPS logs donotcontain the sc-statusfield; therefore, PVCcalculations cannot occur. The field is not included because it would exposepersonal user data (such as bank account information).


15/134


13

Figure 23 Same-server Reporter deployment.

You can install Reporter on the same server to which the ProxySG appliancesends log files if the system resources allow for efficient processing. Refer to theBlue Coat Reporter Sizing Guide.

LEGEND

A: Gateway ProxySG appliance

B: Reporter Server (with network card)

C: Reporter Administrator User

D: Reporter Role User

PROCESS FLOW

1: Enterprise users initiate Web requests (HTTP/HTTPS); they receive content or policy deny notices.

2: Using the mainaccess log format (a group of log field codes), the gateway ProxySG appliance stores

all Web activity in access log files. The ProxySG appliance FTP upload client periodically sends the

raw log files to the Reporter server (a dedicated server that stages the unprocessed log files and

contains the Reporter application).

3: Log files, or sources, are stored in file directories, named according to the source ProxySG appliance.

4: The Reporter Administrator creates a database, which listens for and detects unprocessed log files in

the log source.

5: The processed log data populates the fields in the database.

6: Users access the Reporter application by logging into the Management Console (using the server

network IP address). When a user clicks a report link, Reporter generates the data from the

associated database and displays the report. Administrator users have access to all reports and

configuration options. Role users only have access to log field data that is assigned to them.

Internet

FTP (LAN/WAN)


16/134


14

Standard FTP DeploymentTwo Servers

Enterprises with numerous, larger access log sources require that Reporter isinstalled on a dedicated server, which then uses FTP to retrieve log files fromone or more log file staging servers.

Figure 24 One log server and Reporter server deployment.

Internet

FTP (LAN/WAN)

LEGEND

A: Gateway ProxySG appliance

B: Access Log Staging Server

C: Reporter Server (with network card)

D: Reporter Administrator User

E: Reporter Role User

PROCESS FLOW

1: Enterprise users initiate Web requests (HTTP/HTTPS); they receive content or policy deny notices.

2: Using the mainaccess log format (a group of log field codes), the gateway ProxySG appliances store all Web

activity in access log files. The ProxySG appliance FTP upload clients periodically send the raw log files to the

dedicated log file staging server.

3: Log files, or sources, are stored in file directories, named according to the source ProxySG.

4: The Reporter Administrator creates a database on the Reporter server; the database listens for and detects

unprocessed log files in the log source directories on the log file server.

5: When Reporter detects unprocessed log data in the log source, it retrieves (FTP operation) the log files and

populates the fields in the database.

6: Users access the Reporter application by logging into the Management Console by using a Web browser and

entering the server network IP address and the Reporter port number. When a user clicks a report link, Reporter

generates the data from the associated database and displays the report. Administrator users have access to all

reports and configuration options. Role users only have access to log field data that is assigned to them.


17/134


15

Direct ProxySG Streaming Deployment

You can configure a ProxySG appliance to stream real-timelog data to aReporter system. Unlike the standard FTP deployment, you cannot archiveprocessed log files for back up purpose or future reprocessing. Furthermore, ifReporter encounters an issue and cannot continue to process log data, the

ProxySG stream begins caching the data to one of its local disks. If the ProxySGappliance is able to reconnect to the Reporter server before the cache fills tocapacity, the ProxySG appliance sends the backlog and processing continues. Ifthe ProxySG appliance is not able to reconnect before the ProxySG appliancelocal disk cacheoverflows, the ProxySG appliance begins overwriting the oldestdata file and the data is not recoverable for processing.


18/134


16

Download Access Log Data from the Blue Coat Cloud Security

Service

If your enterprise has a Blue Coat Cloud Security Service (ThreatPulse) accountand is sending Web (HTTP/HTTPS) traffic for policy checks and reporting, youcan configure Reporter to download the cloud access logs for local processing.

This reporting provides flexibility across your enterprise.Communication between Reporter and the cloud service requires an API key,which is created in the cloud service interface. The key secures the link betweenReporter and the your cloud service account.

Reporter downloads cloud log data over a secure (HTTPS) connection to theDestination Directory that you specify (the procedure linked below describes this).

Upon the first successful communication with the cloud service, Reporterdownloads all available log data. After that, Reporter only downloads new logdata.

LEGEND

A: Users at the corporate office: Proxy Forwarding Access Method.

B: Branch office employees: Firewall/VPN Access Method (IPsec).

C: Remote Users: Client Connector.

D: Reporter Administrator/Reporting User

PROCESS FLOW

1: Corporate location users send their Web requests through the gateway ProxySG appliance, which connects to

the cloud service.

2: Branch office employees connect to the cloud service through an on-premise firewall device (IPsec); a remote

user connects through the Client Connector application, which is installed on their system.

3: Reporter receives logs from the ProxySG appliance (FTP) and from the cloud service (API key/HTTPS).

NOTE:The Reporter server must have Web access.


19/134


17

Each access log contains a one hour segment of data. Reporter saves log files inthe Destination Directorywith date-formatted file names similar to:

cloud_###_##############.log.gz.

The second numerical portion of the sequence represents the following date/time information:

Year/Month/Day/Hour/Minute/SecondFor example, a sequence of201211221200means: This log file was collected bythe cloud service at 2012November22nd12pm.

To configure this method, see"Process Access Logs From the Blue Coat CloudSecurity Service"on page 46.

About Optimizing Log Processing ConfigurationsThis section describes some conditions that affect log processing efficiency.

About Access Log Naming Conventions

This section provides suggestions for ProxySG appliance access log namingconventions, especially for deployments that require processing a large numberof log files over a longer duration of time.

For optimal Reporter performance, configure your access logs to use thefollowing filename format:

xxxxxxxxxxxxxxxNddddddddddd.log.gz

where: xrepresents any valid character that can be used in naming a log file

(letters, digits, underscore, dash).

Nrepresents a non-decimal-digit character.

drepresents a decimal digit. This number, preceding the log fileextension, determines the order in which the log files are processed. Thelog file ordering is performed identically for FTP, cloud download, andlocal disk log sources. A date string representing the log line dateswithin the file is preferred. If you mix cloud files with on-premise files,use the 12 digit cloud date syntax described above.

.log.gzis the extension of the (compressed) log file.

DECIMAL DIGIT NOTES

The decimal digit number is the key part of the format.

If this number does not provide a complete ordering on the set of log files,then the log processing speed suffers because of internal log tablethrashing.

Note: The cloud service prevents the downloading of access logs that are lessthan two hours old.


20/134


18

A filename format ofMMDDhhmmssis inadequate because the files processchronologically, except at year-end when they temporarily process out-of-order because of the December (MM= 12) rollover into January (MM= 01)where January files sort before December.

A filename format ofhhmmssis more problematic because log files are

processed out-of-order whenever one day rolls into the next. Given these constraints, to ensure the most efficient log file ordering, format

this eleven-digit number as: YYJJJhhmmss, where:

YY= two-digit year (00 99)

JJJ= three-digit Julian day of the year (001 366)

hh= two-digit hour of the day (00 23)

mm= two-digit minute of the hour (00 59)

ss= two-digit second of the minute (00 59)

Using this format allows Reporter to properly order log files through the

year 2021. The default filename format used for log files on the ProxySG has the

following text and specifiers: SG_%f_%c_%l%m%d%H%M%S.log.gz.

%f= log name (facility)

%c= name of the external certificate used for encryption, if any

%l= the fourth parameter of the ProxySG IP address (101.102.103.104)

%m= two-digit month (01 12)

%d= two-digit day (01 31)

%H= two-digit hour (00 23)

%M= two-digit minute (00 59)

%S= two-digit second (00 59)

.log.gz= extension

The suggested filename format for log files on the ProxySG applianceslightly alters the default and has the following text and specifiers:SG_%f_%c_%l%m%d_%y%j%H%M%S.log.gz.

%y= two-digit year, without century (00 99)

%j= three-digit Julian day within year (001 366)

The value of this naming convention for log files is very evident whenprocessing large numbers of log files (spanning multiple days and months)occurs. The value is less evident when log file generation and processingoccurs regularly (daily or more frequently) so that out-of-order files occurinfrequently. However, when re-processing large sets of log files, thenaming convention is essential.


21/134


19

About Chronological Ordering

Each database creates and manages its own memory resident LogTable. EachLogTable is comprised of hour-tables containing data for each hour thedatabase LogProcessors spend reading log files. These tables constitute some ofthe most active memory in Reporter, and therefore have a significant impact on

overall log processing performance. If all log files were processed inchronological order, there would never be more than one hour-table necessaryin memory. It is common for the log processing process to encounter batches oflog files spanning multiple hours between them. If they are processed out ofchronological order, performance significantly improves by allowing thenumber of hour-tables to grow, provided there is sufficient process memory.Conversely, during low memory conditions, reducing the number of hour-tables prevents unnecessary memory starvation and subsequent diskoperations (swapping files in and out of memory).

Reporter orders log files based on a numeric field in the filename, when it ispresent. The field is part of the filename format described in the AccessLogging chapter (see Configuring the Upload Client) of theBlue Coat SGOSAdministration Guide(in pre-SGOS 5.5.x documention, Access Logging is itsown volume PDF). The default filenames created by the ProxySG contain aMonth/Day/Hour/Minute/Secondtimestamp immediately preceding the .logor.log.gzsuffix; for example: SG_Main_HQ-1_1102081500.log.gz. If the filenameends with .logor .log.gz, the LogProcessor parses it for any purely numericsequence immediately preceding the required suffix. If one is found, it is thenused to sequentially order that batch of log files. You can significantly improveLogProcessor performance by naming the log files with any ordered numericvalues that comply with this format. For example:anyfilenameprefix123.logorsome-other-prefix-84757.log.gz.

About Known Conditions for Efficiency/In-efficiency The many variables involved in processing log files prevents the ability to

present a comprehensive and complete set of recommended configurationsettings. Some of these variables include:

64 bit versus 32 bit operation systems and hardware.

Variant log file sizes, small to extremely large (dozens of gigabytes).

Available memory for Reporter resources.

Disk speed: setting up a stripped array or SAN might improveperformance.

In addition to knowledge of your systems, understanding the followingconditions that both aid and hinder Reporter log processing functionalityenables you to modify configuration options to optimize efficiency.

Known Condition for Efficient Processing

Do not run other applications on the Reporter server.

Adhere to the system guidelines set in theBlue Coat Reporter Sizing Guide.


22/134


20

Known Conditions for In-efficient Processing

Having insufficient memory to retain all of the active data.

Consuming extra time to write processed data and inactive data frommemory to disk.

Reporter runs well, but other errors occur: Data is not available for report generation because it has not been

written to disk yet.

Other applications on user systems suffer from Reporters resource use.

About Database Purging

Most of the database is kept in memory. If the entire database is notoccasionally purged, it would continue to consume more of the processmemory as new log files are processed. As the database grows, configurationsettings that were previously beneficial might become detrimental.

As a general guideline, Blue Coat recommends that databases contain amaximum of 30 days of log data. However, the amount of log dataismorerelevant than the number of days in the data sets.

Reporter also allows the administrator to purge the database based on thenumber of log lines. Log lines can be purged by expiration, automatically(scheduled) and manually. This task can also be performed on demand asthe administrator does not need to schedule the task but can set a custompurge limit.

About the Default Browse Time Calculations

Some reports provide a datapoint called Browse Time. The intention of thisstatistic is toestimatehow long a user spends browsing a particular website orcategory of a website.

The Page View Criteria Used for Browse Time

Reporter calculates this by matching each source IP address and each user inthe logs with a website. After a match occurs, Reporter tracks the activity ofeach user as seen in the access logs.

As Reporter procesess each log line in each log file, it finds and adds up browsetime for each client IP address. If Reporter determines a request is a page view,the transaction is assigned 30 seconds of browse time. However, if another page

view is discovered within 30 seconds in the Page View Combiner (PVC) cachetime window (10 seconds by default), Reporter subtracts the time of theprevious page view from the next and counts the result. If the next page viewoccurs more than 30 seconds after the previous page view, the previous pageview remains 30 seconds. These page view calculations provides a moreaccurate algorithm than the one used in Reporter 8.x versions, especialy forcases where a user is using two browsers at the same time or has backgroundapplications making requests.


23/134


21

Reporter calculates browse timein real timeduring log processing.Furthermore, Reporer can only subtract the time difference from the last pageview if it still exists in the PVC cache. For example, if a Reporter administratorsets the default browse time to 60 seconds per page and leaves the PVC cachetime windows to 10, the 60 second value applies by default unless another pageview is found for the same client IP address and user agent within the 30

second PVC window. Therefore, you could have pages with anywhere betweenzero and 30 seconds or 60 seconds of browse time. Typically, the default browsetime is set to 30 seconds by default, which means all pages have a browse timefrom zero to 30 seconds, but never more.

For related information, see"About the Page View Combiner"on page 10.

Examples

This sub-section provides browse time examples.

Example Notes

Scenario 1: Employee A visits cnn.com for 40 seconds, visits yahoo.com for 20seconds, and then leaves his browser on youtube.com for 2 minutes but doesnot watch a video or click links in the site.

Reporter calculates 30 seconds for cnn.com, 20 seconds for yahoo.com,and 30 seconds for youtube.comfor a total browse time of 80 seconds.

If, however, the same user browses videos on youtube.comevery 29seconds, the resulting browse time is 30 seconds for each video,resulting in a total browse time of 120 seconds.

Scenario 2: Employee A opens two concurrent browsersInternet explorerand Firefoxat the same time and performs the above scenario. The resultis double.

Scenario 3: Employee A uses the same browser. By default all page viewsare given the default browse time, which is 30 seconds (this value isconfigurable). If Reporter processes another page view from the same clientIP address on the same user agent while the first page view is still in thePVC cache (which is also a 30 second window by default), Reporter lowers

the browse time for the first page view to the time difference between thepage views.

Scenario 4: Employee B vists images.google.comfor 5 seconds and thenclicks a picture, views it for 15 seconds, clicks the back button, clicks adifferent picture, and views it for 45 seconds. Reporter records 5 seconds forimages.google.com, then 15 seconds for the the first picture (plus whatevertime it takes to click back and click on the second pic), and then 30 secondsfor the last picture.

Note: These examples assumes the default values of 30 seconds for browsetime with a default PVC cache of 30 seconds or fewer. For example, if a uservisits cnn.com and never loads another page (does not click through the variousarticles links) for three hours, the resulting browse time is 30 seconds.


24/134


22

Report Field/Log Field Names

This section provides a reference table that lists the report field to log fieldassociation. Report fields are what comprise various reports, based on theinformation contained in the access log. The contents of an access log aredetermined by the log field names (which determine what data types are

captured during the ProxySG logging process). Some log field names correlateto absolute data (such as URLs), others derive information from access logvariables (such as browsing duration).

Log Field Best Practices

Certain access log fields are critical to proper Reporter operation.

To prevent Reporter from disregarding some log lines, the Reporter maindatabases require these fields: cs-host,cs-uri-host or cs-uri-hostname

sc-status

cs-uri-scheme

c-ip, x-client-ip, x-client-address, c-dns or x-cs-username-or-ip

rs(Content-Type)

sc-filter-result or x-exception-id

x-virus-id

For the PVC to operate correctly, Reporter requires these additional fields:

cs(Referer) or x-cs(Referer)-uri

x-exception-id or sc-filter-result (x-exception-id preferred)

sc-filter-category, cs-category, or cs-categories

For the PVC to operate correctly for video reports, Reporter requires theseadditional fields: cs-host cs-uri-host or cs-uri-hostname

cs-uri-scheme

c-ip, x-client-ip, x-client-address or c-dns, x-cs-username-or-ip

sc-status

sc-filter-result, or x-exception-id

x-virus-id

cs-method

time-taken

cs-uri-scheme s-session-id

To properly populate all default Dashboard reports, Reporter requires thesefields in addition to those above:

Note: For more PVC information, see"About the Page View Combiner"onpage 10.


25/134


23

cs-username, x-cache-user, cs-userdn, x-radius-splash-username, x-

cs-session-username or x-ldap-attribute(displayName)

cs-category, sc-filter-category or cs-categories

sc-filter-result or x-exception-id

cs-host cs-uri-host or cs-uri-hostname

x-bluecoat-application-name

x-bluecoat-application-operation

To populate all default video reports, Reporter requires these fields: cs-host cs-uri-host or cs-uri-hostname

c-ip, x-client-ip, x-client-address, c-dns or x-cs-username-or-ip

x-cache-info

cs-auth-group or cs-auth-groups

x-rs-streaming-content

Main Logs

In the following table, italicized report field name text indicates thederiveddata.

Report Field Name Log Field Name

cs(Referer) cs(Referer)

browse_time Calculated at run-time from user session and stored as database

field.

c_ip c-ip

cs_auth_group cs-auth-group

cs_bytes cs-bytes

cs_host cs-host

cs_method cs-method

cs_uri_extension cs-uri-extension

cs_uri_path cs-uri-path

cs_url_query cs-url-query

cs_url_scheme cs-url-scheme

cs_user_agent cs(User-Agent)

cs_username cs-username


26/134


24

Reports/Log Field Matrix

This section provides a table that lists which main-format access log fields arerequired to populate eachpre-definedreport in the User Behavior, Security, andBandwidth Usage groups on the Reports tab. Use this reference to understand howlog fields relate to report data and aid in your customization of reports.

date date

date_time date + time

day_of_week Derived from date.

hits Calculated from page_views+ all related log entries.

hour_of_day Derived from time.

month Derived from date.

requests (same as page

views or hits)

Calculated during database generation and stored as database

field.

rs_content_type rs(Content-Type)

s_action s-action

sc_bytes sc-bytes

sc_filter_category cs-categories (or cs-category or sc-filter-

category)

sc_status sc-status

time time

total_bytes cs-bytes + sc-bytes

url Combined from (uri-scheme://cs-host/cs-url-path

[cs-url-query]).

verdict x-exception-id(sc-filter-resultif x-exception-idis

not present).

week Derived from date.

x_virus_id x-virus-id

year Derived from date.

Report Field Name Log Field Name


27/134


25

Log field Output

date + time YYYY-MM-DD + HH:MM:SS(GMT/UTC)

gmttime DD/MM/YYYY:hh:mm:ssGMT

localtime DD/MMM/YYYY:hh:mm:ss +nnnn

timestamp seconds since epoch in utc/gmt

x_timestamp_unix_utc seconds since epoch in utc/gmt

x_timestamp_unix seconds since epoch in local time


28/134


26

Main Log Field Matrix

These reports are URL-centric; they display reports that reflect browsing activity.

Group Report Required Fields

UserBehavior Blocked Web Browsing perUser sc-filter-result, cs-username, cs-bytes, sc-bytes

Web Browsing per Category {cs-categories -or- sc-filter-

category}, cs-bytes, sc-bytes

Web Browsing per Day date, sc-bytes, cs-bytes

Web Browsing per Day of

Week

date, cs-bytes, sc-bytes, time,

time-taken

Web Browsing per Group cs-auth-group, cs-bytes, sc-bytes

Web Browsing per Hour of

Day

time, cs-bytes, sc-bytes, time-

taken

Web Browsing per Month date, cs-bytes, sc-bytes, time,time-taken

Web Browsing per Site cs-host, {cs-categories -or- sc-

filter-category}, cs-bytes, sc-

bytes, time_taken

Web Browsing per User cs-username, cs-bytes, sc-bytes

Web Browsing per User and

Category

cs-username, sc-filter-category or

cs-categories, sc-bytes, cs-bytes

Web Searches cs-uri-query

(Also requires Blue Coat Web Filter

(BCWF) enabled.)


29/134


27

Security Blocked Web Browsing by

User Agent

sc-filter-result, cs(User-Agent),

cs-bytes, sc-bytes

Blocked Web Sites sc-filter-result, cs-host, {sc-

filter-category -or- cs-

categories}, cs-bytes, sc-bytes

Filtering Verdict Trend by Day date, sc-filter-result

Malware Requests Blocked by

Site

cs-bytes, cs-host, sc-bytes, sc-

filter-category, time-taken

Potential Malware Infected

Clients

c-ip, cs-bytes, cs-host, sc-bytes,

sc-filter-category, time-taken

Potential Threats x-virus-id, sc-filter-category

ProxyAV Malware Detected:

Client IP

c-ip, cs-bytes, sc-bytes, time-

taken, x-virus-id


Names

cs-bytes, sc-bytes, time-taken, x-

virus-id


Sites

cs-bytes, cs-uri-path, cs-uri-

query, cs-uri-scheme, sc_bytes,

time-taken, x-virus-id

Risk Groups sc-filter-category

SSL Certificate Categories {cs-username -or- c-ip}, s-action,

x-rs-certificate-hostname, sc-

bytes, cs-uri-port

SSL Certificate Errors x-rs-certificate-observed-errors,

x-rs-certificate-hostname, sc-

bytes, cs-uri-port

Trend of Potential Threats x-virus-id, sc-filter-category

Bandwidth

Usage

Bandwidth Cost per User date, cs-username, sc-bytes, cs-

bytes

Bandwidth Cost per User and

Site

cs-username, cs-host, sc-filter-

category or cs-categories, cs-

bytes, sc-bytes

Bandwidth Used per Day date, sc-bytes, cs-bytes

Bandwidth Used per Day of

Week

date, sc-bytes, cs-bytes

Bandwidth Used per Hour of

Day

date, sc-bytes, cs-bytes

Bandwidth Used per Month date, sc-bytes, cs-bytes

Requests per Content Type rs(Content-Type), cs-bytes, sc-

bytes

Requests per Protocol cs-uri-scheme, cs-bytes, sc-bytes

Web Requests per Client IP c-ip, cs-bytes, sc-bytes


30/134


28

Web Application Reports


Video Usage Reports


Report Field Name Required Fields

Web Application Name x-bluecoat-application-name, hits, page-views,

browse-time, cost-time, total-bytes, cost-bytes,

sc-bytes, cs-bytes, cache-bytes, rs-bytes

Web Application

Operation

x-bluecoat-application-operation, hits, page-

views, browse-time, cost-time, total-bytes, cost-

bytes, sc-bytes, cs-bytes, cache-bytes, rs-bytes

Web Application

Detailed Report

x-bluecoat-application-name, x-bluecoat-

application-operation, c-ip, total-bytes, cost-

bytes, hits, sc-bytes, cs-bytes, page-views,

browse-time, cost-time, cache-bytes

Web Browsing per Web

Application Name and

Client IP

x-bluecoat-application-name, c-ip, total-bytes,

cost-bytes, sc-bytes, cs-bytes, hits, page-views,

browse-time, cost-time, cache-bytes

Web Browsing per Web

Application Name and

User

x-bluecoat-application-name, cs-username, total-

bytes, cost-bytes, sc-bytes, cs-bytes, hits, page-

views, browse-time, cost-time, cache-bytes

Report Field Name Field Name

Client IP Video c-ip, total-bytes, cost-bytes, sc-bytes, cs-

bytes, hits, page-views, browse-time, cost-time,

cache-bytes

Flash Streaming

Bandwidth Cost per Day

date, page-views, browse-time, sc-bytes, rs-

bytes, total-bytes, cs-bytes, cache-bytes

Group Video cs-auth-group, total-bytes, cost-bytes, sc-bytes,

cs-bytes, hits, page-views, browse-time, cost-

time, cache-bytes

Video Application

Delivery Method

x-rs-streaming-content, total-bytes, cost-bytes,

sc-bytes, cs-bytes, hits, page-views, browse-

time, cost-time, cache-bytes

Video Application Type x-cache-info, total-bytes, cost-bytes, sc-bytes,

cs-bytes, hits, page-views, browse-time, cost-

time, cache-bytes


31/134


29

Video Applications x-rs-streaming-content, cs-host, total-bytes, sc-

bytes, cs-bytes, hits, page-views, browse-time,

cost-time, cache-bytes, cost-bytes

Video Page Detail cs-host, filename, c-ip, sc-bytes, cs-bytes, hits,page-views, browse-time, cost-time, cache-bytes,

total-bytes,

Video Site cs-host, total-bytes, sc-bytes, cs-bytes, hits,

page-views, browse-time, cost-time, cache-bytes

Report Field Name Field Name


32/134


30


33/134

31


This chapter describes various maintenance and performance tasks available tothe Reporter administrator, some of which are beyond the scope of the ReporterOnline Help System that is accessible directly in the Reporter ManagementConsole.

This chapter contains the following sections:

Section A: "Reporter Administration Tasks"on page 32

Section B: "Reporter Performance Best Practices"on page 50

Section C: "Advanced Filtering Tasks"on page 53

Section D: "Troubleshooting"on page 55

How Do I...?

How do I...? Tasks

I want to install Reporter as a root user. Procedure:"Linux Root UserInstallation Procedure"on page 32

I want to secure the connection thatReporter uses to communicate with theWeb server.

Procedure: "Securing the Reporter WebServer Transport Protocol"on page 35

What type of information does Blue Coatcollect from the Reporter Improvement

Program? Can I opt out?

"About the Reporter ImprovementProgram"on page 35

I want to configure Reporter to send e-mail to myself and/or others whenspecified events occur.

Procedure:"Connecting Reporter to E-mail Servers"on page 40.

Procedure:"Configuring Reporter toSend Alerts"on page 56.

I have configured the Direct ProxySGUpload Client and want to create back uplog files.

Procedure:"Creating ProxySG PolicyThat Backs Up Access Log Files"onpage 40.

My log files have spaces hardcodedbetween user names (first last) and in

reports the space displays as a %20symbol. Can I fix this?

Procedure:"Processing Log Files WithEncoded Spaces in User Names"on

page 45

I have a Blue Coat Secure Web Service(cloud) account and want to downloadaccess logs from there to use in Reporter.

Procedure:"Process Access Logs Fromthe Blue Coat Cloud Security Service"on page 46

I want to apply customized filteringoptions.

Section C: "Advanced Filtering Tasks"on page 53


34/134


32



This section provides common tasks that Reporter administrators perform tofurther configure Reporter.

Linux Root User Installation Procedure

The Blue Coat Reporter Initial Configuration Guide provides a procedure forinstalling Reporter as a non-root user on a Linux server. This procedure is forinstalling Reporter as a root user.

Install the Reporter Application

Step 1: In a browser, enter:https://bto.bluecoat.com

Step 2: Access the software download page.

Step 3: Download the application files:

a. Click the Downloadtab. The Blue Coat Download page displays.

b. From the Downloadmenu, click Reporter.

c. Before installing any version of Reporter, Blue Coat strongly recommends reading the ReleaseNotes (Please Read link). System compatibility lists, new feature briefs, and any known issues arelisted in this document.

d. Click the download link that matches the system on which you are installing.Note:The Linux##RPMlink is the uncompressed installation file. If you select Linux##, this is thecompressed (gzip) file. Gunzip the file in the /opt/bcdirectory.

Step 4: Open a terminal and navigate to the

directory to which you downloaded theapplication. Invoke the installation script:

rpm -Uhv Reporter*.rpm

Alternate location:rpm -Uhv --

prefix=

Reporter*.rpm

Text similar to the following displays:Preparing #################### [100%]

1:bcreporter #################### [100%]

########################################

# The Blue Coat Reporter installation

completed successfully.

# Please change your current working

directory to ...

########################################


35/134


33


Update Reporter Installation Procedure

The Blue Coat Reporter Initial Configuration Guide provides a procedure forinstalling Reporter as a non-root user on a Linux server. This procedure is forinstalling Reporter as a root user.You update by downloading and running anISO, which is 64-bit.

Step 5: Change the working directory to the Reporter installation directory:cd /opt/bc/reporter

Alternate, if you installed to a different location:cd

Step 6: Run the startup configuration. ./do-startup.sh

Step 7: Supply responses for the prompts: Username [admin]:

Password:

License key:

a. For the Username [admin]prompt, enter thedefault administrator access username.

b. For the Passwordprompt, create a passwordfor the default administrator user.

c. (Optional) If you purchased an Enterprise orPremium license andretrievedthe key, enter it

for the License keyprompt.Note:You can also enter the license key afterinstalling Reporter. Press .

Step 8: Text similar to the following displays: Blue Coat Reporter is already stopped

Starting Blue Coat Reporter: [OK]

Reporter is now up-and-running.

The RPM installation process also adds a start andstop script into the init.dfile, which automaticallystarts Reporter.



Step 1: In a browser, enter:https://bto.bluecoat.com


36/134


34


Step 2: Access the software download page.

Step 3: Download the application files:

a. Click theDownload

tab. The Blue Coat Download page displays.b. From the Downloadmenu, click Reporter.

c. Before installing any version of Reporter, Blue Coat strongly recommends reading the ReleaseNotes (Please Read link). System compatibility lists, new feature briefs, and any known issues arelisted in this document.

d. Click the download link that matches the system on which you are installing.Note:The Linux##RPMlink is the uncompressed installation file. If you select Linux##, this is thecompressed (gzip) file. Gunzip the file in the /opt/bcdirectory.

Step 4: Open a terminal and navigate to thedirectory to which you downloaded theapplication. Invoke the installation script:

rpm -Uhv Reporter*.rpm

Alternate location:rpm -Uhv --

prefix=

Reporter*.rpm

Text similar to the following displays:Preparing #################### [100%]

1:bcreporter #################### [100%]########################################

# The Blue Coat Reporter installation

completed successfully.

# Please change your current working

directory to ...

########################################

Step 5: Change the working directory to the Reporter installation directory:cd /opt/bc/reporter

Alternate, if you installed to a different location:cd

Step 6: Run the startup configuration. ./do-startup.sh



37/134


35


Uninstalling Reporter

The Readme file, which is located in the Reporter root folder/directory, containsprocedures that describe how to uninstall the Reporter application from Windowsand Linux servers.

About the Reporter Improvement Program

After it completes the installation process, Reporter displays a dialog thatdiscusses the Blue Coat Reporter Improvement Program. As described in thedialog, Reporter sends anonymous information to Blue Coat to assist Blue Coat

personnel in analyzing how Reporter is used. This basic information includesyour license, server specifications, system resource use, and Reporterconfiguration and use. No private, enterprise-sensitive data is transmitted.Although this feature is enabled by default, you have the option to declinesending this information. The General Settings > Systems Settings > Server Settingspage provides the Send anonymous system data to help improve Reporteroption; clearthis option and click Save.

Securing the Reporter Web Server Transport Protocol

By default, Reporter communicates with the Web server through the HTTPSprotocol. For increased security, you have the option to configure Reporter to use

HTTPS as the transport protocol. You must either accept the default certificate orspecify the location of a signed server certificate and unencrypted private key(2048-byte or larger key). Consult with your security administrator concerning thecreation of these.

Step 7: Supply responses for the prompts: Username [admin]:

Password:

License key:

a. For the Username [admin]prompt, enter thedefault administrator access username.

b. For the Passwordprompt, create a passwordfor the default administrator user.

c. (Optional) If you purchased an Enterprise orPremium license andretrievedthe key, enter itfor the License keyprompt.Note:You can also enter the license key afterinstalling Reporter. Press .

Step 8: Text similar to the following displays: Blue Coat Reporter is already stopped

Starting Blue Coat Reporter: [OK]

Reporter is now up-and-running.

The RPM installation process also adds a start andstop script into the init.dfile, which automaticallystarts Reporter.


Important: Reporter 9.x doesnotsupport SSL private keys that are passwordprotected.


38/134


36


Proceed to the appropriate section:

"Default Certificate"on page 36

"Selected Certificate"on page 37

Default Certificate

Consider the following if you elect to employ the default certificate. The defaultcertificate is a Reporter generated self-signed test certificate; however, most

browsers correctly warn you to avoid using self-signed certificates because theyare not signed by a reputable certificate authority. To prevent browsers fromrejecting the improper default certificate after you change your host IP address,you also must generate a new default certificate. This is accomplished bychanging the host IP address (using the host's configuration tools), thenconfiguring Reporter to HTTP and restarting Reporter (to remove the old defaultcertificate), configuring back to HTTPS and restarting once again (to create thenew default certificate).

Configure HTTPS with the Default Certificate

Step 1: With administrator credentials, in the Reporter Management Console select General Settings >System Settings > Server Settings.

Step 2: Select the Default Certificate. a. In the Protocolarea, select HTTPS. The areaexpands to displays certificate options.The default Reporter secure port is 8082.Ifyourusers access Reporter with the 8081port,Reporter redirects the connection to the secureport.

b. In the Certificatearea, select Use defaultcertificate.

c. Click Save.

Step 3: Restart Reporter. a. Select General Settings > Shut Down/Restart.b. Click Restart Reporter.


39/134


37


Selected Certificate

Reporter cannot use private keys that are password encrypted. If you already havea custom keypair that uses a password encrypted prviate key for your Reporterhost, then you must create an unencrypted version of the key. The public

certificate will continue to work with either key, but Reporter will only work withthe unencrypted key.

If you need to create a new unencrypted version of the existing keypair,proceed to"Creating an Unencrypted Private Key From an Existing Key".

If you have an existing keypair that isnotencrypted with passwords, proceedto"Configuring Reporter to Use HTTPS with a Selected Certificate"on page38.

Creating an Unencrypted Private Key From an Existing Key

The following procedures describe how to use the OpenSSL application that shipswith Reporter; the procedure contains steps for Windows and Linux operatingsystems.

Note: Blue Coat recommends creating certificate and key pair files in a privatefolder and not in the Reporter installation folders. If Reporter is uninstalled, youlose the files.

Create an Unencrypted Private Key From an Existing Encrypted Key

Step 1: Access the Open SSL application: In Windows, navigate to C:\Program Files\Blue Coat Reporter 9\utilities\ssl; double-click the openssl

application file. In Linux, run the opensslutility (to verify that the opensslpackage is installed on your system,

enter which openssl).

Step 2: Enter the following command: OpenSSL> rsa -inexisting_encryped_private.key -outnew_unencrypted_private.key

Step 3: The utility prompts you for the encrypted key password; enter that.

Step 4: Name the new key with a different name than the existing key, which prevents overwriting thestill valid encrypted key.

Step 5: The files are created and stored on the system. Proceed to the next procedure:"ConfigureReporter to use HTTPS with a Selected Certificate"on page 38.


40/134


38


Configuring Reporter to Use HTTPS with a Selected Certificate

Forcing Renegotiation of SSL sessions

The longer connections exist, the more susceptible are they to man-in-the-middleattacks. You can configure Reporter to renegotiate SSL sessions, which preventsthese attacks by selecting the Force secure renegotiationoption (General Settings >System Settings > Server Settings Web Server Settings > SSL Settingsarea).

However, after this option is enabled, Reporter only supports clients that supportsecure renegotiation.

To support older clients that do not support the secure renegotiation option, clearthis option.

Anonymize DataTo comply with security requirements, sensitive data can be encrypted, or,anonymized. Anonymized data is defined on a per role basis. By default, data isviewed in clear text. Any or all fields can be anonymized. Reports are displayedafter applying the algorithm so that users with that role are able to view theanonymized data instead of viewing the original data.

Configure Reporter to use HTTPS with a Selected Certificate

Step 1: With administrator credentials, in the Reporter Management Console select General Settings >System Settings > Server Settings.

Step 2: Select the generated CSR and keypair files. a. In the Protocolarea, select HTTPS. The areaexpands to display certificate options.Note:If this option is not available, see"Troubleshooting HTTPS Configuration onLinux"on page 59.

b. Select Enter Certificate; the Server Certificateand Private Keyfields become active.

c. Enter the path to the generated certificateand keypair files (or click the folder icons to

navigate to their stored locations and selectthem).

d. (Optional, recommended) Click TestCertificate and Key to test their validity. If thetest fails, work with your securityadministrator to create valid files.

e. Click Save.

Step 3: Restart Reporter. a. Select General Settings > Shut Down/Restart.

b. Click Restart Reporter.


41/134


39


Reporter Management Console Location:

Step 1: Administrator credentials: General Settings > Access Control > Roles.

Step 2: Databases: Click Newto define specific access rights for a role.

Move DatabaseUse this option to move the database to other physical locations. This is usefulwhen the drive on which database is located is running out of space. Beforemoving the database, Reporter calculates the size of the selected database and thespace available at the destination drive location.

Reporter Management Console Location

Step 1: Administrator credentials: General Settings > Data Settings > Databases.

Step 2: Select Move Databasesfrom the Actionsmenu for an existing database.Cancel move database operation restores the database back to the originallocation.

Step 3: Select General Settings > Data Settings> Databases.

Step 4: Select Cancel Databases from the Actions menu.

Internationalized Domain Name (IDN)

Enabling the Internationalized Domain Name (IDN) option converts domainnames back to the original Unicode domain name that were originally entered atthe client browser.Reporter Management Console Location:

Administrator credentials: General Settings > System Settings > Server Settings >ReportGeneration Settings.

Note: The move database operation fails if the destination drive size is lessthan 125% of the database size. While running the Move Database feature,Reporter unloads database and log sources.


42/134


40


Connecting Reporter to E-mail Servers

To enable users to e-mail generated reports to recipients, Reporter must beconfigured to communicate with an SMTP mail server. This is also required toenable Reporter to send administrators alerts when system resources reach

specified use levels (see"Configuring Reporter to Send Alerts"on page 56).

Creating ProxySG Policy That Backs Up Access Log Files

If you configured the ProxySG appliance to continuously stream access log data tothe Reporter server, the data is not stored anywhere after it is processed. If thedatabase becomes corrupt or if you have another scenario that requiresreprocessing of legacy data, you cannot do so unless you configure the ProxySGappliance to send back up files (raw data) to another location.

The following procedure describes how to configure the ProxySG appliance toupload back up files and create a policy that implements the back up operation.

Connect Reporter to an SMTP (E-mail) Server

Step 1: With administrator credentials, in the Reporter Management Console selectGeneral Settings > Reporter Settings > System Settings > External Servers > Email.

Step 2: Enter the primary SMTP server IP address or hostname.

Step 3: Specify the Fromaddress used in e-mails; for example:[email protected](must be a valid e-mail address).

Step 4: (Optional) Enter the SMTP server access credentials if they are required by theserver.

Step 5: (Optional, recommended) Enter information for a backup SMTP server shouldthe primary server become unavailable.

Step 6: Click Save.

Note: This procedure is valid only in SGOS 5.x; in SGOS 4.x, you cannotsimultaneously stream and forward logs.

Configure the ProxySG to Store Back Up Raw Access Log Data

Step 1: In the ProxySG appliance Management Console, select the Configuration > Access Logging >Logs > Logtab.

Step 2: Click New. The Create Log dialog displays.


43/134


41


Step 3: Configure the new log settings. a. In the Log Namefield, enter a name for thenew log file. For example, SG_HTTP_Backup.

b. From the Log Formatdrop-down list, selectbcreportermain_v1 (or whichever format youuse for Reporter logs).

c. (Optional) In the Descriptionfield, describethe new log.

d. Click OKto close the dialog

e. Click Applyto commit the new log file.

Step 4: Select the Configuration > Access Logging >Logs > Upload Clienttab.

Select the upload client:

a. From the Logdrop-down list, select thenewly-created log.

b. From the Upload Clientdrop-down list,select FTP Client.

c. Click Settings. The FTP Client Settingsdialog displays.



44/134


42


Step 5: Configure the ProxySG to communicatewith the FTP server to be used for log

archiving.

Server connection options:

a. From the Settings for drop-down list, selectPrimary FTP Server.

b. In the Hostfield, enter the IP address orhostname of the FTP server; change thedefault port only if a different port is used.

c. In the Pathfield, enter the destinationfolder to be used for this log archive (toprevent log data duplication, do not pointto the same directory that is used forproduction data).

d. In the Usernamefield, enter the name

required to access this FTP server.e. If a server access password is required,

click Change Primary Passwordand enterthe information.

f. Click OKto close the dialog.

Step 6: Select the Configuration > Access Logging >Logs > Upload Scheduletab.

Upload schedule options:

a. From the Logdrop-down list, select thenew-created log.

b. In the Upload Typearea, select Periodically.

c. In the Upload the Log Filearea, set theinterval at which the ProxySG uploads thelog files. Blue Coat recommends once perhour.

d. Click Apply.

Step 7: Launch the Visual Policy Manager (VPM): Configuration > Policy > Visual Policy Managertab;click Launch.

Step 8: In a Web Access Layer, click Add Rule; or, select Policy > Add Web Access Layer. As policy bestpractice, do not create a new Web Access Layerif one already exists.



45/134


43


Step 9: Add the FTP, HTTP, and HTTPS serviceobjects.

Steps:

a. Right-click a Service column cell; select Set.

The Set Service Object dialog displays.

b. Click Newand select Service Name. TheAdd Service Name Object dialog displays.

c. From the Service Namedrop-down list,select FTPand click OK.

d. Repeat Step band add the HTTP services.

Step 10: Create a combined object for the services. Steps:

a. Still in the Set Service Objects, click Newand select Combined Service Object.

b. Name the object.

c. Select each service and click Add.

d. Click OKto create the combined object.

e. With the combined object selected, click OKto add it to the rule.



46/134


44


Step 11: Create an Action object that enables accesslogging.

a. Right-click the Actioncolumn cell; select Set.The Set Action Object dialog displays.

b. Name the object.

c. Click Newand select Modify Access Logging.The Add Access Logging Object dialogdisplays.

d. Select Enable logging to; from the drop-down list, select thelog file that you createdin Step 3.

e. Click OKto create the object.

f. With the object selected, click OKto add it tothe rule.

The rule is complete.

Step 12: To implement the policy, click Install Policy.



47/134


45


Processing Log Files With Encoded Spaces in User Names

If Reporter generates reports that display %20in between user names (first andlast), it means that your access logs have encoded spaces. For example:

first lastversus a user name first.last

Although Reporter currently supports the Mainaccess log format. A hard-codedspace is a valid character in the Extended Log File Format (ELFF). The followingprocedure describes how to manually configure the databases to display hard-coded spaces as actual spaces rather than %20symbols.

Manually Editing the Database File to Process Encoded Spaces

Step 1: Create a new database, but donotassign alog source.

General Settings > Reporter Settings > Data

Settings > Databases

Step 2: Stop the Reporter service. General Settings > Reporter Settings > Shut Down/Restart

Step 3: Using a text editor, open the newly-created database configuration file.

By default, this file is located in the Blue CoatReporter 9 > Settings > Databasesdirectory.

Database configuration files are named withnumber and letter strings, not intuitive names.Look for the date and time stamp of the databaseyou created for this process.

Step 4: Inside of the database configuration file, search for the log field named cs_username, which islocated near the bottom of the file. The construct is similar to the following:

cs_username = {type = "flat"

index = "0"

name = "cs-username"

db_field = "cs_username"

} # cs_username

Step 5: Add encoded_spaces = "true"to the cs_usernameconstruct:cs_username = {

type = "flat"

index = "0"

name = "cs-username"

db_field = "cs_username"

encoded_spaces = "true"

} # cs_username

Step 6: Save the database configuration file.

Step 7: Re-start Reporter.

Step 8: Add a log source to the database to begin processing log data.


48/134


46


Process Access Logs From the Blue Coat Cloud Security Service

Reporter can download access logs from the Blue Coat Cloud Security Service(ThreatPulse) and process the data, which enables unified reporting. To learnmore, see "Download Access Log Data from the Blue Coat Cloud Security Service"

on page 16.This procedure describes how to create the required cloud service API and how toconfigure Reporter to receive the log downloads.

Prerequisites

This procedure assumes that you have an account for the Web Securitymodule

in ThreatPulse. To receive log files from the cloud service, the Reporter server must have Web

access. If you already have Reporter deployed and that deployment inhibitsallowing Web access to the Reporter server, consider installing anotherinstance of Reporter on a different server at the external edge of the network.Then automate or otherwise move the log files to the existing server (toachieve unified reporting).

ThreatPulse Configuration

This section describes how to create the required API key.

Note: ThreatPulse refers to the Blue Coat cloud service product name. TheBlue Coat Cloud Security Service is a solution that includes all productsproduced by Blue Coat in the cloud.

Create an API Key in the ThreatPulse User Interface

Step 1: Access the ThreatPulse portal. a. In a browser, enter:https://portal.threatpulse.com

b. Log in with your account credentials.

Step 2: In Service mode, select Account Maintenance > Account Provisioning > API Keys.


49/134


47


Reporter Configuration

This section describes how to configure Reporter to use the cloud API key tovalidate and begin access log downloads to specified location.

Step 3: Create an API key for Reporter.

a. Click Add API Key. The service displays the Create API Keys dialog.

b. Define a Usernameand Password. You will enter these during the Reporter configuration.

c. Click Add.

Step 4: Enable the key: select the key and click Enable.

Proceed to the next section to configure Reporter.

Create an API Key in the ThreatPulse User Interface

Configure Reporter to Download Access Logs from the Cloud Service

Step 1: With administrator credentials, in the Reporter Management Console (on the server where thelogs are to be staged) select General Settings > Data Settings > Cloud Download.


50/134


48


Step 2: Select Enable Cloud Download; the other options become available to edit.

a. Select the Destination Directory, which is the folder that stages the cloud service access logs.

b. (Optional) If you have not previously created a folder for this purpose, click the Create New Foldericon.

Step 3: Set the Scheduleof how often Reporter

checks for new logs in the cloud service.The shortest increment is one hour, as all cloud logfiles contain one hour of data.

By design, the Blue Coat cloud service prevents thedownloading of logs that are less than two hours old.Furthermore, given that the minimum time chunk isone hour, allow some time for the data to accumulatein the Destination Directory.

Step 4: Enter the API key. a. Enter the Blue Coat cloud service APIUsernameand Passwordthat you created inthe ThreatPulse user interface.

b. Click Test Username and Password. If the test

fails, check the API key in the ThreatPulseuser interface (Service mode> AccountMaintenance > API Keys). Also check theexternal connection.

Step 5: Click Save.

Configure Reporter to Download Access Logs from the Cloud Service


51/134


49


Create a Database or Assign Log Source

Now that you have a new source of log data, you can create a separate databaseand generate reports based on the cloud source only, add the newlog source to anexisting database to provide true unified reporting, or both.

Also, see"About Optimizing Log Processing Configurations"on page 17.


52/134


50

Section B: Reporter Performance Best Practices

Section B: Reporter Per

Reporter Administrators Guide 9.x.b

Documents

Transcript of Reporter Administrators Guide 9.x.b