ARCSIGHT FLEXCONNECTOR TRAINING LEVEL 02€¦ · SNMP Connector Collect logs from snmp traps....

Created and Presented by

Balahasan V. | SIEM SME Accenture

AESA, AEIA

ARCSIGHT FLEXCONNECTOR TRAINING LEVEL 02

Introduction

Brief about Flex.

Planning your Flex.

Types of Flex Connector and related Parameters.

Sample Examples.

Detailed Configuration File Structure Topics.

Basic Flex Concepts

Declaring Regex and Configuring FlexAgentWizard /Regex Wizard.

Token Declaration

Event Mapping

Severity Mapping

Understanding the Regex Usage

Few Examples on Different Flex Types

Little Advanced Concepts of Flex

Submessages

Conditional Mapping.

Extra Processor

Multi-Line Regex

Parser Overrides.

Extra Mapping Files.

Merge operations.

Custom Categorizations.

Key Value Parsers

Creating Map Files.

Defining deviceEventClassId.

Additional Data Mapping

CounterACT Connector and REST API

Flex Active List Import and Flex Asset Import.

Few Sample Agent Property file Important Configurations

Topics Covered

Custom Defined Smart Connector.

Collects and normalizes data from unsupported devices.

Fully functioning agent, including categorization, zoning, aggregation, batching and priority calculation features.

Installed through the ArcSight smart connector installer.

Run smart connector installer

Select desired Flex connector type

What Is An ArcSight FlexConnector

Flex Connector Log-file

FlexConnecor Regex log-file

Flex Connector Regex Folder log file

Flex Connector Syslog

Flex Connector Time-based Database

Flex Connector ID-based Database

FlexConnector Multi-Database

FlexConnector SNMP

FlexConnector XML Folder Log file

FlexConnector Scanner for Text, XML, Database

Rest API

CounterACT Connectors

Types of FlexConnector

Configuration file holds the flex parser which will be used to parse the raw logs

There are 4 steps to creating a FlexConnector configuration file

Define a parsing mechanism

Identify and name tokens (Tokenization)

Map tokens to ArcSight schema (Normalization)

Map device severity to ArcSight severity

Advanced Configuration Properties

Flex Connector Configuration File

Base Directory : /current/user/agent/flexagent/

Log-file : < vendor >.sdkfilereader.properties

Regex log-file & Folder : < vendor >.sdkrfilereader.properties

Time based DB & Multi-DB : < vendor >.sdktbdatabase.properties

ID Based DB : < vendor >.sdkibdatabase.properties

Syslog : syslog/< vendor >.subagent.sdkrfilereader.properties

SNMP : < vendor >/sdksnmp.#.snmptrap.rpoperties

XML Folder log file : < vendor >.xqueryparser.properties

Scanner Text/XML/DB :

< vendor >.< scanner/vulns/openports/uris >.sdkrfilereader.properties

< vendor >.< scanner/vulns/openports/uris >.xqueryparser.properties

< vendor >.sdkdatabase.properties

Rest API : < vendor >.jsonparser.properties

CounterACT : < file_name >.counteract.properties

Flex Configuration File Location (Important)

Flex Connector Installation

Log file Flexconnector For fixed format ,delimited log file (real time log collection)

Regex Log File For variable format log file(real time log collection)

Regex folder Follower To read logs in batch mode

Regex Multiple Folder Follower Read logs from multiple folder (Real time and Batch mode)

Time-based DB Flexconnector Read event info from tables based on timestamp value

ID based DB Flexconnector Read event info from tables based on ID value

Multiple DB Flexconnector Read logs from multiple database(Time based as well as ID based)

Selecting A Flex Connector Type

SNMP Connector Collect logs from snmp traps.

SYSLOG Connector Security events from syslog messages.

XML Connector Read logs from XML-based files in a folder.

Scanner Connector To import the scan results from a scanner device.

Rest API Provides a configurable method to collect security events when you use cloud-based applications such as Box, Salesforce, or Google Apps…

CounterACT User can then execute commands on third party devices from within ArcSight and send the output of those commands back to the console.(Allowing the third party device to be controllable from ArcSight Console itself Amazing Feature isn’t it)

Selecting A Flex Connector Type

Log file Example: 08/09/2050-11:33:00,1.1.1.1,52123,2.2.2.2,80,Invalid URL

08/09/2050-12:43:00,3.3.3.3,49123,2.2.2.2,80,Buffer Overflow

Regex Log File Sep 10 15:28:49 beach sshd[24939]: Failed password for rajiv from 192.168.10.27 port 33654 ssh2

Sep 10 15:28:51 beach PAM_unix[24948]: (ssh) session opened for user rajiv by (uid=525)

Time Based Database

Examples

ID Based Database

Syslog

My application: Intruder Detected from 1.1.1.1 to 2.2.2.2 High

Examples

Parsing Mechanism

Token declaration

Event mapping with ArcSight Schema

Severity Mapping

Submessages

Additional Data and Conditional Mapping.

Extra Processor

Multi-Line Regex

Parser Overrides.

Extra Mapping Files.

Merge operations.

Custom Categorizations.

Creating Map Files.

Creating Key Value Parsers.

Defining deviceEventClassId.

Configuring FlexAgentWizard and Regex Wizard.

Detailed Configuration File Structure

Ex: Sample Log

28/09/11 08:15:00 SRC=194.168.0.12 DST=195.172.0.12 SPT=4236

DPT=80

Declaring Regex:

Requires clear idea what u need to parse and which field u need to map with

ArcSight CEF Field.

Below example gives how the Message is broken and parsed with the corresponding regular expression.

•Date & Time : (\\d+\\/\\d+\\/\\d+)\\s+(\\d+\:\\d+\:\\d+)

•Src and Dst Ip : (\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})

•Src and Dst Port : (\\d+)

Overall :

Regex=(\\d+\\/\\d+\\/\\d+)\\s+(\\d+\:\\d+\:\\d+)\\s+SRC\=(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})\\s+DST\=(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})\\s+SPT\=(\\d+)\\s+DPT\=(\\d+)

Parsing Mechanism Declaring Regex

//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d//d//d//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)

Ways of Configuring Regex And Testing(3 Examples)

Regex Testing with

Cutom Apps(ex: Notepad++)

Regex creation using

Flex Creation Wizard

(only delimited)

Regex creation using

FlexAgent Regex Tool

token. count Number of tokens present in each line of the file

token[x].name User defined name for the tokens

token[x].type Data type of the token

token[x].format Format of the token or Modified Type

Token Declaration

Integer

Date

IPAddress

IPv6Address

Long

MacAddress

RegexToken

String

Time

TimeStamp

Reference Snap: Token Types

Token Types

Mapping the parsed token to ArcSight Event fields(400+ field event schema )

Type of token must match the ArcSight Event field Type.

In addition to the tokens that are parsed from each input record, you can also configure built-in tokens for specific Flex Connectors.

For Example

token[0].name=Msg

token[0].type=String

token[1].name=MyIP

token[1].type=IPAddress

event.sourceAddress=MyIP

event.message=Msg

event.deviceCustomDate1=_SYSLOG_TIMESTAMP (* Built-in token )

Event Mapping

Severity is an important part of the Threat Level Formula as well as for usage in reports that make use of device / event Severity.

Assume Token1 values are 23,46,69,82,95

It can be some string values too or Both string and Integer ..

Assume Token1 values as Error, Warning, Informational , Critical, Notification

Example 1

event.deviceSeverity=Token1

severity.map.veryhigh.if.deviceSeverity=95

severity.map.high.if.deviceSeverity=82

severity.map.low.if.deviceSeverity=23

severity.map.medium.if.deviceSeverity=46,69

Example 2

event.deviceSeverity=Token1

severity.map.veryhigh.if.deviceSeverity=Critical

severity.map.high.if.deviceSeverity=Error

severity.map.low.if.deviceSeverity=Informational

severity.map.medium.if.deviceSeverity=Warning, Notification

Severity Mapping

do.unparsed.events=true

Regex=(\\d+\\/\\d+\\/\\d+)\\s+(\\d+\:\\d+\:\\d+)\\s+SRC\=(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})\\s+DST\=(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})\\s+SPT\=(\\d+)\\s+DPT\=(\\d+)\\s+Sev\=(\\d+)\\s+URL\=(.*?)

token.count=5

token[0].name=Time_of_the_event

token[0].type=TimeStamp

token[0].format=dd/MM/yy HH:mm:ss

token[1].name=SrcIp


token[2].name=DstIp


token[3].name=Sev

token[3].type=Integer

token[4].name=URL


event.deviceReceiptTime=Time_of_the_event

event.sourceAddress=SrcIp

event.destinationAddress=DstIp

event.deviceSeverity=Sev

event.requestUrl=URL

event.deviceVendor=__getVendor(“MyVendor”)

event.deviceProduct=__stringConstant(“MyProduct”)

severity.map.veryhigh.if.deviceSeverity=404,500

severity.map.medium.if.deviceSeverity=303,302

Sample Configuration File

//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*

Example Type 01: Regex

Defining Regex is the main task in the Parsing mechanism from simple to complex Logs using the Regular Expressions for various Log files.

Example Type 02: Query

Defining query is the main task in the Parsing mechanism for Database Logs using the SQL Queries to retrieve the Data from Database schema.

Example Type 03: Expression

Defining node expression is one of the main task in the Parsing mechanism for XML Logs using beginning node location expression to process the Expression. A root node is at the top of the tree, hop nodes are in between, and trigger nodes are at the bottom.

Understanding The Parser Expression Usage

Regular Expression

Regex Examples

Getting the Sample Logs and Analyzing for Events of Interest from

Device/Application documents and understand its Nature.

Checking the Logging Mechanism Enabled on the End Device with Audit Level.

Choosing the Method of Logs Collection based on logging mechanism and gather the Events and define the possible Use cases.

Ex: Batch mode or Real Time.

Choosing the Suitable Flex Connector Type for Parsing.

Checking out the Log Rotation Policies of both End Device and Connector.

Defining your Flex Configuration file and Agent Properties file.

Use this in Test Environment to check your Flex Connector working without any issues. And check whether all the events are parsed properly by enabling raw events and comparing them with the Unparsed Events.

Normalization and Further Content Development.

Choosing your Categorization files, Additional Mapping, Key Value Parsers, Map files etc., and placing in the Exact Location.

Ensure the Following event severity, deviceEventClassId, categorization, deviceCustom and Flex field Labels.

Things To Remember While Defining The Flex Connectors

Example

Time based Connector

ID based Connector

Database Flex Connector

Configuration File –Time Based

version.order Specifies the order in which parser files are executed.

version.query This property enables you to perform a test query against the database to validate the database version.

version.id If the version.query succeeds, the deviceVersion token is set to the version.id.

Query Retrieves the rows(events) that were inserted between the last time query was run and the current time.

timestamp.field Specifies the field to use to determine when to run the next query.

uniqueid.fields Specifies the field to use to distinguish rows with the same timestamp field.

Configuration File –Time Based

Configuration File - Id BASED

maxid.query Specifies the query to use to retrieve the maximum ID present in the database when the query is run.

id.field Specifies the field to use to determine when to run the next query.

uniqueid.fields Specifies the field to use to distinguish rows with the same ID field.

query.limit Specifies the maximum number of rows to return when a query is run.

Configuration File –ID Based

Configuration File – XML Flex

namespace.count Specifies the number of namespaces that your XML log file Uses.

namespace.prefix Specifies the namespace prefix to use.

namespace.uri Specifies the Uniform Resource Identifier (URI) for the namespace.

hop.node.count Specifies the number of hop nodes.

hop.node.name Specifies the names for the hop nodes.

hop.node.expression Specifies the XPath/XQuery path expressions to select the nodes.

trigger.node.expression These are the nodes that trigger events.

token[x].expression Specifies the XPath/XQuery path expression that is traversed to obtain the value for the token.

token[x].node Specifies the context node—root node, hop node, or trigger node—relative to which the path expression is evaluated.

extraevent.count Specifies the number of extra events.

extraevent[x].filename Specifies the file name of the additional configuration file that this parser should use.

extraevent[x].name Specifies a name to associate with the extra events.

Configuration File – XML Flex

Example

• Nov 28 22:02:42 10.0.111.2 %PIX-6-106015: Deny TCP (no connection)from 3.3.3.3/4532 to 4.4.4.4/80 flags RST on interface outside

• Nov 28 22:06:10 10.0.111.2 %PIX-3-305005: No translation groupfound for tcp src inside:10.0.112.9/37 dst outside:4.5.6.7/3562

• Nov 29 01:46:42 10.0.111.2 %PIX-6-305005: Translation built for gaddr 1.2.3.4 to laddr 10.0.111.9

• Nov 29 01:35:15 10.0.111.2 %PIX-4-500004: Invalid transport fieldfor protocol=6, from 2.2.2.2/0 to 3.3.3.3/0

Single log source may contain more than one message format. We don’t have to define 4 different parsers for single source.

Need For Sub Messages

Message divided in two portions.(common to all messages and one that varies with each message format)

• Nov 28 22:02:42 10.0.111.2 %PIX-6-106015: Deny TCP (no connection)from 199.248.65.116/3564 to 10.0.111.22/80 flags RST on interface outside

Into: (Static Event content) • Nov 28 22:02:42 10.0.111.2 %PIX-6-106015:

And: (Variable Event content) • Deny TCP (no connection) from 199.248.65.116/3564 to 10.0.111.22/80

flags RST on interface outside

To define the sub-message we need to perform these steps:

1. Define the corresponding sub-message ID.

2. Define the regular expression(s) to use.

3. Define the mappings to event fields.

Sub Messages

regex=(\S+ \d+ \d+:\d+:\d+) (\S+) %PIX-(\d)-(\d+): (.*)

token.count=5 token[0].name=Timestamp token[0].type=TimeStamp token[0].format=MMM dd HH\:mm\:ss token[1].name=PixIP token[1].type=IPAddress token[2].name=PixSeverity token[2].type=String token[3].name=SubmessageIdToken token[3].type=String token[4].name=SubmessageToken token[4].type=String

Example

submessage.messageid.token=SubmessageIdToken

identifies the token that will hold the message identifier.

submessage.token=SubmessageToken

token that contains the actual sub-message.

submessage.count=1

count of sub-message IDs (106015).

Following will be internally equivalent to 2 Tokens

Submessage[0].messageid=106015

submessage[0].pattern.count=1

submessage[0].pattern[0].regex=Deny (\\S+) \$no connection\$\\s(\\d+\.\\d+\.\\d+\.\\d+)

submessage[0].pattern[0].fields=event.transportProtocol, event.sourceAddress

submessage[0].pattern[0].types=String,IPAddress

submessage[0].pattern[0].formats=null,null

The format can also be defined using one sub-message property. Can be used for Different Time Zone Mappings.

Example Continued

//S//S//S

Event id is 532 type A with parameter 3.3.3.3

Event id is 533 type A with parameter root

Event id is 534 type A with parameter 3.3.3.3

Scenario:

Event id is 532 or 534, set the ArcSight event field event.sourceAddress to 3.3.3.3 and

if the event id is 533,

set the event.sourceUserName to root.

Conditional mappings enable you to map tokens that can contain

different types of information, based on the characteristic of the

event.

Need For Conditional Mapping

regex=Event id is (\\d+) type (\\S+) with parameter (\\S+)

token.count=3

token[0].name=EVENTID

token[1].name=TYPE

token[2].name=PARAMETER

#Standard mappings

event.deviceEventClassId=EVENTID

event.deviceEventCategory=TYPE

#Conditional mappings

conditionalmap.count=1

conditionalmap[0].field=event.deviceEventClassId

conditionalmap[0].mappings.count=2

conditionalmap[0].mappings[0].values=532,534

conditionalmap[0].mappings[0].event.sourceAddress=PARAMETER

conditionalmap[0].mappings[1].values=533

conditionalmap[0].mappings[1].event.sourceUserName=PARAMETER

Example

submessage[3].messageid=conditionalmapsample


submessage[3].pattern[0].regex=Event id is (\\d+) type (\\S+) with parameter (\\S+)

submessage[3].pattern[0].fields=event.deviceEventClassId

submessage[3].pattern[0].conditionalmap.count=2

submessage[3].pattern[0].conditionalmap[0].field=event.deviceEventClassId

submessage[3].pattern[0].conditionalmap[0].mappings.count=2

submessage[3].pattern[0].conditionalmap[0].mappings[0].values=532,534

submessage[3].pattern[0].conditionalmap[0].mappings[0].event.destinationAddress=$3

submessage[3].pattern[0].conditionalmap[0].mappings[1].values=533

submessage[3].pattern[0].conditionalmap[0].mappings[1].event.destinationUserName=$3

submessage[3].pattern[0].conditionalmap[1].token=$2

submessage[3].pattern[0].conditionalmap[1].mappings.count=1

submessage[3].pattern[0].conditionalmap[1].mappings[0].values=B

submessage[3].pattern[0].conditionalmap[1].mappings[0].event.destinationAddress=$3

In the above example, there are three groups:

$1 -- (\\d+)

$2 -- (\\S+)

$3 -- (\\S+)

Example 2(conditional Mapping In Submessages)

To chain two configuration files together Useful if you need to use two or more different types of FlexConnectors for the same data.

Extra processors are particularly useful when an event has more than one type of data in it and cannot be parsed by a single parser. This property is also referred to as parser linking.

Can be useful when you use Regular expression to parse data that was obtained from a time-based SQL database.

Configuration files need to be placed in the \user\agent\flexagent folder.

Example Scenario

When you use the same Log File for logging different versions of Same Application Server with varying Formats.

Need For Extra Processor

extraprocessor.count=1 (the number of extra processors)

extraprocessor[0].type=regex (extra processor type)

extraprocessor[0].filename=netiq/netiq (extra process file name)

extraprocessor[0].field=event.message

extraprocessor[0].flexagent (extra processor variable)=true (extra processor parameter or conditional value)

extraprocessor[0].clearfieldafterparsing=false

Example

Extra Processor Type

extraprocessor.count=2

extraprocessor[0].type=regex

extraprocessor[0].field=event.name

extraprocessor[0].filename=securitymanager/Name-Name


extraprocessor[0].flexagent=true

extraprocessor[1].type=regex

extraprocessor[1].field=event.name

extraprocessor[1].filename=scm/Name-Name


extraprocessor[1].flexagent=true

Example

Multiline parsing provides a mechanism for providing hints so that the parser can reconstruct messages that have been broken into multiple lines. Because some files may contain events that are split into multiple lines.

Ex Scenario:

|01/01/2005 11:00:50|1.1.1.1|7663|2.2.2.2|80|this

is

a

message

that

takes

multiple

lines|

01/01/2005 11:00:51|1.1.1.1|7663|2.2.2.2|80|this

is another large message that takes

multiple lines|

Need For Multiline Parser

multiline.starts.regex=\|\d+/\d+/\d+ \d+:\d+:\d+\|.*

multiline.ends.regex=.*\|$

Output:-

|01/01/2005 11:00:50|1.1.1.1|7663|2.2.2.2|80|this is a message that takes multiple lines|

Sample Output

multiline.starts.regex=\\|\\d+/\\d+/\\d+ \\d+\:\\d+\:\\d+\\|.*

regex=\\|(.*?)\\|(\\S+)\\|(\\d+)\\|(\\S+)\\|(\\d+)\\|(.*)|

token.count=6

token[0].name=Timestamp

token[0].type=TimeStamp

token[0].format=MM/dd/yyyy(\\s)HH\:mm\:ss

token[1].name=SourceAddress


Multiline Regex Configuration File

//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|

To support multi-line messages, we need to define the message start and end in the configuration file.

Multiline Regex

Some SmartConnector parsers map sensitive information such as source and

destination user names, host names, addresses, etc… inappropriately, i.e.

Windows Event Log SmartConnectors.

Parser Override(Parser Versioning) which enables each SmartConnector to

Parse raw events in many different ways using different Parser Versions, and thus

generate ArcSight security events with different types of mappings.

It support the current parser mappings so as to not break existing content for

users, but also to support newly corrected mappings so as to allow new and

accurate content to be developed.

Need For Parser Overrides

A SmartConnector feature that allows a SmartConnector to support multiple

versions of parsers.

Allows users to configure their SmartConnectors with any available parser version of their choice, depending on their ArcSight security event mapping requirements.

Each SmartConnector is designed to have its own internal parameter fcp.version to represent its current Parser Version.

Each SmartConnector can support a total of 8 Parser Versions

fcp.version range from 0(Base Parser Version) through 7.

To identify the Parser Version with which a raw event has been parsed, observe the last digit of the Agent Version field of the ArcSight security event. i.e.,

For Parser Version 0, the Agent Version will be 5.1.2.5823.0

For Parser Version 1, the Agent Version will be 5.1.2.5823.1

Parser Override Pg.01

Parser Override Pg.02

Extra mappings is another property of the sub-message that can be used to directly add additional mapping properties.

Example:

submessage[3].pattern[0].extramappings=event.name=

__stringConstant("Unparsed event")

|event.deviceProduct=__stringConstant("Unknown")

In this above Example, you might have multiple Submessages Patterns. If u don’t want to miss any events and want to map particular set events in one category say all unknown events in this one category but you are not using the submessage[2].pattern[0].fields directly.

Need For Extra Mapping

# Default sub-message descriptor


submessage[3].pattern[0].regex=(.*)

submessage[3].pattern[0].extramappings.delimiter=@

submessage[3].pattern[0].fields=event.message

submessage[3].pattern[0].extramappings=

event.name\=__stringConstant("Unparsed event")

@event.deviceProduct\=__stringConstant("Unknown")

Extra Mapping Configuration

Some devices will send information about a single event in multiple log lines. Even though in some cases it would be fine to send each line as a single event, in some other instances it is necessary to merge the information of all the events into a single one.

Ex:

[18/Jul/2005:12:30:20 -0400] conn=8 op=0 msgId=82 - BIND uid=admin

[18/Jul/2005:12:30:25 -0400] conn=7 op=-1 msgId=-1 - LDAP connection from

10.0.20.122 to 10.0.20.122

[18/Jul/2005:12:30:30 -0400] conn=8 op=0 msgId=82 - RESULT err=0

We can say I have Multiline Parser, In this instance you can’t say i can use

Multiline, coz here 1st line is input and 3rd line is output, so you can’t use the

Multiline Parser to solve this issue. Here in cases like this Merge Operation will

help you solve this mistery. So we can deploy the merge operation for the events

Which have connection in above (Conn,msgId) but with different operations.

Need For Merge Operations

Each merge operation defines:

Which events to include in the merge operation

When to start a merge operation

When to end a merge operation

The fields that identify which events belong to the same group

Note: Currently ONLY the regular expression based agents support this feature.

You need to use the predefined set of merge operation property variables.

Defining Merging Operation

merge.count Defines the number of merge operations.

merge[{mergeindex}].traceenabled When set to TRUE all operations regarding

event merging will be logged for this merge operation.

merge[{mergeindex}].pattern.count Defines how many patterns will be defined.

merge[{mergeindex}].pattern[{patternindex}].token Defines the token that will be used for

this pattern.

merge[{mergeindex}].pattern[{patternindex}].regex Defines the regular expression to use for

this pattern.

merge[{mergeindex}].starts.count Defines how many start patterns will be defined.

merge[{mergeindex}].starts[{patternindex}].token Defines the token that will be used for

this start pattern.

merge[{mergeindex}].starts[{patternindex}].regex Defines the regular expression to use for

this start pattern.

merge[{mergeindex}].starts[{patternindex}].endspreviousmerge If set to true then it means

that if the start message is found within an already merged event, then the merge processor

should end the previous merge and start a new one.

Merge Operation Property Pg.01

merge[{mergeindex}].ends.count Merge operations require end patterns to

define which events will end the merge operation.

merge[{mergeindex}].ends[{patternindex}].token Defines the token that will

be used for this end pattern.

merge[{mergeindex}].ends[{patternindex}].regex Defines the regular

expression to use for this end pattern.

merge[{mergeindex}].timeout Defines the timeout in milliseconds for the

merging operation.

merge[{mergeindex}].id.tokens Defines the list of tokens that will be used to

group the events.

merge[{mergeindex}].id.delimiter Defines an optional delimiter to use.

merge[{mergeindex}].sendpartialevents It specifies if each event in the merge

operation must be sent individually as it is merged with other events.

merge[{mergeindex}].capacity An event merging operation requires a

cache of events that hold the merged results.

Merge Operation Property Pg.02

merge.count=1

merge[0].pattern.count=2

merge[0].pattern[0].token=NAME1

merge[0].pattern[0].regex=(BIND|UNBIND|MOD|RESULT)

merge[0].pattern[1].token=NAME2

merge[0].pattern[1].regex=(BIND|UNBIND|MOD|RESULT)2

merge[0].starts.count=1

merge[0].starts[0].token=NAME3

merge[0].starts[0].regex=(BIND|UNBIND|MOD)

merge[0].ends.count=2

merge[0].ends[0].token=NAME4

merge[0].ends[0].regex=RESULT

merge[0].ends[1].token=NAME5

merge[0].ends[1].regex=RESULT2

merge[0].timeout=60000

merge[0].id.tokens=conn|msgId

merge[0].id.tokens.delimiter=|

merge[0].sendpartialevents=true

merge[0].capacity=100

Merge Properties

Sample Logs :

[18/Jul/2005:12:30:20 -0400] conn=8 op=0 msgId=82 - BIND uid=admin

[18/Jul/2005:12:30:25 -0400] conn=7 op=-1 msgId=-1 - LDAP connection from

10.0.20.122 to 10.0.20.122

[18/Jul/2005:12:30:30 -0400] conn=8 op=0 msgId=82 - RESULT err=0

Merger Property:

merge.count=1

merge[0].pattern.count=1

merge[0].pattern[0].token=OperationName OperationName set to BIND or RESULT.

merge[0].pattern[0].regex=(BIND|RESULT)

merge[0].starts.count=1

merge[0].starts[0].token=OperationName OperationName set to BIND(start the merge)

merge[0].starts[0].regex=BIND

merge[0].ends.count=1

merge[0].ends[0].token=OperationName OperationName set to RESULT(End merge)

merge[0].ends[0].regex=RESULT

merge[0].id.tokens=Connection,Operation,MessageId (Defining Fields which must be Identical)

merge[0].timeout=60000

Merge Operation Example

In Event Mapping Section:

event.deviceReceiptTime=Date

event.name=__oneOf(mergedevent.name,OperationName)

Gets to Sub message from the name Field for Mapping the Merged Operation.

event.deviceAction=ResultCode

event.destinationUserId=UserId

Merge Operation Example Pg 2

The FlexConnector developer can control categorization by creating or modifying

Existing categorization files. Categorization files are comma-separated value (CSV)

text files, placed in a folder named for the device vendor under the directory:

ARCSIGHT_HOME/user/agent/acp/categorizer/current//.csv (Note: This Overrides Existing Categorization)

Examples:

event.deviceSeverity,set.event.categoryObject,set.event.categoryBehavior,set.event.categoryTechnique,set.event.categoryDeviceGroup,set.event.categorySignificance, set.event.categoryOutcome

666,/Host/Resource,/Access/Start,,/Application,/Normal,/Success

event.deviceAction,set.event.categoryObject,set.event.categoryBehavior,set.event.categoryDeviceGroup,set.event.categorySignificance,set.event.categoryOutcome

OPEN,/Host/Application/Service,/Communicate/Query,/Firewall,/Normal,/Success

Custom Categorizations

Key-value parsers divide log lines into key-value pairs (key=value), extract the key-value pairs into tokens, and then the tokens are mapped to event fields.

Key-value parsers are used with keyvalue extra processors and syslog subagents Use key-value parsers for secondary processing.

The configuration file name for key- value parsers is

vendor.subagent.sdkkeyvaluefilereader.properties.

Ex: TIME=28/09/11 08:15:00 SRC=194.168.0.12 DST=195.172.0.12 SPT=4236 DPT=80

Key-value parsers have the following properties:

key.delimiter key.delimiter=\\s

key.value.delimiter key.value.delimiter==

key.regexp key.regexp=([^\\s]+)

text.qualifier text.qualifier=“

trim.message True trims the leading and trailing white spaces of the log line.

trim.tokens True trims the leading and trailing white spaces of each token.

trim.keys True trims the leading and trailing white spaces of each key.

Key Value Parsers

Map files are a way that we can set ArcSight event fields based on the information in another field. Essentially the Map file is a CSV file that functions much the same way as a categorization file in that it uses getters followed by setters.

The map files are always located under

/current/user/agent/map/map.X.properties

Can have multiple map files as long as they are named using a sequential number

Allow customers to perform custom field mappings

Allow override of standard parser values

A very simple example is if you do not use DNS for hostname to IP resolution. This can be handled in the map file. The structure of the map file would look something similar to below:

range.event.sourceAddress,set.event.deviceCustomString4,

set.event.deviceCustomString4Label

10.100.0.0-10.100.0.100,QATestLab Building2,Location

10.100.0.101-10.100.0.200,Building1,Location

Creating Map Files

The deviceEventClassId is a method that ArcSight uses to create a unique identifier for each event.

For example all the ArcSight internal agent messages are in the format “agent:xxx” where xxx represent a number. When tracking events using rules we are able to use these numbers as they are unique for each event.

Ex:

event.deviceEventClassId=__concatenateDeleting("Nessus=",NessusID,

"#",Name,"#",Risk,"#",INFO,"%CVE=",CVE,"%Bugtraq=",Bugtraq,"%|#=/@")

%, |, #, =, @, which are used asdelimiters in parsers.

Defining DeviceEventClassId

In some environments it is useful to map certain additional data names to normal ArcSight schema fields. The mapping can vary based on the device vendor and product and can be controlled from the

ArcSight Console, with the mappings stored on the SmartConnector machine. The Get Additional Data Names command specifies the additional data names Assigned to each

device vendor or product combination since the SmartConnector started running.

The Map Additional Data Name Field used must be a valid ArcSight event field.

Additional Data Mapping

Action connectors are built to allow integrations between ArcSight and third party devices for the purpose of allowing the third party device to be controllable from within the ArcSight console.

The user can then execute commands on third party devices from within ArcSight and send the output of those commands back to the console. The remote command can be executed as an action in the correlation rules engine, or as a right click on the action connector. The command is executed from the host that the connector resides on.

While Installing select the Flex CounterACT connector from the list of available connectors. After selecting the connector you will need to enter the name of the Configuration File (the extension will be added automatically). Complete the wizard.

Need For CounterACT Connector

Create a file named “.counteract.properties” in the

\current\user\agent\flexagent directory. This file will contain the commands that you want to be able to execute. Here is an example of such file:

command.count The number of commands that will be supported.

command[x].name The internal name that you want for the command. command[x].displayname The Command display name in the ArcSight console.

command[x].parameter.count Number of parameters that the command will receive.

command[x].parameter[x].name The internal name of the parameter.

command[x].parameter[x].displayname The parameter display name shown in the ArcSight console

command[x].action This is the command line executable that will be executed. This property should be provided as a template with variables that will be replaced by the actual values. A couple of variables are provided by default:

ARCSIGHT_HOME: The absolute path where the connector is running

PLATFORM: A platform code (win32/linux/solaris) Typically used if you have scripts for different OSs

PLATFORM_BINARY_EXT: Set to .bat for win32 and set to .sh for linux and solaris

CounterACT Config Commands

command.count=1

command[0].name=nmapit

command[0].displayname=NMap

command[0].parameter.count=1

command[0].parameter[0].name=ipaddress

command[0].parameter[0].displayname=Ip Address

command[0].action=C:\\NMAP\\NMAP.EXE ${ipaddress}

You can make use of the CounterACT Commands in 2 ways:

From Connector Will pop out the Command Parameter.

From a Rule Will pop out the Fields for your Command Parameter.

It is possible to parse this output and modify the return event to extract the Output you are looking for using a module called SecondLevelRegexParser.

To use the second level parser feature, create the file user/agent/fcp/additionalregexparsing/ngflexcounteract/regex.0.sdkrfilereader.properties

CounterACT Example

CounterACT Command Execution

The REST FlexConnector provides a configurable method to collect security events when you use cloud-based applications such as Box, Salesforce, or Google Apps.

The REST FlexConnector framework allows you to develop FlexConnectors to collect events from vendors by configuring:

OAuth2 for authentication with the vendor.

REST API endpoints exposed by the vendor for event collection.

JSON parsers for parsing and mapping data (retrieved from the REST APIs).

Refer the Rest Flex Connector Development Guide for More Information on this and how to configure one.

Rest API

Register Your Connector Application

Box OAuth2 Registration and Values.

Salesforce OAuth2 Registration and Values.

Google Apps OAuth2 Registration and Values.

Create OAuth2 Client Properties File

Determine Which Events URL (REST API Endpoint) to Use

REST API End Points General Information

Querying Based On Timestamp, Rate Limiting

Box REST API

Salesforce REST APIs

Google Apps REST API

Create a JSON Parser File

Defining the JSON Structure

Defining the JSON Parser

Viewing the Raw JSON Data

REST FlexConnector Configuration Support Tool (restutil)

REST Flexconnector Development Tasks

Enter the name of the parser file, provided the parser file is copied into the user\agent\flexagent dir

Enter the events URL. This is the REST API endpoint which is used by the connector to get the events.

Browse for the OAuth2 Client Properties File. You must create this file from values you obtain when you register your connector application, as well as providing a redirect_uri.

Rest Flex Installation

Create any regular flex connector to read the data corresponding to the

Active List.

Define Tokens only and do not map to fields.

Map tokens to additional data.

Additional Data field name can be anything.

Define the properties to invoke Model Import feature.

Define the property to invoke the custom Velocity Macro file that converts the data into the ArcSight Archive and place it in the user/agent/fcp directory.

Edit the agent.properties and add the following

agent.component[34].maxeventsbeforebuild=20000

agent.component[34].buildmodeldelay=90000

Flex Active List Import

comments.start.with=#

delimiter=,

token.count=1

token[0].name=IP


additionaldata.enabled=true

additionaldata.IP_ADDRESS=IP

additionaldata.CREATE_DATE=__concatenate(__longToString(__currentTimestampInSeconds()),"000")

event.deviceVendor=__stringConstant(ArcSight)

event.deviceProduct=__stringConstant(FlexArchiveImport)

event.deviceCustomString1Label=__stringConstant(model.sender)

event.deviceCustomString1=__stringConstant(DVLabs)

event.deviceCustomString2Label=__stringConstant(model.template)

event.deviceCustomString2=__stringConstant(ips.vm)

Flex Active List Import Example

The SmartConnector for Asset Import lets you define a comma-separated (.csv) file that imports asset modeling details in a batch.

If your asset inventory changes regularly, you can set up a process to update and export this list at regular intervals to update the assets in ArcSight ESM.

Enter the file path to the folder where the CSV files to be imported are stored for the connector to automatically import the assets into ArcSight ESM.

Assign your Asset Import connector to the ArcSight Network or Networks represented by the assets modeled in your CSV file.

The CSV File contains following headers or Fields:address, macAddress, hostname, location, category

Flex Asset Import

agents[x].usenonlockingwindowsfilereader Does not lock the log file

read by the connector on the Windows platform.

agents[x].startatend The default is true. Useful when log files to be

processed already exist and contain data at connector startup or when the log file

rotation takes place.

agents[x].wildcard Specifies a file extension. The Regex FlexConnector

processes only files with the specified file extension.

agents[x].processfoldersrecursively Specifies whether to process log files in

the subfolders of a specified folder.

agents[x].mode Specifies the action to perform on a log file after

the FlexConnector has processed it. (RenameFileInTheSameDirectory, DeleteFile,

PersistFile)

agents[x].foldertable[x].processingmode The Flex Log Processing mode either

on real time or Batch File Mode.

Few Sample Agent Property File Important Configurations

Advanced Regex Usages and Scenarios.

Agent Property file Important Configurations.

Collection of all Other Useful Files (Token Operations List, ArcSight CEF Fields….)

Basic Troubleshooting.

Lab Exercises for Practice using Regex.

Using Replay Connectors to test the Sample Logs.

Start Open Forum for Flex Connector Building and Suggestions.(Trust me if I am a coder will create a Site to Test and Generate Online Flex )

Q & A.

Modifications based on user Suggestions.

Future Concepts

Multiple References:

ArcSight Flex Dev Guide.

Protect 724 Posts.

Other Flex Documents and Discussions.

Merge Operations: Hector Aguilar – Macias, Girish Mantry.

Rate the ArcSight content of these Document in the same ArcSight Forum Thread where it is uploaded. Your appreciation and Suggestions are always helpful and motivating. Next Update will provide more useful Snaps for Flex.

Thank You

Copyright V.B 2013

REFERENCES

ARCSIGHT FLEXCONNECTOR TRAINING LEVEL 02€¦ · SNMP Connector Collect logs from snmp traps....

Documents

Transcript of ARCSIGHT FLEXCONNECTOR TRAINING LEVEL 02€¦ · SNMP Connector Collect logs from snmp traps....