Basic System Troubleshooting DPI

7/18/2019 Basic System Troubleshooting DPI

http://slidepdf.com/reader/full/basic-system-troubleshooting-dpi 1/36

10-1Basic System Troubleshooting

Viettel ACTE Technical Training



In this module, we will introduce you to the basic troubleshooting steps for

Allot systems. We will start with the first questions that you should ask

yourself before beginning any troubleshooting process. Next we will seehow to verify that all the system elements are functioning and

communicating as they should. We will then see how to check if

connections are being correctly classified and bandwidth is being allocated

according to the configuration and our expectations. We will review some

common tasks such as verifying the key, checking software version and

more. At the end of this module we will present how to proceed with the

next step: how to create a snapshot file, search the online knowledgebase

and open a support case.





Before starting to troubleshooting a scenario in the network, you should

ask yourself three basic questions.

Firstly, what exactly is wrong? What do you expect to see and how is this

different than actual behavior? When did it start?

Secondly, you should check the environment. Have any recent changes

been made to the network environment or to the Allot solution itself? Make

sure to check the Allot solution architecture as well as other network

elements which may affect Allot solution.

Finally, make sure you check the events log and alarm pane in the

NetXplorer GUI to see if the management system holds any clues as to

the problem being experienced.





We will now review several key steps which can be taken to ensure that

the system is functioning correctly.





You should take particular care when working in a firewall environment, to

ensure that the appropriate ports are open in between all of the network

elements. If not, then the different elements of the Allot solution will not beable to communicate and functionality will be impaired. Here we see a

summary of the different communication protocols used.

Communication between the NX Client and the NX Server can take place

either over TCP:80 (HTTP), or over TCP:443 (HTTPS). GUI Browsing to

the server is performed by the Java RMI protocol. Java communication

between the NX Client and Server requires that TCP ports 1098, 1099 and

4446 are open. In addition, TCP:4457 and TCP:50010 is used for

communicating alarms and port TCP:3873 for catalogs.

The transfer of monitoring and reporting data between NE/SG and theNetXplorer databases is performed over TCP:80 by default. It is possible

to configure port TCP:443 instead. Communication between the NX

Server and NE/SG for configuration purposes is performed by SNMP over

UDP:161. UDP:162 is used for sending events from the NE/SG to the

NetXplorer. In addition, UDP:123 must be open to enable NTP clock

synchronization. In case you define an SNMP client to retrieve SNMP

KPIs, this information will be sent over UDP:161.

Finally, the databases on the NX server used TCP ports 50000, 50001,

50002. NOTE: when working with additional components such as SMP,SP or CSC additional ports may be required. Check the appendix section

of the NX Installation & Administration guide for more details.





All Allot solution elements must be synchronized to the same time. Time

zones may differ between one element and another, yet absolute time

must be the same.

When a NetEnforcer or Service Gateway is added to the NetXplorer it is

configured to use the NetXplorer server as its NTP server (with stratum

level 13)

It is recommended however to synchronize the NetEnforcer or Service

Gateway, the NetXplorer server, distributed collector and any other Allot

solution element with two servers – an external NTP server and the NX

server (in case connectivity with the internet is lost). The server with the

lowest stratum will always take precedence.

When there is no synchronization between the different elements it may

lead to unexpected graph behavior, or to problems in saving policy

changes. NTP related issued are discussed in much greater detail in the

ACPP Allot advanced training course.





To define an external NTP on a NetXplorer Server, edit the following file:

/etc/ntp.conf on a Linux server and c:\Allot\ntp\ntp.conf on a Windowsserver. Modify this file and add this line:

server <NTP SERVER IP> minpoll 6 maxpoll 8 iburst

The minimum time interval between sending packets is defined

using minpoll. The maximum time interval between sending the packets is

defined by maxpoll. These options specify the minimum and maximum poll

intervals for NTP messages, as a power of 2 in seconds. (minpoll 6 = 2^6

= 64sec, maxpoll 8 = 2^8 = 256sec ). iburst speeds up the initial

synchronization by sending a burst of eight packets instead of the usual

one, The packet spacing is normally 2s.

When working on a Windows based NX server, you should also:

1. Disable automatic synchronization with the internet time server. From

the control panel open Date & Time. On the Internet Time tab uncheck

the Automatic synchronization with internet time server check box

2. Disable the Windows time synchronization service. Open the Services

control panel, Double click Windows Time service. The Properties

dialog appears. Now change the startup type to Disabled

3. Finally, restart the NTP service on the NetXplorer server. On a Linux

server type the command: service ntpd restart. On a Windows server:open the Services control panel. Right click the Startup type column of

the Network Time Protocol service and select Restart





Next, you will need to configure each NE/SG synchronize with the external

NTP Server.

To do so use the go config ips command with the – ts parameter:

go config ips – ts <ntp1:ntp2:ntp3>

ntp1, ntp2, and ntp3 represent IPs of different NTP servers that the NE/SG

can synchronize with. The NetEnforcer or Service Gateway will

automatically synchronize with the NTP server that has the lowest stratum

value (stratum levels define the accuracy of the NTP server).

For example, if you have two external NTP servers with IP addresses of

10.31.68.48 and 10.0.120.1, the command would be: go config ips – ts

10.31.68.48:10.0.120.1Remember, the NX server is not a reliable NTP server, and it is strongly

recommended to use external NTP servers if they are available.

Note: in case the time difference between the NE/SG and the new

configured NTP server is more than 30 seconds backwards, the NE/SG

may reboot in order to synchronize.





In order to verify that the NE/SG is up and running, open its configuration

window. In the Navigation pane, select and right-click the NE/SG in the

network tree and select Configuration from the popup menu. TheConfiguration window for the selected entity is displayed.

On the “General” tab, the “Status” field will show you if the system is active

or in bypass. For the Service Gateway you can also check the status of

each blade via the “Slots&Boards” tab. Choose a blade from the graphical

representation of the screen. Below the graphic you will see each sensor

and its current reading as well as the overall “board status” for that blade.





Alternatively, you can check system and blades status using a CLI

command. The go config view network command can be run on any

NE/SG. In the “system status” field, you will see if the system is “active” orin “bypass” mode. In the SG output you will see a column called “card

status” which indicates the status of each blade in the system.

NOTE: Instructions for connecting to the in-line platform and logging into

the CLI are provided in Module 2: Introducing In-Line Platforms.





Now that we know the system is up and running, we will check what is

happening to the connections flowing through it. How are they classified?

What is the allocated bandwidth?





When you want to check connection classification, NetXplorer monitoring

is the place to start. Open an NE/Line/Pipe/VC graph to see if each rule is

getting the expected bandwidth. Open a host graph when you are lookingfor information on a specific IP. You can then drill down or limit the graph

to see additional details.





To check connection classification via CLI, use the acstat command. The

acstat CLI command is a tool for troubleshooting classification of traffic by

the NetEnforcer or Service Gateway. The information can be viewed eitheras a total number of connections, in an extended and detailed form, or in a

specific, filtered format. Full details of acstat usage are discussed in the

advanced ACPP Allot training course.

In order to view the total number of connections on an NE/SG, type the

CLI command acstat. This will show you the current total number of

connections, and will also break them down into protocol type categories:

TCP, UDP, any IP and non IP.

The output will be displayed per XLR, which is the processor of the NE/CC

on the SG.

Running acstat on a multi blade devices will display the total number of

connections per Core Controller, and per each XLR on each Core

Controller. (The CC-200 of the SG-Sigma has one XLR and the CC-300 of

the SG-Sigma E has 2 XLRs).





Running acstat –ix displays an extended view of all connections.

You can see the protocol that this connection was classified as, itsinternal and external IP address and port, and the VC each connection

was classified to.

The state of the connection is also displayed. The options are:

DROP – Allot’s DART engine has decided to drop this session as per

policy configuration.

WIRE4EVER – Allot’s DART engine has completed the matching process,

and a decision was made about the service of this session.

PARSED – Allot’s DART engine completed the matching process, and is

keeping track on the session in order to retrieve more information.Example: FTP sessions in active mode will stay “parsed”, since we are

waiting for the data connections

UNWIRED – Allot’s DART engine has seen the “Syn” packet but has not

yet completed the matching process. This means that it is in the middle of

identification.

Other fields supply enhanced information about the connection. Details

about these fields are discussed in the advanced ACPP Allot training

course.

In case you see connections are not classified as you expect them to be –

this is the time to contact Allot Support.





Another display option for the acstat command is to display connections

per rule. You can list connections per active VCs: acstat -lvc, Pipes:

acstat –lpipe, or Lines: acstat -lline.

Here we can see an example of the command acstat –lvc which displays

all active VCs in the system. You can see the total number of connections

classified to this rule, and which connections were accepted / dropped.

In the column furthest to the left, you can see the rule QID. This is an ID

number which is assigned to each specific rule. We will see later on when

this ID can be used for troubleshooting. This ID is a set of 5 numbers:

• The first number is the ID of the defined Line.

• The second number is the ID of the defined pipe or pipe template• If the pipe is created from a pipe template, the pipe ID (template ID) will

be followed by the pipe instance ID.

• The next (4th) number is the ID of the virtual channel within the pipe.

• If the VC is created from a VC template, the VC ID (template ID) will be

followed by a 5th number representing the VC instance within the

template.

A Pipe ID will only include the first 3 numbers, and a Line ID will only

include the first number.

Note: The fallback rule ID will always be 1.





When you want to check if bandwidth was allocated correctly as per the

enforcement policy you have configured, NetXplorer monitoring is the

place to start. Open an NE/Line/Pipe/VC graph to check the rate of trafficwhich is flowing through each of the policy entities. Open a utilization

graph for the rules configured with a maximum bandwidth QoS, to check

the rule behavior. You can then drill down or limit the graph to see

additional details.





To check current bandwidth allocation via CLI, use the acmon command.

This command is a central tool for troubleshooting quality of service

issues.

Running this command will display the inbound/outbound traffic per

physical interface.

You can use this command to verify that all links see traffic.

This command will run continuously until stopped. You can stop it using

the keyboard ‘Ctrl’ button together with the ‘c’ button.

As with the acstat command, acmon has different filter and display

options. We will review one of them now.





In case you want to check the allocated bandwidth at a specific moment

for a specific rule, you can type one of the following:

• acmon –v <VC ID>

• acmon – p <pipe ID>

• acmon –l <line ID>

The ID is the same ID as seen earlier with the acstat command. The

output is displayed per Core Controller (for the Service Gateway) and per

XLR. It runs once and then stops.





Now we will see few common tasks which should be performed before

contacting Allot Support.





The first thing to verify is the existence of a valid license key. The NE/SG

license key can be checked by selecting the NE/SG from the network tree,

right clicking and choosing “configuration”. The details of the license andits expiration date are listed in the “Identification and Key” tab. Here you

can see the key expiration date. Verify that the key expiration date is valid,

and that all features you purchased are enabled.

Note: The NE/SG license expiration date is synchronized with the NE/SG

support contract expiration date. If a support contract has expired for a

particular in-line platform then APU will be disabled for it. Protocol Pack

updates can only be pushed from the NetXplorer to in-line platforms for

which APU is enabled.





Alternatively, the in-line platform license can be checked via CLI. The go

config view key command can be entered on any Service Gateway or

NetEnforcer. The output displays the activation key, followed by a list offeatures. For each feature you will see whether or not they are enabled or

disabled. If the feature you require is listed as disabled, this is because the

key entered does not enable it.

To get a new license for an additional feature, you will be asked to provide

your box key. This is done by entering the boxkey command, as seen on

the screen.





The NX License key can be checked by selecting “NetXplorer Application

Server Registration” from the tools menu. Here you can see the key

expiration date. Verify that the key expiration date is valid, and that allfeatures you purchased are enabled.

Note: The NX license expiration date is synchronized with the NX support

contract expiration date. In case the NX license has expired, APU will be

disabled and protocol updates cannot be downloaded to the NetXplorer.





Next check what software version is running on your NE/SG. Via the GUI

open the configuration window of the NE/SG, and go to the Identification &

Key tab. At the bottom of this dialog box, you can see the software versionand protocol pack currently used by the NE/SG. Alternately, you can check

the software version using the CLI command actype.





In order to check the NetXplorer software version, go to the Tools menu

and choose About NetXplorer. The window displayed here will be shown,

with the current software version of the NetXplorer.





Finally, in case you need additional assistance with troubleshooting the

Allot Solution, now is the time to contact Allot Support. In this section we

will see how to do that.





A “snapshot” is a zip file that can be produced for both the NE/SG and the

NetXplorer.

The snapshot contains log files, Virtual Channel definitions, system

settings and much more.

The snapshot gives us a precise picture of what was happening inside

NE/SG and/or NetXplorer when a particular event occurred and as such, it

is an essential troubleshooting tool for customer support.





To create a snapshot on a Linux server, enter create_snapshot_logs.sh

This script takes all the relevant logs and prepares a snapshot file that canbe sent via e-mail. Please note that this file can be large at times (approx.

9MB). The snapshot will be created in

/opt/allot/tmp/snapshot_<date>.tar.gz

On a Windows server, the \allot\bin directory contains a batch file called

create_snapshot_logs.bat. The snapshot will be created under

\allot\tmp\snapshot_<date>.tar.gz





Using the CLI command snapshot, you can generate a snapshot

command on an in-line platform.

For a NetEnforcer the file name will be: snapshot.date_time.tgz.

For a Service Gateway, the file name will be:

snapshot.system.date_time.tgz and it will include logs from all blades.

The snapshot file will be created in the following directory:

/usr/local/SWG/snapshots/.





In order to use the Allot Online Knowledge base, first you have to login to

Allot Support Area. In order to do so, open Allot support page:

http://www.allot.com/support.html. Type in your user (email address) andpassword.

In case you don’t have user and password yet, register at the bottom of

the screen and you will receive your login details by email once you have

been verified as an Allot partner or customer.





Here you can see the Support Area home page. Other than the knowledge

base, you can find in the support area information about your in-line

platforms, register new products, generate new keys for your products andmore. We will focus on the Knowledgebase and Support Cases.

In order to open the Knowledgebase, choose the knowledgebase tab.





The Allot Knowledgebase has 4 main parts:

Free Text search: enter any phrase here to search the answer to yourquestion.

FAQs: Frequently Asked Questions. This section holds some of the most

common questions seen by Allot support teams. View this section to see if

your question was asked before.

Documentation: This sections hold all Allot official documentation. This

includes Operation Guides, Installation and Administration Guides,

Hardware Guides and more for all Allot Products. A documentation CD will

accompany every shipped product. However Allot user guides are

regularly updated. Check here for the most up to date version of the guide

you seek.

Recently Updated: All items recently updated or created will appear here.

In case you still cannot find the answer to your question, contact Allot

Support.





You can also use the Allot Support Area to generate new keys for your

Allot products. You will normally want to generate new keys when you

upgrade to a newer version which requires a new key, or in order to test anew feature.

From the registration page, you can click one of 4 buttons:

• NetEnforcer Key – which will lead you to a page displaying a new

permanent key for your in-line platform. The key will have the same

add-ons as per the original purchase.

• NetEnforcer Temp Key – which will lead you to a key generation page

for your in-line platform. Here you will be able to change some

configuration parameters of the key. The key generated via this page

will be temporary.

• NetXplorer Key – which will lead you to a page displaying a new

permanent key for your NetXplorer. The key will have the same add-ons

as per the original purchase.

• NetXplorer Temp Key – which will lead you to a key generation page

for your NetXplorer. Here you will be able to change some configuration

parameters of the key. The key generated via this page will be

temporary.

NOTE: In order to be able to generate a key your product must have avalid support contract.





Here we see the permanent key generation page. You see the key string

itself, which can be copied from here to the in-line platform or the

NetXplorer. You can also see the S/W version of the generated key. Incase you want to update the software version, make sure to do so from

the registration page, before you click the ‘New Key’ button.

In addition, the support contact end date is displayed here. Note: The Allot

Protocol Update (APU) expiration date will be set to the end date of the

support contract.





Here we see the temporary key generation page.

Via this page you can change the software version of the key, or chooseadditional add-ons for it. The key string will appear only after you click the

button ‘Generate Temporary Key’. The generated key will be valid for one

month. This means all key features will expire after a month.

Note: it is possible to generate up to 3 temporary keys for the same

software version.





Finally we will see how to open a new case with Allot Customer Support.

Go to the ‘Cases’ tab and click the ‘New Case’ button. This page will be

displayed.

Fill in the serial number or boxkey of the NE/SG or NetXplorer you want

report in Registration. Specify the issue in the Subject filed. Supply full

details of the issue in the Description field. Share all the troubleshooting

steps you have performed so far. Supply additional information in the

Case Details section.

It is important to attach snapshot in order to allow Allot Support teams to

fully investigate the issue. Click Submit.

The case will now be seen by one of Allot Support teams around the

world.





Which CLI command will you use to troubleshoot the following?


Basic System Troubleshooting DPI

Documents

Transcript of Basic System Troubleshooting DPI