Troubleshooting Firewalls

© 2008, Cisco Systems, Inc. All rights reserved.Networkers SEC-3020 – Troubleshooting Firewalls

1

© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 1BRKSEC-3020 – Troubleshooting Firewalls14470_04_2008_c3


Troubleshooting Firewalls

BRKSEC-3020


2


Agenda

Packet Flow

Understanding the Architecture

Failover

Troubleshooting

Case Studies

Tools

Best Practices

Note: Cisco IOS® firewall is covered in SEC-3000 (Troubleshooting Cisco IOS Security Features) and will not be covered in this presentation


Packet Flow


3


Understanding the Packet Flow

To effectively troubleshoot a problem, one must first understand the packet path through the network

Attempt to isolate the problem down to a single device

Then perform a systematic walk of the packet path through the device to determine where the problem could be

For problems relating to the Cisco ASA/PIX®/FWSM, always

Determine the flow: SRC IP, DST IP, SRC port, DST port,and protocol

Determine the interfaces through which the flow passesNote: All firewall issues can be simplified to two interfaces (ingress and egress) and the rules tied to both


Accounting

Example Flow

FlowSRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCPDST IP: 198.133.219.25 DST Port: 80

InterfacesSource: Inside Destination: Outside

With the Flow Defined, Examination of

Configuration Issues Boils Down to Just the Two Interfaces: Inside

and Outside

Eng

Client: 10.1.1.9

Server: 198.133.219.25

Inside

Servers DMZ

Partner O

utside

Hosting

Packet Flow


4


Understanding the Packet Flow

Once the device and flow have been identified, walk the path of the packet through the device

The packet path through the firewall is illustrated in the next several slides

For troubleshooting, pay careful attention to where the packet can be dropped in the decision-making process


Packet Processing Flow Diagram

The diagram below will be referenced on the following slides; it is shown here enlarged for reference


5


Packet Processing: Ingress Interface

Packet arrives on ingress interface

Input counters incremented

Software input queue is an indicator of load

“No buffers” indicates packet drops, typically due to bursty trafficASA-5540# show interface gb-ethernet1interface gb-ethernet1 "inside" is up, line protocol is upHardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214IP address 10.1.1.1, subnet mask 255.255.255.0MTU 1500 bytes, BW 1 Gbit full duplex

5912749 packets input, 377701207 bytes, 0 no bufferReceived 29519 broadcasts, 0 runts, 0 giants0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort286298 packets output, 18326033 bytes, 0 underrunsinput queue (curr/max blocks): hardware (0/25) software (0/0)output queue (curr/max blocks): hardware (0/3) software (0/0)


Packet Processing: Locate Connection

Check first for existing connection

If connection exists, flow is matched; bypass ACL check

If no existing connection TCP non-SYN packet, drop and log

TCP SYN or UDP packet, pass to ACL checks

Established Connection:

ASA-5540# show connTCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO

Syslog Because of No Connection, and Non-SYN Packet:

ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface inside


6


Packet Processing: ACL Check

First packet in flow is processed through interface ACLs

ACLs are first match

First packet in flow matches ACE, incrementing hit count by one

Denied packets are dropped and logged

Packet Permitted by ACL:

ASA-5540B# show access-list insideaccess-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)

Syslog When Packet Is Denied by ACL:

ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside"


Packet Processing: Match Translation

First packet in flow must match a translation rule*

A quick route lookup is done only to determine egress interface

Translation rule can be to NAT, or not to NAT

NAT order of operations dictates what happens with overlapping translation rules

Once translation rule is matched, connection is created

Translation Exists:ASA-5540# show xlate debugNAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00

Syslogs When No Translation Rule Found: (305005—No NAT; 305006—No Global)ASA-3-305005: No translation group found for tcp src inside:10.1.1.9/11039 dst outside:198.133.219.25/80ASA-3-305006: regular translation creation failed for tcp src inside:10.1.1.9/11040 dst outside:198.133.219.25/80


7


Translation and NAT Order of Operations

1. nat 0 access-list (nat-exempt)2. Match existing xlates3. Match static commands (Cisco ASA/PIX first match;

FWSM best match)Static NAT with and without access-listStatic PAT with and without access-list

4. Match nat commandsnat <id> access-list (first match)nat <id> <address> <mask> (best match)

If the ID is 0, create an identity xlateUse global pool for dynamic NATUse global pool for dynamic PAT

First Match


Packet Processing: Inspections/Sec Checks

Inspections are applied to ensure protocol compliance

(Optional) Customized AIC inspections

NAT embedded IPs in payload

Additional security checks are applied to the packet

(Optional) Packets passed to Content Security and Control (CSC) Module

Syslog from Packets Denied by Security Check:

ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside

ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP


8


Packet Processing: NAT IP Header

Translate the IP address in the IP header

Translate the port if performing PAT

Update checksums

(Optional) Following the above, pass packet to IPS (AIP) module


Packet Processing: Egress Interface

Packet is “virtually” forwarded to egress interface (i.e., not forwarded to the driver yet)

Egress interface is determined first by translation rules

If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used todetermine egress interface

Example:

static (inside,outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0D

MZ

Inside Outside

172.16.0.0/16 172.16.12.0/24

172.16.12.4

Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics


9


Packet Processing: L3 Route Lookup

Once on egress interface, an interface route lookup is performed

Only routes pointing out the egress interface are eligible

Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface

Syslog from Packet on Egress Interface with No Route Pointing Out Interface:

ASA-6-110001: No route to 209.165.202.130 from 10.1.1.9


Packet Processing: L2 Address Lookup

Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed

Layer 2 rewrite of MAC header

If Layer 2 resolution fails—no syslog

show arp will not display an entry for the L3 next hop

debug arp will indicate if we are not receiving an ARP reply


10


Packet Processing: Transmit Packet

Packet is transmitted on wire

Interface counters will increment on interface

Output hardware and software queues indicate buffering at driver level, interface is busy

ASA-5540# show interface gb-ethernet0interface gb-ethernet0 "outside" is up, line protocol is upHardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626cIP address 172.18.124.64, subnet mask 255.255.255.0MTU 1500 bytes, BW 1 Gbit full duplex

3529518 packets input, 337798466 bytes, 0 no bufferReceived 32277 broadcasts, 0 runts, 0 giants0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort5585431 packets output, 359059032 bytes, 0 underrunsinput queue (curr/max blocks): hardware (0/25) software (0/0)output queue (curr/max blocks): hardware (0/2) software (0/0)


Agenda

Packet Flow


Failover

Troubleshooting

Case Studies

Tools

Best Practices


11


Cisco ASA/PIX—Understanding the Architecture

Cisco ASA/PIX platforms process all packets in software (via the central CPU)

All packets are processed first in…usually also first out

No software limits on the number of ACEs (rules) that can be configured.

Each ACE takes a minimum of 212 bytes of RAM.

Cisco ASA platforms have software imposed connection limits; Cisco PIX platforms do not (bound by RAM)


FWSM—Understanding the Architecture

Packets processed in hardware have zero impact on CPU

Similarly, if the CPU is pegged at 100%, this has zero impact on packets processed in hardware

FWSM Process Most Packets in Hardware, with Some Packets Needing to be Processed in Software—via the Control Point (CP)


12


Fast PathFlow Identification,Security Checks andNAT in Hardware

FWSM Architectural Overview

C6K Backplane Interface

Session ManagerNP 3

Control Point (CP)Central CPU

Fast PathNP 1

Software

Hardware

FWSM

Control PointACL Compilation,Fixups, Syslog, AAA in Software

Session Manager Session Establishment and Teardown, AAA Cache, ACLs

Fast PathNP 2


FWSM—ACL Rule Limits

ACL rule limits are about the only hardware limit users run into

In multimode, ACL resources are divided in 13 equal partitions (12 active, one backup)

If you have less than 12 contexts, wasted reserved space

FWSM 2.3 introducedresource acl-partition—set the number of ACL partitionsallocate-acl-partition—assigns a context to a specific partition

FWSM 3.2 introducedresource-rule—allows further customization of a partition

FWSM 4.0 introduced resource partition— customize the size of individual partitions

* See the Appendix for examples of each of these commands


13


FWSM—Hardware LimitsFWSM has several hardware limits that should be considered in your network design

Limits are hard set, but vary based on single or multimode

Some limits include:

XX

X

XX

3.2 / 4.0 Configurable

3747 (576)5621 (1,537)

32K (32K)999,990 (999,990)

256K (256K)2,498 (384)

2K (2K)4K (4K)

8,744 (1,345)100,567 (14,801)

4.0 (Multimode)

2764 (425)3942 (606)Filter Statements4147 (1,417)32 (32 per)Fixup/Inspect Rules

32K (32K)999,990 (999,990)

256K (256K)1,843 (283)

2K (2K)4K (4K)

6,451 (992)72,806 (11,200)

3.1 (Multimode)

2K (2K)Static NAT Statements1K (1K)Global Statements

3,942 (606)AAA Rules56,627 (9,704)ACEs

32K (32K)Route table entries999,990 (999,990)Connections

256K (256K)NAT Translations3,942 (606)Policy NAT ACEs

2.3 (Multimode)

Increase over 2.3

*Complete list in FWSM docs, Appendix A (Specifications)

Increase over 3.1


Classifier in Multimode

FWSM has a single MAC address for all interfaces

Cisco ASA/PIX has single MAC for ‘shared’ interfaces (physical interfaces have unique MACs)

Cisco ASA/PIX 7.2 introduces an option to change this

When the firewall receives a packet, it must ‘classify’ it to determine where to send the packet

Packets are classified based on the followingUnique ingress interface/VLAN

Packet’s destination IP matches a global IP


14



Inbound traffic is ‘classified’ to context CTX3, based on the global IP in the static

VLA

N 3

—10

.14.

3.x

Inside

10.1.2.2

Inside

10.1.1.2

Inside

10.1.3.2

Inbound PacketOutside

VLAN 4

VLAN 5

VLAN 6

FWSM

CTX1

CTX2

CTX3

MSFC

.1

.2

.3

192.168.5.410.14.3.89SRC IPDST IP

static (inside,outside) 10.14.3.89 10.1.3.2

Shared Interface

Example



If the firewall is unable to classify a packet, the following syslog message is generated in the Admin context*

%FWSM-6-106025: Failed to determine security context for packet: vlan3 tcp src 192.168.5.4/1025 dest 10.14.3.25/80

*Added to FWSM 3.1


15


Agenda

Packet Flow


Failover

Troubleshooting

Case Studies

Tools

Best Practices


Failover Basics

Active/standby vs. primary/secondary

Serial vs. LAN failover

Stateful failover (optional)

A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall

Both firewalls swap MAC and IP addresses when a failover occurs

Level 1 syslogs will give reason of failover

Secondary(Standby)

Primary(Active)

LAN/Serial

Stateful

Internet

Corp


16


PIX(config)# show failoverFailover On Cable status: N/A - LAN-based failover enabledFailover unit PrimaryFailover LAN Interface: Failover Ethernet5 (up)Unit Poll frequency 1 seconds, holdtime 3 secondsInterface Poll frequency 5 seconds, holdtime 25 secondsMonitored Interfaces 2 of 250 maximumVersion: Ours 7.2(3), Mate 7.2(2)Last Failover at: 18:30:43 UTC Apr 12 2007

This host: Primary - ActiveActive time: 5371 (sec)Interface outside (10.36.8.36): Normal Interface inside (10.5.5.144): Normal

Other host: Secondary - Standby ReadyActive time: 0 (sec)Interface outside (10.36.8.37): Normal Interface inside (10.5.5.145): Normal

Stateful Failover Logical Update StatisticsLink : Failover Ethernet5 (up)Stateful Obj xmit xerr rcv rerr General 86 0 73 0 sys cmd 74 0 73 0

Verifying Failover Configuration

Interface Monitoring


What Triggers a Failover?

Power loss/reload (this includes crashes) on the active firewall

SSM interface/module failure

The standby becoming “healthier” than the active firewall


17


What Triggers a Failover?

Two consecutive “hello” messages missed on any monitored interface forces the interface into testing mode

Both units first verify the link status on the interface

Next, both units execute the following tests Network activity test

ARP test

Broadcast ping test

The first test passed causes the interface on that unit to be marked “healthy”; only if all tests “fail” will the interface be marked “failed”


What to Do After a Failover

Always check the syslogs to determine root cause

Example: switch port failed on inside interface of active firewall

ASA-4-411002: Line protocol on Interface inside, changed state to downASA-1-105007: (Primary) Link status ‘Down’ on interface 1ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier

ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active

Syslogs from Primary (Active) Firewall

Syslogs from Secondary (Standby) Firewall


18


ASA# show failover state

State Last Failure Reason Date/TimeThis host - Primary

Failed Ifc Failure 12:56:00 UTC May 6 2007Inside: Failed

Other host - SecondaryActive None

====Configuration State===Sync Done

====Communication State===Mac set


Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover state

This information is not saved across reboots


Agenda

Packet Flow


Failover

Troubleshooting

Case Studies

Tools

Best Practices


19


Troubleshooting Tools

Syslogs

Debug commands

Show commands

Packet capture

Packet tracer


Buffered

Uses of Syslogs

Primary mechanism to record traffic to and through the firewall

The best troubleshooting tool available

SSH Client

Internet

Archival Purposes Debugging Purposes

Syslog Server

SNMP Server

Console

Trap .Syslog


20


ASA Syslog Level vs. Number of Messages

267 (1603)

337 (1336)

218 (999)

281 (781)

363 (500)

50 (137)

87 (87)

0

Ver. 8.1

266 (1585)

335 (1319)

216 (984)

280 (768)

361 (488)

49 (127)

78 (78)

0

Ver. 8.0

302 (1221)234 (939)95 (308)Informational6

258 (1479)217 (1156)15 (323)Debugging7

267 (713)179 (544)56 (192)Warnings4

000Emergencies0

Number of Messages (SUM)

206 (919)

334 (446)

35 (112)

77 (77)

Ver. 7.2

161 (705)21 (213)Notifications5

274 (365)74 (136)Errors3

29 (91)21 (62)Critical2

62 (62) 41 (41)Alerts1

Ver. 7.0Ver. 6.3DescriptionLog

Level


FWSM Syslog Level vs. Number of Messages

226 (1272)

255 (1046)

178 (791)

199 (613)

318 (414)

29 (96)

67 (67)

0

Ver. 4.0

248 (1015)245 (1007)116 (446)Informational6

225 (1240)225 (1232)23 (469)Debugging7

196 (598)194 (595)131 (304)Warnings4

000Emergencies0

Number of Messages (SUM)

169 (767)

306 (402)

29 (96)

67 (67)

Ver. 3.2

167 (762)26 (330)Notifications5

305 (401)94 (173)Errors3

29 (96)21 (79)Critical2

67 (67)58 (58)Alerts1

Ver. 3.1Ver. 2.3DescriptionLog

Level


21


What Are Modifiable Syslog Levels?

Modifiable syslog levelsAllows one to move any syslog message to any level

ProblemYou want to record what exec commands are being executed on the firewall; syslog ID 111009 records this information, but by default it is at level 7 (debug)

%PIX-7-111009: User ‘johndoe’executed cmd: show run

The problem is we don’t want to log all 1602 other syslogs that are generated at debug level

[no] logging message <syslog_id> level <level>

Levels0—Emergency

1—Alert

2—Critical

3—Errors

4—Warnings

5—Notifications

6—Informational

7—Debugging


How to Create Modifiable Syslog Levels

Lower syslog message 111009 to level 3 (error)ASA(config)# logging message 111009 level 3

OrASA(config)# logging message 111009 level error

Now our syslog looks as follows%ASA-3-111009: User ‘johndoe’ executed cmd: show run

To restore the default syslog levelASA(config)# no logging message 111009 level error

OrASA(config)# logging message 111009 level 7

[no] logging message <syslog_id> level <level>

Solution


22


Debug Commands

1. Debugs should not be the first choice to troubleshoot a problem

2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution

3. Debugs are not conditional*

4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug

* Crypto Conditional Debugging was added to Cisco ASA/PIX 8.0


Debug ICMP Trace

Valuable tool used to troubleshoot connectivity issues

Provides interface and translation information to quickly determine flow

Echo-replys must be explicitly permitted through ACL, or ICMP inspection must be enabled

http://www.cisco.com

ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22

ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2

Example debug icmp trace output

Internet


23


Logging Debugs to Syslog

ProblemLog only debug output to syslog

SolutionCreate a logging list with only syslog ID 711001

Enable debug output to syslogs

Log on the logging list

ASA(config)# logging list Networkers message 711001ASA(config)# logging debug-traceASA(config)# logging trap Networkers


Show Output Filters

Use output filters to filter the output of show command to only the information you want to see

To use them, at the end of show <Command>, use the pipe character “|” followed by

begin Start displaying the output beginning at the first match of the RegEx, and continue to display the remaining output

include Display any line that matches the RegEx

exclude Display any line that does not match the RegEx

grep Same as include

grep –v Same as exclude

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>


24


Example: Show Output Filters

ExamplesDisplay the interface stats starting with the ‘inside’ interface

show interface | begin inside

Display the access-list entries that contain address 10.1.1.5show access-list | grep 10.1.1.5

Display the config, except for the access-listsshow run | exclude access-list

Display only access-list entries that have non-zero hitcountsshow access-list | grep –v hitcnt=0

Display a count of the number of connections each host hasshow local-host | include host|count/limit

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>

Note: You must include a space on either side of the pipe for the command to be accepted; also, trailing spaces are counted


Show CPU Usage

Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets

FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled

The show cpu usage command displays the CPU over time as a running average

pixfirewall# show cpu usageCPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%

*First introduced in Cisco PIX OS version 6.0(1)/FWSM 1.1(1)


25


Show Traffic

The show traffic command displays the traffic received and transmitted out each interface of the firewall

ASA# show traffic outside:

received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec

transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec

inside: received (in 124.650 secs):

261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec

transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec


Show Xlate and Show Xlate DebugThe show xlate command displays information about the translations through the firewall

You can limit the output to just the local or global IP

ASA# show xlate debug2 in use, 2381 most usedFlags: D - DNS, d - dump, I - identity, i - inside, n - no random,

o - outside, r - portmap, s - static

NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:02:03 timeout 3:00:00

TCP PAT from inside:10.9.9.3/4101 to outside:172.18.124.65/1024 flags r idle 0:00:08 timeout 0:00:30

ASA# show xlate2 in use, 2381 most usedGlobal 172.18.124.68 Local 10.1.1.9PAT Global 172.18.124.65(1024) Local 10.9.9.3(4101)

“debug” Adds interface names, idle and xlate

timeouts


26


ASA# show conn2 in use, 64511 most used

TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIOUDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags –

Show Conn and Show Conn Detail

ASA# show conn detail2 in use, 64511 most usedFlags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,

B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump,E - outside back connection, F - outside FIN, f - inside FIN,G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,i - incomplete, J - GTP, j - GTP data, K - GTP t3-responsek - Skinny media, M - SMTP data, m - SIP media, n - GUPO - outbound data, P - inside back connection, q - SQL*Net data,R - outside acknowledged FIN,R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS,X - inspected by service module

TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101,flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127

UDP outside:172.18.124.1/123 dmz:10.1.1.9/123,flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431

“detail” Adds uptime and timeout in 7.2(4), 8.0(4)

Idle Time,Bytes Transferred

Connection Flags

‘real’ Interface names added in

7.2(4), 8.0(4)


3 ACK5 Data1 SYN 4Data 2SYN+ACK

Example—Connection Build Up

1. Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags “saA”

2. The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show “A”

3. The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered “up” (U flag)

4. The outside device sends the first data packet; the connection is updated and an “I” is added to the flags to indicate the firewall received Inbounddata on that connection

5. Finally, the inside device has sent a data packet and the connection is updated to include the “O” flag

UsaAAUIOUIConnection Flags

Client Server

OutsideInside


27


1 FIN 2FIN+ACK UfFRUf3 ACK UfFRr

Example—Connection Teardown

1. Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an “f” to indicate that the FIN was received on the Inside interface

2. The outside device immediately responds to the FIN packet with aFIN+ACK; the connection flags are updated to reflect this, and now show “UfFR”

3. The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists

Connection Flags

Client Server

OutsideInside


Outbound Connection Inbound Connection

Connection Flags—Quick Reference


28


TCP Connection Termination Reasons

If a TCP connection is built through the firewall, it will always have a teardown reason

The TCP teardown syslog is logged at level 6

If you are having problems with connections abnormally closing, temporally increase your logging level (or move the syslog down), and check the teardown reason

ASA-6-302014: Teardown TCP connection number for intf_name:real_IP/real_port to intf_name:real_IP/real_portduration time bytes number [reason] [(user)]


TCP Connection Termination Reasons—Quick Reference

Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed TimeoutFIN Timeout

The Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active UnitFailover Primary Closed

Flow Was Terminated by Application InspectionDeny Terminate

Connection Ended Because It Was Idle Longer Than the Configured Idle TimeoutConn-Timeout

Flow Was Terminated by TCP InterceptFlow Terminated by TCP Intercept

Flow Was Reset by IPSFlow Reset by IPS

Flow Was Terminated by IPSFlow Terminated by IPS

Flow Was Terminated by Inspection FeatureFlow Closed by Inspection

Back Channel Initiation from Wrong SideSYN Control

Flow Was Terminated Due to IPS Card DownIPS Fail-Close

Connection Timed Out Because It Was Idle Longer Than the Timeout ValueIdle Timeout

SYN Packet Not ValidInvalid SYN

DescriptionReason


29


TCP Connection Termination Reasons—Quick Reference (Cont.)

Invalid TCP SYN PacketTCP Invalid SYN

Normal Close Down SequenceTCP Fins

Connection Terminated Because of Bad TCP RetransmissionTCP Bad Retransmission

Force Termination After Two Minutes Awaiting Three-Way Handshake CompletionSYN Timeout

Connection Terminated Due to a Variation in the TCP Window Size

TCP Unexpected Window Size Variation

Detected a Partially Overlapping SegmentTCP Segment Partial Overlap

TCP Reset Was Sent From the Outside HostTCP Reset-O

TCP Reset Was Sent From the Inside HostTCP Reset-I

User Executed the ‘Clear Xlate’ CommandXlate Clear

Catch-All ErrorUnknown

Connection Denied by URL Filtering ServerUauth Deny

Flow Terminated Because Tunnel Is DownTunnel Has Been Torn Down

DescriptionReason


show local-host

A local-host entry is created for any IP tracked through the firewall

It groups the xlates, connections, and AAA information

Very useful for seeing the connections terminating on servers

ASA# show local-hostInterface inside: 1131 active, 2042 maximum active, 0 deniedlocal host: <10.1.1.9>,

TCP connection count/limit = 1/unlimitedTCP embryonic count = 0TCP intercept watermark = 50UDP connection count/limit = 0/unlimited

AAA:user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10)

absolute timeout: 0:05:00inactivity timeout: 0:00:00

Xlate(s):Global 172.18.124.69 Local 10.1.1.9

Conn(s):TCP out 198.133.219.25:80 in 10.1.1.9:11055 idle 0:00:10 Bytes 127 flags UIO


30


show service-policyThe show service-policy command is used to quickly see what inspection policies are applied and the packets matching them

ASA# show service-policyGlobal policy: Service-policy: global_policyClass-map: inspection_defaultInspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0Inspect: ftp, packet 43, drop 0, reset-drop 0Inspect: h323 h225, packet 0, drop 0, reset-drop 0Inspect: h323 ras, packet 0, drop 0, reset-drop 0Inspect: http, packet 562, drop 0, reset-drop 0Inspect: netbios, packet 0, drop 0, reset-drop 0Inspect: rsh, packet 0, drop 0, reset-drop 0Inspect: rtsp, packet 0, drop 0, reset-drop 0Inspect: skinny, packet 349, drop 0, reset-drop 0Inspect: esmtp, packet 0, drop 0, reset-drop 0...

Interface outside:Service-policy: VoIPClass-map: voice_markedPriority:Interface outside: aggregate drop 0, aggregate transmit 349


show service-policy flow

Use to determine what policies a given flow will match in the Modular Policy Framework (MPF)

Eventually all policies will be in MPF

ASA# show service-policy flow tcp host 10.0.0.2 host 10.1.1.2 eq 23

Global policy: Service-policy: global_policy

Interface outside:Service-policy: outside_policyClass-map: inbound_classMatch: access-list telnet_inboundAccess rule: permit tcp host 10.1.1.2 host 10.0.0.2 eq telnet

Action:Output flow: set connection timeout tcp 0:05:00


31


show asp drop

Packets dropped in the Accelerated Security Path (ASP) will increment a counter

Frame drop counters are per packet, flow drops are per flow

Some counters have corresponding syslogsASA# show asp drop

Frame drop:Invalid encapsulation (invalid-encap) 10897Invalid tcp length (invalid-tcp-hdr-length) 9382Invalid udp length (invalid-udp-length) 10No valid adjacency (no-adjacency) 5594No route to host (no-route) 1009Reverse-path verify failed (rpf-violated) 15Flow is denied by access rule (acl-drop) 25247101First TCP packet not SYN (tcp-not-syn) 36888Bad TCP flags (bad-tcp-flags) 67148TCP option list invalid (tcp-bad-option-list) 731TCP MSS was too large (tcp-mss-exceeded) 10942Bad TCP Checksum (bad-tcp-cksum) 893

*Drop counters are documented in the CMD Ref, under ‘show asp drop’


Packet Capture

Capture command first introduced in Cisco PIX 6.2; FWSM 2.3; it deprecates the “debug packet” command7.2(3) and 8.0(3) added a ‘real-time’ optionASDM 6.0 adds a ‘capture wizard’Capture sniffs packets on an interface that match an ACL, or match lineKey steps

– Create an ACL that will match interesting traffic– Define the capture and bind it to an access-list and interface– View the capture on the firewall, or copy it off in .pcap format

capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type <type>] [interface <if-name>] [packet-length <bytes>][circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>][match <prot> {host <sip> | <sip> <mask> | any}

[eq | lt |gt <port>] {host <dip> | <dip> <mask> | any}[eq | lt | gt <port>]]

[real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]


32


Packet Capture (Cont.)

Traffic can be captured both before and after it passes through the firewall; one capture on the inside interface, one capture on the outside interface

Capture buffer saved in RAM (default size 512KB)

Default is to stop capturing when buffer is full

Default packet length is 1518 bytes

Copy captures off via TFTP or HTTPS

OutsideInside

Capture In Capture Out


Where Packets Are Captured in Packet Flow

Packets are captured at the first and last points they can be in the flow

Ingress packets are captured before any packet processing has been done on them

Egress packets are captured after all processing (excluding L2 source MAC rewrite)

Ingress Packets Captured

Egress Packets Captured


33


Capture Command: Example

Problem: User on the inside with an IP of 10.1.3.2 is having a problem accessing www.cisco.com (198.133.219.25); the user is getting PATed to 192.168.2.2

OutsideInside

Capture In Capture Out

Internet

www.cisco.com

198.133.219.2510.1.3.2 10.1.3.2 192.168.2.2

Step 1: Create ACL for Both Inside and Outside Interface

Step 2: Create Captures on Both Inside and Outside Interface

Step 3: Have Inside User Access www.cisco.com

Step 4: Copy the Captures Off to a TFTP Server

Step 5: Analyze Captures with Sniffer Program


Capture Command: Example

Step 1: Create ACL for both inside and outside interface! Outside Capture ACLAccess-list 100 permit tcp host 192.168.2.2 host 198.133.219.25 eq 80Access-list 100 permit tcp host 198.133.219.25 eq 80 host 192.168.2.2

! Inside Capture ACLAccess-list 101 permit tcp host 10.1.3.2 host 198.133.219.25 eq 80Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.3.2

Step 2: Create captures on both inside and outside interfacecapture out access-list 100 interface outside packet-length 1518capture in access-list 101 interface inside packet-length 1518

Step 3: Have inside user access www.cisco.comStep 4: Copy the captures off to a TFTP server

! ASA ver 7.0+ / FWSM 3.0+ copy capturecopy /pcap capture:out tftp://10.1.3.5/out.pcapcopy /pcap capture:in tftp://10.1.3.5/in.pcap

! PIX ver 6.x / FWSM 2.3 copy capture copy capture:out tftp://10.1.3.5/out.pcap pcapcopy capture:in tftp://10.1.3.5/in.pcap pcap

Or copy using https:https://<FW_IP>/capture/out/pcap


34


Packet Capture: Example

Step 5: Analyze captures with sniffer program

Outside CAP

Inside CAPOutbound SYN, No SYN+ACK


Capturing Packets Dropped by the ASP

Capture all packets dropped by the ASPASA# capture drops type asp-drop all

Capture on a specific drop reasonASA# capture drops type asp-drop invalid-tcp-hdr-length

ASA# capture drop type asp-drop ?

acl-drop Flow is denied by configured ruleall All packet drop reasonsbad-crypto Bad crypto return in packetbad-ipsec-natt Bad IPSEC NATT packetbad-ipsec-prot IPSEC not AH or ESPbad-ipsec-udp Bad IPSEC UDP packetbad-tcp-cksum Bad TCP checksumbad-tcp-flags Bad TCP flags


35


Packet Capture: Limitations on FWSM

Capture functionality is available on the FWSM starting in 2.3

However, only packets processed by the control point could be captured

FWSM 3.1(1) added support to capture packets in hardware

Only ingress packets were captured

FWSM 3.1(5) both ingress and egress transient packets can be captured which flow through hardware

Capture requires an ACL to be appliedCapture copies the matched packets in hardware to the control point where they are captured; be careful not to flood the control point with too much traffic

Control Point (CP) Central CPU


Session ManagerNP 3

Fast PathNP 1

Fast PathNP 1

FWSM


Packet Tracer

Packet tracer is the future of troubleshooting configuration issues (and many other issues)

Introduced in version 7.2 and ASDM 5.2

A packet can be traced by:Defining the packet characteristics via the CLI

Capturing the packets using the ‘trace’ option


36


Packet Tracer: Overview

A packet tagged with the ‘trace’ option is injected into the interface, and processed in the data-plane

Each action taken on the packet is recorded in the packet itself

When the packet reaches the egress interface, or is dropped, it is punted to the control-plane

The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configuration


Packet Tracer: Creating Packet via CLI

From the CLI, define the input interface along with source and destination IPs and portspacket-tracer input <intf> <proto> <src_ip> <src_port> <dst_ip> <dst_port>

Example—Trace the flow from inside host 10.1.1.2 to http://www.cisco.com (198.133.219.25)

ASA# packet-tracer input inside tcp 10.1.1.2 1025 198.133.219.25 80


37


Packet Tracer: Example OutputASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80

Phase: 1Type: FLOW-LOOKUPSubtype: Result: ALLOWConfig:Additional Information:Found no matching flow, creating a new flow

Phase: 2Type: ACCESS-LISTSubtype: logResult: ALLOWConfig:access-group in in interface insideaccess-list in extended permit tcp any any eq www

Additional Information:

Phase: 3Type: INSPECTSubtype: np-inspectResult: ALLOWConfig:class-map match-all inspection_defaultmatch default-inspection-traffic

policy-map global_policyclass inspection_defaultinspect http

service-policy global_policy globalAdditional Information:


Packet Tracer: Example Output (Cont.)

...

Phase: 10Type: NATSubtype: Result: ALLOWConfig:nat (inside) 1 10.1.1.0 255.255.255.0

Additional Information:Dynamic translate 10.1.1.2/4 to 209.165.201.3/516 using netmask 255.255.255.255

...

Phase: 15Type: ROUTE-LOOKUPSubtype: output and adjacencyResult: ALLOWConfig:Additional Information:found next-hop 209.165.201.1 using egress ifc outsideadjacency Activenext-hop mac address 000a.f331.83c0 hits 0

>>>>Packet successfully forwarded to fast path<<<<


38


Packet Tracer: Tracing Captured Packet

Create a capture using the ‘trace’ option

Find the packet in the capture you want traced

Then select that packet to be tracedASA# show capture inside trace packet-number 4 .

ASA# capture inside access-list web interface inside trace .

ASA# show capture inside68 packets captured1: 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: S 2: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: S ack3: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: . ack 4: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: P ack5: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: . Ack

...


Packet Tracer: ASDM

ASDM includes a nice GUI front-end to the packet tracer tool

It is located off the Tools menu

Input the packets characteristics in the top half

Actions taken on the packet are shown in the bottom half, along with associated config and links back to modify that config entry in ASDM


39


Link Back to Edit Rule

Matching Config

Define Packet

Action

Final Result

Packet Tracer: ASDM (Screen Shot)


Agenda

Packet Flow


Failover

Troubleshooting

Case Studies

Tools

Best Practices


40


Case Study

Intermittent Access to Web Server


Case Study: Intermittent Access to Web Server

Problem

Most external clients are not able to load company’s web page

10.1.1.50

Internet

ASA-5510

HTTP Requests to 192.168.1.50

Clients

Web Server

NATed to 10.1.1.50


41



Traffic Spike



show perfmon indicates high number of embryonic connections

ASA-5510# show perfmon

PERFMON STATS: Current AverageXlates 0/s 0/sConnections 2059/s 299/sTCP Conns 2059/s 299/sUDP Conns 0/s 0/sURL Access 0/s 0/sURL Server Req 0/s 0/sTCP Fixup 0/s 0/sTCP Intercept Established Conns 0/s 0/sTCP Intercept Attempts 0/s 0/sTCP Embryonic Conns Timeout 1092/s 4/sHTTP Fixup 0/s 0/sFTP Fixup 0/s 0/sAAA Authen 0/s 0/sAAA Author 0/s 0/sAAA Account 0/s 0/s

VALID CONNS RATE in TCP INTERCEPT: Current AverageN/A 95.00%

ASA-5510#


42



Issue show conn to see ‘who’ is creating the connections

ASA-5510# show conn54764 in use, 54764 most usedTCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aBTCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aBTCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aBTCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aBTCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aBTCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aBTCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aBTCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aBTCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aBTCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aBTCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aBTCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aBTCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aBTCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aBTCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aBTCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB. . .

Random Sources Embryonic Conns



Connection Count Jumps

Traffic Permitted

SYN Flood Detected


43



Apply TCP Intercept to stop the SYN flood attack

access-list 140 extended permit tcp any host 192.168.1.50 eq www!class-map protectdescription Protect web server from attacksmatch access-list 140!policy-map interface_policyclass protectset connection embryonic-conn-max 100

!service-policy interface_policy interface outside



TCP Intercept applied

Few clients represent 50+ % of traffic


44



Apply per-client-max option to limit the number of connections any single client can establish

access-list 140 extended permit tcp any host 192.168.1.50 eq www!class-map protectdescription Protect web server from attacksmatch access-list 140!policy-map interface_policyclass protectset connection embryonic-conn-max 100 per-client-max 25

!service-policy interface_policy interface outside



TCP Intercept

per-client-max


45



Attacks Still Occurring

Attacks Being Mitigated by ASA


Case Study

Poor Voice Quality


46


Case Study: Poor Voice Quality

Problem

Poor Outbound Voice Quality at SOHO sites

100 Mbps 100 Mbps Cable Modem

2 Mbps WAN

ASA-5505

Outbound RTP Stream



Solution: Traffic ShapingWhat is Traffic Shaping, and why is it needed here?Why won’t Policing work?Why won’t Priority Queuing alone work?

100 Mbps 100 Mbps

Cable Modem

2 Mbps

WAN

ASA-5505

Shape to2 Mbps


47


Case Study: Poor Voice Quality –Configuration Example (Traffic Shaping)

class-map voice-trafficmatch dscp af13 ef!policy-map qos_class_policyclass voice-trafficpriority

!policy-map qos_outside_policyclass class-default shape average 2000000 service-policy qos_class_policy

!service-policy qos_outside_policy interface outside

SolutionPrioritize voice traffic and shape all traffic down to 2 Mbps on the outside interface.

To view statistics on the operation of the shaper, use the command show service-policy shape



Things to keep in mind:Shaping can only be applied to the class ‘class-default’Shaping only works in the outbound direction on an interfaceThe shaping value is in bits per second, and must be a multiple of 8000The shaping policy is applied to all sub-interfaces on a physical interfaceNot supported on the ASA-5580 platformNot supported in Transparent or Multi-context mode


48


Agenda

Packet Flow


Failover

Troubleshooting

Case Studies

Tools

Best Practices


Tools

ASDM

Output interpreter

Online learning modules


49


ASDMRun as a standalone application using the ADSM Launcher

This allows for one-stop access to multiple firewalls

ASDM 6.0 adds “Upgrade Wizard” to upgrade ASA and ASDM software direct from cisco.com

ASDM 6.1 works with both ASA 8.1 and 8.0 releases

ASDM 6.1F works with FWSM 4.0, 3.2 and 3.1 releases


ASDM Home Page

CPU, Memory, Conns/Sec, Interface Traffic

Real-Time Syslogs

Device Information


50


Using ASDM for Monitoring Great forMonitoring

Trends

Up to Four Different Graphs Can Be Displayed


ASDM: Editing Rules from the Log Viewer

Select Log Entry from Viewer

Right-Click on Message to View or Edit Associated Rule


51


ASDM: Syslogs Explained


ASDM 6.0

Drag-and-drop and in-place editing for simplified policy editing

User interface customization with dockable windows and toolbars

New Firewall Dashboardthat provides at-a-glance status of firewall services

Live ACL hitcount in firewall rule table for easy policy auditing


52


Output Interpreter

Great Tool for Catching Configuration Errors

Linked Off the Technical Support and

Documentation—Tools and Resources

Section on CCO

Paste in the “show run”Output and Hit “Submit”


Output Interpreter: Example Output

Warning: Unapplied

Crypto Map

Warning: Invalid Crypto Map

https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl

Warning: Unused Statics


53


Online Learning Modules on CCO

Great way to learn about new features in the ASA

Located on CCO

From http://www.cisco.com select: Products and Solutions

Security

ASA 5500 Series Adaptive Security Appliances

Training resources

Online learning modules

Direct linkhttp://www.cisco.com/en/US/partner/products/ps6120/tsd_products_support_online_learning_modules_list.html


Agenda

Packet Flow


Failover

Troubleshooting

Case Studies

Tools

Best Practices


54


Cisco ASA/PIX/FWSM Best Practices

Enable ip verify reverse-path on all interfaces

Set embryonic and maximum connection counts on static and nat statements; for 7.2.1+ use per-client-max

Configure logging to syslog server

Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you don’t want to see

Disable telnet access! Use SSH for management access

Enable authentication for management access (console/SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallback


Cisco ASA/PIX/FWSM Best Practices

Restrict DMZ access inbound to your internal networks

Baseline CPU load, connection counts, xlate counts, and traffic (per interface)

Run the latest maintenance release in your train

Upgrade major feature trains only when you need new features, or after train has ‘matured’


55


Cisco ASA/PIX Release Process

7.0(1)

7.0(2)

Maintenance Trains

7.0(4)

EoL

7.0(5)

7.0(2.1) 7.0(2.2) 7.0(2.3)InterimImages

7.1(1)

Major Feature Releases

7.2(1) 8.0(2)

7.2(2) 7.2(3)

8.0(3)

Time

8.2(1)

7.0(6) 7.0(7) 7.0(9)

Future

Bug Fixes Waterfall

Down

GD

7.1(2)

8.1(1)

7.2(4) 7.2(5)

8.0(4)

8.1(2)

7.0(8)GD


Cisco PIX - End of Sale / End of Life Milestones

End of Sale: July 28, 2008

Last day of sale for software, accessories,and licenses: January 28, 2009

End of Software Maintenance Releases: July 28, 2009

End of Support / End of Life: July 27, 2013

Cisco has Announced the End-of-Saleand End-of-Life Dates for CiscoPIX Security Appliances


56


Q and A


Recommended Reading

Continue your Cisco Live learning experience with further reading from Cisco Press

Check the Recommended Reading flyer for suggested books

I recommend: Cisco ASA: All-in-One Firewall, IPS, and VPN Adaptive Security Appliance

Available Onsite at the Cisco Company Store


57


Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Passport points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008.

Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.



58


Appendix

Lucky You

This appendix contains some extra information which you may find useful, but I just didn’t have enough time to cover in the lecture

Enjoy… :-)


Appendix

Release trains

Gotcha’s upgrading Cisco PIX to 7.0

Cisco PIX password recovery

Case study– Out of Order Packet Buffering

– TCP MSS issue

– Out of memory

– High CPU

Online Tools (Network Professionals Connection, Bug Toolkit)

Information to include when opening a TAC case


59


Cisco PIX Release Process

6.0(1)

6.0(2)

Maintenance Trains

6.1(2)

6.0(3)

6.1(3) 6.1(4) 6.1(5)

6.0(4)

6.0(2.100) 6.0(2.101) 6.0(2.122)InterimImages

CSCdy54228

Bug Fixes Committedto Multiple Trains

6.1(1)

Major Feature Releases

6.2(1) 6.3(1)

6.2(2) 6.2(3)

6.3(2) 6.3(3)

6.2(4)

6.3(4)

Time


Feature Releases

Cisco PIX/ASA/FWSM Code Base History

6.0(1) 6.1(1) 6.2(1) 6.3(1)

Cisco PIX

1.1(3)

FWSM1.1(1)

Feature Releases

1.1(2)Maintenance

Releases

2.2(1)

Port FeaturesBug Fixes

2.3(1)

2.3(2)

Time

7.0(1)

3.1(1)

7.1(1) 7.2(1)

3.2(1)

Cisco PIX/ASAIn Sync

4.0(1)

3.1(2) 3.1(6)GD

3.1(10)SafeHarbor

3.2(2) 3.2(4)SafeHarbor


60


Gotcha’s in Upgrading to 7.0

Cisco PIX-515s (non-E) and Cisco PIX-535 should be upgraded from monitor mode

Upgrading from monitor mode requires you to copy the 7.0 image over twice

Once from monitor mode (to boot 7.0 and format flash)

Once after 7.0 is up and flash has been formatted (to save image in flash)

The upgrade process automatically converts your pre-7.0 config to the new 7.0 CLI

If there were any errors during the config conversion process, view them by issuing

show startup-config errors


Cisco PIX Password Recovery

Password recovery can only be performed by uploading the password recovery utility to the Cisco PIX from monitor mode via TFTP

Password recovery will also remove any AAA commands

A password recovery utility is created for each major Cisco PIX release (6.1, 6.2, 6.3); however, the utilities are backwards compatible

Example: np63.bin for Cisco PIX 6.3 and prior releases

http://www.cisco.com/warp/public/110/34.shtml


61


monitor> interface 1monitor> address 14.36.1.88monitor> file np63.binmonitor> gateway 14.36.1.1monitor> server 172.18.108.26monitor> tftptftp [email protected] via 14.36.1.1............... Received 92160 bytes

Do you wish to erase the passwords? [yn] yThe following lines will be removed from the configuration:

enable password 8Ry2YjIyt7RRXU24 encryptedpasswd 2KFQnbNIdI.2KYOU encryptedaaa authentication serial console LOCALaaa authentication telnet console LOCALaaa authentication ssh console LOCALaaa authentication enable console LOCAL

Do you want to remove the commands listed above from the configuration? [yn] yPasswords and aaa commands have been erased.

Rebooting..

Example: Cisco PIX Password Recovery

InsideTFTP

172.18.108.26 14.36.1.88

Example


Case Study

Out-of-order packet buffering


62


Case Study: Out-of-Order Packets

Inspections require ordered packets

Packets sent to the SSM (AIP and CSC) require ordered packets

Cisco ASA/PIX will buffer up to three packets by default

Buffering can be increased on ASA by using the ‘queue-limit’ option under the tcp-map


Case Study: Out-of-Order Packets

Some networks have high numbers of out-of-order packets; often caused by asymmetric traffic flows

If the out-of-order packet buffer isn’t large enough, traffic is dropped and packets must be retransmitted

OutsideInside

Client Server

Packet 10

10.16.9.2192.168.1.30

Packet 12Packet 13Packet 14Packet 15

Buffer

Dropped by Firewall

Packet 11Dropped on Network

Problem


63


Case Study: Out-of-Order PacketBuffering Example

How to detect?ASA# show asp dropFrame drop:...TCP packet SEQ past window 46331TCP packet buffer full 90943...

access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0!tcp-map OOO-Buffer

queue-limit 6!class-map tcp-options

match access-list OOB-nets!policy-map global_policy

class tcp-optionsset connection advanced-options OOO-Buffer

!service-policy global_policy global

How to fix?


Case Study: Out-of-Order PacketBuffering Example

How to verify?ASA# show service-policy

Global policy: Service-policy: global_policyClass-map: inspection_default...

Class-map: tcp-optionsSet connection policy: Set connection advanced-options: OOB-BufferRetransmission drops: 0 TCP checksum drops : 0 Exceeded MSS drops : 0 SYN with data drops: 0 Out-of-order packets: 2340 No buffer drops : 0


64


Case Study

TCP MSS (Maximum Segment Size)


Case Study: TCP MSS

MSS is the Maximum Segment Size—or the maximum amount of data that can be sent in a single packet

The MSS is set in the SYN packets

The device that receives the MSS advertisement cannot send more data in a single packet to the peer than specified by the MSS


65


Case Study: TCP MSS

Some servers have broken TCP stacks and ignore the MSS advertised by the Client

The firewall will drop packets that exceed the advertised MSS

OutsideInside

Client Server

SYN MSS=1380

SYN+ACK MSS=1400

DATA=1390

10.16.9.2192.168.1.30

Problem


Case Study: TCP MSS Example

How to detect?ASA# show asp dropFrame drop:

TCP MSS was too large 943

%ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390

access-list MSS-hosts permit tcp any host 10.16.9.2!tcp-map mss-map

exceed-mss allow!class-map mss

match access-list MSS-hosts!policy-map global_policy

class mssset connection advanced-options mss-map

!service-policy global_policy global

How to fix?


66


Case Study: TCP MSS Example

How to verify?ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518

ASA# show capture mss-capture0 packets captured0 packets shown

How else could you verify?


Case Study

Out of Memory


67


Case Study: Out of Memory

Users are unable to access the Internet

No new connections are working

All old (long lived) connections continue to work

Step 1: Check the Syslogs%PIX-3-211001: Memory allocation Error%PIX-3-211001: Memory allocation Error

Step 2: Check the Amount of Free Memory AvailableHardware: PIX-515E, 64 MB RAM

pixfirewall# show memoryFree memory: 714696 bytesUsed memory: 66394168 bytes------------- ----------------Total memory: 67108864 bytes

Problem



Cisco PIX image (run from RAM)

Configuration

IPSec database

Xlates (translations)

Connections

What Can Eat Up 64 MB on a

Cisco PIX-515E?

Step 4: Let’s Check the Translationspixfirewall# show xlate251 in use, 258 most usedPAT Global 209.165.201.26(2379) Local 10.1.1.132(52716) PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276)Global 209.165.201.25 Local 10.1.1.102PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783) PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197) PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)

Step 3: What Eats Up Memory (RAM) on the Cisco PIX?

Varied Source IPs

A Small Global Pool Is Used, Overloading to

a PAT Address


68


pixfirewall# show conn147456 in use, 147456 most usedTCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIUTCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIUTCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saATCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags OIUTCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIUTCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saATCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags OIUTCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIUTCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIUTCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags OIUTCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags OIUTCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIUTCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags OIUTCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saA. . .


Step 5: Check the Connections

Q: Why is the connection count so high?



Vast majority of traffic is coming in the inside interface and going out the outside interface

pixfirewall# show trafficoutside:

received (in 25.000 secs):1475 packets 469050 bytes59 pkts/sec 18762 bytes/sec

transmitted (in 25.000 secs):167619 packets 9654480 bytes6704 pkts/sec 386179 bytes/sec

inside:received (in 25.000 secs):

180224 packets 10410480 bytes7208 pkts/sec 416419 bytes/sec

transmitted (in 25.000 secs):1050 packets 118650 bytes42 pkts/sec 4746 bytes/sec

Out

side

Insi

deTraf

fic F

low

Take a Look at the Traffic Load


69


pixfirewall# show conn count147456 in use, 147456 most used

pixfirewall# show xlate count251 in use, 258 most used


Step 6: Review What We Know and Take Action

Conn Count Is Very High, but xlate Count Is LowMany connections per xlate

Probably one, or a few hosts, are generating the vast majority of connections

Most likely due to a virus on the host(s)


pixfirewall# show local-host | include host|count/limitlocal host: <10.1.1.131>,

TCP connection count/limit = 0/unlimitedUDP connection count/limit = 0/unlimited

local host: <10.1.1.51>,TCP connection count/limit = 2/unlimitedUDP connection count/limit = 0/unlimited

local host: <10.1.1.236>,TCP connection count/limit = 0/unlimitedUDP connection count/limit = 0/unlimited

. . .local host: <10.1.1.99>,

TCP connection count/limit = 146608/unlimitedUDP connection count/limit = 0/unlimited


Step 7: Find the Host(s) Generating All the Connections

Only Show Lines That Have the Word “host” or

“count/limit” in Them

Host 10.1.1.99 is eating up all the connections, and they are TCP-based connections


70


pixfirewall# show local-host 10.1.1.99Interface inside: 250 active, 250 maximum active, 0 deniedlocal host: <10.1.1.99>,

TCP connection count/limit = 146608/unlimitedTCP embryonic count = 146606UDP connection count/limit = 0/unlimited

Xlate(s):Global 209.165.201.21 Local 10.1.1.99

Conn(s):TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags saATCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags saATCP out 64.100.205.160:135 in 10.1.1.99:7774 idle 0:01:43 Bytes 0 flags saATCP out 64.101.182.19:135 in 10.1.1.99:39193 idle 0:01:43 Bytes 0 flags saATCP out 64.102.218.45:135 in 10.1.1.99:16462 idle 0:01:43 Bytes 0 flags saATCP out 64.100.21.120:135 in 10.1.1.99:30322 idle 0:01:43 Bytes 0 flags saATCP out 64.101.25.195:135 in 10.1.1.99:41116 idle 0:01:43 Bytes 0 flags saATCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags saATCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags saATCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags saA. . .


Step 8: Now That We Found the Host, Let’s Look at the Connections It Is Generating

Note: All Connections Are Embryonic

Connections to Random Destinations on TCP/135–

MS Blaster



Cisco PIX provides two methods to limit the number of connections per host

TCP intercept

Max connections

TCP intercept won’t help because the source address is valid

Limiting the maximum number of connections each internal host can have is the only option

Question: Which One Can Be Used Here?


71



Step 9: Limit Infected Host(s) Impact on Network

Configure the MAX TCP connections for NATed hosts to be 50

Note: The local-host must be cleared before the new connection limits are applied

pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0

pixfirewall(config)# clear local-host 10.1.1.99

pixfirewall(config)# show local-host 10.1.1.99Interface inside: 250 active, 250 maximum active, 0 deniedlocal host: <10.1.1.99>,

TCP connection count/limit = 50/50TCP embryonic count = 50TCP intercept watermark = unlimitedUDP connection count/limit = 0/unlimited. . .

The Infected Host Is Limited to 50 TCP

Connections



Things look much better now

Question: How could we configure the Cisco PIX so the connectionlimit was only applied to the one host (10.1.1.99) which was infected with the virus?

pixfirewall# show conn count126 in use, 147456 most used

pixfirewall# show memoryFree memory: 47716152 bytesUsed memory: 19392712 bytes------------- ----------------Total memory: 67108864 bytes

nat (inside) 1 10.1.1.99 255.255.255.255 50 0nat (inside) 1 0.0.0.0 0.0.0.0 0 0

Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit


72


Case Study

High CPU Usage


High CPU Usage on the Cisco PIX

A quick overview of the “show processes” command

pixfirewall(config)# show processes

PC SP STATE Runtime SBASE Stack ProcessHsi 001eab19 008a5a74 00557910 0 008a4aec 3628/4096 arp_timerLsi 001f00bd 00a28dbc 00557910 0 00a27e44 3832/4096 FragDBGCLwe 00119abf 02d280dc 0055b070 0 02d27274 3688/4096 dbgtraceLwe 003e4425 02d2a26c 00557dd8 74440 02d28324 6936/8192 LoggerCrd 001e26fb 0533940c 00557d88 6070290 05338484 3684/4096 557pollLsi 00300a29 04c0f504 00557910 0 04c0e57c 3944/4096 xlate clean

The Name of the Process

Number of msec This Process Has Been on the CPU

Problem: Cisco PIX CPU Running Very High

For more information on the output of the “show processes” command, see http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml


73



Step 1: Determine What Process Is Eating the CPU

Take the difference in output of two “show processes” over a period of time

The following output was a diff of the processes taken one minute apart

Process_Name Runtime (msec)Logger 25940 pix/intf3 18410 557poll 9250 i82543_timer 4180 i82542_timer 2230

In One Minute, These Processes Account for

44 Seconds of CPU Time ~ 73%

The Interface Polling Processes Always Run, and Are Not Counted in

the CPU Usage


pixfirewall(config)# show logSyslog logging: enabled

Standby logging: disabledConsole logging: disabledMonitor logging: disabledBuffer logging: level alerts, 0 messages loggedTrap logging: level warnings, 5919412 messages logged

Logging to lab 172.18.173.123History logging: disabled

. . .pixfirewall(config)# show logSyslog logging: enabled

Buffer logging: level alerts, 0 messages loggedTrap logging: level warnings, 6172472 messages logged

Logging to lab 172.18.173.123


Step 2: Focus on the Processes with High CPU Time

Logging is taking up much of the CPU; let’s review what we have configured to log

This Is Cumulative Since the Cisco PIX Was Last Rebooted

Notice the Change Over a Few Minutes


74


pixfirewall(config)# show logBuffer logging: level warnings, 31527 messages loggedTrap logging: level warnings, 6453127 messages logged

Logging to lab 172.18.173.123

. . .400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab


Enable buffered logging to same level as syslog server, and examine the buffered messages

Cisco PIX’s Interface Address

Syslog Server Is Controlled by a Different Group


pixfirewall(config)# show run | grep auditip audit name IDS info action alarmip audit interface lab IDS


Syslog service was down on the syslog server

ICMP unreachable was generated by syslog server for each syslog message the Cisco PIX sent it

Cisco PIX’s IDS configuration also logged every ICMP unreachable message, creating the exponentially increasing problem

SyslogServer

Outside

Syslog MessageICMP Unreachable

IDS Syslog Message

Lab

Examine IDS Configuration


75


ip audit signature 2001 disableor

no logging message 400011


Bring back up syslog service on server

Take server offline

Configure Cisco PIX to not log IDS ICMP unreachable messages

pixfirewall# show run | grep signatureip audit signature 2001 disable

pixfirewall# show cpu usageCPU utilization for 5 seconds = 2%; 1 minute: 50%; 5 minutes: 99%

Solution



Examine the DIFF of two show processes taken over a one minute interval

Find the process taking up the highest amount of CPU (excluding the polling processes)

Take actions to lower that process’s CPU time

Reexamine the CPU output, and repeat as necessary

Summary


76


FWSM

Additional Architecture Information


FWSM and ACLs

ACLs on the FWSM are compiled on the control point and pushed down into hardware (NP 3)

During compile time, CPU should stay at ~ 99%ACL compile uses all free CPU cycles

Allows compile to complete in shortest time possible

Once compile is complete, rules are attempted to be pushed into hardware

Successful downloadAccess Rules Download Complete: Memory Utilization: 49%

Failed download (exceeded HW memory)ERROR: Unable to add, access-list config limit reached


77


FWSM and ACLs (Multimode)

Use show np 3 acl stats to see the current ACL resource utilization in that context

FWSM/admin(config)# show np 3 acl stats----------------------------

ACL Tree Statistics ----------------------------Rule count : 9584Bit nodes (PSCB's): 8760Leaf nodes : 8761Total nodes : 17521 (max 24260)Leaf chains : 6912Total stored rules: 15673Max rules in leaf : 3Node depth : 32----------------------------

Note: One ACE does not equal one node

Total Number of ACEs

This Is the Hardware Limit



Use show np 3 acl tree to see which ACL tree a context is mapped to

FWSM# show np 3 acl tree--------------------------------------------ACL Tree Instance <-> Context Name (ID) Map--------------------------------------------Tree Instance 0 Context (001) adminTree Instance 1 Context (002) coreTree Instance 2 Context (003) EngineeringTree Instance 3 Context (004) Accounting--------------------------------------------

ACL Tree Number

Context Name


78


FWSM(config)# context AccountingFWSM(config-context)# allocate-acl-partition 0FWSM(config-context)# show np 3 acl tree

--------------------------------------------ACL Tree Instance <-> Context Name (ID) Map--------------------------------------------Tree Instance 0 Context (001) adminTree Instance 1 Context (002) coreTree Instance 2 Context (003) EngineeringTree Instance 0 Context (004) Accounting--------------------------------------------


Use the command resource acl-partition <num-of-partitions> to reduce the number of active partitions created; default is 12

Use the command allocate-acl-partition <num> to assign a context to a specific ACL tree

Both Use Tree 0


FWSM—Resource Rule

FWSM 3.2 introduced resource-rule—allows further customization of a partition

show resource-rule—displays information about the current rule allocation

resource rule nat 10000 acl 2200 filter 400 fixup 595 est 70 aaa 555 console 283

FWSM# show resource rule

Default Configured AbsoluteCLS Rule Limit Limit Max

-----------+---------+----------+---------Policy NAT 1843 1843 10000ACL 74188 74188 74188Filter 2764 2764 5528Fixup 4147 4147 10000Est Ctl 460 460 460Est Data 460 460 460AAA 6451 6451 10000Console 1843 1843 3686

-----------+---------+----------+---------Total 92156 92156

Partition Limit - Configured Limit = Available to allocate92156 - 92156 = 0


79


FWSM—Resource PartitionFWSM 4.0 introduced

resource partition—allows customization of the size of individual partitions (multi-context mode)

FWSM(config)# resource partition 10FWSM(config-partition)# size 1000WARNING: The rule max has been reset based on partition size 1000.The <size> command leads to re-partitioning of ACL Memory.It will not take effect until you save the configuration and reboot.

FWSM# show resource rule partition 10



-----------+---------+----------+---------Total 19219 19219

Partition Limit - Configured Limit = Available to allocate

19219 - 19219 = 0

FWSM# show resource rule partition 10



-----------+---------+----------+---------Total 1000 1000

Partition Limit - Configured Limit = Available to allocate

1000 - 1000 = 0

Before After


FWSM and Control Point

The traffic that makes it to the control point is traffic that requires layer 7 fixup (embedded NAT, or cmd inspection)

FTPVoIP (SIP/SKINNY/H.323/RTSP)DNSXDMCPetc.

Traffic sourced from, or destined to, the FWSM also goes through the control point

SyslogsAAA (RADIUS/TACACS+)URL filtering (WebSense/N2H2)Management traffic (telnet/SSH/HTTPS/SNMP)Failover communicationsRouting protocols (OSPF/ RIP)etc.



Session ManagerNP 3

Fast PathNP 1

Fast PathNP 2

FWSM


80


FWSM and Network Processors

The Session Manager—NP 3Processes first packet in a flowACL checksTranslation creationEmbryonic/established connection countsTCP/UDP ChecksumsSequence number randomizationTCP interceptetc.

The fast path—NP 1 and 2Performs per packet session lookupMaintains connection tablePerforms NAT/PATTCP checks Fragmentation reassemblyetc.



Fast PathNP 1

Fast PathNP 2

FWSM

Session ManagerNP 3



Fast PathNP 1

Fast PathNP 2

FWSM

Session ManagerNP 3


FWSM - Enabling the Completion Unit

Due to the FWSM’s NP architecture, there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the firewall.

This issue might be encountered when performing TCP throughput testing, or passing high speed TCP flows through the FWSM

Examples: CIFS, FTP, AFP, backups

FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt npcompletion-unit to ensure the firewall maintains the packet order (by enabling a hardware knob on the NPs called the Completion Unit).

In multiple mode enter this command in the admin context configuration; It will then be enabled for all contexts on the firewall.

4 3 2 1 4 2 3 1


81


Online Tools

Networking Professionals Connection

Bug Toolkit



Online Open Forum to Ask

Questions

Anyone Can Ask a Question, and

Anyone Can Answer

Regular “Ask the Expert” Events

on Certain Topics


82



http://www.cisco.com/go/netpro

Online Open Forum to Ask

Questions

Anyone Can Ask a Question, and

Anyone Can Answer

Regular “Ask the Expert” Events

on Certain Topics


Bug Toolkit

On the Support Tools and Resources Page


83


Bug Toolkit—Product Selection

Select Security, Then Cisco ASA

5500 Series


Bug Toolkit—Advanced Search

Version

Search Keywords

Severity

Status


84


Bug Toolkit—Search Results

Select Link to View Details of Bug


Bug Toolkit—Bug Details

First Fixed-In Releases


85


Opening a TAC Case

If after using all your troubleshooting tools you still cannot resolve the problem, please open a TAC case

http://www.cisco.com/techsupport/servicerequest/

At a minimum include:Detailed problem description

Output from “show tech”

Optionally include:Syslogs captured during time of problem

Sniffer traces from both interfaces using the “capture” command (capturing only the relevant packets, and saved in pcap format)


Failover Prompt Display Configuration

The firewall’s prompt maybe changed to display certain keywordUsage

prompt <keyword> [<keyword> ...]

Syntaxkeywords:Hostname Configures the prompt to display the hostnameDomain Configures the prompt to display the domainContext Configures the prompt to display the current context (multi-mode only)Priority Configures the prompt to display the 'failover lan unit' settingState Configures the prompt to display the current traffic handling stateSlot Configures the prompt to display the slot location (when applicable)

ExampleFWSM(config)# prompt hostname domain priority state slot

FWSM/cisco.com/sec/actNoFailover/4(config)#


86


ASA# show failover history==========================================================================From State To State Reason==========================================================================Disabled Negotiation Set by the CI config cmdNegotiation Just Active No Active unit foundJust Active Active Drain No Active unit foundActive Drain Active Applying Config No Active unit foundActive Applying Config Active Config Applied No Active unit foundActive Config Applied Active No Active unit foundActive Failed Interface check==========================================================================


Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover history

This information is not saved across reboots


Disabling Debug Commands

Starting with Cisco PIX 6.3/FWSM 2.3, all debugs can be disabled simultaneously by issuing no debug allor undebug all (un all for short)

undebug all

ASA# show debugdebug icmp tracedebug sip

ASA# un allASA# show debugASA# . .


87


Syslog Enhancements in 7.0 and 3.1

Logging via e-mail to an SMTP server

Event classes to group syslogs

Logging to an FTP server

Logging to Flash

Logging debug messages encapsulated in syslogs

Troubleshooting Firewalls

Documents

Transcript of Troubleshooting Firewalls