PIX Troubleshooting Firewalls PPT

Troubleshooting Firewalls

Copyright 2005

Eric Stuhl

Senior Network Consultant

Chesapeake NetCraftsmen

[email protected]

Agenda

• Packet Flow

• Understanding the Architecture

• Failover

• Troubleshooting

• Case Studies

Copyright 2005

• Case Studies

• Tools

• Best Practices

Packet Flow

Copyright 2005

Understanding the Packet Flow

• To effectively troubleshoot a problem, one must first understand the packet path through the network

• Attempt to isolate the problem down to a single device

• Then perform a systematic walk of the packet path through the device to determine where the problem could be

• For problems relating to the Cisco ASA/PIX®/FWSM, always

– Determine the flow: SRC IP, DST IP, SRC port, DST port,

Copyright 2005

– Determine the flow: SRC IP, DST IP, SRC port, DST port,and protocol

– Determine the interfaces through which the flow passes

Note: All firewall issues can be simplified to two interfaces (ingress and egress) and the rules tied to both

Example Flow

• Flow– SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP

– DST IP: 198.133.219.25 DST Port: 80

• Interfaces– Source: Inside Destination: Outside

Copyright 2005

Accounting

With the Flow Defined, Examination of

Configuration Issues Boils Down to Just the Two Interfaces: Inside

and Outside

Eng

Client: 10.1.1.9

Server: 198.133.219.25

Serv

ers

Ou

tsid

e

Packet Flow

Understanding the Packet Flow

• Once the device and flow have been identified, walk the path of the packet through the device

• The packet path through the firewall is illustrated in the next several slides

• For troubleshooting, pay careful attention to where the packet can be dropped in the decision-making

Copyright 2005

the packet can be dropped in the decision-making process

Packet Processing Flow Diagram

• The diagram below will be referenced on the following slides; it is shown here enlarged for reference

Copyright 2005

Packet Processing: Ingress Interface

• Packet arrives on ingress interface

• Input counters incremented

• Software input queue is an indicator of load

Copyright 2005

• Software input queue is an indicator of load

• “No buffers” indicates packet drops, typically due to bursty traffic

ASA-5540# show interface gb-ethernet1interface gb-ethernet1 "inside" is up, line protocol is upHardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214IP address 10.1.1.1, subnet mask 255.255.255.0MTU 1500 bytes, BW 1 Gbit full duplex

5912749 packets input, 377701207 bytes, 0 no bufferReceived 29519 broadcasts, 0 runts, 0 giants0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort286298 packets output, 18326033 bytes, 0 underrunsinput queue (curr/max blocks): hardware (0/25) software (0/0)output queue (curr/max blocks): hardware (0/3) software (0/0)

Packet Processing: Locate Connection

• Check first for existing connection

• If connection exists, flow is matched; bypass ACL check

• If no existing connection

Copyright 2005

• If no existing connection – TCP non-SYN packet, drop and log

– TCP SYN or UDP packet, pass to ACL checks

Established Connection:

ASA-5540# show connTCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO

Syslog Because of No Connection, and Non-SYN Packet:

ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface inside

Packet Processing: ACL Check

• First packet in flow is processed through interface ACLs

• ACLs are first match

• First packet in flow matches ACE, incrementing hit count by

Copyright 2005

• First packet in flow matches ACE, incrementing hit count by one

• Denied packets are dropped and logged

Packet Permitted by ACL:

ASA-5540B# show access-list insideaccess-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)

Syslog When Packet Is Denied by ACL:

ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside"

Packet Processing: Match Translation

• First packet in flow must match a translation rule*

• A quick route lookup is done only to determine egress interface

• Translation rule can be to NAT, or not to NAT

• NAT order of operations dictates what happens with overlapping translation

Copyright 2005

• NAT order of operations dictates what happens with overlapping translation rules

• Once translation rule is matched, connection is created

Translation Exists:ASA-5540# show xlate debugNAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00

Syslogs When No Translation Rule Found: (305005—No NAT; 305006—No Global)ASA-3-305005: No translation group found for tcp src inside:10.1.1.9/11039 dst outside:198.133.219.25/80ASA-3-305006: regular translation creation failed for tcp src inside:10.1.1.9/11040 dst outside:198.133.219.25/80

Translation and NAT Order of Operations

1. nat 0 access-list (nat-exempt)

2. Match existing xlates

3. Match static commands (Cisco ASA/PIX first match;

Copyright 2005

3. Match static commands (Cisco ASA/PIX first match; FWSM best match)

– Static NAT with and without access-list

– Static PAT with and without access-list

4. Match nat commands

– nat <id> access-list (first match)

– nat <id> <address> <mask> (best match)

• If the ID is 0, create an identity xlate

• Use global pool for dynamic NAT

• Use global pool for dynamic PAT

Firs

t Ma

tch

Packet Processing: Inspections/Sec Checks

• Inspections are applied to ensure protocol compliance

• (Optional) Customized AIC inspections

• NAT embedded IPs in payload

Copyright 2005

• NAT embedded IPs in payload

• Additional security checks are applied to the packet

• (Optional) Packets passed to Content Security and Control (CSC) Module

Syslog from Packets Denied by Security Check:

ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside

ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP

Packet Processing: NAT IP Header

• Translate the IP address in the IP header

• Translate the port if performing PAT

• Update checksums

Copyright 2005

• Update checksums

• (Optional) Following the above, pass packet to IPS (AIP) module

Packet Processing: Egress Interface

• Packet is “virtually” forwarded to egress interface (i.e., not forwarded to the driver yet)

• Egress interface is determined first by translation rules

Copyright 2005

• Egress interface is determined first by translation rules

• If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface

• Example:

static (inside,outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0

DM

Z

Inside Outside

172.16.0.0/16172.16.12.0/24

172.16.12.4

Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics

Packet Processing: L3 Route Lookup

• Once on egress interface, an interface route lookup is performed

Copyright 2005

• Only routes pointing out the egress interface are eligible

• Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface

Syslog from Packet on Egress Interface with No Route Pointing Out Interface:

ASA-6-110001: No route to 209.165.202.130 from 10.1.1.9

Packet Processing: L2 Address Lookup

• Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed

• Layer 2 rewrite of MAC header

Copyright 2005

• Layer 2 rewrite of MAC header

• If Layer 2 resolution fails—no syslog

• show arp will not display an entry for the L3 next

hop

• debug arp will indicate if we are not receiving an

ARP reply

Packet Processing: Transmit Packet

• Packet is transmitted on wire

• Interface counters will increment on interface

• Output hardware and software queues indicate

Copyright 2005

• Output hardware and software queues indicate buffering at driver level, interface is busy

ASA-5540# show interface gb-ethernet0interface gb-ethernet0 "outside" is up, line protocol is upHardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626cIP address 172.18.124.64, subnet mask 255.255.255.0MTU 1500 bytes, BW 1 Gbit full duplex

3529518 packets input, 337798466 bytes, 0 no bufferReceived 32277 broadcasts, 0 runts, 0 giants0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort5585431 packets output, 359059032 bytes, 0 underrunsinput queue (curr/max blocks): hardware (0/25) software (0/0)output queue (curr/max blocks): hardware (0/2) software (0/0)

Agenda

• Packet Flow


• Failover

• Troubleshooting

• Case Studies

Copyright 2005

• Case Studies

• Tools

• Best Practices

Cisco ASA/PIX—Understanding the Architecture

• Cisco ASA/PIX platforms process all packets in software (via the central CPU)

– All packets are processed first in…usually also first out

• No software limits on the number of ACEs (rules) that can be configured.

Copyright 2005

that can be configured.

– Each ACE takes a minimum of 212 bytes of RAM.

• Cisco ASA platforms have software imposed connection limits; Cisco PIX platforms do not (bound by RAM)

Classifier in Multimode

• FWSM has a single MAC address for all interfaces

• Cisco ASA/PIX has single MAC for ‘shared’ interfaces (physical interfaces have unique MACs)

– Cisco ASA/PIX 7.2 introduces an option to change this

• When the firewall receives a packet, it must ‘classify’ it to determine where to send the packet

Copyright 2005

‘classify’ it to determine where to send the packet

• Packets are classified based on the following

– Unique ingress interface/VLAN

– Packet’s destination IP matches a global IP


• Inbound traffic is ‘classified’ to context CTX3, based on the global IP in the static

Inside

FWSM

.1

DST IP SRC IP

Example

Copyright 2005

VL

AN

3—

10.1

4.3

.x

Inside

10.1.2.2

10.1.1.2

Inside

10.1.3.2

Inbound Packet

Outside

VLAN 4

VLAN 5

VLAN 6

CTX1

CTX2

CTX3

MSFC

.1

.2

.3

10.14.3.89 192.168.5.4

static (inside,outside) 10.14.3.89 10.1.3.2

Shared Interface


• If the firewall is unable to classify a packet, the following syslog message is generated in the Admin context*

%FWSM-6-106025: Failed to determine security context for

Copyright 2005

%FWSM-6-106025: Failed to determine security context for

packet: vlan3 tcp src 192.168.5.4/1025 dest 10.14.3.25/80

*Added to FWSM 3.1

Agenda

• Packet Flow


• Failover

• Troubleshooting

• Case Studies

Copyright 2005

• Case Studies

• Tools

• Best Practices

Failover Basics

• Active/standby vs. primary/secondary

• Serial vs. LAN failover

• Stateful failover (optional)

• A failover only occurs when either firewall

Internet

Copyright 2005

when either firewall determines the standby firewall is healthier than the active firewall

• Both firewalls swap MAC and IP addresses when a failover occurs

• Level 1 syslogs will give reason of failover

Secondary(Standby)

Primary(Active)

LAN/Serial

Stateful

Corp

PIX(config)# show failoverFailover On Cable status: N/A - LAN-based failover enabledFailover unit PrimaryFailover LAN Interface: Failover Ethernet5 (up)Unit Poll frequency 1 seconds, holdtime 3 secondsInterface Poll frequency 5 seconds, holdtime 25 secondsMonitored Interfaces 2 of 250 maximumVersion: Ours 7.2(3), Mate 7.2(2)Last Failover at: 18:30:43 UTC Apr 12 2007

Verifying Failover Configuration

Interface Monitoring

Copyright 2005

Last Failover at: 18:30:43 UTC Apr 12 2007This host: Primary - Active

Active time: 5371 (sec)Interface outside (10.36.8.36): Normal Interface inside (10.5.5.144): Normal

Other host: Secondary - Standby ReadyActive time: 0 (sec)Interface outside (10.36.8.37): Normal Interface inside (10.5.5.145): Normal

Stateful Failover Logical Update StatisticsLink : Failover Ethernet5 (up)Stateful Obj xmit xerr rcv rerr General 86 0 73 0 sys cmd 74 0 73 0

Monitoring

What Triggers a Failover?

• Power loss/reload (this includes crashes) on the active firewall

• SSM interface/module failure

• The standby becoming “healthier” than the active firewall

Copyright 2005

What Triggers a Failover?

• Two consecutive “hello” messages missed on any monitored interface forces the interface into testing mode

• Both units first verify the link status on the interface

• Next, both units execute the following tests

Copyright 2005

• Next, both units execute the following tests

– Network activity test

– ARP test

– Broadcast ping test

• The first test passed causes the interface on that unit to be marked “healthy”; only if all tests “fail” will the interface be marked “failed”

What to Do After a Failover

• Always check the syslogs to determine root cause

• Example: switch port failed on inside interface of active firewall

Syslogs from Primary (Active) Firewall

Copyright 2005

ASA-4-411002: Line protocol on Interface inside, changed state to down

ASA-1-105007: (Primary) Link status ‘Down’ on interface 1

ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier

ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active

Syslogs from Primary (Active) Firewall

Syslogs from Secondary (Standby) Firewall

ASA# show failover state

What to Do After a Failover

• Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover state

• This information is not saved across reboots

Copyright 2005

ASA# show failover state

State Last Failure Reason Date/TimeThis host - Primary

Failed Ifc Failure 12:56:00 UTC May 6 2007Inside: Failed

Other host - SecondaryActive None

====Configuration State===Sync Done

====Communication State===Mac set

Agenda

• Packet Flow


• Failover

• Troubleshooting

• Case Studies

Copyright 2005

• Case Studies

• Tools

• Best Practices

Troubleshooting Tools

• Syslogs

• Debug commands

• Show commands

• Packet capture

• Packet tracer

Copyright 2005

• Packet tracer

Uses of Syslogs

• Primary mechanism to record traffic to and through the firewall

• The best troubleshooting tool available

Archival Purposes Debugging Purposes

Copyright 2005

Buffered

SSH Client

InternetSyslog Server

SNMP Server

Console

Trap .Syslog

ASA Syslog Level vs. Number of Messages

Log Level

Description

Number of Messages (SUM)

Ver. 6.3 Ver. 7.0 Ver. 7.2 Ver. 8.0 Ver. 8.1

0 Emergencies 0 0 0 0 0

1 Alerts 41 (41) 62 (62) 77 (77) 78 (78) 87 (87)

Copyright 2005

1 Alerts 41 (41) 62 (62) 77 (77) 78 (78) 87 (87)

2 Critical 21 (62) 29 (91) 35 (112) 49 (127) 50 (137)

3 Errors 74 (136) 274 (365) 334 (446) 361 (488) 363 (500)

4 Warnings 56 (192) 179 (544) 267 (713) 280 (768) 281 (781)

5 Notifications 21 (213) 161 (705) 206 (919) 216 (984) 218 (999)

6 Informational 95 (308) 234 (939) 302 (1221) 335 (1319) 337 (1336)

7 Debugging 15 (323) 217 (1156) 258 (1479) 266 (1585) 267 (1603)

FWSM Syslog Level vs. Number of Messages

Log Level

Description

Number of Messages (SUM)

Ver. 2.3 Ver. 3.1 Ver. 3.2 Ver. 4.0

0 Emergencies 0 0 0 0

1 Alerts 58 (58) 67 (67) 67 (67) 67 (67)

Copyright 2005

1 Alerts 58 (58) 67 (67) 67 (67) 67 (67)

2 Critical 21 (79) 29 (96) 29 (96) 29 (96)

3 Errors 94 (173) 305 (401) 306 (402) 318 (414)

4 Warnings 131 (304) 194 (595) 196 (598) 199 (613)

5 Notifications 26 (330) 167 (762) 169 (767) 178 (791)

6 Informational 116 (446) 245 (1007) 248 (1015) 255 (1046)

7 Debugging 23 (469) 225 (1232) 225 (1240) 226 (1272)

What Are Modifiable Syslog Levels?

• Modifiable syslog levels

– Allows one to move any syslog message to any level

• Problem

[no] logging message <syslog_id> level <level>

Levels

0—Emergency

1—Alert

Copyright 2005

• Problem

– You want to record what exec commands are being executed on the firewall; syslog ID 111009 records this information, but by default it is at level 7 (debug)

%PIX-7-111009: User ‘johndoe’

executed cmd: show run

The problem is we don’t want to log all 1602 other syslogs that are generated at debug level

1—Alert

2—Critical

3—Errors

4—Warnings

5—Notifications

6—Informational

7—Debugging

How to Create Modifiable Syslog Levels

• Lower syslog message 111009 to level 3 (error)– ASA(config)# logging message 111009 level 3

– Or

– ASA(config)# logging message 111009 level error

[no] logging message <syslog_id> level <level>

Solution

Copyright 2005

• Now our syslog looks as follows– %ASA-3-111009: User ‘johndoe’ executed cmd: show run

• To restore the default syslog level– ASA(config)# no logging message 111009 level error

– Or

– ASA(config)# logging message 111009 level 7

http://www.cisco.com/en/US/docs/security/asa/asa80/system/m

essage/logmsgs.html

Debug Commands

1. Debugs should not be the first choice to troubleshoot a problem

2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution

3. Debugs are not conditional*

Copyright 2005

3. Debugs are not conditional*

4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug

* Crypto Conditional Debugging was added to Cisco ASA/PIX 8.0

Debug ICMP Trace

• Valuable tool used to troubleshoot connectivity issues

• Provides interface and translation information to

http://www.cisco.com

Internet

Copyright 2005

• Provides interface and translation information to quickly determine flow

• Echo-replys must be explicitly permitted through ACL, or ICMP inspection must be enabled

ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22

ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2

Example debug icmp trace output

Logging Debugs to Syslog

• Problem

– Log only debug output to syslog

• Solution

– Create a logging list with only syslog ID 711001

– Enable debug output to syslogs

– Log on the logging list

Copyright 2005

– Log on the logging list

ASA(config)# logging list C-MUG message 711001

ASA(config)# logging debug-trace

ASA(config)# logging trap C-MUG

Show Output Filters

• Use output filters to filter the output of show command to only the information you want to see

• To use them, at the end of show <Command>, use the pipe character “|” followed by

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>

Copyright 2005

pipe character “|” followed by– begin Start displaying the output beginning at the first

match of the RegEx, and continue to display the remaining output

– include Display any line that matches the RegEx

– exclude Display any line that does not match the RegEx

– grep Same as include

– grep –v Same as exclude

Example: Show Output Filters

Examples

• Display the interface stats starting with the ‘inside’ interface

– show interface | begin inside

• Display the access-list entries that contain address 10.1.1.5

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>

Copyright 2005

– show access-list | grep 10.1.1.5

• Display the config, except for the access-lists– show run | exclude access-list

• Display only access-list entries that have non-zero hitcounts– show access-list | grep –v hitcnt=0

• Display a count of the number of connections each host has– show local-host | include host|count/limit

Note: You must include a space on either side of the pipe for the command to be accepted; also, trailing spaces are counted

Show CPU Usage

• Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets

• FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near

Copyright 2005

during ACL compilation CPU is expected to be near 100% until ACL is compiled

• The show cpu usage command displays the CPU

over time as a running average

pixfirewall# show cpu usage

CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%

*First introduced in Cisco PIX OS version 6.0(1)/FWSM 1.1(1)

Show Traffic

• The show traffic command displays the traffic

received and transmitted out each interface of the firewall

ASA# show traffic

outside:

received (in 124.650 secs):

295468 packets 167218253 bytes

Copyright 2005


2370 pkts/sec 1341502 bytes/sec

transmitted (in 124.650 secs):



inside:







Show Xlate and Show Xlate Debug

• The show xlate command displays information

about the translations through the firewall

• You can limit the output to just the local or global IP

ASA# show xlate2 in use, 2381 most usedGlobal 172.18.124.68 Local 10.1.1.9

Copyright 2005

ASA# show xlate debug2 in use, 2381 most usedFlags: D - DNS, d - dump, I - identity, i - inside, n - no random,

o - outside, r - portmap, s - static

NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:02:03 timeout 3:00:00

TCP PAT from inside:10.9.9.3/4101 to outside:172.18.124.65/1024 flags r idle 0:00:08 timeout 0:00:30

Global 172.18.124.68 Local 10.1.1.9PAT Global 172.18.124.65(1024) Local 10.9.9.3(4101)

“debug” Adds interface names, idle and xlate

timeouts

ASA# show conn2 in use, 64511 most used

TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIOUDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags –

Show Conn and Show Conn Detail

ASA# show conn detail“detail” Adds uptime and

timeout in 7.2(4), 8.0(4)

Idle Time,Bytes Transferred

Connection Flags

‘real’ Interface names added in

7.2(4), 8.0(4)

Copyright 2005

ASA# show conn detail2 in use, 64511 most usedFlags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,

B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump,E - outside back connection, F - outside FIN, f - inside FIN,G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,i - incomplete, J - GTP, j - GTP data, K - GTP t3-responsek - Skinny media, M - SMTP data, m - SIP media, n - GUPO - outbound data, P - inside back connection, q - SQL*Net data,R - outside acknowledged FIN,R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS,X - inspected by service module

TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101,flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127

UDP outside:172.18.124.1/123 dmz:10.1.1.9/123,flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431

timeout in 7.2(4), 8.0(4)

Example—Connection Build Up

1. Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags “saA”

2. The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show “A”

3. The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered “up” (U flag)

Copyright 2005

3ACK

5Data

1SYN

4Data

2SYN+ACK

considered “up” (U flag)

4. The outside device sends the first data packet; the connection is updated and an “I” is added to the flags to indicate the firewall received Inbound data on that connection

5. Finally, the inside device has sent a data packet and the connection is updated to include the “O” flag

UsaAAUIOUI

Connection Flags

Client Server

OutsideInside

Example—Connection Teardown

1. Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an “f” to indicate that the FIN was received on the Inside interface

2. The outside device immediately responds to the FIN packet with a FIN+ACK; the connection flags are updated to reflect this, and now show “UfFR”

3. The inside device responds to the FIN+ACK with a final ACK and the

Copyright 2005

1FIN

2FIN+ACK UfFRUf3

ACK UfFRr

3. The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists

Connection Flags

Client Server

OutsideInside

Outbound Connection Inbound Connection

Connection Flags—Quick Reference

Copyright 2005

TCP Connection Termination Reasons

• If a TCP connection is built through the firewall, it will always have a teardown reason

• The TCP teardown syslog is logged at level 6

• If you are having problems with connections abnormally closing, temporally increase your logging level (or move the syslog down), and check

Copyright 2005

logging level (or move the syslog down), and check the teardown reason

ASA-6-302014: Teardown TCP connection number for

intf_name:real_IP/real_port to intf_name:real_IP/real_port

duration time bytes number [reason] [(user)]

TCP Connection Termination Reasons—Quick Reference

Reason Description

Conn-TimeoutConnection Ended Because It Was Idle Longer Than the Configured Idle Timeout

Deny Terminate Flow Was Terminated by Application Inspection

Failover Primary ClosedThe Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit

FIN TimeoutForce Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout

Copyright 2005

Flow Closed by Inspection Flow Was Terminated by Inspection Feature

Flow Terminated by IPS Flow Was Terminated by IPS

Flow Reset by IPS Flow Was Reset by IPS

Flow Terminated by TCP Intercept

Flow Was Terminated by TCP Intercept

Invalid SYN SYN Packet Not Valid

Idle TimeoutConnection Timed Out Because It Was Idle Longer Than the Timeout Value

IPS Fail-Close Flow Was Terminated Due to IPS Card Down

SYN Control Back Channel Initiation from Wrong Side

TCP Connection Termination Reasons—Quick Reference (Cont.)

Reason Description

SYN TimeoutForce Termination After Two Minutes Awaiting Three-Way Handshake Completion

TCP Bad Retransmission Connection Terminated Because of Bad TCP Retransmission

TCP Fins Normal Close Down Sequence

TCP Invalid SYN Invalid TCP SYN Packet

TCP Reset-I TCP Reset Was Sent From the Inside Host

Copyright 2005

TCP Reset-I TCP Reset Was Sent From the Inside Host

TCP Reset-O TCP Reset Was Sent From the Outside Host

TCP Segment Partial Overlap Detected a Partially Overlapping Segment

TCP Unexpected Window Size Variation

Connection Terminated Due to a Variation in the TCP Window Size

Tunnel Has Been Torn Down Flow Terminated Because Tunnel Is Down

Uauth Deny Connection Denied by URL Filtering Server

Unknown Catch-All Error

Xlate Clear User Executed the ‘Clear Xlate’ Command

show local-host

• A local-host entry is created for any IP tracked through the firewall

• It groups the xlates, connections, and AAA information

• Very useful for seeing the connections terminating on servers

Copyright 2005

ASA# show local-host 10.1.1.9 detailInterface inside: 1131 active, 2042 maximum active, 0 deniedlocal host: <10.1.1.9>,

TCP connection count/limit = 1/unlimitedTCP embryonic count = 0TCP intercept watermark = 50UDP connection count/limit = 0/unlimited

AAA:user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10)

absolute timeout: 0:05:00inactivity timeout: 0:00:00

Xlate(s):Global 172.18.124.69 Local 10.1.1.9

Conn(s):TCP out 198.133.219.25:80 in 10.1.1.9:11055 idle 0:00:10 Bytes 127 flags UIO

show service-policy

• The show service-policy command is used to quickly

see what inspection policies are applied and the packets matching them

ASA# show service-policyGlobal policy: Service-policy: global_policyClass-map: inspection_defaultInspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0

Copyright 2005

Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0Inspect: ftp, packet 43, drop 0, reset-drop 0Inspect: h323 h225, packet 0, drop 0, reset-drop 0Inspect: h323 ras, packet 0, drop 0, reset-drop 0Inspect: http, packet 562, drop 0, reset-drop 0Inspect: netbios, packet 0, drop 0, reset-drop 0Inspect: rsh, packet 0, drop 0, reset-drop 0Inspect: rtsp, packet 0, drop 0, reset-drop 0Inspect: skinny, packet 349, drop 0, reset-drop 0Inspect: esmtp, packet 0, drop 0, reset-drop 0...

Interface outside:Service-policy: VoIPClass-map: voice_markedPriority:Interface outside: aggregate drop 0, aggregate transmit 349

show service-policy flow

• Use to determine what policies a given flow will match in the Modular Policy Framework (MPF)

• Eventually all policies will be in MPF

ASA# show service-policy flow tcp host 10.0.0.2 host 10.1.1.2 eq 23

Copyright 2005

ASA# show service-policy flow tcp host 10.0.0.2 host 10.1.1.2 eq 23

Global policy:

Service-policy: global_policy

Interface outside:

Service-policy: outside_policy

Class-map: inbound_class

Match: access-list telnet_inbound

Access rule: permit tcp host 10.1.1.2 host 10.0.0.2 eq telnet

Action:

Output flow: set connection timeout tcp 0:05:00

show asp drop

• Packets dropped in the Accelerated Security Path (ASP) will increment a counter

• Frame drop counters are per packet, flow drops are per flow

• Some counters have corresponding syslogs

ASA# show asp drop

Copyright 2005

Frame drop:

Invalid encapsulation (invalid-encap) 10897

Invalid tcp length (invalid-tcp-hdr-length) 9382

Invalid udp length (invalid-udp-length) 10

No valid adjacency (no-adjacency) 5594

No route to host (no-route) 1009

Reverse-path verify failed (rpf-violated) 15

Flow is denied by access rule (acl-drop) 25247101

First TCP packet not SYN (tcp-not-syn) 36888

Bad TCP flags (bad-tcp-flags) 67148

TCP option list invalid (tcp-bad-option-list) 731

TCP MSS was too large (tcp-mss-exceeded) 10942

Bad TCP Checksum (bad-tcp-cksum) 893

*Drop counters are documented in the CMD Ref, under ‘show asp drop’

Packet Capture

• Capture command first introduced in Cisco PIX 6.2; FWSM 2.3; it deprecates the “debug packet” command

capture <capture-name> [access-list <acl-name>] [buffer <buf-size>]

[ethernet-type <type>] [interface <if-name>] [packet-length <bytes>]

[circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>]

[match <prot> {host <sip> | <sip> <mask> | any}

[eq | lt |gt <port>] {host <dip> | <dip> <mask> | any}

[eq | lt | gt <port>]]

[real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]

Copyright 2005

FWSM 2.3; it deprecates the “debug packet” command• 7.2(3) and 8.0(3) added a ‘real-time’ option• ASDM 6.0 adds a ‘capture wizard’• Capture sniffs packets on an interface that match

an ACL, or match line• Key steps

– Create an ACL that will match interesting traffic– Define the capture and bind it to an access-list and interface– View the capture on the firewall, or copy it off in .pcap format

Packet Capture (Cont.)

• Traffic can be captured both before and after it passes through the firewall; one capture on the inside interface, one capture on the outside interface

• Capture buffer saved in RAM (default size 512KB)

• Default is to stop capturing when buffer is full

Copyright 2005

• Default is to stop capturing when buffer is full

• Default packet length is 1518 bytes

• Copy captures off via TFTP or HTTPS

OutsideInside

Capture In Capture Out

Where Packets Are Captured in Packet Flow

Ingress Packets Captured

Egress Packets Captured

Copyright 2005

• Packets are captured at the first and last points they can be in the flow

• Ingress packets are captured before any packet processing has been done on them

• Egress packets are captured after all processing (excluding L2 source MAC rewrite)

Capture Command: Example

• Problem: User on the inside with an IP of 10.1.3.2 is having a problem accessing www.cisco.com (197.133.219.25); the user is getting PATed to 192.168.2.2

OutsideInside

Capture In Capture Out

Internet

www.cisco.com

Copyright 2005

OutsideInside Internet

198.133.219.2510.1.3.210.1.3.2 192.168.2.2

Step 1: Create ACL for Both Inside and Outside Interface

Step 2: Create Captures on Both Inside and Outside Interface

Step 3: Have Inside User Access www.cisco.com

Step 4: Copy the Captures Off to a TFTP Server

Step 5: Analyze Captures with Sniffer Program

Capture Command: Example

• Step 1: Create ACL for both inside and outside interface– ! Outside Capture ACL

Access-list 100 permit tcp host 192.168.2.2 host 198.133.219.25 eq 80Access-list 100 permit tcp host 198.133.219.25 eq 80 host 192.168.2.2

! Inside Capture ACLAccess-list 101 permit tcp host 10.1.3.2 host 198.133.219.25 eq 80Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.3.2

• Step 2: Create captures on both inside and outside interface– capture out access-list 100 interface outside packet-length 1518

Copyright 2005

– capture out access-list 100 interface outside packet-length 1518capture in access-list 101 interface inside packet-length 1518

• Step 3: Have inside user access www.cisco.com

• Step 4: Copy the captures off to a TFTP server– ! ASA ver 7.0+ / FWSM 3.0+ copy capture

copy /pcap capture:out tftp://10.1.3.5/out.pcapcopy /pcap capture:in tftp://10.1.3.5/in.pcap

! PIX ver 6.x / FWSM 2.3 copy capture copy capture:out tftp://10.1.3.5/out.pcap pcapcopy capture:in tftp://10.1.3.5/in.pcap pcap

– Or copy using https:

– https://<FW_IP>/capture/out/pcap

Packet Capture: Example

• Step 5: Analyze captures with sniffer program

Outside CAP

Copyright 2005

Inside CAPOutbound SYN, No SYN+ACK

Capturing Packets Dropped by the ASP

• Capture all packets dropped by the ASP– ASA# capture drops type asp-drop all

• Capture on a specific drop reason– ASA# capture drops type asp-drop invalid-tcp-hdr-length

Copyright 2005

ASA# capture drop type asp-drop ?

acl-drop Flow is denied by configured rule

all All packet drop reasons

bad-crypto Bad crypto return in packet

bad-ipsec-natt Bad IPSEC NATT packet

bad-ipsec-prot IPSEC not AH or ESP

bad-ipsec-udp Bad IPSEC UDP packet

bad-tcp-cksum Bad TCP checksum

bad-tcp-flags Bad TCP flags

Packet Tracer

• Packet tracer is the future of troubleshooting configuration issues (and many other issues)

• Introduced in version 7.2 and ASDM 5.2

• A packet can be traced by:

– Defining the packet characteristics via the CLI

– Capturing the packets using the ‘trace’ option

Copyright 2005

– Capturing the packets using the ‘trace’ option

Packet Tracer: Overview

• A packet tagged with the ‘trace’ option is injected into the interface, and processed in the data-plane

• Each action taken on the packet is recorded in the packet itself

• When the packet reaches the egress interface, or is dropped, it is punted to the control-plane

Copyright 2005

or is dropped, it is punted to the control-plane

• The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configuration

Packet Tracer: Creating Packet via CLI

• From the CLI, define the input interface along with source and destination IPs and portspacket-tracer input <intf> <proto> <src_ip> <src_port> <dst_ip>

<dst_port>

• Example—Trace the flow from inside host 10.1.1.2 to

Copyright 2005

• Example—Trace the flow from inside host 10.1.1.2 to http://www.cisco.com (198.133.219.25)

ASA# packet-tracer input inside tcp 10.1.1.2 1025 198.133.219.25 80

Packet Tracer: Example Output

ASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 2

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Copyright 2005

Result: ALLOW

Config:

access-group in in interface inside

access-list in extended permit tcp any any eq www


Phase: 3

Type: INSPECT

Subtype: np-inspect

Result: ALLOW

Config:

class-map match-all inspection_default

match default-inspection-traffic

policy-map global_policy

class inspection_default

inspect http

service-policy global_policy global


Packet Tracer: Example Output (Cont.)

...

Phase: 10

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (inside) 1 10.1.1.0 255.255.255.0


Dynamic translate 10.1.1.2/4 to 209.165.201.3/516 using netmask 255.255.255.255

Copyright 2005

...

Phase: 15

Type: ROUTE-LOOKUP

Subtype: output and adjacency

Result: ALLOW

Config:


found next-hop 209.165.201.1 using egress ifc outside

adjacency Active

next-hop mac address 000a.f331.83c0 hits 0

>>>>Packet successfully forwarded to fast path<<<<

Packet Tracer: Tracing Captured Packet

• Create a capture using the ‘trace’ option

• Find the packet in the capture you want traced

ASA# capture inside access-list web interface inside trace .

ASA# show capture inside

68 packets captured

Copyright 2005

• Then select that packet to be traced

ASA# show capture inside trace packet-number 4 .

68 packets captured

1: 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: S

2: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: S ack

3: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: . ack

4: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: P ack

5: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: . Ack

...

Packet Tracer: ASDM

• ASDM includes a nice GUI front-end to the packet tracer tool

• It is located off the Tools menu

• Input the packets characteristics in the top half

• Actions taken on the packet are shown in the bottom half, along with associated config and links

Copyright 2005

bottom half, along with associated config and links back to modify that config entry in ASDM

Define Packet

Packet Tracer: ASDM (Screen Shot)

Copyright 2005

Link Back to Edit Rule

Matching Config

Action

Final Result

Agenda

• Packet Flow


• Failover

• Troubleshooting

• Case Studies

Copyright 2005

• Case Studies

• Tools

• Best Practices

Case Study

• Intermittent Access to Web Server

Copyright 2005

Case Study: Intermittent Access to Web Server

Problem

• Most external clients are not able to load company’s web page

Copyright 2005

10.1.1.50

Internet

ASA-5510

HTTP Requests to 192.168.1.50

Clients

Web Server

NATed to 10.1.1.50


Traffic Spike

Copyright 2005


• show perfmon indicates high number of embryonic connections

ASA-5510# show perfmon

PERFMON STATS: Current AverageXlates 0/s 0/sConnections 2059/s 299/s

Copyright 2005

Connections 2059/s 299/sTCP Conns 2059/s 299/sUDP Conns 0/s 0/sURL Access 0/s 0/sURL Server Req 0/s 0/sTCP Fixup 0/s 0/sTCP Intercept Established Conns 0/s 0/sTCP Intercept Attempts 0/s 0/sTCP Embryonic Conns Timeout 1092/s 4/sHTTP Fixup 0/s 0/sFTP Fixup 0/s 0/sAAA Authen 0/s 0/sAAA Author 0/s 0/sAAA Account 0/s 0/s

VALID CONNS RATE in TCP INTERCEPT: Current AverageN/A 95.00%

ASA-5510#


• Issue show conn to see ‘who’ is creating the connections

ASA-5510# show conn54764 in use, 54764 most usedTCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aBTCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB

Random Sources Embryonic Conns

Copyright 2005

TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aBTCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aBTCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aBTCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aBTCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aBTCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aBTCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aBTCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aBTCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aBTCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aBTCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aBTCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aBTCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aBTCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aBTCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB. . .


Traffic Permitted

Copyright 2005

Connection Count Jumps

Permitted

SYN Flood Detected


• Apply TCP Intercept to stop the SYN flood attack

access-list 140 extended permit tcp any host 192.168.1.50 eq www!

Copyright 2005

!class-map protectdescription Protect web server from attacksmatch access-list 140!policy-map interface_policyclass protectset connection embryonic-conn-max 100

!service-policy interface_policy interface outside


Few clients represent

Copyright 2005

TCP Intercept applied

Few clients represent 50+ % of traffic


• Apply per-client-max option to limit the number of connections any single client can establish

access-list 140 extended permit tcp any host 192.168.1.50 eq www!

Copyright 2005

!class-map protectdescription Protect web server from attacksmatch access-list 140!policy-map interface_policyclass protectset connection embryonic-conn-max 100 per-client-max 25

!service-policy interface_policy interface outside


per-client-max

Copyright 2005

TCP Intercept


Attacks Being Mitigated by ASA

Copyright 2005

Attacks Still Occurring

Case Study

• Poor Voice Quality

Copyright 2005

Case Study: Poor Voice Quality

Problem

• Poor Outbound Voice Quality at SOHO sites

Outbound RTP Stream

Copyright 2005

100 Mbps 100 Mbps Cable Modem

2 Mbps WAN

ASA-5505


Solution: Traffic Shaping

• What is Traffic Shaping, and why is it needed here?

• Why won’t Policing work?

• Why won’t Priority Queuing alone work?

Copyright 2005

• Why won’t Priority Queuing alone work?

100 Mbps 100 Mbps

Cable Modem

2 Mbps

WAN

ASA-5505

Shape to2 Mbps

Case Study: Poor Voice Quality –Configuration Example (Traffic Shaping)

class-map voice-traffic

match dscp af13 ef

!

policy-map qos_class_policy

Solution

� Prioritize voice traffic and shape all traffic down to 2 Mbps on the outside interface.

Copyright 2005

policy-map qos_class_policy

class voice-traffic

priority

!

policy-map qos_outside_policy

class class-default

shape average 2000000

service-policy qos_class_policy

!

service-policy qos_outside_policy interface outside

� To view statistics on the operation of the shaper, use the command show service-policy shape


Things to keep in mind:

� Shaping can only be applied to the class ‘class-default’

� Shaping only works in the outbound direction on an interface

� The shaping value is in bits per second, and must be a

Copyright 2005

� The shaping value is in bits per second, and must be a multiple of 8000

� The shaping policy is applied to all sub-interfaces on a physical interface

� Not supported on the ASA-5580 platform

� Not supported in Transparent or Multi-context mode

Agenda

• Packet Flow


• Failover

• Troubleshooting

• Case Studies

Copyright 2005

• Case Studies

• Tools

• Best Practices

ASDM

• Run as a standalone application using the ADSM Launcher

• This allows for one-stop access to multiple firewalls

• ASDM 6.0 adds “Upgrade Wizard” to upgrade ASA and ASDM software direct from cisco.com

• ASDM 6.1 works with both ASA 8.1 and 8.0 releases

• ASDM 6.1F works with FWSM 4.0, 3.2 and 3.1 releases

Copyright 2005

• ASDM 6.1F works with FWSM 4.0, 3.2 and 3.1 releases

ASDM Home Page

Device Information

Copyright 2005

CPU, Memory, Conns/Sec, Interface Traffic

Real-Time Syslogs

Using ASDM for Monitoring

Up to Four Different Graphs Can Be Displayed

Copyright 2005

Great forMonitoring

Trends

ASDM: Editing Rules from the Log Viewer

Select Log Entry

Copyright 2005

Select Log Entry from Viewer

Right-Click on Message to View or Edit Associated Rule

ASDM 6.0

� Drag-and-drop and in-place editing for simplified policy editing

� User interface customization with dockable windows and toolbars

Copyright 2005

� New Firewall Dashboardthat provides at-a-glance status of firewall services

� Live ACL hitcount in firewall rule table for easy policy auditing

Output Interpreter

Great Tool for Catching

Linked Off the Technical Support and

Documentation—Tools and Resources

Section on CCO

Copyright 2005

Great Tool for Catching Configuration Errors

Paste in the “show run” Output and Hit “Submit”

Output Interpreter: Example Output

Warning: Unused Statics

Copyright 2005

Warning: Unapplied

Crypto Map

Warning: Invalid Crypto Map

https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl

Online Learning Modules on CCO

• Great way to learn about new features in the ASA

• Located on CCO

• From http://www.cisco.com select: Products and Solutions

– Security

• ASA 5500 Series Adaptive Security Appliances

Copyright 2005

• ASA 5500 Series Adaptive Security Appliances

– Training resources

» Online learning modules

• Direct link

– http://www.cisco.com/en/US/partner/products/ps6120/tsd_products_support_online_learning_modules_list.html

Agenda

• Packet Flow


• Failover

• Troubleshooting

• Case Studies

Copyright 2005

• Case Studies

• Tools

• Best Practices

Cisco ASA/PIX/FWSM Best Practices

• Enable ip verify reverse-path on all interfaces

• Set embryonic and maximum connection counts on static and nat statements; for 7.2.1+ use per-client-max

• Configure logging to syslog server

• Move messages you want to see to lower levels, instead

Copyright 2005

• Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you don’t want to see

• Disable telnet access! Use SSH for management access

• Enable authentication for management access (console/SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallback

Cisco ASA/PIX/FWSM Best Practices

• Restrict DMZ access inbound to your internal networks

• Baseline CPU load, connection counts, xlate counts, and traffic (per interface)

• Run the latest maintenance release in your train

• Upgrade major feature trains only when you need

Copyright 2005

• Upgrade major feature trains only when you need new features, or after train has ‘matured’

Cisco ASA/PIX Release Process

7.0(2)

Maintenance

7.0(4)

EoL

7.0(5)

7.0(2.1) 7.0(2.2) 7.0(2.3)InterimImages

7.0(6) 7.0(7) 7.0(9)

FutureGD

7.1(2)

7.0(8)

GD

Copyright 2005

7.0(1)

Maintenance Trains

7.1(1)

Major Feature Releases

7.2(1) 8.0(2)

7.2(2) 7.2(3)

8.0(3)

Time

8.2(1)

Bug Fixes Waterfall

Down

8.1(1)

7.2(4) 7.2(5)

8.0(4)

8.1(2)

Cisco PIX - End of Sale / End of Life Milestones

Cisco has Announced the End-of-Saleand End-of-Life Dates for CiscoPIX Security Appliances

Copyright 2005

� End of Sale: July 28, 2008

� Last day of sale for software, accessories,and licenses: January 28, 2009

� End of Software Maintenance Releases: July 28, 2009

� End of Support / End of Life: July 27, 2013

Gotcha’s in Upgrading to 7.0

• Cisco PIX-515s (non-E) and Cisco PIX-535 should be upgraded from monitor mode

• Upgrading from monitor mode requires you to copy the 7.0 image over twice

– Once from monitor mode (to boot 7.0 and format flash)

Copyright 2005

– Once from monitor mode (to boot 7.0 and format flash)

– Once after 7.0 is up and flash has been formatted (to save image in flash)

• The upgrade process automatically converts your pre-7.0 config to the new 7.0 CLI

• If there were any errors during the config conversion process, view them by issuing– show startup-config errors

Cisco PIX Password Recovery

• Password recovery can only be performed by uploading the password recovery utility to the Cisco PIX from monitor mode via TFTP

• Password recovery will also remove any AAA commands

• A password recovery utility is created for each

Copyright 2005

• A password recovery utility is created for each major Cisco PIX release (6.1, 6.2, 6.3); however, the utilities are backwards compatible

– Example: np63.bin for Cisco PIX 6.3 and prior releases

http://www.cisco.com/warp/public/110/34.shtml

monitor> interface 1

monitor> address 14.36.1.88

monitor> file np63.bin

monitor> gateway 14.36.1.1

monitor> server 172.18.108.26

monitor> tftp

tftp [email protected] via 14.36.1.1............... Received 92160 bytes

Example: Cisco PIX Password Recovery

InsideTFTP

172.18.108.26 14.36.1.88

Example

Copyright 2005

Do you wish to erase the passwords? [yn] y

The following lines will be removed from the configuration:

enable password 8Ry2YjIyt7RRXU24 encrypted

passwd 2KFQnbNIdI.2KYOU encrypted

aaa authentication serial console LOCAL

aaa authentication telnet console LOCAL

aaa authentication ssh console LOCAL

aaa authentication enable console LOCAL

Do you want to remove the commands listed above from the configuration? [yn] y

Passwords and aaa commands have been erased.

Rebooting..

Case Study: Out-of-Order Packets

• Inspections require ordered packets

• Packets sent to the SSM (AIP and CSC) require ordered packets

• Cisco ASA/PIX will buffer up to three packets by default

• Buffering can be increased on ASA by using the

Copyright 2005

• Buffering can be increased on ASA by using the ‘queue-limit’ option under the tcp-map

Case Study: Out-of-Order Packets

• Some networks have high numbers of out-of-order packets; often caused by asymmetric traffic flows

• If the out-of-order packet buffer isn’t large enough, traffic is dropped and packets must be retransmitted

Problem

Copyright 2005

OutsideInside

Client Server

Packet 10

10.16.9.2192.168.1.30

Packet 12

Packet 13

Packet 14

Packet 15

Buffer

Dropped by Firewall

Packet 11Dropped on Network

Case Study: Out-of-Order PacketBuffering Example

• How to detect?ASA# show asp drop

Frame drop:

...

TCP packet SEQ past window 46331

TCP packet buffer full 90943

...

� How to fix?

Copyright 2005

access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0

!

tcp-map OOO-Buffer

queue-limit 6

!

class-map tcp-options

match access-list OOB-nets

!


class tcp-options

set connection advanced-options OOO-Buffer

!


� How to fix?

Case Study: Out-of-Order PacketBuffering Example

• How to verify?

ASA# show service-policy

Global policy:

Service-policy: global_policy

Class-map: inspection_default

...

Class-map: tcp-options

Copyright 2005

Class-map: tcp-options

Set connection policy:

Set connection advanced-options: OOB-Buffer

Retransmission drops: 0 TCP checksum drops : 0

Exceeded MSS drops : 0 SYN with data drops: 0

Out-of-order packets: 2340 No buffer drops : 0

Case Study: TCP MSS

• MSS is the Maximum Segment Size—or the maximum amount of data that can be sent in a single packet

• The MSS is set in the SYN packets

• The device that receives the MSS advertisement cannot send more data in a single packet to the

Copyright 2005

cannot send more data in a single packet to the peer than specified by the MSS

Case Study: TCP MSS

• Some servers have broken TCP stacks and ignore the MSS advertised by the Client

• The firewall will drop packets that exceed the advertised MSS

Problem

Copyright 2005

OutsideInside

Client Server

SYN MSS=1380

SYN+ACK MSS=1400

DATA=1390

10.16.9.2192.168.1.30

Case Study: TCP MSS Example

• How to detect?ASA# show asp drop

Frame drop:

TCP MSS was too large 943

%ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to

inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390

� How to fix?

Copyright 2005

access-list MSS-hosts permit tcp any host 10.16.9.2

!

tcp-map mss-map

exceed-mss allow

!

class-map mss

match access-list MSS-hosts

!


class mss

set connection advanced-options mss-map

!


� How to fix?

Case Study: TCP MSS Example

• How to verify?ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518

ASA# show capture mss-capture

0 packets captured

0 packets shown

� How else could you verify?

Copyright 2005

� How else could you verify?

Case Study: Out of Memory

• Users are unable to access the Internet

• No new connections are working

• All old (long lived) connections continue to work

Step 1: Check the Syslogs

Problem

Copyright 2005

Step 1: Check the Syslogs

%PIX-3-211001: Memory allocation Error

%PIX-3-211001: Memory allocation Error

Step 2: Check the Amount of Free Memory Available

Hardware: PIX-515E, 64 MB RAM

pixfirewall# show memory

Free memory: 714696 bytes

Used memory: 66394168 bytes

------------- ----------------

Total memory: 67108864 bytes


• Cisco PIX image (run from RAM)

• Configuration

• IPSec database

• Xlates (translations)

• Connections

What Can Eat Up 64 MB on a

Cisco PIX-515E?

Step 3: What Eats Up Memory (RAM) on the Cisco PIX?

Copyright 2005

Step 4: Let’s Check the Translations

pixfirewall# show xlate

251 in use, 258 most used

PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716)

PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276)

Global 209.165.201.25 Local 10.1.1.102

PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783)

PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197)

PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)

Varied Source IPs

A Small Global Pool Is Used, Overloading to

a PAT Address

pixfirewall# show conn


TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIU


TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saA




Step 5: Check the Connections

Copyright 2005











. . .

� Q: Why is the connection count so high?


pixfirewall# show traffic

outside:







Ou

tsid

e

Take a Look at the Traffic Load

Copyright 2005

• Vast majority of traffic is coming in the inside interface and going out the outside interface


inside:







Insid

e

Tra

ffic

Flo

w

pixfirewall# show conn count


pixfirewall# show xlate count



Step 6: Review What We Know and Take Action

Copyright 2005

Conn Count Is Very High, but xlate Count Is Low

� Many connections per xlate

� Probably one, or a few hosts, are generating the vast majority of connections

� Most likely due to a virus on the host(s)

pixfirewall# show local-host | include host|count/limit

local host: <10.1.1.131>,

TCP connection count/limit = 0/unlimited

UDP connection count/limit = 0/unlimited

local host: <10.1.1.51>,




Step 7: Find the Host(s) Generating All the Connections

Only Show Lines That Have the Word “host” or

“count/limit” in Them

Copyright 2005


local host: <10.1.1.236>,



. . .

local host: <10.1.1.99>,



� Host 10.1.1.99 is eating up all the connections, and they are TCP-based connections

pixfirewall# show local-host 10.1.1.99

Interface inside: 250 active, 250 maximum active, 0 denied

local host: <10.1.1.99>,


TCP embryonic count = 146606


Step 8: Now That We Found the Host, Let’s Look at the Connections It Is Generating

Note: All Connections

Copyright 2005



Xlate(s):

Global 209.165.201.21 Local 10.1.1.99

Conn(s):











. . .

Note: All Connections Are Embryonic

Connections to Random Destinations on TCP/135–

MS Blaster


• Cisco PIX provides two methods to limit the number of connections per host

– TCP intercept

– Max connections

Question: Which One Can Be Used Here?

Copyright 2005

• TCP intercept won’t help because the source address is valid

• Limiting the maximum number of connections each internal host can have is the only option

Question: Which One Can Be Used Here?


Step 9: Limit Infected Host(s) Impact on Network

• Configure the MAX TCP connections for NATed hosts to be 50

• Note: The local-host must be cleared before the new connection limits are applied

pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0

Copyright 2005

pixfirewall(config)# clear local-host 10.1.1.99

pixfirewall(config)# show local-host 10.1.1.99

Interface inside: 250 active, 250 maximum active, 0 denied

local host: <10.1.1.99>,

TCP connection count/limit = 50/50


TCP intercept watermark = unlimited


. . .

The Infected Host Is Limited to 50 TCP

Connections


pixfirewall# show conn count


pixfirewall# show memory

Free memory: 47716152 bytes

Used memory: 19392712 bytes

------------- ----------------

Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit

Copyright 2005

• Things look much better now

• Question: How could we configure the Cisco PIX so the connection limit was only applied to the one host (10.1.1.99) which was infected with the virus?

------------- ----------------

Total memory: 67108864 bytes

nat (inside) 1 10.1.1.99 255.255.255.255 50 0

nat (inside) 1 0.0.0.0 0.0.0.0 0 0

PIX Troubleshooting Firewalls PPT

Documents

Transcript of PIX Troubleshooting Firewalls PPT