Troubleshooting Firewalls
Copyright 2005
Eric Stuhl
Senior Network Consultant
Chesapeake NetCraftsmen
Agenda
• Packet Flow
• Understanding the Architecture
• Failover
• Troubleshooting
• Case Studies
Copyright 2005
• Case Studies
• Tools
• Best Practices
Packet Flow
Copyright 2005
Understanding the Packet Flow
• To effectively troubleshoot a problem, one must first understand the packet path through the network
• Attempt to isolate the problem down to a single device
• Then perform a systematic walk of the packet path through the device to determine where the problem could be
• For problems relating to the Cisco ASA/PIX®/FWSM, always
– Determine the flow: SRC IP, DST IP, SRC port, DST port,
Copyright 2005
– Determine the flow: SRC IP, DST IP, SRC port, DST port,and protocol
– Determine the interfaces through which the flow passes
Note: All firewall issues can be simplified to two interfaces (ingress and egress) and the rules tied to both
Example Flow
• Flow– SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP
– DST IP: 198.133.219.25 DST Port: 80
• Interfaces– Source: Inside Destination: Outside
Copyright 2005
Accounting
With the Flow Defined, Examination of
Configuration Issues Boils Down to Just the Two Interfaces: Inside
and Outside
Eng
Client: 10.1.1.9
Server: 198.133.219.25
Serv
ers
Ou
tsid
e
Packet Flow
Understanding the Packet Flow
• Once the device and flow have been identified, walk the path of the packet through the device
• The packet path through the firewall is illustrated in the next several slides
• For troubleshooting, pay careful attention to where the packet can be dropped in the decision-making
Copyright 2005
the packet can be dropped in the decision-making process
Packet Processing Flow Diagram
• The diagram below will be referenced on the following slides; it is shown here enlarged for reference
Copyright 2005
Packet Processing: Ingress Interface
• Packet arrives on ingress interface
• Input counters incremented
• Software input queue is an indicator of load
Copyright 2005
• Software input queue is an indicator of load
• “No buffers” indicates packet drops, typically due to bursty traffic
ASA-5540# show interface gb-ethernet1interface gb-ethernet1 "inside" is up, line protocol is upHardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214IP address 10.1.1.1, subnet mask 255.255.255.0MTU 1500 bytes, BW 1 Gbit full duplex
5912749 packets input, 377701207 bytes, 0 no bufferReceived 29519 broadcasts, 0 runts, 0 giants0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort286298 packets output, 18326033 bytes, 0 underrunsinput queue (curr/max blocks): hardware (0/25) software (0/0)output queue (curr/max blocks): hardware (0/3) software (0/0)
Packet Processing: Locate Connection
• Check first for existing connection
• If connection exists, flow is matched; bypass ACL check
• If no existing connection
Copyright 2005
• If no existing connection – TCP non-SYN packet, drop and log
– TCP SYN or UDP packet, pass to ACL checks
Established Connection:
ASA-5540# show connTCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO
Syslog Because of No Connection, and Non-SYN Packet:
ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface inside
Packet Processing: ACL Check
• First packet in flow is processed through interface ACLs
• ACLs are first match
• First packet in flow matches ACE, incrementing hit count by
Copyright 2005
• First packet in flow matches ACE, incrementing hit count by one
• Denied packets are dropped and logged
Packet Permitted by ACL:
ASA-5540B# show access-list insideaccess-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)
Syslog When Packet Is Denied by ACL:
ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside"
Packet Processing: Match Translation
• First packet in flow must match a translation rule*
• A quick route lookup is done only to determine egress interface
• Translation rule can be to NAT, or not to NAT
• NAT order of operations dictates what happens with overlapping translation
Copyright 2005
• NAT order of operations dictates what happens with overlapping translation rules
• Once translation rule is matched, connection is created
Translation Exists:ASA-5540# show xlate debugNAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00
Syslogs When No Translation Rule Found: (305005—No NAT; 305006—No Global)ASA-3-305005: No translation group found for tcp src inside:10.1.1.9/11039 dst outside:198.133.219.25/80ASA-3-305006: regular translation creation failed for tcp src inside:10.1.1.9/11040 dst outside:198.133.219.25/80
Translation and NAT Order of Operations
1. nat 0 access-list (nat-exempt)
2. Match existing xlates
3. Match static commands (Cisco ASA/PIX first match;
Copyright 2005
3. Match static commands (Cisco ASA/PIX first match; FWSM best match)
– Static NAT with and without access-list
– Static PAT with and without access-list
4. Match nat commands
– nat <id> access-list (first match)
– nat <id> <address> <mask> (best match)
• If the ID is 0, create an identity xlate
• Use global pool for dynamic NAT
• Use global pool for dynamic PAT
Firs
t Ma
tch
Packet Processing: Inspections/Sec Checks
• Inspections are applied to ensure protocol compliance
• (Optional) Customized AIC inspections
• NAT embedded IPs in payload
Copyright 2005
• NAT embedded IPs in payload
• Additional security checks are applied to the packet
• (Optional) Packets passed to Content Security and Control (CSC) Module
Syslog from Packets Denied by Security Check:
ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside
ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP
Packet Processing: NAT IP Header
• Translate the IP address in the IP header
• Translate the port if performing PAT
• Update checksums
Copyright 2005
• Update checksums
• (Optional) Following the above, pass packet to IPS (AIP) module
Packet Processing: Egress Interface
• Packet is “virtually” forwarded to egress interface (i.e., not forwarded to the driver yet)
• Egress interface is determined first by translation rules
Copyright 2005
• Egress interface is determined first by translation rules
• If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface
• Example:
static (inside,outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0
DM
Z
Inside Outside
172.16.0.0/16172.16.12.0/24
172.16.12.4
Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics
Packet Processing: L3 Route Lookup
• Once on egress interface, an interface route lookup is performed
Copyright 2005
• Only routes pointing out the egress interface are eligible
• Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface
Syslog from Packet on Egress Interface with No Route Pointing Out Interface:
ASA-6-110001: No route to 209.165.202.130 from 10.1.1.9
Packet Processing: L2 Address Lookup
• Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed
• Layer 2 rewrite of MAC header
Copyright 2005
• Layer 2 rewrite of MAC header
• If Layer 2 resolution fails—no syslog
• show arp will not display an entry for the L3 next
hop
• debug arp will indicate if we are not receiving an
ARP reply
Packet Processing: Transmit Packet
• Packet is transmitted on wire
• Interface counters will increment on interface
• Output hardware and software queues indicate
Copyright 2005
• Output hardware and software queues indicate buffering at driver level, interface is busy
ASA-5540# show interface gb-ethernet0interface gb-ethernet0 "outside" is up, line protocol is upHardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626cIP address 172.18.124.64, subnet mask 255.255.255.0MTU 1500 bytes, BW 1 Gbit full duplex
3529518 packets input, 337798466 bytes, 0 no bufferReceived 32277 broadcasts, 0 runts, 0 giants0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort5585431 packets output, 359059032 bytes, 0 underrunsinput queue (curr/max blocks): hardware (0/25) software (0/0)output queue (curr/max blocks): hardware (0/2) software (0/0)
Agenda
• Packet Flow
• Understanding the Architecture
• Failover
• Troubleshooting
• Case Studies
Copyright 2005
• Case Studies
• Tools
• Best Practices
Cisco ASA/PIX—Understanding the Architecture
• Cisco ASA/PIX platforms process all packets in software (via the central CPU)
– All packets are processed first in…usually also first out
• No software limits on the number of ACEs (rules) that can be configured.
Copyright 2005
that can be configured.
– Each ACE takes a minimum of 212 bytes of RAM.
• Cisco ASA platforms have software imposed connection limits; Cisco PIX platforms do not (bound by RAM)
Classifier in Multimode
• FWSM has a single MAC address for all interfaces
• Cisco ASA/PIX has single MAC for ‘shared’ interfaces (physical interfaces have unique MACs)
– Cisco ASA/PIX 7.2 introduces an option to change this
• When the firewall receives a packet, it must ‘classify’ it to determine where to send the packet
Copyright 2005
‘classify’ it to determine where to send the packet
• Packets are classified based on the following
– Unique ingress interface/VLAN
– Packet’s destination IP matches a global IP
Classifier in Multimode
• Inbound traffic is ‘classified’ to context CTX3, based on the global IP in the static
Inside
FWSM
.1
DST IP SRC IP
Example
Copyright 2005
VL
AN
3—
10.1
4.3
.x
Inside
10.1.2.2
10.1.1.2
Inside
10.1.3.2
Inbound Packet
Outside
VLAN 4
VLAN 5
VLAN 6
CTX1
CTX2
CTX3
MSFC
.1
.2
.3
10.14.3.89 192.168.5.4
static (inside,outside) 10.14.3.89 10.1.3.2
Shared Interface
Classifier in Multimode
• If the firewall is unable to classify a packet, the following syslog message is generated in the Admin context*
%FWSM-6-106025: Failed to determine security context for
Copyright 2005
%FWSM-6-106025: Failed to determine security context for
packet: vlan3 tcp src 192.168.5.4/1025 dest 10.14.3.25/80
*Added to FWSM 3.1
Agenda
• Packet Flow
• Understanding the Architecture
• Failover
• Troubleshooting
• Case Studies
Copyright 2005
• Case Studies
• Tools
• Best Practices
Failover Basics
• Active/standby vs. primary/secondary
• Serial vs. LAN failover
• Stateful failover (optional)
• A failover only occurs when either firewall
Internet
Copyright 2005
when either firewall determines the standby firewall is healthier than the active firewall
• Both firewalls swap MAC and IP addresses when a failover occurs
• Level 1 syslogs will give reason of failover
Secondary(Standby)
Primary(Active)
LAN/Serial
Stateful
Corp
PIX(config)# show failoverFailover On Cable status: N/A - LAN-based failover enabledFailover unit PrimaryFailover LAN Interface: Failover Ethernet5 (up)Unit Poll frequency 1 seconds, holdtime 3 secondsInterface Poll frequency 5 seconds, holdtime 25 secondsMonitored Interfaces 2 of 250 maximumVersion: Ours 7.2(3), Mate 7.2(2)Last Failover at: 18:30:43 UTC Apr 12 2007
Verifying Failover Configuration
Interface Monitoring
Copyright 2005
Last Failover at: 18:30:43 UTC Apr 12 2007This host: Primary - Active
Active time: 5371 (sec)Interface outside (10.36.8.36): Normal Interface inside (10.5.5.144): Normal
Other host: Secondary - Standby ReadyActive time: 0 (sec)Interface outside (10.36.8.37): Normal Interface inside (10.5.5.145): Normal
Stateful Failover Logical Update StatisticsLink : Failover Ethernet5 (up)Stateful Obj xmit xerr rcv rerr General 86 0 73 0 sys cmd 74 0 73 0
Monitoring
What Triggers a Failover?
• Power loss/reload (this includes crashes) on the active firewall
• SSM interface/module failure
• The standby becoming “healthier” than the active firewall
Copyright 2005
What Triggers a Failover?
• Two consecutive “hello” messages missed on any monitored interface forces the interface into testing mode
• Both units first verify the link status on the interface
• Next, both units execute the following tests
Copyright 2005
• Next, both units execute the following tests
– Network activity test
– ARP test
– Broadcast ping test
• The first test passed causes the interface on that unit to be marked “healthy”; only if all tests “fail” will the interface be marked “failed”
What to Do After a Failover
• Always check the syslogs to determine root cause
• Example: switch port failed on inside interface of active firewall
Syslogs from Primary (Active) Firewall
Copyright 2005
ASA-4-411002: Line protocol on Interface inside, changed state to down
ASA-1-105007: (Primary) Link status ‘Down’ on interface 1
ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier
ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active
Syslogs from Primary (Active) Firewall
Syslogs from Secondary (Standby) Firewall
ASA# show failover state
What to Do After a Failover
• Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover state
• This information is not saved across reboots
Copyright 2005
ASA# show failover state
State Last Failure Reason Date/TimeThis host - Primary
Failed Ifc Failure 12:56:00 UTC May 6 2007Inside: Failed
Other host - SecondaryActive None
====Configuration State===Sync Done
====Communication State===Mac set
Agenda
• Packet Flow
• Understanding the Architecture
• Failover
• Troubleshooting
• Case Studies
Copyright 2005
• Case Studies
• Tools
• Best Practices
Troubleshooting Tools
• Syslogs
• Debug commands
• Show commands
• Packet capture
• Packet tracer
Copyright 2005
• Packet tracer
Uses of Syslogs
• Primary mechanism to record traffic to and through the firewall
• The best troubleshooting tool available
Archival Purposes Debugging Purposes
Copyright 2005
Buffered
SSH Client
InternetSyslog Server
SNMP Server
Console
Trap .Syslog
ASA Syslog Level vs. Number of Messages
Log Level
Description
Number of Messages (SUM)
Ver. 6.3 Ver. 7.0 Ver. 7.2 Ver. 8.0 Ver. 8.1
0 Emergencies 0 0 0 0 0
1 Alerts 41 (41) 62 (62) 77 (77) 78 (78) 87 (87)
Copyright 2005
1 Alerts 41 (41) 62 (62) 77 (77) 78 (78) 87 (87)
2 Critical 21 (62) 29 (91) 35 (112) 49 (127) 50 (137)
3 Errors 74 (136) 274 (365) 334 (446) 361 (488) 363 (500)
4 Warnings 56 (192) 179 (544) 267 (713) 280 (768) 281 (781)
5 Notifications 21 (213) 161 (705) 206 (919) 216 (984) 218 (999)
6 Informational 95 (308) 234 (939) 302 (1221) 335 (1319) 337 (1336)
7 Debugging 15 (323) 217 (1156) 258 (1479) 266 (1585) 267 (1603)
FWSM Syslog Level vs. Number of Messages
Log Level
Description
Number of Messages (SUM)
Ver. 2.3 Ver. 3.1 Ver. 3.2 Ver. 4.0
0 Emergencies 0 0 0 0
1 Alerts 58 (58) 67 (67) 67 (67) 67 (67)
Copyright 2005
1 Alerts 58 (58) 67 (67) 67 (67) 67 (67)
2 Critical 21 (79) 29 (96) 29 (96) 29 (96)
3 Errors 94 (173) 305 (401) 306 (402) 318 (414)
4 Warnings 131 (304) 194 (595) 196 (598) 199 (613)
5 Notifications 26 (330) 167 (762) 169 (767) 178 (791)
6 Informational 116 (446) 245 (1007) 248 (1015) 255 (1046)
7 Debugging 23 (469) 225 (1232) 225 (1240) 226 (1272)
What Are Modifiable Syslog Levels?
• Modifiable syslog levels
– Allows one to move any syslog message to any level
• Problem
[no] logging message <syslog_id> level <level>
Levels
0—Emergency
1—Alert
Copyright 2005
• Problem
– You want to record what exec commands are being executed on the firewall; syslog ID 111009 records this information, but by default it is at level 7 (debug)
%PIX-7-111009: User ‘johndoe’
executed cmd: show run
The problem is we don’t want to log all 1602 other syslogs that are generated at debug level
1—Alert
2—Critical
3—Errors
4—Warnings
5—Notifications
6—Informational
7—Debugging
How to Create Modifiable Syslog Levels
• Lower syslog message 111009 to level 3 (error)– ASA(config)# logging message 111009 level 3
– Or
– ASA(config)# logging message 111009 level error
[no] logging message <syslog_id> level <level>
Solution
Copyright 2005
• Now our syslog looks as follows– %ASA-3-111009: User ‘johndoe’ executed cmd: show run
• To restore the default syslog level– ASA(config)# no logging message 111009 level error
– Or
– ASA(config)# logging message 111009 level 7
http://www.cisco.com/en/US/docs/security/asa/asa80/system/m
essage/logmsgs.html
Debug Commands
1. Debugs should not be the first choice to troubleshoot a problem
2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution
3. Debugs are not conditional*
Copyright 2005
3. Debugs are not conditional*
4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug
* Crypto Conditional Debugging was added to Cisco ASA/PIX 8.0
Debug ICMP Trace
• Valuable tool used to troubleshoot connectivity issues
• Provides interface and translation information to
http://www.cisco.com
Internet
Copyright 2005
• Provides interface and translation information to quickly determine flow
• Echo-replys must be explicitly permitted through ACL, or ICMP inspection must be enabled
ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22
ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2
Example debug icmp trace output
Logging Debugs to Syslog
• Problem
– Log only debug output to syslog
• Solution
– Create a logging list with only syslog ID 711001
– Enable debug output to syslogs
– Log on the logging list
Copyright 2005
– Log on the logging list
ASA(config)# logging list C-MUG message 711001
ASA(config)# logging debug-trace
ASA(config)# logging trap C-MUG
Show Output Filters
• Use output filters to filter the output of show command to only the information you want to see
• To use them, at the end of show <Command>, use the pipe character “|” followed by
show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
Copyright 2005
pipe character “|” followed by– begin Start displaying the output beginning at the first
match of the RegEx, and continue to display the remaining output
– include Display any line that matches the RegEx
– exclude Display any line that does not match the RegEx
– grep Same as include
– grep –v Same as exclude
Example: Show Output Filters
Examples
• Display the interface stats starting with the ‘inside’ interface
– show interface | begin inside
• Display the access-list entries that contain address 10.1.1.5
show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
Copyright 2005
– show access-list | grep 10.1.1.5
• Display the config, except for the access-lists– show run | exclude access-list
• Display only access-list entries that have non-zero hitcounts– show access-list | grep –v hitcnt=0
• Display a count of the number of connections each host has– show local-host | include host|count/limit
Note: You must include a space on either side of the pipe for the command to be accepted; also, trailing spaces are counted
Show CPU Usage
• Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets
• FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near
Copyright 2005
during ACL compilation CPU is expected to be near 100% until ACL is compiled
• The show cpu usage command displays the CPU
over time as a running average
pixfirewall# show cpu usage
CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%
*First introduced in Cisco PIX OS version 6.0(1)/FWSM 1.1(1)
Show Traffic
• The show traffic command displays the traffic
received and transmitted out each interface of the firewall
ASA# show traffic
outside:
received (in 124.650 secs):
295468 packets 167218253 bytes
Copyright 2005
295468 packets 167218253 bytes
2370 pkts/sec 1341502 bytes/sec
transmitted (in 124.650 secs):
260901 packets 120467981 bytes
2093 pkts/sec 966449 bytes/sec
inside:
received (in 124.650 secs):
261478 packets 120145678 bytes
2097 pkts/sec 963864 bytes/sec
transmitted (in 124.650 secs):
294649 packets 167380042 bytes
2363 pkts/sec 1342800 bytes/sec
Show Xlate and Show Xlate Debug
• The show xlate command displays information
about the translations through the firewall
• You can limit the output to just the local or global IP
ASA# show xlate2 in use, 2381 most usedGlobal 172.18.124.68 Local 10.1.1.9
Copyright 2005
ASA# show xlate debug2 in use, 2381 most usedFlags: D - DNS, d - dump, I - identity, i - inside, n - no random,
o - outside, r - portmap, s - static
NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:02:03 timeout 3:00:00
TCP PAT from inside:10.9.9.3/4101 to outside:172.18.124.65/1024 flags r idle 0:00:08 timeout 0:00:30
Global 172.18.124.68 Local 10.1.1.9PAT Global 172.18.124.65(1024) Local 10.9.9.3(4101)
“debug” Adds interface names, idle and xlate
timeouts
ASA# show conn2 in use, 64511 most used
TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIOUDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags –
Show Conn and Show Conn Detail
ASA# show conn detail“detail” Adds uptime and
timeout in 7.2(4), 8.0(4)
Idle Time,Bytes Transferred
Connection Flags
‘real’ Interface names added in
7.2(4), 8.0(4)
Copyright 2005
ASA# show conn detail2 in use, 64511 most usedFlags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,
B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump,E - outside back connection, F - outside FIN, f - inside FIN,G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,i - incomplete, J - GTP, j - GTP data, K - GTP t3-responsek - Skinny media, M - SMTP data, m - SIP media, n - GUPO - outbound data, P - inside back connection, q - SQL*Net data,R - outside acknowledged FIN,R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS,X - inspected by service module
TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101,flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127
UDP outside:172.18.124.1/123 dmz:10.1.1.9/123,flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431
timeout in 7.2(4), 8.0(4)
Example—Connection Build Up
1. Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags “saA”
2. The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show “A”
3. The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered “up” (U flag)
Copyright 2005
3ACK
5Data
1SYN
4Data
2SYN+ACK
considered “up” (U flag)
4. The outside device sends the first data packet; the connection is updated and an “I” is added to the flags to indicate the firewall received Inbound data on that connection
5. Finally, the inside device has sent a data packet and the connection is updated to include the “O” flag
UsaAAUIOUI
Connection Flags
Client Server
OutsideInside
Example—Connection Teardown
1. Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an “f” to indicate that the FIN was received on the Inside interface
2. The outside device immediately responds to the FIN packet with a FIN+ACK; the connection flags are updated to reflect this, and now show “UfFR”
3. The inside device responds to the FIN+ACK with a final ACK and the
Copyright 2005
1FIN
2FIN+ACK UfFRUf3
ACK UfFRr
3. The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists
Connection Flags
Client Server
OutsideInside
Outbound Connection Inbound Connection
Connection Flags—Quick Reference
Copyright 2005
TCP Connection Termination Reasons
• If a TCP connection is built through the firewall, it will always have a teardown reason
• The TCP teardown syslog is logged at level 6
• If you are having problems with connections abnormally closing, temporally increase your logging level (or move the syslog down), and check
Copyright 2005
logging level (or move the syslog down), and check the teardown reason
ASA-6-302014: Teardown TCP connection number for
intf_name:real_IP/real_port to intf_name:real_IP/real_port
duration time bytes number [reason] [(user)]
TCP Connection Termination Reasons—Quick Reference
Reason Description
Conn-TimeoutConnection Ended Because It Was Idle Longer Than the Configured Idle Timeout
Deny Terminate Flow Was Terminated by Application Inspection
Failover Primary ClosedThe Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit
FIN TimeoutForce Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout
Copyright 2005
Flow Closed by Inspection Flow Was Terminated by Inspection Feature
Flow Terminated by IPS Flow Was Terminated by IPS
Flow Reset by IPS Flow Was Reset by IPS
Flow Terminated by TCP Intercept
Flow Was Terminated by TCP Intercept
Invalid SYN SYN Packet Not Valid
Idle TimeoutConnection Timed Out Because It Was Idle Longer Than the Timeout Value
IPS Fail-Close Flow Was Terminated Due to IPS Card Down
SYN Control Back Channel Initiation from Wrong Side
TCP Connection Termination Reasons—Quick Reference (Cont.)
Reason Description
SYN TimeoutForce Termination After Two Minutes Awaiting Three-Way Handshake Completion
TCP Bad Retransmission Connection Terminated Because of Bad TCP Retransmission
TCP Fins Normal Close Down Sequence
TCP Invalid SYN Invalid TCP SYN Packet
TCP Reset-I TCP Reset Was Sent From the Inside Host
Copyright 2005
TCP Reset-I TCP Reset Was Sent From the Inside Host
TCP Reset-O TCP Reset Was Sent From the Outside Host
TCP Segment Partial Overlap Detected a Partially Overlapping Segment
TCP Unexpected Window Size Variation
Connection Terminated Due to a Variation in the TCP Window Size
Tunnel Has Been Torn Down Flow Terminated Because Tunnel Is Down
Uauth Deny Connection Denied by URL Filtering Server
Unknown Catch-All Error
Xlate Clear User Executed the ‘Clear Xlate’ Command
show local-host
• A local-host entry is created for any IP tracked through the firewall
• It groups the xlates, connections, and AAA information
• Very useful for seeing the connections terminating on servers
Copyright 2005
ASA# show local-host 10.1.1.9 detailInterface inside: 1131 active, 2042 maximum active, 0 deniedlocal host: <10.1.1.9>,
TCP connection count/limit = 1/unlimitedTCP embryonic count = 0TCP intercept watermark = 50UDP connection count/limit = 0/unlimited
AAA:user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10)
absolute timeout: 0:05:00inactivity timeout: 0:00:00
Xlate(s):Global 172.18.124.69 Local 10.1.1.9
Conn(s):TCP out 198.133.219.25:80 in 10.1.1.9:11055 idle 0:00:10 Bytes 127 flags UIO
show service-policy
• The show service-policy command is used to quickly
see what inspection policies are applied and the packets matching them
ASA# show service-policyGlobal policy: Service-policy: global_policyClass-map: inspection_defaultInspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0
Copyright 2005
Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0Inspect: ftp, packet 43, drop 0, reset-drop 0Inspect: h323 h225, packet 0, drop 0, reset-drop 0Inspect: h323 ras, packet 0, drop 0, reset-drop 0Inspect: http, packet 562, drop 0, reset-drop 0Inspect: netbios, packet 0, drop 0, reset-drop 0Inspect: rsh, packet 0, drop 0, reset-drop 0Inspect: rtsp, packet 0, drop 0, reset-drop 0Inspect: skinny, packet 349, drop 0, reset-drop 0Inspect: esmtp, packet 0, drop 0, reset-drop 0...
Interface outside:Service-policy: VoIPClass-map: voice_markedPriority:Interface outside: aggregate drop 0, aggregate transmit 349
show service-policy flow
• Use to determine what policies a given flow will match in the Modular Policy Framework (MPF)
• Eventually all policies will be in MPF
ASA# show service-policy flow tcp host 10.0.0.2 host 10.1.1.2 eq 23
Copyright 2005
ASA# show service-policy flow tcp host 10.0.0.2 host 10.1.1.2 eq 23
Global policy:
Service-policy: global_policy
Interface outside:
Service-policy: outside_policy
Class-map: inbound_class
Match: access-list telnet_inbound
Access rule: permit tcp host 10.1.1.2 host 10.0.0.2 eq telnet
Action:
Output flow: set connection timeout tcp 0:05:00
show asp drop
• Packets dropped in the Accelerated Security Path (ASP) will increment a counter
• Frame drop counters are per packet, flow drops are per flow
• Some counters have corresponding syslogs
ASA# show asp drop
Copyright 2005
Frame drop:
Invalid encapsulation (invalid-encap) 10897
Invalid tcp length (invalid-tcp-hdr-length) 9382
Invalid udp length (invalid-udp-length) 10
No valid adjacency (no-adjacency) 5594
No route to host (no-route) 1009
Reverse-path verify failed (rpf-violated) 15
Flow is denied by access rule (acl-drop) 25247101
First TCP packet not SYN (tcp-not-syn) 36888
Bad TCP flags (bad-tcp-flags) 67148
TCP option list invalid (tcp-bad-option-list) 731
TCP MSS was too large (tcp-mss-exceeded) 10942
Bad TCP Checksum (bad-tcp-cksum) 893
*Drop counters are documented in the CMD Ref, under ‘show asp drop’
Packet Capture
• Capture command first introduced in Cisco PIX 6.2; FWSM 2.3; it deprecates the “debug packet” command
capture <capture-name> [access-list <acl-name>] [buffer <buf-size>]
[ethernet-type <type>] [interface <if-name>] [packet-length <bytes>]
[circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>]
[match <prot> {host <sip> | <sip> <mask> | any}
[eq | lt |gt <port>] {host <dip> | <dip> <mask> | any}
[eq | lt | gt <port>]]
[real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]
Copyright 2005
FWSM 2.3; it deprecates the “debug packet” command• 7.2(3) and 8.0(3) added a ‘real-time’ option• ASDM 6.0 adds a ‘capture wizard’• Capture sniffs packets on an interface that match
an ACL, or match line• Key steps
– Create an ACL that will match interesting traffic– Define the capture and bind it to an access-list and interface– View the capture on the firewall, or copy it off in .pcap format
Packet Capture (Cont.)
• Traffic can be captured both before and after it passes through the firewall; one capture on the inside interface, one capture on the outside interface
• Capture buffer saved in RAM (default size 512KB)
• Default is to stop capturing when buffer is full
Copyright 2005
• Default is to stop capturing when buffer is full
• Default packet length is 1518 bytes
• Copy captures off via TFTP or HTTPS
OutsideInside
Capture In Capture Out
Where Packets Are Captured in Packet Flow
Ingress Packets Captured
Egress Packets Captured
Copyright 2005
• Packets are captured at the first and last points they can be in the flow
• Ingress packets are captured before any packet processing has been done on them
• Egress packets are captured after all processing (excluding L2 source MAC rewrite)
Capture Command: Example
• Problem: User on the inside with an IP of 10.1.3.2 is having a problem accessing www.cisco.com (197.133.219.25); the user is getting PATed to 192.168.2.2
OutsideInside
Capture In Capture Out
Internet
www.cisco.com
Copyright 2005
OutsideInside Internet
198.133.219.2510.1.3.210.1.3.2 192.168.2.2
Step 1: Create ACL for Both Inside and Outside Interface
Step 2: Create Captures on Both Inside and Outside Interface
Step 3: Have Inside User Access www.cisco.com
Step 4: Copy the Captures Off to a TFTP Server
Step 5: Analyze Captures with Sniffer Program
Capture Command: Example
• Step 1: Create ACL for both inside and outside interface– ! Outside Capture ACL
Access-list 100 permit tcp host 192.168.2.2 host 198.133.219.25 eq 80Access-list 100 permit tcp host 198.133.219.25 eq 80 host 192.168.2.2
! Inside Capture ACLAccess-list 101 permit tcp host 10.1.3.2 host 198.133.219.25 eq 80Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.3.2
• Step 2: Create captures on both inside and outside interface– capture out access-list 100 interface outside packet-length 1518
Copyright 2005
– capture out access-list 100 interface outside packet-length 1518capture in access-list 101 interface inside packet-length 1518
• Step 3: Have inside user access www.cisco.com
• Step 4: Copy the captures off to a TFTP server– ! ASA ver 7.0+ / FWSM 3.0+ copy capture
copy /pcap capture:out tftp://10.1.3.5/out.pcapcopy /pcap capture:in tftp://10.1.3.5/in.pcap
! PIX ver 6.x / FWSM 2.3 copy capture copy capture:out tftp://10.1.3.5/out.pcap pcapcopy capture:in tftp://10.1.3.5/in.pcap pcap
– Or copy using https:
– https://<FW_IP>/capture/out/pcap
Packet Capture: Example
• Step 5: Analyze captures with sniffer program
Outside CAP
Copyright 2005
Inside CAPOutbound SYN, No SYN+ACK
Capturing Packets Dropped by the ASP
• Capture all packets dropped by the ASP– ASA# capture drops type asp-drop all
• Capture on a specific drop reason– ASA# capture drops type asp-drop invalid-tcp-hdr-length
Copyright 2005
ASA# capture drop type asp-drop ?
acl-drop Flow is denied by configured rule
all All packet drop reasons
bad-crypto Bad crypto return in packet
bad-ipsec-natt Bad IPSEC NATT packet
bad-ipsec-prot IPSEC not AH or ESP
bad-ipsec-udp Bad IPSEC UDP packet
bad-tcp-cksum Bad TCP checksum
bad-tcp-flags Bad TCP flags
Packet Tracer
• Packet tracer is the future of troubleshooting configuration issues (and many other issues)
• Introduced in version 7.2 and ASDM 5.2
• A packet can be traced by:
– Defining the packet characteristics via the CLI
– Capturing the packets using the ‘trace’ option
Copyright 2005
– Capturing the packets using the ‘trace’ option
Packet Tracer: Overview
• A packet tagged with the ‘trace’ option is injected into the interface, and processed in the data-plane
• Each action taken on the packet is recorded in the packet itself
• When the packet reaches the egress interface, or is dropped, it is punted to the control-plane
Copyright 2005
or is dropped, it is punted to the control-plane
• The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configuration
Packet Tracer: Creating Packet via CLI
• From the CLI, define the input interface along with source and destination IPs and portspacket-tracer input <intf> <proto> <src_ip> <src_port> <dst_ip>
<dst_port>
• Example—Trace the flow from inside host 10.1.1.2 to
Copyright 2005
• Example—Trace the flow from inside host 10.1.1.2 to http://www.cisco.com (198.133.219.25)
ASA# packet-tracer input inside tcp 10.1.1.2 1025 198.133.219.25 80
Packet Tracer: Example Output
ASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80
Phase: 1
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow
Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Copyright 2005
Result: ALLOW
Config:
access-group in in interface inside
access-list in extended permit tcp any any eq www
Additional Information:
Phase: 3
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map match-all inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect http
service-policy global_policy global
Additional Information:
Packet Tracer: Example Output (Cont.)
...
Phase: 10
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 1 10.1.1.0 255.255.255.0
Additional Information:
Dynamic translate 10.1.1.2/4 to 209.165.201.3/516 using netmask 255.255.255.255
Copyright 2005
...
Phase: 15
Type: ROUTE-LOOKUP
Subtype: output and adjacency
Result: ALLOW
Config:
Additional Information:
found next-hop 209.165.201.1 using egress ifc outside
adjacency Active
next-hop mac address 000a.f331.83c0 hits 0
>>>>Packet successfully forwarded to fast path<<<<
Packet Tracer: Tracing Captured Packet
• Create a capture using the ‘trace’ option
• Find the packet in the capture you want traced
ASA# capture inside access-list web interface inside trace .
ASA# show capture inside
68 packets captured
Copyright 2005
• Then select that packet to be traced
ASA# show capture inside trace packet-number 4 .
68 packets captured
1: 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: S
2: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: S ack
3: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: . ack
4: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: P ack
5: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: . Ack
...
Packet Tracer: ASDM
• ASDM includes a nice GUI front-end to the packet tracer tool
• It is located off the Tools menu
• Input the packets characteristics in the top half
• Actions taken on the packet are shown in the bottom half, along with associated config and links
Copyright 2005
bottom half, along with associated config and links back to modify that config entry in ASDM
Define Packet
Packet Tracer: ASDM (Screen Shot)
Copyright 2005
Link Back to Edit Rule
Matching Config
Action
Final Result
Agenda
• Packet Flow
• Understanding the Architecture
• Failover
• Troubleshooting
• Case Studies
Copyright 2005
• Case Studies
• Tools
• Best Practices
Case Study
• Intermittent Access to Web Server
Copyright 2005
Case Study: Intermittent Access to Web Server
Problem
• Most external clients are not able to load company’s web page
Copyright 2005
10.1.1.50
Internet
ASA-5510
HTTP Requests to 192.168.1.50
Clients
Web Server
NATed to 10.1.1.50
Case Study: Intermittent Access to Web Server
Traffic Spike
Copyright 2005
Case Study: Intermittent Access to Web Server
• show perfmon indicates high number of embryonic connections
ASA-5510# show perfmon
PERFMON STATS: Current AverageXlates 0/s 0/sConnections 2059/s 299/s
Copyright 2005
Connections 2059/s 299/sTCP Conns 2059/s 299/sUDP Conns 0/s 0/sURL Access 0/s 0/sURL Server Req 0/s 0/sTCP Fixup 0/s 0/sTCP Intercept Established Conns 0/s 0/sTCP Intercept Attempts 0/s 0/sTCP Embryonic Conns Timeout 1092/s 4/sHTTP Fixup 0/s 0/sFTP Fixup 0/s 0/sAAA Authen 0/s 0/sAAA Author 0/s 0/sAAA Account 0/s 0/s
VALID CONNS RATE in TCP INTERCEPT: Current AverageN/A 95.00%
ASA-5510#
Case Study: Intermittent Access to Web Server
• Issue show conn to see ‘who’ is creating the connections
ASA-5510# show conn54764 in use, 54764 most usedTCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aBTCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB
Random Sources Embryonic Conns
Copyright 2005
TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aBTCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aBTCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aBTCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aBTCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aBTCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aBTCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aBTCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aBTCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aBTCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aBTCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aBTCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aBTCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aBTCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aBTCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aBTCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB. . .
Case Study: Intermittent Access to Web Server
Traffic Permitted
Copyright 2005
Connection Count Jumps
Permitted
SYN Flood Detected
Case Study: Intermittent Access to Web Server
• Apply TCP Intercept to stop the SYN flood attack
access-list 140 extended permit tcp any host 192.168.1.50 eq www!
Copyright 2005
!class-map protectdescription Protect web server from attacksmatch access-list 140!policy-map interface_policyclass protectset connection embryonic-conn-max 100
!service-policy interface_policy interface outside
Case Study: Intermittent Access to Web Server
Few clients represent
Copyright 2005
TCP Intercept applied
Few clients represent 50+ % of traffic
Case Study: Intermittent Access to Web Server
• Apply per-client-max option to limit the number of connections any single client can establish
access-list 140 extended permit tcp any host 192.168.1.50 eq www!
Copyright 2005
!class-map protectdescription Protect web server from attacksmatch access-list 140!policy-map interface_policyclass protectset connection embryonic-conn-max 100 per-client-max 25
!service-policy interface_policy interface outside
Case Study: Intermittent Access to Web Server
per-client-max
Copyright 2005
TCP Intercept
Case Study: Intermittent Access to Web Server
Attacks Being Mitigated by ASA
Copyright 2005
Attacks Still Occurring
Case Study
• Poor Voice Quality
Copyright 2005
Case Study: Poor Voice Quality
Problem
• Poor Outbound Voice Quality at SOHO sites
Outbound RTP Stream
Copyright 2005
100 Mbps 100 Mbps Cable Modem
2 Mbps WAN
ASA-5505
Case Study: Poor Voice Quality
Solution: Traffic Shaping
• What is Traffic Shaping, and why is it needed here?
• Why won’t Policing work?
• Why won’t Priority Queuing alone work?
Copyright 2005
• Why won’t Priority Queuing alone work?
100 Mbps 100 Mbps
Cable Modem
2 Mbps
WAN
ASA-5505
Shape to2 Mbps
Case Study: Poor Voice Quality –Configuration Example (Traffic Shaping)
class-map voice-traffic
match dscp af13 ef
!
policy-map qos_class_policy
Solution
� Prioritize voice traffic and shape all traffic down to 2 Mbps on the outside interface.
Copyright 2005
policy-map qos_class_policy
class voice-traffic
priority
!
policy-map qos_outside_policy
class class-default
shape average 2000000
service-policy qos_class_policy
!
service-policy qos_outside_policy interface outside
� To view statistics on the operation of the shaper, use the command show service-policy shape
Case Study: Poor Voice Quality
Things to keep in mind:
� Shaping can only be applied to the class ‘class-default’
� Shaping only works in the outbound direction on an interface
� The shaping value is in bits per second, and must be a
Copyright 2005
� The shaping value is in bits per second, and must be a multiple of 8000
� The shaping policy is applied to all sub-interfaces on a physical interface
� Not supported on the ASA-5580 platform
� Not supported in Transparent or Multi-context mode
Agenda
• Packet Flow
• Understanding the Architecture
• Failover
• Troubleshooting
• Case Studies
Copyright 2005
• Case Studies
• Tools
• Best Practices
Tools
• ASDM
• Output interpreter
• Online learning modules
Copyright 2005
ASDM
• Run as a standalone application using the ADSM Launcher
• This allows for one-stop access to multiple firewalls
• ASDM 6.0 adds “Upgrade Wizard” to upgrade ASA and ASDM software direct from cisco.com
• ASDM 6.1 works with both ASA 8.1 and 8.0 releases
• ASDM 6.1F works with FWSM 4.0, 3.2 and 3.1 releases
Copyright 2005
• ASDM 6.1F works with FWSM 4.0, 3.2 and 3.1 releases
ASDM Home Page
Device Information
Copyright 2005
CPU, Memory, Conns/Sec, Interface Traffic
Real-Time Syslogs
Using ASDM for Monitoring
Up to Four Different Graphs Can Be Displayed
Copyright 2005
Great forMonitoring
Trends
ASDM: Editing Rules from the Log Viewer
Select Log Entry
Copyright 2005
Select Log Entry from Viewer
Right-Click on Message to View or Edit Associated Rule
ASDM: Syslogs Explained
Copyright 2005
ASDM 6.0
� Drag-and-drop and in-place editing for simplified policy editing
� User interface customization with dockable windows and toolbars
Copyright 2005
� New Firewall Dashboardthat provides at-a-glance status of firewall services
� Live ACL hitcount in firewall rule table for easy policy auditing
Output Interpreter
Great Tool for Catching
Linked Off the Technical Support and
Documentation—Tools and Resources
Section on CCO
Copyright 2005
Great Tool for Catching Configuration Errors
Paste in the “show run” Output and Hit “Submit”
Output Interpreter: Example Output
Warning: Unused Statics
Copyright 2005
Warning: Unapplied
Crypto Map
Warning: Invalid Crypto Map
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
Online Learning Modules on CCO
• Great way to learn about new features in the ASA
• Located on CCO
• From http://www.cisco.com select: Products and Solutions
– Security
• ASA 5500 Series Adaptive Security Appliances
Copyright 2005
• ASA 5500 Series Adaptive Security Appliances
– Training resources
» Online learning modules
• Direct link
– http://www.cisco.com/en/US/partner/products/ps6120/tsd_products_support_online_learning_modules_list.html
Agenda
• Packet Flow
• Understanding the Architecture
• Failover
• Troubleshooting
• Case Studies
Copyright 2005
• Case Studies
• Tools
• Best Practices
Cisco ASA/PIX/FWSM Best Practices
• Enable ip verify reverse-path on all interfaces
• Set embryonic and maximum connection counts on static and nat statements; for 7.2.1+ use per-client-max
• Configure logging to syslog server
• Move messages you want to see to lower levels, instead
Copyright 2005
• Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you don’t want to see
• Disable telnet access! Use SSH for management access
• Enable authentication for management access (console/SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallback
Cisco ASA/PIX/FWSM Best Practices
• Restrict DMZ access inbound to your internal networks
• Baseline CPU load, connection counts, xlate counts, and traffic (per interface)
• Run the latest maintenance release in your train
• Upgrade major feature trains only when you need
Copyright 2005
• Upgrade major feature trains only when you need new features, or after train has ‘matured’
Cisco ASA/PIX Release Process
7.0(2)
Maintenance
7.0(4)
EoL
7.0(5)
7.0(2.1) 7.0(2.2) 7.0(2.3)InterimImages
7.0(6) 7.0(7) 7.0(9)
FutureGD
7.1(2)
7.0(8)
GD
Copyright 2005
7.0(1)
Maintenance Trains
7.1(1)
Major Feature Releases
7.2(1) 8.0(2)
7.2(2) 7.2(3)
8.0(3)
Time
8.2(1)
Bug Fixes Waterfall
Down
8.1(1)
7.2(4) 7.2(5)
8.0(4)
8.1(2)
Cisco PIX - End of Sale / End of Life Milestones
Cisco has Announced the End-of-Saleand End-of-Life Dates for CiscoPIX Security Appliances
Copyright 2005
� End of Sale: July 28, 2008
� Last day of sale for software, accessories,and licenses: January 28, 2009
� End of Software Maintenance Releases: July 28, 2009
� End of Support / End of Life: July 27, 2013
Gotcha’s in Upgrading to 7.0
• Cisco PIX-515s (non-E) and Cisco PIX-535 should be upgraded from monitor mode
• Upgrading from monitor mode requires you to copy the 7.0 image over twice
– Once from monitor mode (to boot 7.0 and format flash)
Copyright 2005
– Once from monitor mode (to boot 7.0 and format flash)
– Once after 7.0 is up and flash has been formatted (to save image in flash)
• The upgrade process automatically converts your pre-7.0 config to the new 7.0 CLI
• If there were any errors during the config conversion process, view them by issuing– show startup-config errors
Cisco PIX Password Recovery
• Password recovery can only be performed by uploading the password recovery utility to the Cisco PIX from monitor mode via TFTP
• Password recovery will also remove any AAA commands
• A password recovery utility is created for each
Copyright 2005
• A password recovery utility is created for each major Cisco PIX release (6.1, 6.2, 6.3); however, the utilities are backwards compatible
– Example: np63.bin for Cisco PIX 6.3 and prior releases
http://www.cisco.com/warp/public/110/34.shtml
monitor> interface 1
monitor> address 14.36.1.88
monitor> file np63.bin
monitor> gateway 14.36.1.1
monitor> server 172.18.108.26
monitor> tftp
tftp [email protected] via 14.36.1.1............... Received 92160 bytes
Example: Cisco PIX Password Recovery
InsideTFTP
172.18.108.26 14.36.1.88
Example
Copyright 2005
Do you wish to erase the passwords? [yn] y
The following lines will be removed from the configuration:
enable password 8Ry2YjIyt7RRXU24 encrypted
passwd 2KFQnbNIdI.2KYOU encrypted
aaa authentication serial console LOCAL
aaa authentication telnet console LOCAL
aaa authentication ssh console LOCAL
aaa authentication enable console LOCAL
Do you want to remove the commands listed above from the configuration? [yn] y
Passwords and aaa commands have been erased.
Rebooting..
Case Study
• Out-of-order packet buffering
Copyright 2005
Case Study: Out-of-Order Packets
• Inspections require ordered packets
• Packets sent to the SSM (AIP and CSC) require ordered packets
• Cisco ASA/PIX will buffer up to three packets by default
• Buffering can be increased on ASA by using the
Copyright 2005
• Buffering can be increased on ASA by using the ‘queue-limit’ option under the tcp-map
Case Study: Out-of-Order Packets
• Some networks have high numbers of out-of-order packets; often caused by asymmetric traffic flows
• If the out-of-order packet buffer isn’t large enough, traffic is dropped and packets must be retransmitted
Problem
Copyright 2005
OutsideInside
Client Server
Packet 10
10.16.9.2192.168.1.30
Packet 12
Packet 13
Packet 14
Packet 15
Buffer
Dropped by Firewall
Packet 11Dropped on Network
Case Study: Out-of-Order PacketBuffering Example
• How to detect?ASA# show asp drop
Frame drop:
...
TCP packet SEQ past window 46331
TCP packet buffer full 90943
...
� How to fix?
Copyright 2005
access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0
!
tcp-map OOO-Buffer
queue-limit 6
!
class-map tcp-options
match access-list OOB-nets
!
policy-map global_policy
class tcp-options
set connection advanced-options OOO-Buffer
!
service-policy global_policy global
� How to fix?
Case Study: Out-of-Order PacketBuffering Example
• How to verify?
ASA# show service-policy
Global policy:
Service-policy: global_policy
Class-map: inspection_default
...
Class-map: tcp-options
Copyright 2005
Class-map: tcp-options
Set connection policy:
Set connection advanced-options: OOB-Buffer
Retransmission drops: 0 TCP checksum drops : 0
Exceeded MSS drops : 0 SYN with data drops: 0
Out-of-order packets: 2340 No buffer drops : 0
Case Study
• TCP MSS (Maximum Segment Size)
Copyright 2005
Case Study: TCP MSS
• MSS is the Maximum Segment Size—or the maximum amount of data that can be sent in a single packet
• The MSS is set in the SYN packets
• The device that receives the MSS advertisement cannot send more data in a single packet to the
Copyright 2005
cannot send more data in a single packet to the peer than specified by the MSS
Case Study: TCP MSS
• Some servers have broken TCP stacks and ignore the MSS advertised by the Client
• The firewall will drop packets that exceed the advertised MSS
Problem
Copyright 2005
OutsideInside
Client Server
SYN MSS=1380
SYN+ACK MSS=1400
DATA=1390
10.16.9.2192.168.1.30
Case Study: TCP MSS Example
• How to detect?ASA# show asp drop
Frame drop:
TCP MSS was too large 943
%ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to
inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390
� How to fix?
Copyright 2005
access-list MSS-hosts permit tcp any host 10.16.9.2
!
tcp-map mss-map
exceed-mss allow
!
class-map mss
match access-list MSS-hosts
!
policy-map global_policy
class mss
set connection advanced-options mss-map
!
service-policy global_policy global
� How to fix?
Case Study: TCP MSS Example
• How to verify?ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518
ASA# show capture mss-capture
0 packets captured
0 packets shown
� How else could you verify?
Copyright 2005
� How else could you verify?
Case Study
• Out of Memory
Copyright 2005
Case Study: Out of Memory
• Users are unable to access the Internet
• No new connections are working
• All old (long lived) connections continue to work
Step 1: Check the Syslogs
Problem
Copyright 2005
Step 1: Check the Syslogs
%PIX-3-211001: Memory allocation Error
%PIX-3-211001: Memory allocation Error
Step 2: Check the Amount of Free Memory Available
Hardware: PIX-515E, 64 MB RAM
pixfirewall# show memory
Free memory: 714696 bytes
Used memory: 66394168 bytes
------------- ----------------
Total memory: 67108864 bytes
Case Study: Out of Memory
• Cisco PIX image (run from RAM)
• Configuration
• IPSec database
• Xlates (translations)
• Connections
What Can Eat Up 64 MB on a
Cisco PIX-515E?
Step 3: What Eats Up Memory (RAM) on the Cisco PIX?
Copyright 2005
Step 4: Let’s Check the Translations
pixfirewall# show xlate
251 in use, 258 most used
PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716)
PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276)
Global 209.165.201.25 Local 10.1.1.102
PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783)
PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197)
PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)
Varied Source IPs
A Small Global Pool Is Used, Overloading to
a PAT Address
pixfirewall# show conn
147456 in use, 147456 most used
TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIU
TCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIU
TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saA
TCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags OIU
TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIU
Case Study: Out of Memory
Step 5: Check the Connections
Copyright 2005
TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIU
TCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saA
TCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags OIU
TCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIU
TCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIU
TCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags OIU
TCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags OIU
TCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU
TCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags OIU
TCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saA
. . .
� Q: Why is the connection count so high?
Case Study: Out of Memory
pixfirewall# show traffic
outside:
received (in 25.000 secs):
1475 packets 469050 bytes
59 pkts/sec 18762 bytes/sec
transmitted (in 25.000 secs):
167619 packets 9654480 bytes
6704 pkts/sec 386179 bytes/sec
Ou
tsid
e
Take a Look at the Traffic Load
Copyright 2005
• Vast majority of traffic is coming in the inside interface and going out the outside interface
6704 pkts/sec 386179 bytes/sec
inside:
received (in 25.000 secs):
180224 packets 10410480 bytes
7208 pkts/sec 416419 bytes/sec
transmitted (in 25.000 secs):
1050 packets 118650 bytes
42 pkts/sec 4746 bytes/sec
Insid
e
Tra
ffic
Flo
w
pixfirewall# show conn count
147456 in use, 147456 most used
pixfirewall# show xlate count
251 in use, 258 most used
Case Study: Out of Memory
Step 6: Review What We Know and Take Action
Copyright 2005
Conn Count Is Very High, but xlate Count Is Low
� Many connections per xlate
� Probably one, or a few hosts, are generating the vast majority of connections
� Most likely due to a virus on the host(s)
pixfirewall# show local-host | include host|count/limit
local host: <10.1.1.131>,
TCP connection count/limit = 0/unlimited
UDP connection count/limit = 0/unlimited
local host: <10.1.1.51>,
TCP connection count/limit = 2/unlimited
UDP connection count/limit = 0/unlimited
Case Study: Out of Memory
Step 7: Find the Host(s) Generating All the Connections
Only Show Lines That Have the Word “host” or
“count/limit” in Them
Copyright 2005
UDP connection count/limit = 0/unlimited
local host: <10.1.1.236>,
TCP connection count/limit = 0/unlimited
UDP connection count/limit = 0/unlimited
. . .
local host: <10.1.1.99>,
TCP connection count/limit = 146608/unlimited
UDP connection count/limit = 0/unlimited
� Host 10.1.1.99 is eating up all the connections, and they are TCP-based connections
pixfirewall# show local-host 10.1.1.99
Interface inside: 250 active, 250 maximum active, 0 denied
local host: <10.1.1.99>,
TCP connection count/limit = 146608/unlimited
TCP embryonic count = 146606
Case Study: Out of Memory
Step 8: Now That We Found the Host, Let’s Look at the Connections It Is Generating
Note: All Connections
Copyright 2005
TCP embryonic count = 146606
UDP connection count/limit = 0/unlimited
Xlate(s):
Global 209.165.201.21 Local 10.1.1.99
Conn(s):
TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags saA
TCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags saA
TCP out 64.100.205.160:135 in 10.1.1.99:7774 idle 0:01:43 Bytes 0 flags saA
TCP out 64.101.182.19:135 in 10.1.1.99:39193 idle 0:01:43 Bytes 0 flags saA
TCP out 64.102.218.45:135 in 10.1.1.99:16462 idle 0:01:43 Bytes 0 flags saA
TCP out 64.100.21.120:135 in 10.1.1.99:30322 idle 0:01:43 Bytes 0 flags saA
TCP out 64.101.25.195:135 in 10.1.1.99:41116 idle 0:01:43 Bytes 0 flags saA
TCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags saA
TCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags saA
TCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags saA
. . .
Note: All Connections Are Embryonic
Connections to Random Destinations on TCP/135–
MS Blaster
Case Study: Out of Memory
• Cisco PIX provides two methods to limit the number of connections per host
– TCP intercept
– Max connections
Question: Which One Can Be Used Here?
Copyright 2005
• TCP intercept won’t help because the source address is valid
• Limiting the maximum number of connections each internal host can have is the only option
Question: Which One Can Be Used Here?
Case Study: Out of Memory
Step 9: Limit Infected Host(s) Impact on Network
• Configure the MAX TCP connections for NATed hosts to be 50
• Note: The local-host must be cleared before the new connection limits are applied
pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0
Copyright 2005
pixfirewall(config)# clear local-host 10.1.1.99
pixfirewall(config)# show local-host 10.1.1.99
Interface inside: 250 active, 250 maximum active, 0 denied
local host: <10.1.1.99>,
TCP connection count/limit = 50/50
TCP embryonic count = 50
TCP intercept watermark = unlimited
UDP connection count/limit = 0/unlimited
. . .
The Infected Host Is Limited to 50 TCP
Connections
Case Study: Out of Memory
pixfirewall# show conn count
126 in use, 147456 most used
pixfirewall# show memory
Free memory: 47716152 bytes
Used memory: 19392712 bytes
------------- ----------------
Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit
Copyright 2005
• Things look much better now
• Question: How could we configure the Cisco PIX so the connection limit was only applied to the one host (10.1.1.99) which was infected with the virus?
------------- ----------------
Total memory: 67108864 bytes
nat (inside) 1 10.1.1.99 255.255.255.255 50 0
nat (inside) 1 0.0.0.0 0.0.0.0 0 0
Top Related