Troubleshooting Firewalls

Click here to load reader

  • date post

    01-Dec-2014
  • Category

    Documents

  • view

    52
  • download

    4

Embed Size (px)

Transcript of Troubleshooting Firewalls

Troubleshooting FirewallsEric Stuhl Senior Network Consultant Chesapeake NetCraftsmen [email protected]

Copyright 2005

Agenda Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices

Copyright 2005

Packet Flow

Copyright 2005

Understanding the Packet Flow To effectively troubleshoot a problem, one must first understand the packet path through the network Attempt to isolate the problem down to a single device Then perform a systematic walk of the packet path through the device to determine where the problem could be For problems relating to the Cisco ASA/PIX/FWSM, always Determine the flow: SRC IP, DST IP, SRC port, DST port, and protocol Determine the interfaces through which the flow passes

Note: All firewall issues can be simplified to two interfaces (ingress and egress) and the rules tied to both

Copyright 2005

Example Flow Flow SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP DST IP: 198.133.219.25 DST Port: 80

Interfaces Source: Inside Destination: Outside

Client: 10.1.1.9

Servers

Packet FlowEng

Accounting Outside Server: 198.133.219.25

With the Flow Defined, Examination of Configuration Issues Boils Down to Just the Two Interfaces: Inside and Outside

Copyright 2005

Understanding the Packet Flow Once the device and flow have been identified, walk the path of the packet through the device The packet path through the firewall is illustrated in the next several slides For troubleshooting, pay careful attention to where the packet can be dropped in the decision-making process

Copyright 2005

Packet Processing Flow Diagram The diagram below will be referenced on the following slides; it is shown here enlarged for reference

Copyright 2005

Packet Processing: Ingress Interface

Packet arrives on ingress interface Input counters incremented Software input queue is an indicator of load No buffers indicates packet drops, typically due to bursty traffic

ASA-5540# show interface gb-ethernet1 interface gb-ethernet1 "inside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214 IP address 10.1.1.1, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 5912749 packets input, 377701207 bytes, 0 no buffer Received 29519 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 286298 packets output, 18326033 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/3) software (0/0)

Copyright 2005

Packet Processing: Locate Connection

Check first for existing connection If connection exists, flow is matched; bypass ACL check If no existing connection TCP non-SYN packet, drop and log TCP SYN or UDP packet, pass to ACL checks Established Connection:ASA-5540# show conn TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO

Syslog Because of No Connection, and Non-SYN Packet:ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface inside

Copyright 2005

Packet Processing: ACL Check

First packet in flow is processed through interface ACLs ACLs are first match First packet in flow matches ACE, incrementing hit count by one Denied packets are dropped and loggedPacket Permitted by ACL:ASA-5540B# show access-list inside access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)

Syslog When Packet Is Denied by ACL:ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside"Copyright 2005

Packet Processing: Match Translation

First packet in flow must match a translation rule* A quick route lookup is done only to determine egress interface Translation rule can be to NAT, or not to NAT NAT order of operations dictates what happens with overlapping translation rules Once translation rule is matched, connection is created

Translation Exists:ASA-5540# show xlate debug NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00

Syslogs When No Translation Rule Found: (305005No NAT; 305006No Global)ASA-3-305005: No translation group found for tcp src inside:10.1.1.9/11039 dst outside:198.133.219.25/80 ASA-3-305006: regular translation creation failed for tcp src inside:10.1.1.9/11040 dst outside:198.133.219.25/80Copyright 2005

Translation and NAT Order of Operations

1. 2. 3.

nat 0 access-list (nat-exempt) Match existing xlates Match static commands (Cisco ASA/PIX first match; FWSM best match) Static NAT with and without access-list Static PAT with and without access-list

First Match

4.

Match nat commands nat access-list (first match) nat (best match) If the ID is 0, create an identity xlate Use global pool for dynamic NAT Use global pool for dynamic PATCopyright 2005

Packet Processing: Inspections/Sec Checks

Inspections are applied to ensure protocol compliance (Optional) Customized AIC inspections NAT embedded IPs in payload Additional security checks are applied to the packet (Optional) Packets passed to Content Security and Control (CSC) Module

Syslog from Packets Denied by Security Check:ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUPCopyright 2005

Packet Processing: NAT IP Header

Translate the IP address in the IP header Translate the port if performing PAT Update checksums (Optional) Following the above, pass packet to IPS (AIP) module

Copyright 2005

Packet Processing: Egress Interface

Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet) Egress interface is determined first by translation rules If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface Example:Inside172.16.0.0/16

Outside DMZ172.16.12.0/24 172.16.12.4

Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Staticsstatic (inside,outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0 static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0Copyright 2005

Packet Processing: L3 Route Lookup

Once on egress interface, an interface route lookup is performed Only routes pointing out the egress interface are eligible Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interfaceSyslog from Packet on Egress Interface with No Route Pointing Out Interface:ASA-6-110001: No route to 209.165.202.130 from 10.1.1.9

Copyright 2005

Packet Processing: L2 Address Lookup

Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed Layer 2 rewrite of MAC header If Layer 2 resolution failsno syslog show arp will not display an entry for the L3 next hop debug arp will indicate if we are not receiving an ARP reply

Copyright 2005

Packet Processing: Transmit Packet

Packet is transmitted on wire Interface counters will increment on interface Output hardware and software queues indicate buffering at driver level, interface is busyASA-5540# show interface gb-ethernet0 interface gb-ethernet0 "outside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626c IP address 172.18.124.64, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 3529518 packets input, 337798466 bytes, 0 no buffer Received 32277 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 5585431 packets output, 359059032 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/2) software (0/0)Copyright 2005

Agenda Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices

Copyright 2005

Cisco ASA/PIXUnderstanding the Architecture Cisco ASA/PIX platforms process all packets in software (via the central CPU) All packets are processed first inusually also first out

No software limits on the number of ACEs (rules) that can be configured. Each ACE takes a minimum of 212 bytes of RAM.

Cisco ASA platforms have software imposed connection limits; Cisco PIX platforms do not (bound by RAM)

Copyright 2005

Classifier in Multimode FWSM has a single MAC address for all interfaces Cisco ASA/PIX has single MAC for shared interfaces (physical interfaces have unique MACs) Cisco ASA/PIX 7.2 introduces an option to change this

When the firewall receives a packet, it must classify it to determine where to send the packet Packets are classified based on the following Unique ingress interface/VLAN Packets destination IP matches a global IP

Copyright 2005

Classifier in MultimodeExample Inbound traffic is classified to context CTX3, based on the global IP in the staticFWSM Inside 10.1.1.2 Inside VLAN 5 10.1.2.2 Inside VLAN 6 10.1.3.2 static (inside,outside) 10.14.3.89 10.1.3.2Copyright 2005

DST IP VLAN 310.14.3.x CTX1 .1 10.14.3.89 Outside MSFC

SRC IP 192.168.5.4

VLAN 4

Inbound Packet

CTX2

.2

CTX3

.3

Shared Interface

Classifier in Multimode If the firewall is unable to classify a packet, the following syslog message is generated in the Admin context*

%FWSM-6-106025: Failed to d