Troubleshooting GETVPN Deployments
BRKSEC-3051
Wen Zhang - Technical Leader, Services
2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Agenda
GETVPN Solution Overview
What Is GETVPN and Where Does It Fit?
Introduction to GETVPN
Technology Overview
GETVPN Deployment
Configuration and Deployment Considerations
Troubleshooting
Troubleshooting Tools and Techniques
Common Troubleshooting Scenarios
4
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Other Related Sessions
BRKSEC-2054 – Deploying GET to Secure VPNs
BRKSEC-3013 – Advanced IPSec with FlexVPN
BRKSEC-3052 – Troubleshooting DMVPNs
BRKSEC-4054 – Advanced Concepts of DMVPN
CiscoLive 2013
5
GETVPN Solution Overview
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Cisco Group Encrypted Transport - GETVPN
Large-scale any-to-any encrypted communication
Native routing without tunnel overlay
Optimal for QoS and Multicast support - improves application performance
Transport agnostic - private LAN/WAN, FR/ATM, IP, MPLS
Any - to - Any
Connectivity
Real Time Scalable
Any - to - Any
Connectivity
Cisco GET
VPN
What Is GETVPN?
Cisco GETVPN delivers a revolutionary solution for tunnel-less, any-to-any and confidential branch communication
7
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Tunnel-Less VPN - A New Security Model
Scalability—an issue (N^2 problem)
Overlay routing
Any-to-any instant connectivity can’t be done to scale
Limited QoS
Inefficient Multicast replication
WAN
Multicast
Before: IPSec P2P Tunnels After: Tunnel-Less VPN
Scalable architecture for any-to-any connectivity and encryption
No overlays—native routing
Any-to-any instant connectivity
Enhanced QoS
Efficient Multicast replication 8
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
VPN Technology Positioning
Internet/Shared
Network MPLS/Private
Network
EzVPN/FlexVP
N Spoke
GETVPN GM DMVPN/FlexVP
N Spoke DMVPN/FLexVPN
Spoke
IPSec Agg.
WAN Edge Remot Access SW
Clients
GETVPN GM GETVPN GM
Data Center Core
Internet Edge
GET
Encrypted
KS KS
GM GM GM
GM GM
9
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
VPN Technology Positioning (Cont.)
FlexVPN DMVPN GETVPN
Infrastructure Network Public Internet
Transport Public Internet
Transport Private IP Transport
Network Style Converged Site to Site
and Remote Access
Hub-Spoke and Spoke-to-Spoke; (Site-to-Site)
Any-to-Any; (Site-to-Site)
Routing Dynamic Routing or
IKEv2 Route Distribution
Dynamic routing on tunnels
Dynamic routing on IP WAN
Failover Redundancy Route Distribution
Server Clustering
Route Distribution Model
Route Distribution Model + Stateful
Encryption Style Peer-to-Peer
Protection Peer-to-Peer
Protection Group Protection
IP Multicast Multicast replication at
hub Multicast replication at
hub Multicast replication in
IP WAN network
10
Introduction to GETVPN
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Group Encrypted Transport (GETVPN)
Uses three main components
– Secure Group Keys
– Header Preservation
– Key Service
Is based on open standards with patented Cisco technology
Leverages existing IKE, IPSec, and multicast technologies
Takes advantage of the existing routing infrastructure
12
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Group Security Functions
GM
GM
GM
GM
Key Server
Routing
Members
Group Member
Encryption Devices Route Between Secure/ Unsecure Regions Multicast Participation
Key Server
Validate Group Members Manage Security Policy Create Group Keys Distribute Policy/Keys
Routing Member
Forwarding Replication Routing
13
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Group Security Elements
GM
GM
GM
GM
Key Servers
Routing
Members
Key Encryption Key (KEK)
Traffic Encryption Key (TEK)
Group Policy
RFC3547: Group Domain of Interpretation (GDOI)
KS Cooperative Protocol
14
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Basic GETVPN Architecture
Step 1: Group Members (GM) register via GDOI with the Key Server (KS)
KS authenticates and authorizes the GM
KS pushes a set of IPSec SAs for the GM to use
GM1
GM2
GM3 GM4
GM5
GM6
GM7 GM8
GM9 Key Server
15
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Basic GETVPN Architecture
Step 2: Data Plane Encryption
GM exchange encrypted traffic using the group keys
The traffic uses IPSec Tunnel Mode with Header Preservation
GM1
GM2
GM3 GM4
GM5
GM6
GM7 GM8
GM9
Key Server
16
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Basic GETVPN Architecture
Step 3: Periodic Rekey of Keys
KS pushes out replacement IPSec keys before current IPSec keys expire; this is called a Rekey
Key Server
GM1
GM2
GM3 GM4
GM5
GM6
GM7 GM8
GM9
17
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Header Preservation
IPSec Tunnel Mode vs. GETVPN
IP Packet
IP Payload IP Header IPSec Tunnel Mode
ESP New IP Header
IP Payload IP Header
IPSec header inserted by VPN Gateway New IP Address requires overlay routing
IP Packet
IP Payload IP Header ESP Preserved Header GETVPN
IP Payload IP Header
IP header preserved by VPN Gateway Preserved IP Address uses original routing plane
18
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Data Path
GM 1 GM2
Encrypted
Host1 Host2
Encrypted/Authenticated Using Group SA
Original IP
Header Data
Original Src and
Dst Addresses ESP
19
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Rekey Methodology: Multicast Rekey
Rekey Message sent from key server to all group members
IP multicast message provides very efficient distribution
Rekeys resulting from configured KEK and TEK intervals or KS policy change
Key Server
GM1 GM2
GM3 GM4
Single rekey packet sent to multicast enabled core
Core replicates the packets to all GMs
20
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Rekey Methodology: Unicast Rekey
Key Server maintains state of active group members
Group Member sends ACK in response to the rekey messages
Remove Group Member if the GM does not acknowledge three rekeys
Key Server
GM1 GM2
GM3 GM4
21
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Requirement for Time-Based Anti-Replay
Sequence number based anti-replay only works with single sender
Need method to work for all senders using same IPSec SA
– Key Server downloads relative pseudotime and window size to all the GMs
– GMs calculate pseudo-timestamp based on downloaded pseudotime and sends out packet
– Receiving GM verifies packet within window size
– KS periodically refreshes GMs with pseudotime/window size - this means clock does not need to be synchronized between GMs
22
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Time-Based Anti-Replay
If Sender’s pseudotime falls in the below Receiver window, packet accepted
T0 T10 T20
Packet1
Packet2
PTr - W PTr + W PTr
Anti-replay window
Initial
pseudotime
Reject Reject Accept
Packet 1 and Packet 2 have pseudotimeT0, providing loose anti-replay protection (unlike counter-based)
23
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Cooperative Key Servers - HA
Single KS is a single point of failure
Two or more KSs known as COOP KSs manage a common set of keys and security policies for GETVPN group members
Group members can register to any one of the available KSs
GM 1
GM 3
Subnet 1
Subnet 4
Subnet 2
Subnet 3
GM 4
GM 2
Cooperative KS3
Cooperative KS1
IP Network
Cooperative KS2
GDOI Registration
24
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Cooperative Key Servers (Cont.)
One KS is elected as the Primary KS
Cooperative KSs periodically exchange and synchronize group’s database, policy and keys
Primary KS is responsible to generate and distribute group keys
Cooperative KS3 (Secondary)
GM 1
GM 3
Subnet 1
Subnet 4
Subnet 2
Subnet 3
GM 4
GM 2
Cooperative KS1
IP Network
Cooperative KS2 (Secondary)
Announcement Messages
(Primary)
Rekey Messages
25
GETVPN Deployment Configuration
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
COOP Server Exportable RSA Keys
RSA public key distribution from Key Server to Group Member:
– Public key generated in the RSA key pair, is sent to the GM at the registration
– The rekeys are signed by the private key of the KS and GM verifies the signature in the re-key with the public key of the KS
Exporting RSA Key between Key Servers:
– One of the key server in the redundancy group should generate the exportable RSA keys and copy those keys to other key servers
RSA Keys (generated only on KSs) are required for rekey authentication
27
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
crypto keyring gdoi1
pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123
!
crypto isakmp policy 10
encr 3des
authentication pre-share
!
crypto ipsec transform-set 3DES-SHA esp-3des esp-sha-hmac
!
crypto ipsec profile gdoi1
set security-association lifetime seconds 7200
set transform-set 3DES-SHA
!
access-list 150 permit ip any host 225.1.1.1
!
access-list 160 deny eigrp any any
access-list 160 deny pim any any
access-list 160 deny udp any any eq isakmp
access-list 160 deny udp any any eq 848
access-list 160 permit ip any any
Pre-shared Key
IPSec Profile
ISAKMP Policy
Access-list defining the
encryption policy
Access-List used for
defining
rekey (useful in
multicast rekeys only)
IPSec Transform
KS Configuration
28
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
crypto gdoi group getvpn1
identity number 101
server local
!rekey address ipv4 150 !
rekey lifetime seconds 14400
rekey retransmit 10 number 2
rekey authentication mypubkey rsa getvpn1
rekey transport unicast
sa ipsec 1
profile gdoi1
match address ipv4 160
address ipv4 130.23.1.1
redundancy
local priority 10
peer address ipv4 130.1.2.1
!
Encryption ACL
GDOI Group ID
Rekey Address mapping
(only for multicast rekeys)
Source address for rekeys
Rekey Properties
COOP KS Config
KS Configuration (Cont.)
29
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GM Configuration
crypto keyring gdoi
pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123
!
crypto isakmp policy 10
encr 3des
authentication pre-share
group 2
!
crypto gdoi group getvpn1
identity number 101
server address ipv4 130.23.1.1
!
crypto map getvpn10 gdoi
set group getvpn1
!
interface FastEthernet0/0
crypto map getvpn
Crypto map on the interface
GDOI Group
KS Address
GDOI crypto map
Pre-shared Key
ISAKMP Policy
30
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Platform Support
Platform Group Member Key Server
Software Yes Not supported
870 Yes Not supported
1821 Yes Not supported
1841/1900 Yes Yes
2800 (AIM/SSL)/2900 Yes Yes
3800 (AIM-II/AIM-III)/3900 Yes Yes
7200 NPEG1, VAM2+ Yes Yes
7301 NPEG1, VAM2+ Yes Yes
7200 NPEG2, VAM2+ Yes Yes
7200 NPEG2, VSA Yes Yes
Cisco ASR 1000 Yes Yes (since XE3.6)
31
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Scalability and Performance
GETVPN Provides complete segregation of control and data plane
Key Server is responsible to maintain the control plane (key management) and GM is responsible to handle the data plane (actual user traffic)
KS and GM can NOT be configured on same IOS device
KS should be properly sized for number of branches (scale) in the network
GM should be properly sized for traffic throughput at each branch
32
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Deployment Best Practices
IKE/IPSec
Use specific pre-shared keys for all the GMs and KSs instead of using default key
KS
Always use COOP KSs
Set the huge buffer to 65535 and add 10 buffers to permanent buffer list
Configure periodic DPDs between the COOP KSs
Enable GM authorization
Policy
Aggregate the permit access-list entries to reduce the entries
Enable Time-Based Anti-Replay
Avoid re-encrypting traffic which is already encrypted (SSH, HTTPS)
Registration
Distribute GM registration to multiple KSs by arranging the KS order in configuration
Rekey Timers
Set TEK lifetime to 7200 Seconds
Set KEK lifetime to 86400 Seconds
34
GETVPN Troubleshooting
‟A problem well stated is a problem half solved”
• Charles F. Kettering
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Troubleshooting GETVPN
Ultimately all problems manifest at the data plane -“my user application is not working over GETVPN!”
But where really is the problem?
Control Plane
– Events that lead up to SAs getting installed on the GMs
Data plane
– Policy downloaded with SAs installed but traffic is not flowing
37
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Troubleshooting GETVPN High Level Flow
Control Plane Data Plane
COOP
IKE
Registration
Policy Download
Rekey
Troubleshooting Flow
Time Based
Anti-Replay
Fragmentation
MTU Issues
Transport Issues
Crypto
policy/engine
38
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Control Plane
Common Control Plane Issues
– GM registration issues
– Policy download issues
– COOP issues
– Rekey failures
Understand the expected protocol flow and know how to check for them
39
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Control Plane Troubleshooting Tools
GETVPN provides enhanced set of show commands for functionality verification
IOS also provided wide variety of syslog messages to verify proper GETVPN operations, and early insight into potential problems
IPSec and GDOI related debugs can then be enabled for further troubleshooting
GDOI conditional debugs – 15.1(3)T
GDOI event trace – 15.1(3)T
40
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Show crypto gdoi (on KS)
Key Server Role
KEK lifetime remaining
COOP configuration
TEK lifetime remaining
Registered GMs
Group Name : GET
Group Identity : 101
Group Members : 3
IPSec SA Direction : Both
Active Group Server : Local
Redundancy : Configured
Local Address : 130.23.1.1
Local Priority : 10
Local KS Status : Alive
Local KS Role : Primary
Group Rekey Lifetime : 1800 secs
Group Rekey
Remaining Lifetime : 88 secs
Rekey Retransmit Period : 10 secs
Rekey Retransmit Attempts: 3
Group Retransmit
Remaining Lifetime : 0 secs
IPSec SA Number : 1
IPSec SA Rekey Lifetime: 900 secs
Profile Name : gdoi1
Replay method : Count Based
Replay Window Size : 64
SA Rekey
Remaining Lifetime : 446 secs
ACL Configured : access-list 160
42
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Show crypto gdoi ks member (on KS)
KS#show crypto gdoi ks members
Group Member Information :
Number of rekeys sent for group GET: 4
Group Member ID : 131.1.1.1
Group ID : 101
Group Name : getvpn1
Key Server ID : 130.2.1.1
Rekeys sent : 4
Rekey Acks Rcvd : 4
Sent seq num : 1 2 3 4
Rcvd seq num : 1 2 3 4
KS GM is registered with
GM’s IP address
GM rekey history
43
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Show crypto gdoi (on GM)
GROUP INFORMATION
Group Name : GET
Group Identity : 101
Rekeys received : 270
IPSec SA Direction : Both
Active Group Server : 134.50.0.1
Group Server list : 134.50.0.1
GM Reregisters in : 5187 secs
Rekey Received(hh:mm:ss) : 00:02:30
Rekeys received
Cumulative : 270
After registration : 270
Rekey Acks sent : 270
ACL Downloaded From KS 134.50.0.1:
access-list deny eigrp any any
access-list deny tcp any any port = 179
access-list deny udp any port = 848
any port = 848
access-list permit ip any any
KEKPOLICY:
Rekey Transport Type : Unicast
Lifetime (secs) : 12295
Encrypt Algorithm : 3DES
Key Size : 192
Sig Hash Algorithm : HMAC_AUTH_SHA
Sig Key Length (bits) : 1024
TEK POLICY:
FastEthernet0/0:
IPSec SA:
sadirection:outbound
spi: 0x7C45C74A(2084947786)
transform: esp-aes esp-sha-hmac
sa timing: remaining key lifetime
(sec): (5246)
Anti-Replay(Time Based) : 2 sec interval
Active KS
When was
last rekey
received
Remaining
IPSec SA
Lifetime
44
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Control Plane Verification Syslog Messages - KS
Rekey:
GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from address 101.1.1.1 with seq # 1
COOP:
GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.0.9.1 Unreachable in group G1
GDOI-5-COOP_KS_ELECTION: KS entering election mode in group G1 (Previous Primary = NONE)
GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.0.8.1 in group G1 transitioned to Primary (Previous Primary = NONE)
45
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Control Plane Verification Syslog Messages - GM
Registration:
CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1 using address 10.1.13.2
GDOI-5-GM_REKEY_TRANS_2_UNI: Group G1 transitioned to Unicast Rekey
GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for group G1 using address 10.1.13.2
Rekey:
GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to 10.1.13.2 with seq # 3
46
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Control Plane Debugging Challenges
Challenge
Networks are getting bigger and faster, traditional debugs may not scale
Solution
Use IPSec and GDOI conditional debugs to minimize the debugging impact
Use the minimal level of debugs required
Challenge
Problems can be unpredictable with no identifiable trigger
Solution
Syslogs
GDOI Event Trace
47
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GDOI Debug Level Granularity
All feature components can be debugged at 5 levels
Start with the highest level, enable additional levels as needed
Debug Level What you will get
Error Error Conditions
Terse Important messages to the user and protocol issues
Event State transitions and events such as send/receive rekeys
Detail Most detailed debug message information
Packet Dump of detailed packet information
All All of the above
GM1#debug crypto gdoi gm rekey ?
all-levels All levels
detail Detail level
error Error level
event Event level
packet Packet level
terse Terse level
48
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GDOI Conditional Debugs
All IPSec and GDOI debugs can now be triggered with a conditional filter based on group or peer address
Use the unmatched flag to catch debugs with no context information
To enable conditional debugs
1) Set the conditional filter
2) Enable relevant debugs of interest as usual
KS1# debug crypto gdoi condition peer add ipv4 10.1.20.2
% GDOI Debug Condition added.
KS1#
KS1# show crypto gdoi debug-condition
GDOI Conditional Filters:
Peer Address 10.1.20.2
Unmatched NOT set
KS1#debug crypto gdoi ks registration all-levels
GDOI Key Server Registration Debug level: (Packet, Detail, Event, Terse, Error)
KS1
GM500 GM1
MPLS/Private IP
GM145
KS2
49
?
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Best practices when using the debug commands
Turn off console logging
Use NTP to sync up times on all devices
Enable msectimestamping debug and log messages
– service timestamps debug datetime msec
– service timestamps log datetime msec
Send the debugs to a syslog server
If no syslog server is available, use the logging buffer with an increased buffer size
– logging buffered 1000000 debugging
terminal exec prompt timestamp when using the show commands to correlate show commands with the debug output
reload in x to prepare for the worst
50
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GDOI Event Trace
Light weight event buffer to supplement syslogs
Always-on
Flexible output and display options
Event buffer
Continuous real time output
Output to file
Merged output from different feature components
Circular or one-shot buffer
Extensive exit path/error tracing capability
51
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GDOI Event Trace - Example
GM1#show monitor event-trace gdoi?
all Show all the traces in current buffer
back Show trace from this far back in the past
clock Show trace from a specific clock time/date
coop GDOI COOP Event Traces
from-boot Show trace from this many seconds after booting
infra GDOI INFRA Event Traces
latest Show latest trace events since last display
merged Show entries in all event traces sorted by time
registration GDOI Registration event Traces
rekey GDOI Rekey event Traces
GM1#show monitor event-trace gdoi merged all
*May 25 20:20:57.706: Registration_events: GDOI_REG_EVENT: REGISTRATION_STARTED:
GM 10.1.20.2 to KS 10.1.11.2 for group G1
*May 25 20:21:08.970: Registration_events: GDOI_REG_EVENT: REGISTRATION_DONE: GM
10.1.13.2 to KS 10.1.11.2 for group G1
*May 26 00:45:52.878: Rekey_events: GDOI_REKEY_EVENT: REKEY_RCVD: From 10.1.11.2
to 10.1.13.2 with seq no 131 for the group G1
*May 26 00:45:52.878: Rekey_events: GDOI_REKEY_EVENT: ACK_SENT: From 10.1.11.2
to 10.1.13.2 with seq no 131 for the group G1
52
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
crypto gdoi group G1
identity number 3333
server address ipv4 10.1.12.2
server address ipv4 10.1.11.2
!
crypto map gm_map 10 gdoi
set group G1
!
interface Serial1/0
crypto map gm_map
crypto gdoi group G1
identity number 3333
server address ipv4 10.1.11.2
server address ipv4 10.1.12.2
!
crypto map gm_map 10 gdoi
set group G1
!
interface Serial1/0
crypto map gm_map
crypto gdoi group G1
identity number 3333
server local
rekey lifetime seconds 86400
rekey authenmypubkeyrsa get
rekey transport unicast
saipsec 1
profile gdoi-p
match address ipv4ENCPOL
replay time window-size 5
address ipv4 10.1.12.2
redundancy
local priority 2
peer address ipv4 10.1.11.2
crypto gdoi group G1
identity number 3333
server local
rekey lifetime seconds 86400
rekey authenmypubkeyrsa get
rekey transport unicast
sa ipsec 1
profile gdoi-p
match address ipv4ENCPOL
replay counter window-size 64
address ipv4 10.1.11.2
redundancy
local priority 10
peer address ipv4 10.1.12.2
Troubleshooting Methodology
KS1 KS2
GM2 GM1
Ser 1/0: 10.1.20.2 Ser 1/0: 10.1.21.2
Eth 0/0: 192.168.20.1/24 Eth 0/0: 192.168.21.1/24
Ser 1/0: 10.1.11.2 Ser 1/0: 10.1.12.2
MPLS/Private IP
53
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Control Plane Setup Steps
COOP KS IKE Setup
COOP Election and Policy Creation
GM-KS IKE Setup
GM Authorization and Registration
GM Encryption Keys and Policy download
GM Data Encryption and Decryption
Periodic Key Renewal and Distribution (Rekeys)
54
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Common Issues – Control Plane
Encryption Policy
Key Renewal—Rekey
Control Plane Packet Fragmentation Issue
Control Plane Replay Check
55
IKE Setup
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
IKE Setup Between KS and GM
First step in GM registration is IKE setup
On successful negotiation of the IKE process, GM proceeds with the GDOI group registration
IKE SA is established at the time of registration eventually times out as its no longer needed after registration
KS1# show crypto isakmp sa
IPv4 Crypto ISAKMP SA
Dst src state conn-id slot status
10.1.11.2 10.1.20.2 GDOI_IDLE 1013 0 ACTIVE
10.1.12.2 10.1.11.2 GDOI_IDLE 1004 0 ACTIVE
10.1.21.2 10.1.11.2 GDOI_REKEY 0 0 ACTIVE
GM1# show crypto isakmp sa
IPv4 Crypto ISAKMP SA
Dst src state conn-id slot status
10.1.11.2 10.1.20.2 GDOI_IDLE 1073 0 ACTIVE
10.1.20.2 10.1.11.2 GDOI_REKEY 1074 0 ACTIVE
Expires
after IKE
lifetime
56
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
IKE Setup – IKE Failure Symptoms
If a GM fails to register with the KS, it will continue to attempt to register with the KS
Possible causes:
– Network issues between the GM and KS
– IKE negotiation failure
– KS policy issues
*May 24 06:40:15.581: %CRYPTO-5-GM_REGSTER: Start registration to KS
10.1.11.2 for group G1 using address 10.1.20.2
GM1#
*May 24 06:41:25.581: %CRYPTO-5-GM_REGSTER: Start registration to KS
10.1.11.2 for group G1 using address 10.1.20.2
KS1 KS2
GM2 GM1
MPLS/Private IP
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for
group G1 using address 10.1.20.2
57
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Pre-Shared Key Mismatch Troubleshooting
Verify routing information on KS and GM and try ping KS from the GM
After ruling out the connectivity issues, check the IKE SA on the GM
Verify the logs on the Key Server
GM1#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
Dst src state conn-id status
10.1.11.2 10.1.20.2 MM_KEY_EXCH 1038 ACTIVE
IPv6 Crypto ISAKMP SA
IKE SA not getting established; can’t
get to GDOI_IDLE state
KS1#
%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 10.1.20.2 failed its
sanity check or is malformed
58
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Pre-Shared Key Mismatch Solution
Syslog pointing to a mismatched pre-shared key configuration
Can be verified using “debug crypto isakmp”
KS Config:
GM Config:
crypto isakmp key cicso address 10.1.20.2
crypto isakmp key cisco address 10.1.11.2
Correct the pre-shared key configuration
KS1(config)#no crypto isakmp key cicso address 10.1.20.2
KS1(config)#crypto isakmp key cisco add 10.1.20.2
KS1(config)#^Z
59
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Common Issues – Control Plane
IKE Setup
Key Renewal—Rekey
Control Plane Packet Fragmentation Issue
Control Plane Replay Check
60
Encryption Policy
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GM1#show crypto gdoi
<snip>
ACL Downloaded From KS 10.1.11.2:
access-list deny eigrp any any
access-list deny ip 224.0.0.0 0.0.0.255 any
access-list deny ip any 224.0.0.0 0.0.0.255
access-list deny udp any port = 848 any port = 848
access-list permit ip any any
KEK POLICY:
Rekey Transport Type : Unicast
Lifetime (secs) : 2954
<snip>
TEK POLICY:
Serial1/0:
IPSec SA:
sa direction:inbound
spi: 0x2113F73B(554956603)
transform: esp-3desesp-sha-hmac
sa timing:remaining key lifetime (sec): (99)
Anti-Replay(Time Based) : 5 sec interval
<snip>
GM Policy Download
As part of the registration process, KS pushes down the encryption policies and keying material to the GM:
61
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
KS Policy Issues Routing Control Plane Traffic Failure
In most environments, GETVPN runs on the CE devices and PE devices do not participate in GETVPN
Failure to deny control plane traffic (such as routing protocol) on the PE-CE link will cause routing protocol to go down as soon as GM successfully registers
To identify, look at the ACL downloaded at GM:
GM1#show crypto gdoi gm acl
Group Name: G1
ACL Downloaded From KS 10.1.11.2:
access-list deny eigrp any any
access-list deny ip 224.0.0.0 0.0.0.255 any
access-list deny ip any 224.0.0.0 0.0.0.255
access-list deny udp any port = 848 any port = 848
access-list permit ip any any
ACL Configured Locally:
KS1 KS2
GM2 GM1
MPLS/Private IP
BGP
BGP is not denied in the ACL
downloaded from the KS
62
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
KS Policy Issues Control Plane Traffic - Solution
If most of the CEs are running BGP with the PE routers, configure a global KS policy to deny BGP
If only a handful of CEs are running BGP with the PE routers, configure a local GM policy to deny BGP
KS1&2(config)# ip access-list extended ENCPOL
KS1&2(config-ext-nacl)#1 deny tcp any any eq bgp
KS1&2(config-ext-nacl)#2 deny tcp any eq bgp any
GM1#
!
access-list 150 deny tcp any any eq bgp
access-list 150 deny tcp any eq bgp any
!
crypto map gm_map 10 gdoi
set group G1
match address 150
!
GM1#show crypto gdoi gm acl
Group Name: G1
ACL Downloaded From KS 10.1.11.2:
<snip>
access-list permit ip any any
ACL Configured Locally:
Map Name: gm_map
access-list 150 deny tcp any any port = 179
access-list 150 deny tcp any port = 179 any
KS1 KS2
GM2 GM1
MPLS/Private IP
BGP
63
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Common Issues – Control Plane
IKE Setup
Encryption Policy
Control Plane Packet Fragmentation Issue
Control Plane Replay Check
64
Key Renewal - Rekey
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Rekeys
Once the GETVPN network is properly setup and is working, KS is responsible for sending out rekey messages to all the GMs
KS can use unicast or multicast rekeys
Following syslog messages will appear in the log:
PRIMARY KS:
%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from
address 10.1.11.2 with seq # 11
All the GMs:
%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to
10.1.20.2 with seq # 11
65
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Following the Rekey Flow
66
Rekey sent? Rekey
delivered?
Rekey
received?
Rekey received by
IP?
Rekey verified by
IKE?
Rekey Processed
by GDOI?
Rekey
Acknowledged?
KS Network
Transport
GM
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Missing RSA Key Symptoms
When GM registers to the KS, the following message shows up in the syslog:
%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have
expired/been cleared, or didn't go through. Re-register to KS.
%GDOI-1-KS_NO_RSA_KEYS: RSA Key - get : Not found, Required for group G1
As a result KS will not send rekey messages, and GM will re-register when the keys expire
KS1 KS2
GM2 GM1
MPLS/Private IP
67
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Missing RSA Key on the KS Troubleshooting Steps
Check whether KS is sending out the rekeys or not:
KS needs RSA keys to sign the rekey messages; check logs for clues and/or verify the RSA keys
KS1#show crypto gdoi ks rekey
Group G1 (Multicast)
Number of Rekeys sent : 0
Number of Rekeys retransmitted : 0
KEK rekey lifetime (sec) : 86400
Retransmit period : 10
Number of retransmissions : 2
IPSec SA 1 lifetime (sec) : 3600
Remaining lifetime (sec) : 166
Number of registrations after rekey : 22
No rekeys sent
68
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Missing RSA Key on the KS Troubleshooting Steps (Cont.)
Verify RSA key configuration on the KS:
KS1#show running | section gdoi group
crypto gdoi group G1
identity number 3333
server local
rekey address ipv4 102
rekey lifetime seconds 86400
rekey authentication mypubkey rsa get
sa ipsec 1
profile gdoi-p
match address ipv4ENCPOL
no replay
address ipv4 10.1.11.2
Verify the RSA key pair name on the router:
KS1#show crypto key mypubkey rsa | include name
Key name: key1
Key name: key1.server
Labeled RSA key not present
69
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Missing RSA Key on the KS Solution
Generate the required RSA key pair
KS1(config)#crypto key generate rsa label get exportable modulus 1024
The name for the keys will be: getvpn-rsa-key
% The key modulus size is 1024 bits
% Generating 1024 bit RSA keys, keys will be exportable...[OK]
Verify rekey messages are now being sent on the KS
%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from
address 10.1.11.2 with seq # 1
KS1#show crypto gdoi ks rekey
Group G1 (Unicast)
Number of Rekeys sent : 1
<SNIP>
Rekeys are now sent
70
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Multicast Rekey Issues Multicast Rekeys Failing - Symptom
GM is not getting the multicast rekey messages and therefore
continues to re-register with the KS
Rekey starts to work when switched from multicast rekey to
unicast rekey
Possible Causes
– Packet delivery issue within the multicast routing infrastructure
– End-to-end multicast routing enabled?
– mVPN service provided by the MPLS core provider?
71
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Multicast Rekey Failing Troubleshooting
Check KS to verify multicast rekey messages are being sent
Make sure ICMP is excluded from the KS encryption policy and is used as a tool to test multicast
%GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group G1
from address 10.1.11.2 to 226.1.1.1 with seq # 6
KS1#ping 226.1.1.1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 226.1.1.1, timeout is 2 seconds:
Reply to request 0 from 10.1.21.2, 44 ms
No response from
GM1 (10.1.20.2)
KS1 KS2
GM2 GM1
Multicast
Network
10.1.20.2 10.1.21.2
72
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Multicast Rekey Failing Troubleshooting
Check the multicast forwarding path
Check the PIM neighbor
WAN#sh ip pim neighbor
PIM Neighbor Table
Neighbor Interface Uptime/Expires Ver DR
Address Prio/Mode
10.1.11.2 Serial0/0 01:03:54/00:01:16 v2 1 / S
10.1.21.2 Serial3/0 01:13:06/00:01:26 v2 1 / S
WAN#show ip mroute 226.1.1.1
<snip>
(10.1.11.2, 226.1.1.1), 00:13:18/00:02:56, flags: T
Incoming interface: Serial0/0, RPFnbr 0.0.0.0
Outgoing interface list:
Serial3/0, Forward/Sparse-Dense, 00:13:18/00:00:00
Verify the OIL
73
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Multicast Rekey Failing Solution
Enable PIM on the WAN router towards the GM
WAN(config)#int s2/0
WAN(config-if)#ip pim sparse-dense-mode
WAN(config-if)#end
%PIM-5-NBRCHG: neighbor 10.1.20.2 UP on interface
Serial2/0 (vrf default)
Check multicast routing path again
Re-test with multicast ping
Verify GM now receives the multicast rekey messages
74
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Unicast Rekey Failing Transient Network Issues
Due to transient changes in the network, unicast rekey packets might not make it to the GM(s)
If the GMs does not receive the rekey, it will have to re-register
Symptoms:
Missing Following syslog on GM:
%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to 10.1.21.2
with seq # 3
GM shows re-registration syslog:
%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have expired/been
cleared, or didn't go through. Re-register to KS.
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1 using
address 10.1.20.2
75
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Unicast Rekey Failing Troubleshooting and Solution
Verify whether the rekeys are not being sent, not being received or not being processed
KS:
show crypto gdoi ks members
Group Member Information :
Number of rekeys sent for group G1 : 380
Group Member ID : 10.1.20.2
Group ID : 3333
Group Name : G1
Key Server ID : 10.1.11.2
Rekeys sent : 1
Rekeys retries : 0
Rekey Acks Rcvd : 0
Rekey Acks missed : 0
GM:
show crypto gdoi gm rekey
Group G1 (Unicast)
Number of Rekeys received (cumulative) : 0
Number of Rekeys received after registration : 0
Number of Rekey Acks sent : 0
Rekey (KEK) SA information :
dstsrcconn-id my-cookie his-cookie
New : 10.1.20.2 10.1.11.2 1098 44F7FC328302AC61
Current : 10.1.20.2 10.1.11.2 1098 44F7FC328302AC61
Previous: --- --- --- --- ---
Always configure retransmissions to overcome transient issues
rekey retransmit 30 number 3
Make sure UDP port 848 is not blocked in the data path
Unicast rekey dropped
76
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Rekey Fails Signature Validation
Primary KS fails, GM receives rekey from secondary KS, but receives error:
*Apr 27 18:18:19.511: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode
failed with peer at 10.1.12.2
Syslog is not conclusive, let’s see what we can get with some debugs
GM1# debug crypto isakmp
Crypto ISAKMP debugging is on
GM1#
GM1# debug crypto gdoi
GDOI Generic Debug level: (Error, Terse)
*Apr 27 18:18:19.251: ISAKMP (0:1014): received packet from 10.1.12.2 dport 848
sport 848 Global (R) GDOI_REKEY
*Apr 27 18:18:19.251: GDOI:INFRA:(G1:0:1014:HW:0):Received Rekey Message!
*Apr 27 18:18:19.259: GDOI:INFRA:(G1:0:1014:HW:0):Signature Invalid! status = 13
*Apr 27 18:18:19.259: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed
with peer at 10.1.12.2
*Apr 27 18:18:19.259: ISAKMP: Receive GDOI rekey: Processing Failed. IKMP error = 6
Signature validation failed!
77
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Rekey Fails Signature Validation Solution
Problem:
– Secondary KS has its own RSA key pair instead of the exported key pair from the primary
– To verify, compare the RSA key pairs
KS#show crypto key mypubkey rsa
KS1(config)#crypto key generate rsa modulus 1024 exportable label key1
KS2(config)#crypto key import rsa key1 pem terminal <passphrase>
Solution:
Generate exportable RSA key pair on the primary KS
Export RSA key pair to all secondary KSs
KS1 KS2
GM2 GM1
MPLS/Private IP
78
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Common Issues – Control Plane
IKE Setup
Encryption Policy
Key Renewal—Rekey
Control Plane Packet Fragmentation Issue
79
Control Plane Replay Check
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Control Plane Replay Check Detection
Control Plane messages can carry time sensitive information and therefore require replay protection
– Rekey messages from KS to GM
– COOP Announcement messages between KSs
Sequence number check to protect against replayed messages
Pseudotime check to protect against delayed messages with TBAR enabled
Control Plane Replay check added in IOS version 12.4(15)T10, 12.4(22)T3, 12.4(24)T2, 15.0(1)M, and later
80
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Control Plane Replay Check Code interoperability issue
Problem: customer upgraded IOS on a GM to 15.0(1)M for a bug fix, and started to experience KEK rekey failures
The following errors are observed in the syslog
%GDOI-3-GDOI_REKEY_SEQ_FAILURE: Failed to process rekey seq # 1 in seq payload
for group G1, last seq # 11
%GDOI-3-GDOI_REKEY_FAILURE: Processing of REKEY payloads failed on GM 10.1.13.2
in the group G1, with peer at 10.1.11.2
%CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed with peer at
10.1.11.2
81
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Control Plane Replay Check Code interoperability issue - solution
KS does not support control plane replay detection, and resets the rekey sequence # for KEK rekey
GM interprets that as a replayed rekey message
Solution is to upgrade the KS to an IOS version that also supports the control plane replay detection
New behavior
*Apr 6 15:41:26.932: %GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from
10.1.11.2 to 10.1.13.2 with seq # 8
GM1#
*Apr 6 15:42:01.940: %GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from
10.1.11.2 to 10.1.13.2 with seq # 1
TEK Rekey with seq# reset
KEK Rekey
82
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Control Plane Replay Check – IOS Upgrade procedure
Recommended IOS releases
– IOS: 15.2(4)M3
– IOS-XE: 15.1(3)S4
IOS upgrade procedure
– Step 1. Upgrade a secondary KS first, wait until COOP KS election is completed
– Step 2. Repeat step 1 for all secondary KS
– Step 3. Upgrade primary KS
– Step 4. Upgrade Group Members
83
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Common Issues – Control Plane
IKE Setup
Encryption Policy
Key Renewal—Rekey
Control Plane Replay Check
84
Control Plane Packet Fragmentation Issue
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
buffers huge permanent 10
buffers huge size 65535
Control Plane Fragmentation Issues COOP Announcement Packets
In a large network (1500+ GMs), COOP update packet becomes larger than the default maximum buffer size
Default huge buffer size is 18024 bytes
Syslog message appears on the KSs:
Tune buffers to increase huge buffers and add buffers to permanent list:
%SYS-2-GETBUF: Bad getbuffer, bytes= 18872 -Process= "Crypto IKMP", ipl= 0, pid= 183
85
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Control Plane Fragmentation Issues (cont.) COOP Announcement Packets
Large ANN messages are fragmented in transit between KSs
Can have up to 40+ IP fragments
One dropped fragment -> entire ANN dropped
How to identify?
Frag
1
Frag2
Frag3
Frag4
FragN
KS1 KS2
%GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.1.11.1 Unreachable in group G1.
%GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.1.12.2 in group G1 transitioned to
Primary (Previous Primary = 10.1.11.2)
KS1#show ip traffic | section Frags
Frags: 10 reassembled, 3 timeouts, 0 couldn't reassemble
0 fragmented, 0 fragments, 0 couldn't fragment
Need to look at transit path features that may drop fragments, Firewall, VFR, reassembly buffer size, etc.
86
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Troubleshooting GETVPN Data Plane
Ultimately all problems manifest at the data plane -“my user application is not working over GETVPN!”
But where really is the problem?
Control Plane
– Events that lead up to SAs getting installed on the GMs
Data plane
– Policy downloaded with SAs installed but traffic is not flowing
87
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Generic IPSec Data Plane Troubleshooting
Need to have complete understanding of the forwarding path and how to checkpoint it
Which device is the culprit, encrypting or decrypting router?
In which direction is the problem happening, ingress or egress?
Some syslogs may help reveal data plane drops
– Data plane errors are typically rate limited
– Common errors include replay, authentication failures
Heavily dependent upon show commands and counters to trace the packet path
Sniffer capture of limited use due to encryption, however
– ESP-NULL – same crypto processing except packets not encrypted
– DSCP coloring of packets to uniquely identify a flow
88
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Data Plane
IPSec tunnel mode just like IPSec classic so most IPSec troubleshooting techniques still apply, however…
Symmetrical encryption policy requirement
Unique challenges with Header Preservation
– PMTUD
Time Based Anti-Replay
– Extra encapsulation overhead – Fragmentation boundary condition calculation
– Timer Based Anti-Replay failure
89
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Data Plane Troubleshooting Tools
Interface counters
Encryption/decryption counters
Netflow
IP Accounting
ACL
DSCP packet coloring
Embedded Packet Capture (EPC)
90
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
IPSec Data Plane Packet Flow Checkpoints
Encrypting GM
1. Ingress LAN interface Input ACL
Ingress Netflow
Embedded Packet Capture
2. Crypto engine show crypto ipsec sa
show crypto session detail
3. Egress WAN interface Egress Netflow
Embedded Packet Capture
Output IP precedence accounting
4 3
GM2 GM1
Client Server
1
2
6
5
Decrypting GM
4. Ingress WAN interface
Input ACL
Ingress Netflow
Embedded Packet Capture
Input IP precedence accounting
5. Crypto engine show crypto ipsec sa show crypto session detail
6. Egress WAN interface
Egress Netflow
Embedded Packet Capture
Traffic Direction
Private WAN
91
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Importance of a “Controlled Test”
The case for “ping x.x.x.x timeout 0”
Separation from background traffic
– Poor man’s conditional filter
– Packet coloring/marking
– Tools to monitor based on DSCP/Precedence marking
– ESP-NULL
IP characteristics for seemingly application issues
– Ping works but TCP doesn’t?
– Why does IPSec care about TCP, or does it?
92
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Encrypting GM Data Plane Flow
Verify clear traffic being received with Ingress Netflow
Verify encryption operation performed
Lack of per-flow granularity
interface Ethernet0/0
ip address 192.168.13.1 255.255.255.0
ip flow ingress
!
GM1# show ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Et0/0 192.168.13.2 Se1/0 192.168.14.2 06 E443 0017 11
TCP port 23 = telnet
GM1# show crypto session detail
<snip>
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 4, origin: crypto map
Inbound: #pkts dec'ed 162 drop 0 life (KB/Sec) 0/146
Outbound: #pkts enc'ed 170 drop 0 life (KB/Sec) 0/146
93
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Encrypting GM Data Plane Flow – Cont.
Verify encrypted traffic existing GM with egress Netflow
interface Serial/0
ip address 10.1.13.2 255.255.255.252
ip flow egress
!
GM1#show ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Et0/0 192.168.13.2 Se1/0* 192.168.14.2 32 EE5B 2BEF 170
GM1#show crypto ipsec sa
interface: Serial1/0
<snip>
current outbound spi: 0xEE5B2BEF(3998952431)
Protocol 50 = ESP
Active IPSec SA SPI
If per L4 flow granularity is desired, can use inbound precedence coloring and egress precedence accounting
94
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Decrypting GM Data Plane Flow
Verify encrypted traffic arriving on GM with Netflow
GM2#show ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Se1/0 192.168.13.2 Et0/0 192.168.14.2 32 EE5B 2BEF 170
Inbound IPSec SA SPI
Protocol 50 = ESP
Verify traffic decryption
GM2#show crypto session detail
<snip>
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 10, origin: crypto map
Inbound: #pkts dec'ed 170 drop 0 life (KB/Sec) 0/150
Outbound: #pkts enc'ed 162 drop 0 life (KB/Sec) 0/150
Verify clear traffic forwarding post decryption
GM2#show ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Se1/0 192.168.13.2 Et0/0* 192.168.14.2 06 E6CC 0017 170
TCP port 23 = telnet
95
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Common Issues – Data Plane
Other data plane issues common to IPSec
Fragmentation/Path MTU
Asymmetrical Encryption Policy
96
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
KS Policy Issues Data Plane Traffic Failure
Encryption policies (what needs to be encrypted) are defined centrally at the KS
Symmetrical ACLs should be defined to either permit or to deny traffic from getting encrypted
If the traffic is not being encrypted or being blocked, verify we have symmetrical ACL
GM2 GM1
MPLS/Private IP
Ethernet 0/0:
192.168.20.0/24
Ethernet 0/0:
192.168.21.0/24
KS Access-list
ip access-list extended ENCPOL
permit ip 192.168.20.0 0.0.0.255 192.168.21.0 0.0.0.255
permit ip 192.168.21.0 0.0.0.255 192.168.20.0 0.0.0.255
97
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Common Issues – Data Plane
Other data plane issues common to IPSec
Asymmetrical Encryption Policy
Fragmentation/Path MTU
98
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Fragmentation Issues PMTU Discovery
Large packets with the DF bit set may get black-holed in the GETVPN network
GM2 GM1
MTU 1500 MTU 1500
MTU 1000
1400B 1460B
ICMP 3/4
Server sends a large packet with the DF bit set in an attempt to perform network PMTUD
99
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
PMTUD and GETVPN
Encrypting GM adds IPSec overhead and forwards it
Intermediate router drops the packet and sends back icmp3/4 to perform PMTUD, two possibilities
– This ICMP dropped by the encrypt GM because it’s not encrypted based on the encryption policy
– This ICMP gets forwarded to the end host but gets dropped due to unauthenticated payload
Bottom line: PMTUD does not work with the current header preservation implementation of GETVPN
100
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
PMTUD and GETVPN
Solution
Implement ip tcp adjust-mss to reduce the TCP packet segment size
Clear the DF bit in the encapsulating header
interface Ethernet0/0
ip address 192.168.13.1 255.255.255.0
ip policy route-map clear-df-bit
!
route-map clear-df-bit permit 10
match ip address 111
set ip df 0
!
access-list 111 permit tcp any any
DF=1 Data
DF=0 Data DF=0
User Traffic
Encrypting GM
101
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Common Issues – Data Plane
Asymmetrical Encryption Policy
Fragmentation/Path MTU
Other Data Plane Issues Common to IPSec
102
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
IPSec drop due to packet corruption
IPSec integrity check makes IPSec packets a lot more sensitive to packet corruption in the network
Packet corruption symptoms
%CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=695
local=192.168.14.2 remote=192.168.13.2 spi=7C4E759F seqno=00000001
How to prove packets are corrupted in the network?
Enable EPC to capture packets into a circular buffer on both GMs
Use EEM (Embedded Event Manager) to
Synchronize and stop the capture on both routers when the RECVD_PKT_MAC_ERR message is logged
Notify the network operator by email
Retrieve both captures to examine for packet corruption
103
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Troubleshooting Summary
Have a clear and concise problem description
Try to break the problem down to either control or data plane
Understand the expected protocol flow on the control plane and how to check for them
Understand where/how to checkpoint the data plane
Syslog and event trace your friend
There is always TAC!
104
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Maximize your Cisco Live experience with your
free Cisco Live 365 account. Download session
PDFs, view sessions on-demand and participate in
live activities throughout the year. Click the Enter
Cisco Live 365 button in your Cisco Live portal to
log in.
Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily.
Receive 20 Cisco Daily Challenge points for each session evaluation you complete.
Complete your session evaluation online now through either the mobile app or internet kiosk stations.
105
GETVPN Scalability and Troubleshooting Tools
Appendix
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Key Server Scalability
Platform Crypto Card Max Number of GM Time to register to KS
7200/7201 VAM2+ 2000 15 sec *
3845 AIM-VPN/SSL-3 1000 15 sec *
3825 AIM-VPN/SSL-3 500 15 sec
2851 AIM-VPN/SSL-2 200 15 sec
2821 AIM-VPN/SSL-2 100 15 sec
1841 AIM-VPN/SSL-1 50 15 sec
7200/PKI VAM2+ 1000 20 sec **
* GM registration was distributed over two KSs to reduce the registration time
** GM registration was distributed over four KSs to reduce the registration time
108
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GM Performance Attributes (No Features)
PPS Mbps Max IMIX Latency(ms) Avg 100 pps Latency (ms)
871 Anti-Replay 3150 28 <10
No Anti-Replay 3232 28 <5
1841-onboard Anti-Replay 3506 33 <20
No Anti-Replay 3766 35 <35
1841-aim/ssl Anti-Replay 8420 84 <10
No Anti-Replay 8472 84 <20
2821-onboard Anti-Replay 17152 50 <5
No Anti-Replay 17046 50 <1
2821-aim/ssl Anti-Replay 26010 190 <5
No Anti-Replay 25918 190 <5
2851-onboard Anti-Replay 17868 64 <5
No Anti-Replay 19175 65 <10
2851-aim/ssl Anti-Replay 27594 190 <1
No Anti-Replay 27668 190 <1
0.34
0.33
0.25
1.18
1.07
0.68
0.47
109
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GM Performance Attributes (No Features)
PPS Mbps Max IMIX Latency(ms) Avg 100 pps Latency (ms)
3825-onboard Anti-Replay 35,505 283 <1
No Anti-Replay 35,500 283 <5
3825-aim/ssl Anti-Replay 44,170 199 <1
No Anti-Replay 44,452 199 <5
3845-onboard Anti-Replay 46,028 284 <5
No Anti-Replay 46,028 283 <5
3845-aim/ssl Anti-Replay 54,020 200 <1
No Anti-Replay 53,996 200 <1
7200-g1vam2+ Anti-Replay 60,592 266 <5
No Anti-Replay 66,952 266 <5
7200-g2vam2+ Anti-Replay 121,952 283 <5
No Anti-Replay 120,890 283 <1
7200-g2/vsa Anti-Replay
No Anti-Replay 160,000 980 TBD
ASR1000/FP5G Anti-Replay 440,000
No Anti-Replay 470,000 1,890 TBD
ASR1000/FP10G Anti-Replay 976,000 4,200
No Anti-Replay 1,011,000 4,220 <0.270
ASR1000/FP20G Anti-Replay 2,655,000 TBD
No Anti-Replay 2,685,000 8,530 <0.0150.001
0.64
0.66
0.19
TBD
TBD
0.17
0.76
0.81
0.69
110
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GM Performance Attributes (No Features)
Frame Size ASR 1004 (10Gig) 7200 VSA 3845 AIM-
VPN/SSL-3
ISRG2
3945 Onboard
Crypto
ISRG2
2951 Onboard
Crypto
ISRG2
1941
Onboard
Crypto
1400 Byte
4759 Mbps
925 Mbps 200 Mbps
820 Mbps
268 Mbps 154
Mbps
IMIX (90 Bytes 61%,
594 bytes 24%, 1418 15%)
2289 Mbps 780 Mbps 177 Mbps 261Mbps 160 Mbps 64Mbp
s
111
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
GETVPN Verification Common KS Syslog Messages
Syslog Messages Explanation
COOP_CONFIG_MISMATCH The configuration between the primary key server and secondary key server are
mismatched.
COOP_KS_ELECTION The local key server has entered the election process in a group.
COOP_KS_REACH The reachability between the configured cooperative key servers is restored.
COOP_KS_TRANS_TO_PRI The local key server transitioned to a primary role from being a secondary server
in a group.
COOP_KS_UNAUTH An authorized remote server tried to contact the local key server in a group.
Could be considered a hostile event.
COOP_KS_UNREACH The reachability between the configured cooperative key servers is lost. Could
be considered a hostile event.
KS_GM_REVOKED During rekey protocol, an unauthorized member tried to join a group. Could be
considered a hostile event.
KS_SEND_MCAST_REKEY Sending multicast rekey.
KS_SEND_UNICAST_REKEY Sending unicast rekey.
KS_UNAUTHORIZED During GDOI registration protocol, an unauthorized member tried to join a group.
Could be considered a hostile event.
UNAUTHORIZED_IPADDR The registration request was dropped because the requesting device was not
authorized to join the group.
112
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Syslog Messages Explanation
GM_CLEAR_REGISTER The clear crypto gdoi command has been executed by the local
group member.
GM_CM_ATTACH A crypto map has been attached for the local group member.
GM_CM_DETACH A crypto map has been detached for the local group member.
GM_RE_REGISTER IPSec SA created for one group may have been expired or
cleared. Need to reregister to the key server
GM_RECV_REKEY Rekey received
GM_REGS_COMPL Registration complete
GM_REKEY_TRANS_2_MULTI Group member has transitioned from using a unicast rekey
mechanism to using a multicast mechanism.
GM_REKEY_TRANS_2_UNI Group member has transitioned from using a multicast rekey
mechanism to using a unicast mechanism.
PSEUDO_TIME_LARGE A group member has received a pseudotime with a value that is largely
different from its own pseudotime.
REPLAY_FAILED A group member or key server has failed an anti-replay check.
GETVPN Verification Common GM Syslog Messages
113
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Packet marking Techniques
IP TOS byte copied from inner header to the encapsulating delivery header by default
How to mark
– PBR
– MQC
– Local ping
How to monitor
– IP precedence accounting
– ACL counters
114
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
ToS/Precedence/DSCP Reference Chart
7 5 6 4 3 2 1 0
IP Precedence Priority
DSCP
Least
Significant
Bit
ToS Byte
Hex - Decimal ToS
IP Precedence DSCP Binary
20
00
40
48
E0
C0
B8
A0
88
68
60
32
0
64
72
224
192
184
160
128
104
96
1 Priority
0 Routine
2 Immediate
7 Network Control
5 Critical
4 Flash Override
3 Flash
8 CS1
0 Dflt
16 CS2
18 AF21
56 CS7
48 CS6
46 EF
40 CS5
32 CS4
26 AF31
24 CS3
00100000
00000000
01000000
01001000
11100000
11000000
10111000
10100000
10000000
01101000
01100000
6 Internetwork Control
80
136 34 AF41 10001000
115
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Packet marking - Examples
PBR
MQC
interface Ethernet1/0
ip policy route-map mark
!
access-list 150 permit ip host 172.16.1.2 host 172.16.254.2
!
route-map mark permit 10
match ip address 150
set ip precedence flash-override
class-map match-all my_flow
match access-group 150
!
policy-map marking
class my_flow
set ip precedence 4
!
interface Ethernet1/0
service-policy input marking
IP flow in question marked with
precedence 4
116
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Packet marking - Examples
Router Ping
Router#ping ip
Target IP address: 172.16.254.2
Repeat count [5]: 100
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]: 128
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 100-byte ICMPEchos to 172.16.254.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
117
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Packet marking - Monitoring
IP Precedence Accounting
Interface ACL
interface Ethernet0/0
ip address 192.168.1.2 255.255.255.0
ip accounting precedence input
middle_router#show interface precedence
Ethernet0/0
Input
Precedence 4: 100 packets, 17400 bytes
middle_router#sh access-list 144
Extended IP access list 144
10 permit ip any any precedence routine
20 permit ip any any precedence priority
30 permit ip any any precedence immediate
40 permit ip any any precedence flash
50 permit ip any any precedence flash-override (100 matches)
60 permit ip any any precedence critical
70 permit ip any any precedence internet (1 match)
80 permit ip any any precedence network
118
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Using Packet Captures for Data Plane Issues
Packet captures can provide detailed packet information at the bits/bytes level
The new packet capture infrastructure introduced in 12.4(20)T makes this easy to do
– Ability to capture IPv4 and IPv6 packets in the CEF path
– Configurable capture buffer and capture point parameters
– Extensible output filtering and export capabilities
– Support for various WAN encapsulation types
119
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Using IOS Embedded Packet Captures
Router#monitor capture buffer test-buffer
Router#monitor capture buffer test-buffer filter access-list 120
Filter Association succeeded
Router#
Router#monitor capture point ipcef test-capture serial 2/0 both
*Mar 26 20:33:10.896: %BUFCAP-6-CREATE: Capture Point test-capture created.
Router#monitor capture point associate test-capture test-buffer
Router#monitor capture point start test-capture
*Mar 26 20:34:03.108: %BUFCAP-6-ENABLE: Capture Point test-capture enabled.
Router#
Router#monitor capture point stop test-capture
*Mar 26 20:34:21.636: %BUFCAP-6-DISABLE: Capture Point test-capture disabled.
Key Configuration Steps
Create the capture buffer and capture point
Associate the capture point to the buffer
Start/stop the capture
120
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Using IOS Embedded Packet Captures Now we have the packets captured, what’s next?
Router# show monitor capture buffer test-buffer dump
15:34:07.228 EST Mar 26 2009 : IPv4 LES CEF : Se2/0 None
05CECE30: 0F00080045C0002C ....E@.,
05CECE40: 6D170000FE0649DD 02010102 01010114 m...~.I]........
05CECE50: 0017A3530FB6B9523EF1499C 60121020 ..#S.69R>qI.`..
05CECE60: 917A0000 02040218 00 .z.......
Router# monitor capture buffer test-buffer export?
ftp: Location to dump buffer
http: Location to dump buffer
https: Location to dump buffer
rcp: Location to dump buffer
scp: Location to dump buffer
tftp: Location to dump buffer
Dump the packet on the router itself
Dump the packet on the router itself
Or export it out and analyze it in Wireshark
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public
Use EEM and EPC to catch Packet Corruption
event manager applet detect_bad_packet
event syslog pattern "RECVD_PKT_MAC_ERR"
action 1.0 cli command "enable"
action 2.0 cli command "monitor capture point stop test"
action 3.0 syslog msg "Packet corruption detected and capture stopped!"
action 4.0 snmp-trap intdata1 123456 strdata ""
Peer1
Peer2
event manager applet detect_bad_packet
event snmp-notification oid 1.3.6.1.4.1.9.10.91.1.2.3.1.9. oid-val "123456" op
eq src-ip-address 20.1.1.1
action 1.0 cli command "enable"
action 2.0 cli command "monitor capture point stop test"
action 3.0 syslog msg "Packet corruption detected and capture stopped!"
Top Related