Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf ·...

119

Transcript of Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf ·...

Page 1: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services
Page 2: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

Troubleshooting GETVPN Deployments

BRKSEC-3051

Wen Zhang - Technical Leader, Services

2

Page 3: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Agenda

GETVPN Solution Overview

What Is GETVPN and Where Does It Fit?

Introduction to GETVPN

Technology Overview

GETVPN Deployment

Configuration and Deployment Considerations

Troubleshooting

Troubleshooting Tools and Techniques

Common Troubleshooting Scenarios

4

Page 4: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Other Related Sessions

BRKSEC-2054 – Deploying GET to Secure VPNs

BRKSEC-3013 – Advanced IPSec with FlexVPN

BRKSEC-3052 – Troubleshooting DMVPNs

BRKSEC-4054 – Advanced Concepts of DMVPN

CiscoLive 2013

5

Page 5: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

GETVPN Solution Overview

Page 6: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Cisco Group Encrypted Transport - GETVPN

Large-scale any-to-any encrypted communication

Native routing without tunnel overlay

Optimal for QoS and Multicast support - improves application performance

Transport agnostic - private LAN/WAN, FR/ATM, IP, MPLS

Any - to - Any

Connectivity

Real Time Scalable

Any - to - Any

Connectivity

Cisco GET

VPN

What Is GETVPN?

Cisco GETVPN delivers a revolutionary solution for tunnel-less, any-to-any and confidential branch communication

7

Page 7: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Tunnel-Less VPN - A New Security Model

Scalability—an issue (N^2 problem)

Overlay routing

Any-to-any instant connectivity can’t be done to scale

Limited QoS

Inefficient Multicast replication

WAN

Multicast

Before: IPSec P2P Tunnels After: Tunnel-Less VPN

Scalable architecture for any-to-any connectivity and encryption

No overlays—native routing

Any-to-any instant connectivity

Enhanced QoS

Efficient Multicast replication 8

Page 8: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

VPN Technology Positioning

Internet/Shared

Network MPLS/Private

Network

EzVPN/FlexVP

N Spoke

GETVPN GM DMVPN/FlexVP

N Spoke DMVPN/FLexVPN

Spoke

IPSec Agg.

WAN Edge Remot Access SW

Clients

GETVPN GM GETVPN GM

Data Center Core

Internet Edge

GET

Encrypted

KS KS

GM GM GM

GM GM

9

Page 9: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

VPN Technology Positioning (Cont.)

FlexVPN DMVPN GETVPN

Infrastructure Network Public Internet

Transport Public Internet

Transport Private IP Transport

Network Style Converged Site to Site

and Remote Access

Hub-Spoke and Spoke-to-Spoke; (Site-to-Site)

Any-to-Any; (Site-to-Site)

Routing Dynamic Routing or

IKEv2 Route Distribution

Dynamic routing on tunnels

Dynamic routing on IP WAN

Failover Redundancy Route Distribution

Server Clustering

Route Distribution Model

Route Distribution Model + Stateful

Encryption Style Peer-to-Peer

Protection Peer-to-Peer

Protection Group Protection

IP Multicast Multicast replication at

hub Multicast replication at

hub Multicast replication in

IP WAN network

10

Page 10: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

Introduction to GETVPN

Page 11: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Group Encrypted Transport (GETVPN)

Uses three main components

– Secure Group Keys

– Header Preservation

– Key Service

Is based on open standards with patented Cisco technology

Leverages existing IKE, IPSec, and multicast technologies

Takes advantage of the existing routing infrastructure

12

Page 12: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Group Security Functions

GM

GM

GM

GM

Key Server

Routing

Members

Group Member

Encryption Devices Route Between Secure/ Unsecure Regions Multicast Participation

Key Server

Validate Group Members Manage Security Policy Create Group Keys Distribute Policy/Keys

Routing Member

Forwarding Replication Routing

13

Page 13: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Group Security Elements

GM

GM

GM

GM

Key Servers

Routing

Members

Key Encryption Key (KEK)

Traffic Encryption Key (TEK)

Group Policy

RFC3547: Group Domain of Interpretation (GDOI)

KS Cooperative Protocol

14

Page 14: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Basic GETVPN Architecture

Step 1: Group Members (GM) register via GDOI with the Key Server (KS)

KS authenticates and authorizes the GM

KS pushes a set of IPSec SAs for the GM to use

GM1

GM2

GM3 GM4

GM5

GM6

GM7 GM8

GM9 Key Server

15

Page 15: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Basic GETVPN Architecture

Step 2: Data Plane Encryption

GM exchange encrypted traffic using the group keys

The traffic uses IPSec Tunnel Mode with Header Preservation

GM1

GM2

GM3 GM4

GM5

GM6

GM7 GM8

GM9

Key Server

16

Page 16: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Basic GETVPN Architecture

Step 3: Periodic Rekey of Keys

KS pushes out replacement IPSec keys before current IPSec keys expire; this is called a Rekey

Key Server

GM1

GM2

GM3 GM4

GM5

GM6

GM7 GM8

GM9

17

Page 17: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Header Preservation

IPSec Tunnel Mode vs. GETVPN

IP Packet

IP Payload IP Header IPSec Tunnel Mode

ESP New IP Header

IP Payload IP Header

IPSec header inserted by VPN Gateway New IP Address requires overlay routing

IP Packet

IP Payload IP Header ESP Preserved Header GETVPN

IP Payload IP Header

IP header preserved by VPN Gateway Preserved IP Address uses original routing plane

18

Page 18: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Data Path

GM 1 GM2

Encrypted

Host1 Host2

Encrypted/Authenticated Using Group SA

Original IP

Header Data

Original Src and

Dst Addresses ESP

19

Page 19: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Rekey Methodology: Multicast Rekey

Rekey Message sent from key server to all group members

IP multicast message provides very efficient distribution

Rekeys resulting from configured KEK and TEK intervals or KS policy change

Key Server

GM1 GM2

GM3 GM4

Single rekey packet sent to multicast enabled core

Core replicates the packets to all GMs

20

Page 20: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Rekey Methodology: Unicast Rekey

Key Server maintains state of active group members

Group Member sends ACK in response to the rekey messages

Remove Group Member if the GM does not acknowledge three rekeys

Key Server

GM1 GM2

GM3 GM4

21

Page 21: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Requirement for Time-Based Anti-Replay

Sequence number based anti-replay only works with single sender

Need method to work for all senders using same IPSec SA

– Key Server downloads relative pseudotime and window size to all the GMs

– GMs calculate pseudo-timestamp based on downloaded pseudotime and sends out packet

– Receiving GM verifies packet within window size

– KS periodically refreshes GMs with pseudotime/window size - this means clock does not need to be synchronized between GMs

22

Page 22: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Time-Based Anti-Replay

If Sender’s pseudotime falls in the below Receiver window, packet accepted

T0 T10 T20

Packet1

Packet2

PTr - W PTr + W PTr

Anti-replay window

Initial

pseudotime

Reject Reject Accept

Packet 1 and Packet 2 have pseudotimeT0, providing loose anti-replay protection (unlike counter-based)

23

Page 23: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Cooperative Key Servers - HA

Single KS is a single point of failure

Two or more KSs known as COOP KSs manage a common set of keys and security policies for GETVPN group members

Group members can register to any one of the available KSs

GM 1

GM 3

Subnet 1

Subnet 4

Subnet 2

Subnet 3

GM 4

GM 2

Cooperative KS3

Cooperative KS1

IP Network

Cooperative KS2

GDOI Registration

24

Page 24: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Cooperative Key Servers (Cont.)

One KS is elected as the Primary KS

Cooperative KSs periodically exchange and synchronize group’s database, policy and keys

Primary KS is responsible to generate and distribute group keys

Cooperative KS3 (Secondary)

GM 1

GM 3

Subnet 1

Subnet 4

Subnet 2

Subnet 3

GM 4

GM 2

Cooperative KS1

IP Network

Cooperative KS2 (Secondary)

Announcement Messages

(Primary)

Rekey Messages

25

Page 25: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

GETVPN Deployment Configuration

Page 26: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

COOP Server Exportable RSA Keys

RSA public key distribution from Key Server to Group Member:

– Public key generated in the RSA key pair, is sent to the GM at the registration

– The rekeys are signed by the private key of the KS and GM verifies the signature in the re-key with the public key of the KS

Exporting RSA Key between Key Servers:

– One of the key server in the redundancy group should generate the exportable RSA keys and copy those keys to other key servers

RSA Keys (generated only on KSs) are required for rekey authentication

27

Page 27: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

crypto keyring gdoi1

pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123

!

crypto isakmp policy 10

encr 3des

authentication pre-share

!

crypto ipsec transform-set 3DES-SHA esp-3des esp-sha-hmac

!

crypto ipsec profile gdoi1

set security-association lifetime seconds 7200

set transform-set 3DES-SHA

!

access-list 150 permit ip any host 225.1.1.1

!

access-list 160 deny eigrp any any

access-list 160 deny pim any any

access-list 160 deny udp any any eq isakmp

access-list 160 deny udp any any eq 848

access-list 160 permit ip any any

Pre-shared Key

IPSec Profile

ISAKMP Policy

Access-list defining the

encryption policy

Access-List used for

defining

rekey (useful in

multicast rekeys only)

IPSec Transform

KS Configuration

28

Page 28: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

crypto gdoi group getvpn1

identity number 101

server local

!rekey address ipv4 150 !

rekey lifetime seconds 14400

rekey retransmit 10 number 2

rekey authentication mypubkey rsa getvpn1

rekey transport unicast

sa ipsec 1

profile gdoi1

match address ipv4 160

address ipv4 130.23.1.1

redundancy

local priority 10

peer address ipv4 130.1.2.1

!

Encryption ACL

GDOI Group ID

Rekey Address mapping

(only for multicast rekeys)

Source address for rekeys

Rekey Properties

COOP KS Config

KS Configuration (Cont.)

29

Page 29: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GM Configuration

crypto keyring gdoi

pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123

!

crypto isakmp policy 10

encr 3des

authentication pre-share

group 2

!

crypto gdoi group getvpn1

identity number 101

server address ipv4 130.23.1.1

!

crypto map getvpn10 gdoi

set group getvpn1

!

interface FastEthernet0/0

crypto map getvpn

Crypto map on the interface

GDOI Group

KS Address

GDOI crypto map

Pre-shared Key

ISAKMP Policy

30

Page 30: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Platform Support

Platform Group Member Key Server

Software Yes Not supported

870 Yes Not supported

1821 Yes Not supported

1841/1900 Yes Yes

2800 (AIM/SSL)/2900 Yes Yes

3800 (AIM-II/AIM-III)/3900 Yes Yes

7200 NPEG1, VAM2+ Yes Yes

7301 NPEG1, VAM2+ Yes Yes

7200 NPEG2, VAM2+ Yes Yes

7200 NPEG2, VSA Yes Yes

Cisco ASR 1000 Yes Yes (since XE3.6)

31

Page 31: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Scalability and Performance

GETVPN Provides complete segregation of control and data plane

Key Server is responsible to maintain the control plane (key management) and GM is responsible to handle the data plane (actual user traffic)

KS and GM can NOT be configured on same IOS device

KS should be properly sized for number of branches (scale) in the network

GM should be properly sized for traffic throughput at each branch

32

Page 32: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Deployment Best Practices

IKE/IPSec

Use specific pre-shared keys for all the GMs and KSs instead of using default key

KS

Always use COOP KSs

Set the huge buffer to 65535 and add 10 buffers to permanent buffer list

Configure periodic DPDs between the COOP KSs

Enable GM authorization

Policy

Aggregate the permit access-list entries to reduce the entries

Enable Time-Based Anti-Replay

Avoid re-encrypting traffic which is already encrypted (SSH, HTTPS)

Registration

Distribute GM registration to multiple KSs by arranging the KS order in configuration

Rekey Timers

Set TEK lifetime to 7200 Seconds

Set KEK lifetime to 86400 Seconds

34

Page 33: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

GETVPN Troubleshooting

Page 34: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

‟A problem well stated is a problem half solved”

• Charles F. Kettering

Page 35: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Troubleshooting GETVPN

Ultimately all problems manifest at the data plane -“my user application is not working over GETVPN!”

But where really is the problem?

Control Plane

– Events that lead up to SAs getting installed on the GMs

Data plane

– Policy downloaded with SAs installed but traffic is not flowing

37

Page 36: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Troubleshooting GETVPN High Level Flow

Control Plane Data Plane

COOP

IKE

Registration

Policy Download

Rekey

Troubleshooting Flow

Time Based

Anti-Replay

Fragmentation

MTU Issues

Transport Issues

Crypto

policy/engine

38

Page 37: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Control Plane

Common Control Plane Issues

– GM registration issues

– Policy download issues

– COOP issues

– Rekey failures

Understand the expected protocol flow and know how to check for them

39

Page 38: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Control Plane Troubleshooting Tools

GETVPN provides enhanced set of show commands for functionality verification

IOS also provided wide variety of syslog messages to verify proper GETVPN operations, and early insight into potential problems

IPSec and GDOI related debugs can then be enabled for further troubleshooting

GDOI conditional debugs – 15.1(3)T

GDOI event trace – 15.1(3)T

40

Page 39: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Show crypto gdoi (on KS)

Key Server Role

KEK lifetime remaining

COOP configuration

TEK lifetime remaining

Registered GMs

Group Name : GET

Group Identity : 101

Group Members : 3

IPSec SA Direction : Both

Active Group Server : Local

Redundancy : Configured

Local Address : 130.23.1.1

Local Priority : 10

Local KS Status : Alive

Local KS Role : Primary

Group Rekey Lifetime : 1800 secs

Group Rekey

Remaining Lifetime : 88 secs

Rekey Retransmit Period : 10 secs

Rekey Retransmit Attempts: 3

Group Retransmit

Remaining Lifetime : 0 secs

IPSec SA Number : 1

IPSec SA Rekey Lifetime: 900 secs

Profile Name : gdoi1

Replay method : Count Based

Replay Window Size : 64

SA Rekey

Remaining Lifetime : 446 secs

ACL Configured : access-list 160

42

Page 40: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Show crypto gdoi ks member (on KS)

KS#show crypto gdoi ks members

Group Member Information :

Number of rekeys sent for group GET: 4

Group Member ID : 131.1.1.1

Group ID : 101

Group Name : getvpn1

Key Server ID : 130.2.1.1

Rekeys sent : 4

Rekey Acks Rcvd : 4

Sent seq num : 1 2 3 4

Rcvd seq num : 1 2 3 4

KS GM is registered with

GM’s IP address

GM rekey history

43

Page 41: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Show crypto gdoi (on GM)

GROUP INFORMATION

Group Name : GET

Group Identity : 101

Rekeys received : 270

IPSec SA Direction : Both

Active Group Server : 134.50.0.1

Group Server list : 134.50.0.1

GM Reregisters in : 5187 secs

Rekey Received(hh:mm:ss) : 00:02:30

Rekeys received

Cumulative : 270

After registration : 270

Rekey Acks sent : 270

ACL Downloaded From KS 134.50.0.1:

access-list deny eigrp any any

access-list deny tcp any any port = 179

access-list deny udp any port = 848

any port = 848

access-list permit ip any any

KEKPOLICY:

Rekey Transport Type : Unicast

Lifetime (secs) : 12295

Encrypt Algorithm : 3DES

Key Size : 192

Sig Hash Algorithm : HMAC_AUTH_SHA

Sig Key Length (bits) : 1024

TEK POLICY:

FastEthernet0/0:

IPSec SA:

sadirection:outbound

spi: 0x7C45C74A(2084947786)

transform: esp-aes esp-sha-hmac

sa timing: remaining key lifetime

(sec): (5246)

Anti-Replay(Time Based) : 2 sec interval

Active KS

When was

last rekey

received

Remaining

IPSec SA

Lifetime

44

Page 42: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Control Plane Verification Syslog Messages - KS

Rekey:

GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from address 101.1.1.1 with seq # 1

COOP:

GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.0.9.1 Unreachable in group G1

GDOI-5-COOP_KS_ELECTION: KS entering election mode in group G1 (Previous Primary = NONE)

GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.0.8.1 in group G1 transitioned to Primary (Previous Primary = NONE)

45

Page 43: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Control Plane Verification Syslog Messages - GM

Registration:

CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1 using address 10.1.13.2

GDOI-5-GM_REKEY_TRANS_2_UNI: Group G1 transitioned to Unicast Rekey

GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for group G1 using address 10.1.13.2

Rekey:

GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to 10.1.13.2 with seq # 3

46

Page 44: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Control Plane Debugging Challenges

Challenge

Networks are getting bigger and faster, traditional debugs may not scale

Solution

Use IPSec and GDOI conditional debugs to minimize the debugging impact

Use the minimal level of debugs required

Challenge

Problems can be unpredictable with no identifiable trigger

Solution

Syslogs

GDOI Event Trace

47

Page 45: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GDOI Debug Level Granularity

All feature components can be debugged at 5 levels

Start with the highest level, enable additional levels as needed

Debug Level What you will get

Error Error Conditions

Terse Important messages to the user and protocol issues

Event State transitions and events such as send/receive rekeys

Detail Most detailed debug message information

Packet Dump of detailed packet information

All All of the above

GM1#debug crypto gdoi gm rekey ?

all-levels All levels

detail Detail level

error Error level

event Event level

packet Packet level

terse Terse level

48

Page 46: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GDOI Conditional Debugs

All IPSec and GDOI debugs can now be triggered with a conditional filter based on group or peer address

Use the unmatched flag to catch debugs with no context information

To enable conditional debugs

1) Set the conditional filter

2) Enable relevant debugs of interest as usual

KS1# debug crypto gdoi condition peer add ipv4 10.1.20.2

% GDOI Debug Condition added.

KS1#

KS1# show crypto gdoi debug-condition

GDOI Conditional Filters:

Peer Address 10.1.20.2

Unmatched NOT set

KS1#debug crypto gdoi ks registration all-levels

GDOI Key Server Registration Debug level: (Packet, Detail, Event, Terse, Error)

KS1

GM500 GM1

MPLS/Private IP

GM145

KS2

49

?

Page 47: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Best practices when using the debug commands

Turn off console logging

Use NTP to sync up times on all devices

Enable msectimestamping debug and log messages

– service timestamps debug datetime msec

– service timestamps log datetime msec

Send the debugs to a syslog server

If no syslog server is available, use the logging buffer with an increased buffer size

– logging buffered 1000000 debugging

terminal exec prompt timestamp when using the show commands to correlate show commands with the debug output

reload in x to prepare for the worst

50

Page 48: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GDOI Event Trace

Light weight event buffer to supplement syslogs

Always-on

Flexible output and display options

Event buffer

Continuous real time output

Output to file

Merged output from different feature components

Circular or one-shot buffer

Extensive exit path/error tracing capability

51

Page 49: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GDOI Event Trace - Example

GM1#show monitor event-trace gdoi?

all Show all the traces in current buffer

back Show trace from this far back in the past

clock Show trace from a specific clock time/date

coop GDOI COOP Event Traces

from-boot Show trace from this many seconds after booting

infra GDOI INFRA Event Traces

latest Show latest trace events since last display

merged Show entries in all event traces sorted by time

registration GDOI Registration event Traces

rekey GDOI Rekey event Traces

GM1#show monitor event-trace gdoi merged all

*May 25 20:20:57.706: Registration_events: GDOI_REG_EVENT: REGISTRATION_STARTED:

GM 10.1.20.2 to KS 10.1.11.2 for group G1

*May 25 20:21:08.970: Registration_events: GDOI_REG_EVENT: REGISTRATION_DONE: GM

10.1.13.2 to KS 10.1.11.2 for group G1

*May 26 00:45:52.878: Rekey_events: GDOI_REKEY_EVENT: REKEY_RCVD: From 10.1.11.2

to 10.1.13.2 with seq no 131 for the group G1

*May 26 00:45:52.878: Rekey_events: GDOI_REKEY_EVENT: ACK_SENT: From 10.1.11.2

to 10.1.13.2 with seq no 131 for the group G1

52

Page 50: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

crypto gdoi group G1

identity number 3333

server address ipv4 10.1.12.2

server address ipv4 10.1.11.2

!

crypto map gm_map 10 gdoi

set group G1

!

interface Serial1/0

crypto map gm_map

crypto gdoi group G1

identity number 3333

server address ipv4 10.1.11.2

server address ipv4 10.1.12.2

!

crypto map gm_map 10 gdoi

set group G1

!

interface Serial1/0

crypto map gm_map

crypto gdoi group G1

identity number 3333

server local

rekey lifetime seconds 86400

rekey authenmypubkeyrsa get

rekey transport unicast

saipsec 1

profile gdoi-p

match address ipv4ENCPOL

replay time window-size 5

address ipv4 10.1.12.2

redundancy

local priority 2

peer address ipv4 10.1.11.2

crypto gdoi group G1

identity number 3333

server local

rekey lifetime seconds 86400

rekey authenmypubkeyrsa get

rekey transport unicast

sa ipsec 1

profile gdoi-p

match address ipv4ENCPOL

replay counter window-size 64

address ipv4 10.1.11.2

redundancy

local priority 10

peer address ipv4 10.1.12.2

Troubleshooting Methodology

KS1 KS2

GM2 GM1

Ser 1/0: 10.1.20.2 Ser 1/0: 10.1.21.2

Eth 0/0: 192.168.20.1/24 Eth 0/0: 192.168.21.1/24

Ser 1/0: 10.1.11.2 Ser 1/0: 10.1.12.2

MPLS/Private IP

53

Page 51: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Control Plane Setup Steps

COOP KS IKE Setup

COOP Election and Policy Creation

GM-KS IKE Setup

GM Authorization and Registration

GM Encryption Keys and Policy download

GM Data Encryption and Decryption

Periodic Key Renewal and Distribution (Rekeys)

54

Page 52: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Common Issues – Control Plane

Encryption Policy

Key Renewal—Rekey

Control Plane Packet Fragmentation Issue

Control Plane Replay Check

55

IKE Setup

Page 53: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

IKE Setup Between KS and GM

First step in GM registration is IKE setup

On successful negotiation of the IKE process, GM proceeds with the GDOI group registration

IKE SA is established at the time of registration eventually times out as its no longer needed after registration

KS1# show crypto isakmp sa

IPv4 Crypto ISAKMP SA

Dst src state conn-id slot status

10.1.11.2 10.1.20.2 GDOI_IDLE 1013 0 ACTIVE

10.1.12.2 10.1.11.2 GDOI_IDLE 1004 0 ACTIVE

10.1.21.2 10.1.11.2 GDOI_REKEY 0 0 ACTIVE

GM1# show crypto isakmp sa

IPv4 Crypto ISAKMP SA

Dst src state conn-id slot status

10.1.11.2 10.1.20.2 GDOI_IDLE 1073 0 ACTIVE

10.1.20.2 10.1.11.2 GDOI_REKEY 1074 0 ACTIVE

Expires

after IKE

lifetime

56

Page 54: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

IKE Setup – IKE Failure Symptoms

If a GM fails to register with the KS, it will continue to attempt to register with the KS

Possible causes:

– Network issues between the GM and KS

– IKE negotiation failure

– KS policy issues

*May 24 06:40:15.581: %CRYPTO-5-GM_REGSTER: Start registration to KS

10.1.11.2 for group G1 using address 10.1.20.2

GM1#

*May 24 06:41:25.581: %CRYPTO-5-GM_REGSTER: Start registration to KS

10.1.11.2 for group G1 using address 10.1.20.2

KS1 KS2

GM2 GM1

MPLS/Private IP

%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for

group G1 using address 10.1.20.2

57

Page 55: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Pre-Shared Key Mismatch Troubleshooting

Verify routing information on KS and GM and try ping KS from the GM

After ruling out the connectivity issues, check the IKE SA on the GM

Verify the logs on the Key Server

GM1#show crypto isakmp sa

IPv4 Crypto ISAKMP SA

Dst src state conn-id status

10.1.11.2 10.1.20.2 MM_KEY_EXCH 1038 ACTIVE

IPv6 Crypto ISAKMP SA

IKE SA not getting established; can’t

get to GDOI_IDLE state

KS1#

%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 10.1.20.2 failed its

sanity check or is malformed

58

Page 56: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Pre-Shared Key Mismatch Solution

Syslog pointing to a mismatched pre-shared key configuration

Can be verified using “debug crypto isakmp”

KS Config:

GM Config:

crypto isakmp key cicso address 10.1.20.2

crypto isakmp key cisco address 10.1.11.2

Correct the pre-shared key configuration

KS1(config)#no crypto isakmp key cicso address 10.1.20.2

KS1(config)#crypto isakmp key cisco add 10.1.20.2

KS1(config)#^Z

59

Page 57: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Common Issues – Control Plane

IKE Setup

Key Renewal—Rekey

Control Plane Packet Fragmentation Issue

Control Plane Replay Check

60

Encryption Policy

Page 58: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GM1#show crypto gdoi

<snip>

ACL Downloaded From KS 10.1.11.2:

access-list deny eigrp any any

access-list deny ip 224.0.0.0 0.0.0.255 any

access-list deny ip any 224.0.0.0 0.0.0.255

access-list deny udp any port = 848 any port = 848

access-list permit ip any any

KEK POLICY:

Rekey Transport Type : Unicast

Lifetime (secs) : 2954

<snip>

TEK POLICY:

Serial1/0:

IPSec SA:

sa direction:inbound

spi: 0x2113F73B(554956603)

transform: esp-3desesp-sha-hmac

sa timing:remaining key lifetime (sec): (99)

Anti-Replay(Time Based) : 5 sec interval

<snip>

GM Policy Download

As part of the registration process, KS pushes down the encryption policies and keying material to the GM:

61

Page 59: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

KS Policy Issues Routing Control Plane Traffic Failure

In most environments, GETVPN runs on the CE devices and PE devices do not participate in GETVPN

Failure to deny control plane traffic (such as routing protocol) on the PE-CE link will cause routing protocol to go down as soon as GM successfully registers

To identify, look at the ACL downloaded at GM:

GM1#show crypto gdoi gm acl

Group Name: G1

ACL Downloaded From KS 10.1.11.2:

access-list deny eigrp any any

access-list deny ip 224.0.0.0 0.0.0.255 any

access-list deny ip any 224.0.0.0 0.0.0.255

access-list deny udp any port = 848 any port = 848

access-list permit ip any any

ACL Configured Locally:

KS1 KS2

GM2 GM1

MPLS/Private IP

BGP

BGP is not denied in the ACL

downloaded from the KS

62

Page 60: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

KS Policy Issues Control Plane Traffic - Solution

If most of the CEs are running BGP with the PE routers, configure a global KS policy to deny BGP

If only a handful of CEs are running BGP with the PE routers, configure a local GM policy to deny BGP

KS1&2(config)# ip access-list extended ENCPOL

KS1&2(config-ext-nacl)#1 deny tcp any any eq bgp

KS1&2(config-ext-nacl)#2 deny tcp any eq bgp any

GM1#

!

access-list 150 deny tcp any any eq bgp

access-list 150 deny tcp any eq bgp any

!

crypto map gm_map 10 gdoi

set group G1

match address 150

!

GM1#show crypto gdoi gm acl

Group Name: G1

ACL Downloaded From KS 10.1.11.2:

<snip>

access-list permit ip any any

ACL Configured Locally:

Map Name: gm_map

access-list 150 deny tcp any any port = 179

access-list 150 deny tcp any port = 179 any

KS1 KS2

GM2 GM1

MPLS/Private IP

BGP

63

Page 61: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Common Issues – Control Plane

IKE Setup

Encryption Policy

Control Plane Packet Fragmentation Issue

Control Plane Replay Check

64

Key Renewal - Rekey

Page 62: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Rekeys

Once the GETVPN network is properly setup and is working, KS is responsible for sending out rekey messages to all the GMs

KS can use unicast or multicast rekeys

Following syslog messages will appear in the log:

PRIMARY KS:

%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from

address 10.1.11.2 with seq # 11

All the GMs:

%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to

10.1.20.2 with seq # 11

65

Page 63: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Following the Rekey Flow

66

Rekey sent? Rekey

delivered?

Rekey

received?

Rekey received by

IP?

Rekey verified by

IKE?

Rekey Processed

by GDOI?

Rekey

Acknowledged?

KS Network

Transport

GM

Page 64: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Missing RSA Key Symptoms

When GM registers to the KS, the following message shows up in the syslog:

%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have

expired/been cleared, or didn't go through. Re-register to KS.

%GDOI-1-KS_NO_RSA_KEYS: RSA Key - get : Not found, Required for group G1

As a result KS will not send rekey messages, and GM will re-register when the keys expire

KS1 KS2

GM2 GM1

MPLS/Private IP

67

Page 65: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Missing RSA Key on the KS Troubleshooting Steps

Check whether KS is sending out the rekeys or not:

KS needs RSA keys to sign the rekey messages; check logs for clues and/or verify the RSA keys

KS1#show crypto gdoi ks rekey

Group G1 (Multicast)

Number of Rekeys sent : 0

Number of Rekeys retransmitted : 0

KEK rekey lifetime (sec) : 86400

Retransmit period : 10

Number of retransmissions : 2

IPSec SA 1 lifetime (sec) : 3600

Remaining lifetime (sec) : 166

Number of registrations after rekey : 22

No rekeys sent

68

Page 66: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Missing RSA Key on the KS Troubleshooting Steps (Cont.)

Verify RSA key configuration on the KS:

KS1#show running | section gdoi group

crypto gdoi group G1

identity number 3333

server local

rekey address ipv4 102

rekey lifetime seconds 86400

rekey authentication mypubkey rsa get

sa ipsec 1

profile gdoi-p

match address ipv4ENCPOL

no replay

address ipv4 10.1.11.2

Verify the RSA key pair name on the router:

KS1#show crypto key mypubkey rsa | include name

Key name: key1

Key name: key1.server

Labeled RSA key not present

69

Page 67: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Missing RSA Key on the KS Solution

Generate the required RSA key pair

KS1(config)#crypto key generate rsa label get exportable modulus 1024

The name for the keys will be: getvpn-rsa-key

% The key modulus size is 1024 bits

% Generating 1024 bit RSA keys, keys will be exportable...[OK]

Verify rekey messages are now being sent on the KS

%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from

address 10.1.11.2 with seq # 1

KS1#show crypto gdoi ks rekey

Group G1 (Unicast)

Number of Rekeys sent : 1

<SNIP>

Rekeys are now sent

70

Page 68: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Multicast Rekey Issues Multicast Rekeys Failing - Symptom

GM is not getting the multicast rekey messages and therefore

continues to re-register with the KS

Rekey starts to work when switched from multicast rekey to

unicast rekey

Possible Causes

– Packet delivery issue within the multicast routing infrastructure

– End-to-end multicast routing enabled?

– mVPN service provided by the MPLS core provider?

71

Page 69: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Multicast Rekey Failing Troubleshooting

Check KS to verify multicast rekey messages are being sent

Make sure ICMP is excluded from the KS encryption policy and is used as a tool to test multicast

%GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group G1

from address 10.1.11.2 to 226.1.1.1 with seq # 6

KS1#ping 226.1.1.1

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 226.1.1.1, timeout is 2 seconds:

Reply to request 0 from 10.1.21.2, 44 ms

No response from

GM1 (10.1.20.2)

KS1 KS2

GM2 GM1

Multicast

Network

10.1.20.2 10.1.21.2

72

Page 70: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Multicast Rekey Failing Troubleshooting

Check the multicast forwarding path

Check the PIM neighbor

WAN#sh ip pim neighbor

PIM Neighbor Table

Neighbor Interface Uptime/Expires Ver DR

Address Prio/Mode

10.1.11.2 Serial0/0 01:03:54/00:01:16 v2 1 / S

10.1.21.2 Serial3/0 01:13:06/00:01:26 v2 1 / S

WAN#show ip mroute 226.1.1.1

<snip>

(10.1.11.2, 226.1.1.1), 00:13:18/00:02:56, flags: T

Incoming interface: Serial0/0, RPFnbr 0.0.0.0

Outgoing interface list:

Serial3/0, Forward/Sparse-Dense, 00:13:18/00:00:00

Verify the OIL

73

Page 71: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Multicast Rekey Failing Solution

Enable PIM on the WAN router towards the GM

WAN(config)#int s2/0

WAN(config-if)#ip pim sparse-dense-mode

WAN(config-if)#end

%PIM-5-NBRCHG: neighbor 10.1.20.2 UP on interface

Serial2/0 (vrf default)

Check multicast routing path again

Re-test with multicast ping

Verify GM now receives the multicast rekey messages

74

Page 72: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Unicast Rekey Failing Transient Network Issues

Due to transient changes in the network, unicast rekey packets might not make it to the GM(s)

If the GMs does not receive the rekey, it will have to re-register

Symptoms:

Missing Following syslog on GM:

%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to 10.1.21.2

with seq # 3

GM shows re-registration syslog:

%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have expired/been

cleared, or didn't go through. Re-register to KS.

%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1 using

address 10.1.20.2

75

Page 73: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Unicast Rekey Failing Troubleshooting and Solution

Verify whether the rekeys are not being sent, not being received or not being processed

KS:

show crypto gdoi ks members

Group Member Information :

Number of rekeys sent for group G1 : 380

Group Member ID : 10.1.20.2

Group ID : 3333

Group Name : G1

Key Server ID : 10.1.11.2

Rekeys sent : 1

Rekeys retries : 0

Rekey Acks Rcvd : 0

Rekey Acks missed : 0

GM:

show crypto gdoi gm rekey

Group G1 (Unicast)

Number of Rekeys received (cumulative) : 0

Number of Rekeys received after registration : 0

Number of Rekey Acks sent : 0

Rekey (KEK) SA information :

dstsrcconn-id my-cookie his-cookie

New : 10.1.20.2 10.1.11.2 1098 44F7FC328302AC61

Current : 10.1.20.2 10.1.11.2 1098 44F7FC328302AC61

Previous: --- --- --- --- ---

Always configure retransmissions to overcome transient issues

rekey retransmit 30 number 3

Make sure UDP port 848 is not blocked in the data path

Unicast rekey dropped

76

Page 74: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Rekey Fails Signature Validation

Primary KS fails, GM receives rekey from secondary KS, but receives error:

*Apr 27 18:18:19.511: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode

failed with peer at 10.1.12.2

Syslog is not conclusive, let’s see what we can get with some debugs

GM1# debug crypto isakmp

Crypto ISAKMP debugging is on

GM1#

GM1# debug crypto gdoi

GDOI Generic Debug level: (Error, Terse)

*Apr 27 18:18:19.251: ISAKMP (0:1014): received packet from 10.1.12.2 dport 848

sport 848 Global (R) GDOI_REKEY

*Apr 27 18:18:19.251: GDOI:INFRA:(G1:0:1014:HW:0):Received Rekey Message!

*Apr 27 18:18:19.259: GDOI:INFRA:(G1:0:1014:HW:0):Signature Invalid! status = 13

*Apr 27 18:18:19.259: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed

with peer at 10.1.12.2

*Apr 27 18:18:19.259: ISAKMP: Receive GDOI rekey: Processing Failed. IKMP error = 6

Signature validation failed!

77

Page 75: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Rekey Fails Signature Validation Solution

Problem:

– Secondary KS has its own RSA key pair instead of the exported key pair from the primary

– To verify, compare the RSA key pairs

KS#show crypto key mypubkey rsa

KS1(config)#crypto key generate rsa modulus 1024 exportable label key1

KS2(config)#crypto key import rsa key1 pem terminal <passphrase>

Solution:

Generate exportable RSA key pair on the primary KS

Export RSA key pair to all secondary KSs

KS1 KS2

GM2 GM1

MPLS/Private IP

78

Page 76: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Common Issues – Control Plane

IKE Setup

Encryption Policy

Key Renewal—Rekey

Control Plane Packet Fragmentation Issue

79

Control Plane Replay Check

Page 77: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Control Plane Replay Check Detection

Control Plane messages can carry time sensitive information and therefore require replay protection

– Rekey messages from KS to GM

– COOP Announcement messages between KSs

Sequence number check to protect against replayed messages

Pseudotime check to protect against delayed messages with TBAR enabled

Control Plane Replay check added in IOS version 12.4(15)T10, 12.4(22)T3, 12.4(24)T2, 15.0(1)M, and later

80

Page 78: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Control Plane Replay Check Code interoperability issue

Problem: customer upgraded IOS on a GM to 15.0(1)M for a bug fix, and started to experience KEK rekey failures

The following errors are observed in the syslog

%GDOI-3-GDOI_REKEY_SEQ_FAILURE: Failed to process rekey seq # 1 in seq payload

for group G1, last seq # 11

%GDOI-3-GDOI_REKEY_FAILURE: Processing of REKEY payloads failed on GM 10.1.13.2

in the group G1, with peer at 10.1.11.2

%CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed with peer at

10.1.11.2

81

Page 79: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Control Plane Replay Check Code interoperability issue - solution

KS does not support control plane replay detection, and resets the rekey sequence # for KEK rekey

GM interprets that as a replayed rekey message

Solution is to upgrade the KS to an IOS version that also supports the control plane replay detection

New behavior

*Apr 6 15:41:26.932: %GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from

10.1.11.2 to 10.1.13.2 with seq # 8

GM1#

*Apr 6 15:42:01.940: %GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from

10.1.11.2 to 10.1.13.2 with seq # 1

TEK Rekey with seq# reset

KEK Rekey

82

Page 80: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Control Plane Replay Check – IOS Upgrade procedure

Recommended IOS releases

– IOS: 15.2(4)M3

– IOS-XE: 15.1(3)S4

IOS upgrade procedure

– Step 1. Upgrade a secondary KS first, wait until COOP KS election is completed

– Step 2. Repeat step 1 for all secondary KS

– Step 3. Upgrade primary KS

– Step 4. Upgrade Group Members

83

Page 81: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Common Issues – Control Plane

IKE Setup

Encryption Policy

Key Renewal—Rekey

Control Plane Replay Check

84

Control Plane Packet Fragmentation Issue

Page 82: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

buffers huge permanent 10

buffers huge size 65535

Control Plane Fragmentation Issues COOP Announcement Packets

In a large network (1500+ GMs), COOP update packet becomes larger than the default maximum buffer size

Default huge buffer size is 18024 bytes

Syslog message appears on the KSs:

Tune buffers to increase huge buffers and add buffers to permanent list:

%SYS-2-GETBUF: Bad getbuffer, bytes= 18872 -Process= "Crypto IKMP", ipl= 0, pid= 183

85

Page 83: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Control Plane Fragmentation Issues (cont.) COOP Announcement Packets

Large ANN messages are fragmented in transit between KSs

Can have up to 40+ IP fragments

One dropped fragment -> entire ANN dropped

How to identify?

Frag

1

Frag2

Frag3

Frag4

FragN

KS1 KS2

%GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.1.11.1 Unreachable in group G1.

%GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.1.12.2 in group G1 transitioned to

Primary (Previous Primary = 10.1.11.2)

KS1#show ip traffic | section Frags

Frags: 10 reassembled, 3 timeouts, 0 couldn't reassemble

0 fragmented, 0 fragments, 0 couldn't fragment

Need to look at transit path features that may drop fragments, Firewall, VFR, reassembly buffer size, etc.

86

Page 84: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Troubleshooting GETVPN Data Plane

Ultimately all problems manifest at the data plane -“my user application is not working over GETVPN!”

But where really is the problem?

Control Plane

– Events that lead up to SAs getting installed on the GMs

Data plane

– Policy downloaded with SAs installed but traffic is not flowing

87

Page 85: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Generic IPSec Data Plane Troubleshooting

Need to have complete understanding of the forwarding path and how to checkpoint it

Which device is the culprit, encrypting or decrypting router?

In which direction is the problem happening, ingress or egress?

Some syslogs may help reveal data plane drops

– Data plane errors are typically rate limited

– Common errors include replay, authentication failures

Heavily dependent upon show commands and counters to trace the packet path

Sniffer capture of limited use due to encryption, however

– ESP-NULL – same crypto processing except packets not encrypted

– DSCP coloring of packets to uniquely identify a flow

88

Page 86: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Data Plane

IPSec tunnel mode just like IPSec classic so most IPSec troubleshooting techniques still apply, however…

Symmetrical encryption policy requirement

Unique challenges with Header Preservation

– PMTUD

Time Based Anti-Replay

– Extra encapsulation overhead – Fragmentation boundary condition calculation

– Timer Based Anti-Replay failure

89

Page 87: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Data Plane Troubleshooting Tools

Interface counters

Encryption/decryption counters

Netflow

IP Accounting

ACL

DSCP packet coloring

Embedded Packet Capture (EPC)

90

Page 88: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

IPSec Data Plane Packet Flow Checkpoints

Encrypting GM

1. Ingress LAN interface Input ACL

Ingress Netflow

Embedded Packet Capture

2. Crypto engine show crypto ipsec sa

show crypto session detail

3. Egress WAN interface Egress Netflow

Embedded Packet Capture

Output IP precedence accounting

4 3

GM2 GM1

Client Server

1

2

6

5

Decrypting GM

4. Ingress WAN interface

Input ACL

Ingress Netflow

Embedded Packet Capture

Input IP precedence accounting

5. Crypto engine show crypto ipsec sa show crypto session detail

6. Egress WAN interface

Egress Netflow

Embedded Packet Capture

Traffic Direction

Private WAN

91

Page 89: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Importance of a “Controlled Test”

The case for “ping x.x.x.x timeout 0”

Separation from background traffic

– Poor man’s conditional filter

– Packet coloring/marking

– Tools to monitor based on DSCP/Precedence marking

– ESP-NULL

IP characteristics for seemingly application issues

– Ping works but TCP doesn’t?

– Why does IPSec care about TCP, or does it?

92

Page 90: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Encrypting GM Data Plane Flow

Verify clear traffic being received with Ingress Netflow

Verify encryption operation performed

Lack of per-flow granularity

interface Ethernet0/0

ip address 192.168.13.1 255.255.255.0

ip flow ingress

!

GM1# show ip cache flow

<snip>

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts

Et0/0 192.168.13.2 Se1/0 192.168.14.2 06 E443 0017 11

TCP port 23 = telnet

GM1# show crypto session detail

<snip>

IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0

Active SAs: 4, origin: crypto map

Inbound: #pkts dec'ed 162 drop 0 life (KB/Sec) 0/146

Outbound: #pkts enc'ed 170 drop 0 life (KB/Sec) 0/146

93

Page 91: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Encrypting GM Data Plane Flow – Cont.

Verify encrypted traffic existing GM with egress Netflow

interface Serial/0

ip address 10.1.13.2 255.255.255.252

ip flow egress

!

GM1#show ip cache flow

<snip>

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts

Et0/0 192.168.13.2 Se1/0* 192.168.14.2 32 EE5B 2BEF 170

GM1#show crypto ipsec sa

interface: Serial1/0

<snip>

current outbound spi: 0xEE5B2BEF(3998952431)

Protocol 50 = ESP

Active IPSec SA SPI

If per L4 flow granularity is desired, can use inbound precedence coloring and egress precedence accounting

94

Page 92: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Decrypting GM Data Plane Flow

Verify encrypted traffic arriving on GM with Netflow

GM2#show ip cache flow

<snip>

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts

Se1/0 192.168.13.2 Et0/0 192.168.14.2 32 EE5B 2BEF 170

Inbound IPSec SA SPI

Protocol 50 = ESP

Verify traffic decryption

GM2#show crypto session detail

<snip>

IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0

Active SAs: 10, origin: crypto map

Inbound: #pkts dec'ed 170 drop 0 life (KB/Sec) 0/150

Outbound: #pkts enc'ed 162 drop 0 life (KB/Sec) 0/150

Verify clear traffic forwarding post decryption

GM2#show ip cache flow

<snip>

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts

Se1/0 192.168.13.2 Et0/0* 192.168.14.2 06 E6CC 0017 170

TCP port 23 = telnet

95

Page 93: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Common Issues – Data Plane

Other data plane issues common to IPSec

Fragmentation/Path MTU

Asymmetrical Encryption Policy

96

Page 94: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

KS Policy Issues Data Plane Traffic Failure

Encryption policies (what needs to be encrypted) are defined centrally at the KS

Symmetrical ACLs should be defined to either permit or to deny traffic from getting encrypted

If the traffic is not being encrypted or being blocked, verify we have symmetrical ACL

GM2 GM1

MPLS/Private IP

Ethernet 0/0:

192.168.20.0/24

Ethernet 0/0:

192.168.21.0/24

KS Access-list

ip access-list extended ENCPOL

permit ip 192.168.20.0 0.0.0.255 192.168.21.0 0.0.0.255

permit ip 192.168.21.0 0.0.0.255 192.168.20.0 0.0.0.255

97

Page 95: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Common Issues – Data Plane

Other data plane issues common to IPSec

Asymmetrical Encryption Policy

Fragmentation/Path MTU

98

Page 96: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Fragmentation Issues PMTU Discovery

Large packets with the DF bit set may get black-holed in the GETVPN network

GM2 GM1

MTU 1500 MTU 1500

MTU 1000

1400B 1460B

ICMP 3/4

Server sends a large packet with the DF bit set in an attempt to perform network PMTUD

99

Page 97: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

PMTUD and GETVPN

Encrypting GM adds IPSec overhead and forwards it

Intermediate router drops the packet and sends back icmp3/4 to perform PMTUD, two possibilities

– This ICMP dropped by the encrypt GM because it’s not encrypted based on the encryption policy

– This ICMP gets forwarded to the end host but gets dropped due to unauthenticated payload

Bottom line: PMTUD does not work with the current header preservation implementation of GETVPN

100

Page 98: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

PMTUD and GETVPN

Solution

Implement ip tcp adjust-mss to reduce the TCP packet segment size

Clear the DF bit in the encapsulating header

interface Ethernet0/0

ip address 192.168.13.1 255.255.255.0

ip policy route-map clear-df-bit

!

route-map clear-df-bit permit 10

match ip address 111

set ip df 0

!

access-list 111 permit tcp any any

DF=1 Data

DF=0 Data DF=0

User Traffic

Encrypting GM

101

Page 99: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Common Issues – Data Plane

Asymmetrical Encryption Policy

Fragmentation/Path MTU

Other Data Plane Issues Common to IPSec

102

Page 100: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

IPSec drop due to packet corruption

IPSec integrity check makes IPSec packets a lot more sensitive to packet corruption in the network

Packet corruption symptoms

%CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=695

local=192.168.14.2 remote=192.168.13.2 spi=7C4E759F seqno=00000001

How to prove packets are corrupted in the network?

Enable EPC to capture packets into a circular buffer on both GMs

Use EEM (Embedded Event Manager) to

Synchronize and stop the capture on both routers when the RECVD_PKT_MAC_ERR message is logged

Notify the network operator by email

Retrieve both captures to examine for packet corruption

103

Page 101: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Troubleshooting Summary

Have a clear and concise problem description

Try to break the problem down to either control or data plane

Understand the expected protocol flow on the control plane and how to check for them

Understand where/how to checkpoint the data plane

Syslog and event trace your friend

There is always TAC!

104

Page 102: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Maximize your Cisco Live experience with your

free Cisco Live 365 account. Download session

PDFs, view sessions on-demand and participate in

live activities throughout the year. Click the Enter

Cisco Live 365 button in your Cisco Live portal to

log in.

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Cisco Daily Challenge points for each session evaluation you complete.

Complete your session evaluation online now through either the mobile app or internet kiosk stations.

105

Page 103: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services
Page 104: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

GETVPN Scalability and Troubleshooting Tools

Appendix

Page 105: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Key Server Scalability

Platform Crypto Card Max Number of GM Time to register to KS

7200/7201 VAM2+ 2000 15 sec *

3845 AIM-VPN/SSL-3 1000 15 sec *

3825 AIM-VPN/SSL-3 500 15 sec

2851 AIM-VPN/SSL-2 200 15 sec

2821 AIM-VPN/SSL-2 100 15 sec

1841 AIM-VPN/SSL-1 50 15 sec

7200/PKI VAM2+ 1000 20 sec **

* GM registration was distributed over two KSs to reduce the registration time

** GM registration was distributed over four KSs to reduce the registration time

108

Page 106: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GM Performance Attributes (No Features)

PPS Mbps Max IMIX Latency(ms) Avg 100 pps Latency (ms)

871 Anti-Replay 3150 28 <10

No Anti-Replay 3232 28 <5

1841-onboard Anti-Replay 3506 33 <20

No Anti-Replay 3766 35 <35

1841-aim/ssl Anti-Replay 8420 84 <10

No Anti-Replay 8472 84 <20

2821-onboard Anti-Replay 17152 50 <5

No Anti-Replay 17046 50 <1

2821-aim/ssl Anti-Replay 26010 190 <5

No Anti-Replay 25918 190 <5

2851-onboard Anti-Replay 17868 64 <5

No Anti-Replay 19175 65 <10

2851-aim/ssl Anti-Replay 27594 190 <1

No Anti-Replay 27668 190 <1

0.34

0.33

0.25

1.18

1.07

0.68

0.47

109

Page 107: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GM Performance Attributes (No Features)

PPS Mbps Max IMIX Latency(ms) Avg 100 pps Latency (ms)

3825-onboard Anti-Replay 35,505 283 <1

No Anti-Replay 35,500 283 <5

3825-aim/ssl Anti-Replay 44,170 199 <1

No Anti-Replay 44,452 199 <5

3845-onboard Anti-Replay 46,028 284 <5

No Anti-Replay 46,028 283 <5

3845-aim/ssl Anti-Replay 54,020 200 <1

No Anti-Replay 53,996 200 <1

7200-g1vam2+ Anti-Replay 60,592 266 <5

No Anti-Replay 66,952 266 <5

7200-g2vam2+ Anti-Replay 121,952 283 <5

No Anti-Replay 120,890 283 <1

7200-g2/vsa Anti-Replay

No Anti-Replay 160,000 980 TBD

ASR1000/FP5G Anti-Replay 440,000

No Anti-Replay 470,000 1,890 TBD

ASR1000/FP10G Anti-Replay 976,000 4,200

No Anti-Replay 1,011,000 4,220 <0.270

ASR1000/FP20G Anti-Replay 2,655,000 TBD

No Anti-Replay 2,685,000 8,530 <0.0150.001

0.64

0.66

0.19

TBD

TBD

0.17

0.76

0.81

0.69

110

Page 108: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GM Performance Attributes (No Features)

Frame Size ASR 1004 (10Gig) 7200 VSA 3845 AIM-

VPN/SSL-3

ISRG2

3945 Onboard

Crypto

ISRG2

2951 Onboard

Crypto

ISRG2

1941

Onboard

Crypto

1400 Byte

4759 Mbps

925 Mbps 200 Mbps

820 Mbps

268 Mbps 154

Mbps

IMIX (90 Bytes 61%,

594 bytes 24%, 1418 15%)

2289 Mbps 780 Mbps 177 Mbps 261Mbps 160 Mbps 64Mbp

s

111

Page 109: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

GETVPN Verification Common KS Syslog Messages

Syslog Messages Explanation

COOP_CONFIG_MISMATCH The configuration between the primary key server and secondary key server are

mismatched.

COOP_KS_ELECTION The local key server has entered the election process in a group.

COOP_KS_REACH The reachability between the configured cooperative key servers is restored.

COOP_KS_TRANS_TO_PRI The local key server transitioned to a primary role from being a secondary server

in a group.

COOP_KS_UNAUTH An authorized remote server tried to contact the local key server in a group.

Could be considered a hostile event.

COOP_KS_UNREACH The reachability between the configured cooperative key servers is lost. Could

be considered a hostile event.

KS_GM_REVOKED During rekey protocol, an unauthorized member tried to join a group. Could be

considered a hostile event.

KS_SEND_MCAST_REKEY Sending multicast rekey.

KS_SEND_UNICAST_REKEY Sending unicast rekey.

KS_UNAUTHORIZED During GDOI registration protocol, an unauthorized member tried to join a group.

Could be considered a hostile event.

UNAUTHORIZED_IPADDR The registration request was dropped because the requesting device was not

authorized to join the group.

112

Page 110: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Syslog Messages Explanation

GM_CLEAR_REGISTER The clear crypto gdoi command has been executed by the local

group member.

GM_CM_ATTACH A crypto map has been attached for the local group member.

GM_CM_DETACH A crypto map has been detached for the local group member.

GM_RE_REGISTER IPSec SA created for one group may have been expired or

cleared. Need to reregister to the key server

GM_RECV_REKEY Rekey received

GM_REGS_COMPL Registration complete

GM_REKEY_TRANS_2_MULTI Group member has transitioned from using a unicast rekey

mechanism to using a multicast mechanism.

GM_REKEY_TRANS_2_UNI Group member has transitioned from using a multicast rekey

mechanism to using a unicast mechanism.

PSEUDO_TIME_LARGE A group member has received a pseudotime with a value that is largely

different from its own pseudotime.

REPLAY_FAILED A group member or key server has failed an anti-replay check.

GETVPN Verification Common GM Syslog Messages

113

Page 111: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Packet marking Techniques

IP TOS byte copied from inner header to the encapsulating delivery header by default

How to mark

– PBR

– MQC

– Local ping

How to monitor

– IP precedence accounting

– ACL counters

114

Page 112: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

ToS/Precedence/DSCP Reference Chart

7 5 6 4 3 2 1 0

IP Precedence Priority

DSCP

Least

Significant

Bit

ToS Byte

Hex - Decimal ToS

IP Precedence DSCP Binary

20

00

40

48

E0

C0

B8

A0

88

68

60

32

0

64

72

224

192

184

160

128

104

96

1 Priority

0 Routine

2 Immediate

7 Network Control

5 Critical

4 Flash Override

3 Flash

8 CS1

0 Dflt

16 CS2

18 AF21

56 CS7

48 CS6

46 EF

40 CS5

32 CS4

26 AF31

24 CS3

00100000

00000000

01000000

01001000

11100000

11000000

10111000

10100000

10000000

01101000

01100000

6 Internetwork Control

80

136 34 AF41 10001000

115

Page 113: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Packet marking - Examples

PBR

MQC

interface Ethernet1/0

ip policy route-map mark

!

access-list 150 permit ip host 172.16.1.2 host 172.16.254.2

!

route-map mark permit 10

match ip address 150

set ip precedence flash-override

class-map match-all my_flow

match access-group 150

!

policy-map marking

class my_flow

set ip precedence 4

!

interface Ethernet1/0

service-policy input marking

IP flow in question marked with

precedence 4

116

Page 114: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Packet marking - Examples

Router Ping

Router#ping ip

Target IP address: 172.16.254.2

Repeat count [5]: 100

Datagram size [100]:

Timeout in seconds [2]:

Extended commands [n]: y

Source address or interface:

Type of service [0]: 128

Set DF bit in IP header? [no]:

Validate reply data? [no]:

Data pattern [0xABCD]:

Loose, Strict, Record, Timestamp, Verbose[none]:

Sweep range of sizes [n]:

Type escape sequence to abort.

Sending 100, 100-byte ICMPEchos to 172.16.254.2, timeout is 2 seconds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

117

Page 115: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Packet marking - Monitoring

IP Precedence Accounting

Interface ACL

interface Ethernet0/0

ip address 192.168.1.2 255.255.255.0

ip accounting precedence input

middle_router#show interface precedence

Ethernet0/0

Input

Precedence 4: 100 packets, 17400 bytes

middle_router#sh access-list 144

Extended IP access list 144

10 permit ip any any precedence routine

20 permit ip any any precedence priority

30 permit ip any any precedence immediate

40 permit ip any any precedence flash

50 permit ip any any precedence flash-override (100 matches)

60 permit ip any any precedence critical

70 permit ip any any precedence internet (1 match)

80 permit ip any any precedence network

118

Page 116: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Using Packet Captures for Data Plane Issues

Packet captures can provide detailed packet information at the bits/bytes level

The new packet capture infrastructure introduced in 12.4(20)T makes this easy to do

– Ability to capture IPv4 and IPv6 packets in the CEF path

– Configurable capture buffer and capture point parameters

– Extensible output filtering and export capabilities

– Support for various WAN encapsulation types

119

Page 117: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Using IOS Embedded Packet Captures

Router#monitor capture buffer test-buffer

Router#monitor capture buffer test-buffer filter access-list 120

Filter Association succeeded

Router#

Router#monitor capture point ipcef test-capture serial 2/0 both

*Mar 26 20:33:10.896: %BUFCAP-6-CREATE: Capture Point test-capture created.

Router#monitor capture point associate test-capture test-buffer

Router#monitor capture point start test-capture

*Mar 26 20:34:03.108: %BUFCAP-6-ENABLE: Capture Point test-capture enabled.

Router#

Router#monitor capture point stop test-capture

*Mar 26 20:34:21.636: %BUFCAP-6-DISABLE: Capture Point test-capture disabled.

Key Configuration Steps

Create the capture buffer and capture point

Associate the capture point to the buffer

Start/stop the capture

120

Page 118: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Using IOS Embedded Packet Captures Now we have the packets captured, what’s next?

Router# show monitor capture buffer test-buffer dump

15:34:07.228 EST Mar 26 2009 : IPv4 LES CEF : Se2/0 None

05CECE30: 0F00080045C0002C ....E@.,

05CECE40: 6D170000FE0649DD 02010102 01010114 m...~.I]........

05CECE50: 0017A3530FB6B9523EF1499C 60121020 ..#S.69R>qI.`..

05CECE60: 917A0000 02040218 00 .z.......

Router# monitor capture buffer test-buffer export?

ftp: Location to dump buffer

http: Location to dump buffer

https: Location to dump buffer

rcp: Location to dump buffer

scp: Location to dump buffer

tftp: Location to dump buffer

Dump the packet on the router itself

Dump the packet on the router itself

Or export it out and analyze it in Wireshark

Page 119: Troubleshooting GETVPN Deploymentsd2zmdbbm9feqrf.cloudfront.net/2013/usa/pdf/BRKSEC-3051.pdf · Troubleshooting GETVPN Deployments BRKSEC-3051 Wen Zhang - Technical Leader, Services

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSEC-3051 Cisco Public

Use EEM and EPC to catch Packet Corruption

event manager applet detect_bad_packet

event syslog pattern "RECVD_PKT_MAC_ERR"

action 1.0 cli command "enable"

action 2.0 cli command "monitor capture point stop test"

action 3.0 syslog msg "Packet corruption detected and capture stopped!"

action 4.0 snmp-trap intdata1 123456 strdata ""

Peer1

Peer2

event manager applet detect_bad_packet

event snmp-notification oid 1.3.6.1.4.1.9.10.91.1.2.3.1.9. oid-val "123456" op

eq src-ip-address 20.1.1.1

action 1.0 cli command "enable"

action 2.0 cli command "monitor capture point stop test"

action 3.0 syslog msg "Packet corruption detected and capture stopped!"