Network Analytics using Nexus 3000/9000...

Network Analytics using Nexus 3000/9000 Switches

Yogesh Ramdoss, Technical Leader, Cisco Services

BRKDCN-3020

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Fun in bringing things together …. and exploring

3BRKDCN-3020



4BRKDCN-3020

Cisco Open Architecture &

Programmability

ANALYTICS

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5BRKDCN-3020

Analytics…

Analytics is the

discovery,

interpretation, and

communication of

meaningful patterns in

data.

… Wikipedia

Analytics relies on the simultaneous application of data to get useful insights.

Marketing/Portfolio Analytics

Risk Analytics

Security Analytics

Software Analytics

…. Network Analytics


Goal of this session …

• Creates awareness of tools like Latency and Buffer Monitoring built into the Nexus 3000 and 9000 (standalone) platforms – which can be used to get visibility into the applications, network traffic, and also generate analytics.

• Educates on NX-API capabilities, benefits and ease-of-use.

• With Nexus Data Broker as a tap-and-aggregation core, discusses how users are leveraging Cisco and 3rd-party devices/applications to gain network visibility, for early threat detection and to generate analytics.

• Shows how different tools along with Python, XML/JSON, REST and NX-API can be brought together to generate analytics, with a real-world use-case.

6BRKDCN-3020


Let’s do it !!

7BRKDCN-3020

THERE’S NEVER BEEN A BETTER TIME TO ….

Bring data together

…. and ….

Generate Analytics

• Nexus 3000/9000: Built-in Tools

• Nexus 3000/9000: NX-API Usage

• Nexus Data Broker (NDB) and Tools

• Bringing All Together – Analytics

• Summary

Agenda

Nexus 3000/9000: Built-in Tools



Nexus 3000/9000 platforms have so many built-in tools … Which are the ones we are going to look into and why ?

Tools we are choosing are the ones helping to get insights into the device/network performance rather than capturing packet(s) in specific flow for further analysis.

10BRKDCN-3020



• Latency Monitoring – Nexus3500X/3548

• Active Buffer Monitoring – Nexus 3548

• Micro-burst Monitoring – Nexus 3000/9000

Agenda



Latency Monitoring

It is simple ….

Latency could impact applications’ performance and result in bad user experience

Why we need it ?

Accessing

Websites

Video

Conferencing

Online

Games

Trade

Floors and so on…

Video

Streaming

Sometimes, latency may also drive people crazy !!


Active Latency Monitoring – Nexus 3500X

• Real-time view of latency incurred by the frames going through the switch on a per-port basis

• As soon as a frame enters the switch, a timestamp (based on local on-chip time) added to it

• Once it is scheduled to be transmitted, egress port calculates the latency (current time – timestamp on the packet)

• Egress ports maintain the information of frame count along with min/max/total latency

• Total Latency = Sum of latencies for the frames (frame count during each polling interval). Average latency = total latency / number of frames

What it is and How it is implemented ?



Each egress port has the information of frame count (32 bits) and latency register (58 bits) along with min, max and average latency.

What it is and How it is implemented ? (Contd.)

TX Packets: 1000

Min: 180ns

Max: 250ns

Avg: 210ns

Timestamp

Packet

T0

T1 – T0



• Software copies every hour data to the bootflash: and also keeps the latency information in the memory for the last 1 hour, i.e. total of 2 hours of data available

• Software periodic reading could be as low as 1 sec

• After each software read, the record is cleared

What it is and How it is implemented ? (Contd.)



Nexus3500(config)# hardware profile latency monitor

Nexus3500(config)# hardware profile latency monitor sampling 3

Nexus3500(config)# hardware profile latency monitor threshold-avg 500

Nexus3500(config)# hardware profile latency monitor threshold-max 700

Configuration and Results

Nexus3500# show hardware profile latency monitor summary

All latency information provided is measured as FILO (First In Last Out).

05/16/2016 17:42:19

Device instance 0

Total Switch

============

3s 30s 1hr All Time

Min Latency (ns) 390 375 n/a 363

Max Latency (ns) 775 1844 n/a 1950

Avg Latency (ns) 612 721 n/a 754

Std Deviation 205.34 117.23 n/a 69.17

<snip>

How often to sample ?

When to generate syslog ?

current time

for the whole switch



Nexus3500# show hardware profile latency monitor summary brief

Nexus3500# show hardware profile latency monitor summary detail [intf #]

Nexus3500# show hardware profile latency monitor summary [intf #]

Nexus3500# show hardware profile latency monitor summary clear-timestamp

Nexus3500# show hardware profile latency monitor summary sort

Nexus3500# show hardware profile latency monitor summary top

Nexus3500# clear hardware profile latency monitor [intf #]

Configuration and Results (Contd.)

Nexus3500# show hardware profile latency monitor summary

<snip>

Ethernet1/1

============

3s 30s 1hr All Time

Min Latency (ns) 775 762 n/a 762

Max Latency (ns) 775 1757 n/a 1950

Avg Latency (ns) 775 838 n/a 870

Std Deviation n/a 83.87 n/a 100.93

<snip>

for egress port Eth1/1latencies incurred while egressing specific port



• Disabling the latency monitor does not clear existing data

• Clear the latency monitor data before enabling it

• Data is lost when sampling interval is modified

• Data is not preserved across a switch reload

Limitations and Guidelines


Latency Monitoring – Nexus 3548

Requirements:

• PTP deployed in the Data Center – Grand Master GPS Synced and PTP Boundary Clocks

• Feature PTP need to be enabled in Nexus 3548 globally and on specific interfaces

• ERSPAN Header type 3

• For PTPv2, Nexus 3548 6.0(2)A1(1) and later releases

Implementation:

• PTP is hardware-assisted. No performance impact.

• Both layer2 and layer3 ports support PTP.

Precision Time Protocol

PTP grandmaster

Nexus Switch

GPS

Network

Network


Latency Monitoring – Nexus 3548Leveraging Precision Time Protocol (PTP) - Wireshark

Network

Nexus 3548

Nexus 3548

Server w/ Wireshark

Nexus3548(config)# monitor session 1 type erspan-source

Nexus3548(config-erspan-src)# source interface Ethernet 1/25 rx

Nexus3548(config-erspan-src)# destination ip 192.168.100.100

Nexus3548(config-erspan-src)# header-type 3


Latency Monitoring – Nexus 3548Leveraging Precision Time Protocol (PTP) - Corvil

Network

Nexus 3548

Nexus 3548Nexus3548(config)# monitor session 1 type erspan-source

Nexus3548(config-erspan-src)# source interface Ethernet 1/25 rx

Nexus3548(config-erspan-src)# destination ip 192.168.100.100

Nexus3548(config-erspan-src)# header-type 3

Latency Navigator






Agenda


Active Buffer Monitoring

• ASIC has 18 buckets, each bucket corresponds to range of buffer utilization.

Example: (0-384KB), (385KB-768Kb), etc.

• ASIC polls the buffer utilization for all the ports every 4 msec (default)

• Based on buffer utilization for each HW polling interval, bucket counter for

corresponding range is incremented. Example: if port 25 is consuming 500KB of

buffer, bucket #2 (385-768KB) counter is incremented

• This buffer utilization counters maintained for each interface in histogram format

• Each bucket is represented with 8 bits, so saturates after 255 hits and it resets

once software reads the data

Implementation in Nexus 3500 – Hardware Implementation



• Every 1 second, SW polls ASIC to download & clear all histogram counters

• These histogram counters are maintained in the memory for last 60 minutes with

1-second granularity.

• Software also make sure every 1hour it copies the buffer histogram to the

bootflash:, which can be copied to the analyzer for further analysis

• Effectively, this maintains 2 hour worth of Buffer histogram data for all the ports,

latest 1 hour in the memory and second hour in the bootflash:

Implementation in Nexus 3500 – Software Implementation


1 2 3 4 5 6 … 24

48…302928272625

Packet

25BRKDCN-3020

Active Buffer MonitoringBenefits

Shared Buffer

Data collection using

XML interface

Configurable Buffer usage

Threshold to generate

syslog message

Percentage of time buffers were spent empty,

fully occupied with millisecond granularity

Buffer occupancy histogram for default class

on all the 48 ports in the system

Granular data on buffers’ usage

Active Buffer

Monitoring


Kbytes 384 768 1152 1536 1920 2304 2688 3072 3456 3840 … 6144

9/15/2012

3:11:01 PM5 0 5 10 90 140 0 0 0 0 … 0

9/15/2012

3:11:02 PM0 0 0 0 0 10 90 100 50 0 … 0

9/15/2012

3:11:03 PM0 0 0 0 0 0 10 80 110 50 … 0

9/15/2012

3:11:04 PM0 0 0 0 0 100 120 30 0 0 … 0

9/15/2012

3:11:05 PM0 5 10 85 150 0 0 0 0 0 … 0

9/15/2012

3:11:06 PM200 50 0 0 0 0 0 0 0 0 … 0

1 2 3 4 5 6 … 24

48…302928272625

Packet

26BRKDCN-3020

Active Buffer MonitoringAlgoboost Buffer Histogram – HW/SW Polling

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44#

Of

Sam

ple

s

Buffer Buckets

Shared Buffer

Software PollingHardware Polling


Kbytes 384 768 1152 1536 1920 2304 2688 3072 3456 3840 … 6144

9/15/2012

3:11:01 PM5 0 5 10 90 140 0 0 0 0 … 0

9/15/2012

3:11:02 PM0 0 0 0 0 10 90 100 50 0 … 0

9/15/2012

3:11:03 PM0 0 0 0 0 0 10 80 110 50 … 0

9/15/2012

3:11:04 PM0 0 0 0 0 100 120 30 0 0 … 0

9/15/2012

3:11:05 PM0 5 10 85 150 0 0 0 0 0 … 0

9/15/2012

3:11:06 PM200 50 0 0 0 0 0 0 0 0 … 0

1 2 3 4 5 6 … 24

48…302928272625

Packet

Active Buffer MonitoringAlgoboost Buffer Histogram – HW/SW Polling

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44

0

50

100

150

38

4

76

8

11

52

15

36

19

20

23

04

26

88

30

72

34

56

38

40

42

24

46

08

49

92

53

76

57

60

61

44#

Of

Sam

ple

s

Buffer Buckets

Shared Buffer

Software PollingHardware Polling

27BRKDCN-3020



Nexus3548(config)# hardware profile buffer monitor [ unicast | multicast ]

Nexus3548(config)# hardware profile buffer monitor [ unicast | multicast ] threshold <value>

Nexus3548(config)# hardware profile buffer monitor [ unicast | multicast ] sampling <value>

Configuration and Show Commands

Nexus3548# show hardware profile buffer monitor interface ethernet 1/4 detail Detail CLI issued at: 09/10/2015 22:15:42KBytes 384 768 1152 1536 1920 2304 2688 3072 3456 3840 4224 4608 4992 5376 5760 6144 us @ 10Gbps 307 614 921 1228 1535 1842 2149 2456 2763 3070 3377 3684 3991 4298 4605 4912

---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----09/10/2015 22:15:41 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 09/10/2015 22:15:40 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 09/10/2015 22:15:39 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 09/10/2015 22:15:38 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 09/10/2015 22:15:37 34 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 09/10/2015 22:15:36 139 111 0 0 0 0 0 0 0 0 0 0 0 0 0 0 09/10/2015 22:15:35 0 67 179 4 0 0 0 0 0 0 0 0 0 0 0 0 09/10/2015 22:15:34 0 0 0 174 76 0 0 0 0 0 0 0 0 0 0 0 09/10/2015 22:15:33 0 0 0 0 102 148 0 0 0 0 0 0 0 0 0 0 09/10/2015 22:15:32 0 0 0 0 0 30 178 43 0 0 0 0 0 0 0 009/10/2015 22:15:31 0 0 1 0 0 1 0 208 0 0 0 0 0 0 0 0 09/10/2015 22:15:30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 <snip>

How often to sample

?

When to generate syslog ?

“logging level mtc-usd 5” required to generate syslog.

Which traffic ? Only one type at any given time



• By normal means, administrators can access the switch to collect last 1 hour data in the system memory

• Using native NX-OS Python, historic data can be copied to external servers

• Last 1 hour data can be accessed from the switch bootflash: file system. This data can be transported to external server, for example using FTP.

• All Active Buffer Monitoring data have XML equivalents – collections can be automated at any desired interval.

Data Access and Collection for Analytics

Python / XML

Nexus 3548

CorvilNet can correlate latency data with the buffer usage from Active Buffer Monitoring



Does active buffer monitoring impact performance or Latency?

No, this feature doesn't impact latency or performance of the switch

What is the impact of lower Active Buffer Monitoring hardware polling interval?

By default HW polling interval is 4msec. Users can configure this value as low as 10nsec. There is no performance impact because of lower hardware polling interval.

Then, why the default hardware polling interval is set to 4 milliseconds, not more granular ?

The default HW polling of 4msec is chosen to make sure we do not overflow the histogram counters before software polls, every 1 sec (cannot be changed due to CPU/Memory restrictions). If you lower the HW polling interval then it may saturate the hardware counters at 255 samples.

Frequently Asked Questions






Agenda

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 33

Microburst

• Spike of activity – may result in the

system resource exhaustion / saturation

• How short and how high? – Capacity of

“weakest” system in the network

• Not captured by traditional load-

monitoring tools

Why it is important to monitor ?

BRKDCN-3020


Micro-burst Monitoring – Nexus 3000/9000

• Allows monitoring traffic to detect unexpected data bursts

• Detected when egress queue rises above a configured threshold

• It is supported in the following switches/modules, which provides dedicated statistics interface for OOB stats describing timestamp information and instantaneous buffer usage – Nexus 3232C, Nexus 3264Q and Nexus 9500 (9432C-S 100G module).

Introduction

Nexus 3232C

Nexus 3264Q

Nexus 9500


Micro-burst Monitoring – Nexus 3000/9000 Configuration and Show commands

Nexus3264Q(config)# policy-map type queuing micro-burst-monitor

Nexus3264Q(config-pmap-que)# class type queuing c-out-def

Nexus3264Q(config-pmap-c-que)# burst-detect rise-thresh 208 bytes fall-thresh 208 bytes

Nexus3264Q# show queuing burst-detect detail

slot 1

--------------------------------------------------------------------------

Out Of Band Statistics

--------------------------------------------------------------------------

Ethernet |Queue|Pipe |Start Depth| Start Time |Peak Depth|

Interface| | | (bytes) | | (bytes) |

---------------------------------------------------------------------------

Ether1/23| 0 | XPE-A | 23000 | 2015/09/12 16:43:12:227129 | 24174 |

------------------------------------------------------------------------------------

------------------------------------------------------------------------------------

Peak Time |End Depth| End Time |Duration(nsecs)

| (bytes) | |

------------------------------------------------------------------------------------

2015/09/12 16:43:12:239457 | 22850 | 2015/09/12 16:43:12:241236 | 14 msec


Micro-burst Monitoring – Nexus 3000/9000

• Supported only for unicast egress queues, not for multicast, CPU and SPAN queues.

• Fall and rise thresholds needs to be fine-tuned to avoid jitter

• Maximum number of burst records supported in the range of 200-2000. Default 1000.

• More the queues monitored, longer the duration of the burst that can be detected. E.g.,

• 1 – 3 queues: 0.64 microsecond of burst duration

• 8 queues: 9.0 microsecond

• 10 queues: 140 microsecond

Limitations

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

Real-world ExampleLow Video Quality

eth1/1

eth1/3eth1/5

eth1/25

eth1/21

XX X

I see

intermittent

low quality

User1 User2 User3 User4 User5

Same VLAN

me too …

me too …

I am peachy

Network

Video streaming server

BRKDCN-3020

Nexus 3548


Real-world Example

Let us have a closer look at the problem …

• Only specific not all users report low video quality.

• Traffic captures at the affected users indicate intermittent delay in the traffic, but no gap in the sequence.

• If the traffic loss is seen, then there could be micro-burst resulting in resource exhaustion/buffer drops.

• If the switch introduces latency, then most of the times the issue should be same across all the ports.

First thing to check…

Why only specific users ?

Low Video Quality

Nexus 3548


Real-world Example

Ports in issue – Eth 1/1, 1/3 and 1/25. Port with no issue – Eth 1/15 and 1/21.

Nexus 3548 architecture has three Output Buffer blocks – each one serving set of 16 ports, mapped as follows.

Buffer Block #2 Buffer Block #1 Buffer Block #0

Low Video Quality

Let us monitor buffer usage !!


Real-world ExampleActive Buffer Monitoring

Nexus3548# show hardware profile buffer monitor interface ethernet 1/1 detail Detail CLI issued at: 04/18/2016 11:23:19KBytes 384 768 1152 1536 1920 2304 2688 3072 3456 3840 4224 4608 4992 5376 5760 6144 us @ 10Gbps 307 614 921 1228 1535 1842 2149 2456 2763 3070 3377 3684 3991 4298 4605 4912

---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----04/18/2016 11:23:18 245 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 04/18/2016 11:23:17 139 106 0 0 0 0 0 0 0 0 0 0 0 0 0 0 04/18/2016 11:23:16 37 129 79 0 0 0 0 0 0 0 0 0 0 0 0 004/18/2016 11:23:15 0 83 107 67 0 0 0 0 0 0 0 0 0 0 0 004/18/2016 11:23:14 1 0 92 110 46 0 0 0 0 0 0 0 0 0 0 004/18/2016 11:23:13 0 0 0 55 132 59 0 0 0 0 0 0 0 0 0 004/18/2016 11:23:12 0 0 1 0 82 107 58 0 0 0 0 0 0 0 0 0<snip>

The buffer monitoring results indicate slow drain of buffers. Why ??

Last user added to Ethernet 1/25 is operating at 1Gbps, while all other ports mapped to the specific output buffer are operating at 10Gbps.

Workaround:

Add “hardware profile multicast slow-receiver port ethernet 1/25” config command.


Nexus Switches: Built-in Tools

• SPAN / RPSAN / ERSPAN• SPAN on Drop

• SPAN on Latency

• SPAN with ACL Filter

• EthAnalyzer / Inband SPAN

• Rule-based SPAN

• Exception SPAN

• Flexible Netflow

• Nexus 9000: QoS Buffer Monitoring

More Tools to Leverage

Relevant Session(s):

BRKDCT-1890 Network Visibility using Advanced Analytics in Nexus Switches

BRKARC-2011 Overview of Packet Capturing Tools in Cisco Switches and Routers


Let’s do it !!

42BRKDCN-3020


Bring data together

…. and ….

Generate Analytics

Built-in Tools

& CLIs

Nexus 3000/9000: NX-API Usage


Application Programming Interface (API)

• API exposes internal function(s) so that external applications can leverage the functionality without really getting into how this functionality is implemented.

• Set of requirements that govern how specific API (and the functionality it provides) is used by external applications.

• Most of the times APIs come in the form of a library – with specifications for routines, data structures, variables and more.

• Yes, it is important. Because it provides:

• Modularity

• Abstraction

• Automation

What it is and Is that important ?


On-the-Box

Python

EEM

Scheduler

Bash

vi Editor

45BRKDCN-3020

Nexus 3000/9000 – ProgrammabilityOptions

Off-the-Box

Expect/Tcl

NX-API

Container

Guest-shell

LXC

Config-

Management

Puppet

Chef

Ansible

NX-API is an enhancement to the Cisco NX-OS CLI system, so that same set of CLIs are available outside of the device.


NX-API

• HTTP/HTTPS interface to standard NX-OS commands

• Commands are encoded in the HTTP/HTTPS POST payload

• Data encoding formats: XML and JSON

• Supports off-the-box Python scripting

• Open RPC API – Support REST

• Supports RBAC – restricts read/write access

Capabilities and Usage

Nexus9000(config)# feature nxapi

Nexus9000(config)# nxapi http port <port#>

Nexus9000(config)# nxapi https port <port#>

Nexus9000(config)# nxapi certificate <options>

Nexus9000(config)# nxapi sandbox

Nexus9000# show nxapi

nxapi enabled

HTTP Listen on port 80

HTTPS Listen on port 443

Nexus9000# show nxapi <options>


NX-APIComponents

Transport

Security

Integrated to the devices’ authentication system

Strongly recommended to use HTTPS to secure user credentials

Provides session-based cookie (expires in 10 minutes and cannot

be changed)

Performs authentication through a programmable authentication

module (PAM). Use cookies to reduce the number of PAM

authentications.

Supported message formats are XML and JSON for

specific commands.

An XML output can be converted to JSON

No direct map from NX-API XML to Cisco NX-OS

NETCONF

Uses HTTP and HTTPS

CLIs are encoded in the POST body of HTTP/HTTPS

NX-API backend uses NGINX HTTP server. This process (and all child

processes) are under CGROUP protection – CPU and memory resource usage

are capped. If exceeded, process reset and restarted.

Message Format

User

Nexus 3000/9000 Switches


Nexus 3000/9000 – NX-APIDeveloper Sandbox

Open browser and

put in the IP address

of the switch, and

enter credentials

Type in the

commands

as needed

Choose message format

and command type

Hit POST !

Magic !! Request

and Response code

are automatically

populated


Programmability – Sample NX-API Script

49BRKDCN-3020

CLI requestedmessage format

Type of command

command success

VRF where route for

the given IP is found

next-hop IP address

In JSON format


Programmability – Sample NX-API Script

50BRKDCN-3020

CLI requestedmessage format

Type of command

VRF where route for

the given IP is found

next-hop IP address

In XML format


Nexus 3000/9000 – NX-API RESTREST – something we all know

HTTP GET

HTMLWeb Server

User

HTTP GET

JSON / XML

Web Browsing REST NX-API

Talks about how data should be presented to the end user

Talks about how data should be presented to applications.

As the name suggests, it is REpresentational State Transfer

Application Server

Nexus Switches


Nexus 3000/9000 – NX-API REST

• In REST, everything is an object

• All elements are accessible –Config, Faults, Events, Operational Data and Statistics.

• Features supported: BGP, VLAN, LACP, ACL, QoS, UDLD, MAC, DHCP, DNS, RBAC, AAA, SVI, NTP and VRRP.

Object Based Programmability

NGINX

Server

REST

Client


Nexus 3000/9000 – NX-API REST

• It operates in forgiving mode – missing attributes are substituted by default values in the internal data management engine (DME)

• It terminates on a single data model – relief from programming and interfacing with individual components.

• It is event-driven – notification generated for an action/event. Customizable.

• It is secure – password-based authentication. Usage of cookies.

REST NX-API Sandbox going to be during Q3CY16

Characteristics


I am full !!

Alright !

Got it !!

me too...

Sure thing !!

I am full !!


Nexus 9000 Programmability

• Create Super Commands – chaining multiple commands by passing interesting data from one to the next, with useful end results. Very helpful in repetitive debug / troubleshooting commands.

• Resource monitoring – resources like TCAM or VLAN usage, interface statistics/errors etc.

• Consistency checker – VLANs, vPC and more

• Configuration backup and rollover.

• Orchestration

Few use-cases of NX-API / REST NX-API

Relevant Session(s):

LTRDCT-1225 Nexus 9000 DevOps & Programmability Options

BRKDCT-1302 Network Programmability and Automation using Nexus 9000


Let’s do it !!

56BRKDCN-3020


Bring data together

…. and ….

Generate Analytics

NX-API

Built-in Tools

& CLIs

Nexus Data Broker and Tools


Tools

Cisco Nexus Data Broker (NDB)High-level Overview

Production Network

Cisco®

SPAN ports

Cisco Nexus Data Broker

3rd-Party Tools

and Applications

Cisco Tools and

Applications

Traffic Filter and Forward

NDB Controller

58

OpenFlow

or NX-API

REST API or HTTP/S

BRKDCN-3020


Cisco Nexus Data Broker Centralized Deployment

Tools TAP and Cisco® SPAN Aggregation Production Network

Custom

Tools

Optical

TAPs

SPAN

Cisco Nexus 3000 or

9000 Series Switches

Central

tapping point

Java and REST

Cisco Nexus

Data Broker

Traffic filtered and forwarded to

one or more

monitoring tools

With Cisco Nexus® Data Broker

3rd-Party Tools

and Appliances

Cisco Tools and

Appliances

59BRKDCN-3020


Production NetworkTools TAP and Cisco® SPAN Aggregation

SPAN and

ERSPAN

Optical

TAPs

Cisco Nexus

Data Broker and

OpenFlow

REST API for

northbound

application

integration

3rd-Party Tools and

Appliances

Traffic filtered and forwarded

to one or more monitoring tools

With Cisco Nexus® Data Broker

Cisco Nexus

3000 Series or 9300

platform switches

Cisco Tools and

Appliances

Custom

Tools

60

Cisco Nexus Data Broker Embedded Mode Deployment

BRKDCN-3020



Configure connections

Manage devices

Configure traffic filters

Define TAP and Cisco® SPAN ports

View monitoring topology

Troubleshoot

What it can do for us ?

AAA/Security functions

RBAC Capabilities

Traffic load-balancing

Traffic mapping Multipoint-to-multipoint (MP2MP)

Any-to-Multipoint (A2MP)

Clustering between controllers

61

Relevant Session:

BRKDCT-1349 Application Traffic Visibility and Analysis with Cisco Nexus Data Broker

BRKDCN-3020


Nexus Data Broker and ToolsCisco Tools and Applications



Cisco

Prime NAM Cisco NAM

Appliance

SourceFire

IDS

Lancope

Netflow

Generation

Appliance…so on..


Nexus Data Broker and ToolsCisco Tools and Applications



Cisco

Prime NAM Cisco NAM

Appliance

SourceFire

IDS

Lancope

Netflow

Generation

Appliance……..


• Wired and Wireless

Access

• Campus Core and

Distribution

• Data Center Core and

Aggregation

• Server Access

• Virtual Machine/Cloud

Consistent Visibility Across

the Network

64BRKDCN-3020

Cisco Prime NAMConsistent Application Visibility

Cisco WAAS

Visibility

Voice Quality

Cisco Prime

NAM


Cisco Prime NAMCharacterize Applications’ Performance

CiscoUnified Fabric

ClientsClient

Network WAN

ResponseRequest

Application

Servers

Cisco Prime

NAM

Traffic Analysis

Packets/Bits

Packets/sec

Bits/sec

Transaction

Analysis

Data Transfer Time

Server Response Time

Network Time

Relevant Session:

BRKNMS-2444 Improve Application Delivery with Cisco AVC in the Data Center and Cloud


Netflow Generation Appliance (NGA)

• NGA introduces a cross-device approach to flow analysis, facilitating hop-by-hop flow visibility across multiple network segments.

• Helps to address following challenges in IT:

• Security

• Billing

• Capacity Planning

• Resource Optimization

• QoS Monitoring/Validation

• Operations

Introduction

Cisco NGA 3140


Netflow Generation Appliance (NGA)Architecture and Use-cases

Production Network

Netflow Collectors • Visibility into the traffic flows before and after implementing network services

• Profiling of server access network traffic

• Visibility of hosted application traffic and performance

• Traffic monitoring and profiling in Storage and VM environment

and many more …

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCN-3020

Nexus Data Broker and Tools3rd-Party Tools and Applications



PlixerCallRex

Riverbed

Corvil

NetScout

Splunk

68


Nexus Data Broker and Tools3rd-Party Tools and Applications



PlixerCallRex

Riverbed

Corvil

NetScout

Splunk



• Corvil provides several products / solutions that transforms network data into useful insights, in real-time, with speed and accuracy. E.g., IT Operations, Big Data, Security, Trading

What I see often in the field … is for IT Operations

3rd-Party Tools and Applications - Corvil



…. IT Operations

3rd-Party Tools and Applications - Corvil



• As we all know, splunk has several products and solutions which helps to collectand analyze the data generated by the networking devices, and also giving insights to drive operational performance and results.

• What I see often in the field … is Splunk for Cisco Networks.

• It supports:

Cisco Catalyst series switches

Cisco Nexus series switches

Cisco ASR, ISR and CRS routing platforms

Cisco IOS-based Metro and Industrial Ethernet devices

Cisco WLC - WLAN Controller

3rd-Party Tools and Applications – Splunk


Nexus Data Broker and Tools3rd-Party Tools and Applications – Splunk


Let’s do it !!

75BRKDCN-3020


Bring data together

…. and ….

Generate Analytics

NX-API


Built-in Tools

& CLIs

Bringing All Together - Analytics


Bringing All Together …

• Lab Topology

• Components and Programmability• NDB – HTTP / REST API

• NAM – REST API

• Nexus 3000/9000 – NX-API

• Script Results

• More Use-cases

Agenda

77


Lab Topology

78BRKDCN-3020

REST API or HTTPREST API

NX-API

Server

Nexus9216

Nexus3172Nexus3172

Nexus3172

Nexus9216

Production Network

Nexus3000 Nexus3000

Nexus9000

NDB Controller

OpenFlow or NX-API

Nexus Data Broker

Cisco Prime NAM 2320

Tools

replicated traffic

Edge

Port

SPAN capture points

SPAN capture points

Delivery

Port


Bringing All Together - AnalyticsREST API / HTTP with NDB Controller

OpenFlow or NX-API

REST API or HTTP

HTTP to NDB Controller – building the network ...

API guide is embedded in the product, For APIs, click on this icon


Bringing All Together - AnalyticsREST API embedded with NDB Controller



1. Identify the switches to configure

2. Categorize the ports:

• Monitoring Device

• Edge Port – SPAN

• Edge Port – TAP

• Production Port

3. Assign VLAN to identify the traffic

4. Configure monitoring device – Device Name, Switch/Port connected to, Icon and Block Rx.

5. Add traffic filters and associate to flows

NDB Configuration Steps



(1) Perform HTTP GET, and get Domain, nonce (random number used for security), pkey (NAM’s public key) and Session ID.

(2) Hash the password locally using SHA1

(3) Encode the hashed password with MD5, nonce, Domain and Username

(4) Perform HTTP to authenticate the session – send Session ID, Username, Encoded Password and pkey.

(5) Once successfully authenticated, subsequent requests to NAM should include the HTTP Cookie - to avoid repeated auth.

(6) Apply API to get the data – using XML and CVS Data Query (SQL format)

REST API with NAM Appliance – Using Python


Bringing All Together - AnalyticsREST API with NAM Appliance – Using Python

payload = {'api': 'true'}

r = requests.get('http://' + ip_address + '/auth/login.php', params=payload)

(1) Do HTTP GET to access API

salt = “04581273”

password_hash_string = salt + username + password

sha1_hash_object = hashlib.sha1(password_hash_string)

password_hash = sha1_hash_object.hexdigest()

(2) Hash password using SHA1

md5_hash = domain + nonce + username + password_hash

md5_hash_object = hashlib.md5(md5_hash)

encoded_pw = md5_hash_object.hexdigest()

(3) Encode the hashed password

payload = {'sessid': sessid, 'username': username, 'pwdigest': encoded_pw, 'pkey': pkey}

r = requests.get('http://' + ip_address + '/auth/authenticate.php', params=payload)

(4) Authenticate the session …


Bringing All Together - AnalyticsREST API with NAM Appliance – Using Python

nam_ip = "10.122.140.122"

sessid = auth(nam_ip, "ciscoweb", "ciscoweb")

sessid_cookie = dict(PHPSESSID=sessid)

(5) Session cookie for subsequent communication

cur_time = calendar.timegm(time.gmtime())

start_time = cur_time - 900 #15 minutes in seconds

xml_start = "<query-data>\n\t<query>\n\t\t”

query = "SELECT host, SUM(inOctets), SUM(outOctets), SUM(inOctets)+SUM(outOctets)\n\t\tFROM Hosts\n\t\tWHERE TIME >= " + str(start_time) + " AND TIME <= " + str(cur_time) +

"\n\t\tGROUP BY host\n\t\tORDER BY SUM(inOctets)+SUM(outOctets) DESC\n\t\tLIMIT 2, 1”

xml_end = "\n\t</query>\n</query-data>”

xml = xml_start + query + xml_end

r = requests.post('http://' + nam_ip + '/nbi/nbi-csvquery', data=xml, cookies=sessid_cookie)

(6) Leverage API, using XML and CVS Data Query

CVS Data Query API to get TWO top-talkers based on the send/receive traffic in last 15 minutes.

HTTP POST with XML Query. Returns 2 IP addresses.

“auth” performing 4 steps mentioned earlier



Requirements:• List of management IP address of all the switches.

• CDP is enabled on all the switches and working.

• Same credentials valid across all the devices

Algorithm:a. Access the switch and check “sh ip route <top-talker>”

b. If it is not “directly attached” prefix, find the next-hop IP and interface from the results, and go to step (e).

c. If it is “directly attached”, then do “show ip arp” and “show mac address-table …” to find the physical interface. If CDP neighbor on this interface returns empty, then the top-talker should be on this port.

d. If CDP is non-empty, then access neighboring switch and track the mac-address to a physical port. Repeat until host port is identified. Exit.

e. Do “show cdp neighbor …” and find the neighbor’s IP address from CDP details. Go to step (a).

NX-API with Nexus 3000/9000


Bringing All Together - AnalyticsNX-API with Nexus 3000/9000

def get_output(command, ip_address):

nxapi = NXAPI()

nxapi.set_target_url("http://" + ip_address + "/ins")

nxapi.set_username("admin")

nxapi.set_password("cisco!123")

nxapi.set_msg_type("cli_show")

nxapi.set_out_format("json")

nxapi.set_cmd(command)

headers, resp = nxapi.send_req()

resp_obj = json.loads(resp)

if resp_obj["ins_api"]["outputs"]["output"]["code"] == "400":

print "ERROR: Error while parsing cli request.”

return

else:

return(resp_obj["ins_api"]["outputs"]["output”]["body"])

send show commands and get response …


Bringing All Together - AnalyticsNX-API with Nexus 3000/9000

def show_cdp_nei(interface, ip_address):

resp_obj = get_output("show cdp nei int " + interface + " detail", ip_address)["TABLE_cdp_neighbor_detail_info"]["ROW_cdp_neighbor_detail_info"]

# Returns an arrray with one neighbor:

# [ Local Interface, Remote Switch Name, Remote Platform, Remote Interface, Remote MGMT Address ]

return [resp_obj["intf_id"].encode('utf8'), resp_obj["device_id"].encode('utf8'), resp_obj["platform_id"].encode('utf8'), resp_obj["port_id"].encode('utf8'), resp_obj["v4mgmtaddr"].encode('utf8')]

Finding CDP neighbor on a given switch / interface …

Checking

IP routes

def show_ip_route_vrf(route, vrf, switch_mgmt_ip_address):

routes = []

resp_obj = get_output("show ip route " + route + " vrf " + vrf, switch_mgmt_ip_address)

<snip>

new_resp_obj = resp_obj["TABLE_vrf"]["ROW_vrf"]["TABLE_addrf"]["ROW_addrf"]["TABLE_prefix"]["ROW_prefix”]


Bringing All Together - AnalyticsWe did it !!

lansw@davola:~/yramdoss/CLUS2016$ python find_top_talkers.py

REST API to NAM Appliance …

Top Talkers (based on total In and Out Octets):

172.16.22.2

172.16.12.7

NX-API to Nexus Switches …

Finding where host 172.16.22.2 lives.

Host is in Vlan 22, has MAC address 0010.9400.0005, and lives off of port Ethernet1/2 on switch N3K-C3172PQ-10GE-29-15.

Finding where host 172.16.12.7 lives.

Host is in Vlan 12, has MAC address 0010.9400.0002, and lives off of port Ethernet1/1 on switch N3K-C3172PQ-10GE-29-16.


We did it !!

89BRKDCN-3020


Bring data together

…. and ….

Generate Analytics

Built-in Tools

& CLIs

NX-API


Python


Analytics – We Can Do More

(1) With REST, verify if end-to-end network latency is above specific threshold.

(2) Leverage NX-API to get interface statistics from the switches/routers - check if there are drops.

(3) Check what changed in the traffic path – less/more L2 or L3 links ? physical layer issues ? How about MTU on all the potential paths ?

90BRKDCN-3020

Use-case #1 - Intermittent application slowness



91BRKDCN-3020

Server Farm #1

LAN #1

LAN #2

S11 S21 S31

S12 S22 S32

A3

A2

A1

REST API

end-to-end latency is higher than threshold

Server

Use-case #1 - Intermittent application slowness (Contd.)

ERSPAN



92BRKDCN-3020

Server Farm #1

LAN #1

LAN #2

S11 S21 S31

S12 S22 S32

A3

A2

A1

NX-API

Drops detected in S31 and S32, on ports connected to LAN


Server



93BRKDCN-3020

Server Farm #1

LAN #1

LAN #2

S11 S21 S31

S12 S22 S32

A3

A2

A1

Lower MTU on newly-added L3 links




(1) Via REST API, from NAM, obtain statistics based on applications. NAM implements application classification system and uses “Application Tag” to uniquely identify applications.

(2) Find top-talkers by IP address(es) using the traffic statistics.

(3) Leverage NX-API to find the location (Switch, Interface, VLAN) of the user(s) using IP/MAC addresses.

94BRKDCN-3020

Use-case #2 – Finding top-talker(s) by traffic profiling…

Data Transfer

11%

31%

18%

9%

11%

10%

7%3%

SAP

Custom App

Bit Torrent

Social Media

Unified Communication

Monitoring & Operations Applications

Others

Summary


Summary

Built-in Tools

Latency Monitoring


Micro-burst Monitoring

NX-API Usage

Capabilities

Components

Commands and Usage

Sandbox and Sample Scripts

REST NX-API

BRKDCN-3020

Nexus Data Broker

Overview and Deployment

Capabilities

Cisco Prime NAM and NGA

3rd-Party Tools: Corvil, Splunk

Bringing All Together

HTTPS/REST API to NDB

REST API to NAM Appliance

NX-API to Nexus 3000/9000

96



Cisco Open Architecture &

Programmability

ANALYTICS

97


Take Aways …

The switching platforms have lots of tools that are developed keeping ALL OF YOU in mind.

BRKDCN-3020

They are rich with several programmability options, and all are very easy to use. IT IS OPEN !!

Cisco’s products / solutions enable and empower EACH ONE OF YOU to integrate them with your day-to-day operations and generate analytics.

98


References

• Nexus 9000 Programmability Guides

• Cisco Prime NAM REST API Guide 6.1(1)

• Cisco Nexus Data Broker – Data sheets and literature

• IEEE 1588 PTP and Analytics on the Cisco Nexus 3548 Switch

• Cisco Prime NAM2300 Series Appliances Installation and Configuration Guide

• Latency Monitoring on Cisco Nexus Switches: Troubleshoot Network Latency

• Nexus 9000 GitHub Repository

• Cisco NX-API REST Interface

BRKDCN-3020 99

http://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/products-programming-reference-guides-list.html

http://www.cisco.com/c/dam/en/us/td/docs/net_mgmt/network_analysis_module_software/6-1/developer/guide/nam-rest-api-guide-12.pdf

http://www.cisco.com/c/en/us/products/cloud-systems-management/nexus-data-broker/index.html

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-3000-series-switches/white-paper-c11-731501.html#_Toc386062607

http://www.cisco.com/c/en/us/td/docs/net_mgmt/network_analysis_module_appliance/2300/installation/guide/2300-series-install-config/adeovr.html

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white-paper-c11-733025.html

https://github.com/datacenter/nexus9000

https://opennxos.cisco.com/static/nexus/getting_started/programmability/chapters/cisco_nx-api_rest_interface1898.xhtml


Relevant Sessions….

• LTRDCT-1225 Nexus 9000 DevOps & Programmability Options

• BRKDCT-1302 Network Programmability and Automation using Nexus 9000

• BRKDCT-1349 Application Traffic Visibility and Analysis with Cisco Nexus Data Broker

• BRKDCT-1890 Network Visibility using Advanced Analytics in Nexus Switches

• BRKNMS-2444 Improve Application Delivery with Cisco AVC in the Data Center and Cloud

• BRKDCT-2459 Programmability and Automation on Cisco Nexus Platforms

• BRKDCT-3101 Nexus9000 (Standalone) Architecture and Troubleshooting

• BRKARC-2011 Overview of Packet Capturing Tools in Cisco Switches and Routers

100BRKDCN-3020


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

101BRKDCN-3020

CiscoLive.com/Online

CiscoLive.com/us

http://ciscolive.com/Online

http://ciscolive.com/Online


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

102BRKDCN-3020

Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com

Thank you


R&S Related Cisco Education OfferingsCourse Description Cisco Certification

CCIE R&S Advanced Workshops (CIERS-1 &

CIERS-2) plus

Self Assessments, Workbooks & Labs

Expert level trainings including: instructor led workshops, self

assessments, practice labs and CCIE Lab Builder to prepare candidates

for the CCIE R&S practical exam.

CCIE® Routing & Switching

• Implementing Cisco IP Routing v2.0

• Implementing Cisco IP Switched

Networks V2.0

• Troubleshooting and Maintaining

Cisco IP Networks v2.0

Professional level instructor led trainings to prepare candidates for the

CCNP R&S exams (ROUTE, SWITCH and TSHOOT). Also available in

self study eLearning formats with Cisco Learning Labs.

CCNP® Routing & Switching

Interconnecting Cisco Networking Devices:

Part 2 (or combined)

Configure, implement and troubleshoot local and wide-area IPv4 and IPv6

networks. Also available in self study eLearning format with Cisco Learning

Lab.

CCNA® Routing & Switching

Interconnecting Cisco Networking Devices:

Part 1

Installation, configuration, and basic support of a branch network. Also

available in self study eLearning format with Cisco Learning Lab.

CCENT® Routing & Switching

106

For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact [email protected]

http://learningnetwork.cisco.com/

mailto:[email protected]


Data Center / Virtualization Cisco Education OfferingsCourse Description Cisco Certification

Introducing Cisco Data Center Networking (DCICN);

Introducing Cisco Data Center Technologies (DCICT)

Learn basic data center technologies and skills to build a

data center infrastructure.

CCNA® Data Center

Implementing Cisco Data Center Unified Fabric (DCUFI);

Implementing Cisco Data Center Unified Computing (DCUCI)

Designing Cisco Data Center Unified Computing (DCUDC)

Designing Cisco Data Center Unified Fabric (DCUFD)

Troubleshooting Cisco Data Center Unified Computing

(DCUCT)

Troubleshooting Cisco Data Center Unified Fabric (DCUFT)

Obtain professional level skills to design, configure,

implement, troubleshoot data center network infrastructure.

CCNP® Data Center

Product Training Portfolio: DCNMM, DCAC9K, DCINX9K,

DCMDS, DCUCS, DCNX1K, DCNX5K, DCNX7K

Gain hands-on skills using Cisco solutions to configure,

deploy, manage and troubleshoot unified computing, policy-

driven and virtualized data center network infrastructure.

Designing the FlexPod® Solution (FPDESIGN);

Implementing and Administering the FlexPod® Solution

(FPIMPADM)

Learn how to design, implement and administer FlexPod

solutions

Cisco and NetApp Certified

FlexPod® Specialist

107






Network Programmability Cisco Education OfferingsCourse Description Cisco Certification

Integrating Business Applications with Network

Programmability (NIPBA);

Integrating Business Applications with Network

Programmability for Cisco ACI (NPIBAACI)

Learn networking concepts, and how to deploy and troubleshoot

programmable network architectures with these self-paced courses.

Cisco Business Application

Engineer Specialist Certification

Developing with Cisco Network Programmability

(NPDEV);

Developing with Cisco Network Programmability

for Cisco ACI (NPDEVACI)

Learn how to build applications for network environments and effectively

bridge the gap between IT professionals and software developers.

Cisco Network Programmability

Developer Specialist Certification

Designing with Cisco Network Programmability

(NPDES);

Designing with Cisco Network Programmability

for Cisco ACI (NPDESACI)

Learn how to expand your skill set from traditional IT infrastructure to

application integration through programmability.


Design Specialist Certification

Implementing Cisco Network Programmability

(NPENG);

Implementing Cisco Network Programmability

for Cisco ACI (NPENGACI)

Learn how to implement and troubleshoot open IT infrastructure

technologies.


Engineer Specialist Certification



108



Network Analytics using Nexus 3000/9000...

Documents

Transcript of Network Analytics using Nexus 3000/9000...