NSX-T Deep Dive: Kubernetes Networkingadmin@k8s -master:~$ kubectl run nginx foo --image=nginx n foo...

CNET1270BU

NSX-T Deep Dive: Kubernetes Networking

Yasen Simeonov, VMware, Inc.

#vmworld #CNET1270BU

Disclaimer

©2019 VMware, Inc.

This presentation may contain product features or functionality that are current

under development.

This overview of new technology represents no commitment from VMware to d

these features in any generally available product.

Features are subject to change, and must not be included in contracts, purcha

or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed

or presented, have not been determined.

The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment

to deliver any items presented herein.

Public

Users

VMs, Containers, Microservices

VMware partners (VMC)

Private Data Centers

TelcoNetwork

Private Cloud(VCF)

VCN

10kcustomers

to date

82%of Fortune 100

enterprises

70%of all Fortune

global 500 telcos

Gartner

MQ Leader

WAN Edge

Infrastructure

Ties it all together.

©2019 VMware, Inc.

Virtual Cloud Networking

ResourcesHow to get started

©2019 VMware, Inc.

Design Guides

Demos

Take a Hands-on Lab Join VMUG, VM

Communities (VM

LEARN

VMware.com/go/NSXtechzone

CONNECTTRY

Agenda

©2019 VMware, Inc.

Kubernetes Overview

NSX-T Integration with Kubernetes

Demo: The new topology

Summary

Kubernetes Overview

©2019 VMware, Inc.

Networking focused

Kubernetes Components

K8s Cluster Con

Master(s) and N

K8s Master Com

• API Server

• Scheduler

• Controller M

• Dashboard

K8s Node Com

• Kubelet

• Kube-Proxy

• Containers R

K8s masterK8s master

K8s

Master

Controller

Manager

K8s API

Server

Key-Value

Store

dashboard

Scheduler

K8s nodeK8s node

K8s nodeK8s node

K8s Nodes

kubelet c runtime

Kube-proxy

> _Kubectl

CLI

K8s Master(s)

©2019 VMware, Inc.

Kubernetes Namespace

Namespaces ar

divide cluster re

amongst users

They can be tho

Tenants

They are a way

Resources Quo

Networking Mu

and Name uniq

Namespace: fooBase URI: /api/v1/namespaces/foo

‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master

‚redis‘ service:/api/v1/namespaces/foo/services/redis-master

©2019 VMware, Inc.

Namespace: barBase URI: /api/v1/namespaces/bar

‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master

‚redis‘ service:/api/v1/namespaces/bar/services/redis-master

Kubernetes Pod

A Pod is a grou

more container

an IP address a

VolumePod

10.24.0.0/16

10.24.0.2

pause container (‘owns’ the IP stack)

nginx tcp/80

mgmt tcp/22

loggingudp/514

IPC

External IP Traffic

©2019 VMware, Inc.

K8s

MasterReplication Controller:

• The replication controller en

'desired' state of a collection

makes sure that 4 Pods area

in the cluster

Replica Set:

• Replica Set is the next-gene

Replication Controller. Repli

Based selectors while replica

controllers use Equity-Based

Kubernetes RC & RS

Kubernetes Replication Controller (rc) and Replica Set (rs

Replication Controller

/ Replica Set

Pods

©2019 VMware, Inc.

c

Daemon Sets:

• A DaemonSet ensures that a

nodes run a copy of a Pod.

• As nodes are added to the c

added to them.

• As nodes are removed from

those Pods are garbage coll

• Deleting a Daemon Set will

pods it created

• Daemon Sets are used to re

Units in a lot of cases today

Kubernetes Daemon Set

Kubernetes Daemon Set

K8s

Node

InfraPod

K8s

Node

InfraPod

K8s

Node

InfraPod

K8s

Node

InfraPod

K8s

Master

Deamon Set

©2019 VMware, Inc.

Kubernetes Service

A Kubernetes S

a logical set of

selected with m

Serves multiple

• Service Disc

• East/West lo

the Cluster (T

• External load

L4 TCP/UDPLoadBalancer)

• External acc

service throu

IPs (Type: Node

Redis Slave

Pods

redis-slave svc

10.24.0.5

ClusterIP

Web Front-End

Pods

10.24.2.7

▶kubectl describe svc redis-slave

Name: redis-slaveNamespace: default

Labels: name=redis-slave

Selector: name=redis-slave

Type: LoadBalancer

IP: 172.30.0.24

LoadBalancer Ingress: 134.247.200.20

Port: <unnamed> 6379/TCPEndpoints: 10.24.0.5:6379,

10.24.2.7:6379

DNS:

redis-slave.<ns>.cluster.local ➔172.30.0.24

ExternalIP

134.247.200.20 172.30.0.24

DNS:

redis-slave.external.com ➔ 134.247.200.20

©2019 VMware, Inc.

Kubernetes Ingress

A Kubernetes I

is a L7 LoadBal

that binds a hos

url to aService

The LoadBalan

can be impleme

external Load B

a K8s Pod

Web Front-End

Pods (shop svc)Web Front-End

Pods (special-offers svc)

http://www.bikeshop.com/shop

http://www.bikeshop.com/special-offers

LoadBalancer Datapath

(External or K8s Pods)

▶kubectl describe ingress bikeshop-ingress-shop

©2019 VMware, Inc.

Name: Namespace:

bikeshop-shop bikeshop100.64.240.9,134.247.200.1Address:

Default backend: default-http-backend:80 (<none>)

Backends--------

Rules:Host----www.bikeshop.com

Path----/shop

web-svc-1:80 (<none>)

External IP: 134.247.200.1

DNS: *.bikeshop.com ➔134.247.200.1

http://www.bikeshop.com/shop

http://www.bikeshop.com/special-offers

http://www.bikeshop.com/

NSX-T Data Center & KubernetesHow NSX-T does Kubernetes Networking

©2019 VMware, Inc.

Key Design Goals of the NSX-T Data Center Kubernetes

Don't stand in the

way of the developer

!

Provide solutions to

map the Kubernetes

constructs to

enterprise

networking

constructs

Secure Containers,

VMs and any other

endpoints with

overarching Firewall

Policies

Provide

troubl

tools to

containe

in the

©2019 VMware, Inc.

NSX-T K8s Integration – Namespaces & PodsDynamic per Namespace Topology

admin@k8s-master:~$ kubectl create namespace foo namespace ”foo" created

admin@k8s-master:~$ kubectl create namespacebar

namespace ”bar" created

admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foo deployment "nginx-foo" created

admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bar

deployment "nginx-bar" created

NSX / K8s topology

Namespace: foo

10.24.0.0/24 10.24.1.0/24

10.24

NAT

boundary

K8s nodesK8s Masters

©2019 VMware, Inc.

NSX-T K8s Integration – Routed Namespaces

admin@k8s-master:~$ vim no-nat-namespace.yaml

apiVersion: v1kind: Namespacemetadata:

name: no-nat-namespace annotations:

ncp/no_snat: "true“

admin@k8s-master:~$ kubectl create –fno-nat-namespace.yamlnamespace ”no-nat-namespace"created

admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx –nno-nat-namespacedeployment "nginx-k8s" created

Namespace: no-nat-namespace

NSX / K8s topology

114.4.10.0/26

Direct Routing

114.4.10.6


©2019 VMware, Inc.

NSX-T K8s Integration – Shared T1TopologyShared T1 for all Namespaces in the Cluster - Both for Policy and MP

NSX / K8s topology

10.24.0.0/24 10.24.2.0


T1

admin@k8s-master:~$ kubectl create namespace foo namespace ”foo" created

admin@k8s-master:~$ kubectl create namespace bar namespace ”bar" created

admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foo deployment "nginx-foo" created

admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created

AA

NAT boundary

N

©2019 VMware, Inc.

Policy SupportSimplified UI

©2019 VMware, Inc.

NCP is a softwa

component pro

VMware in form

container imag

run as a K8s Po

NCP is build in

way, so that ind

adapters can b

different CaaS a

systems at som

K8s / NSX-T Data Center ComponentsNSX Container Plugin (NCP)

NC

M

Infra

K8s / OS

Adapter

CloudFoundry

Adapter

NSX Container Plugin

More…

NSX

Manager

API

Client

NSX

Manager

NS: fooNS: bar

NSX/ K8s topology

K8s master

etcd

API-

Server

Scheduler

©2019 VMware, Inc.

Node

VM

DFW

eth2

Node

VM

DFW

eth0

Minion Mgmt.

IP Stack

eth0

Minion Mgmt.

IP Stack

mgmt network

OVS

mgmt network

Vla

n10

vla

n11

Sub VIF

eth2

vla

n10

vla

n11

OVS

NSX CNI

Plugin & Node Agent

Pods

PodsNSX CNI

Plugin & NodeAgent

Sub VIF

Sub VIF Sub VIF

K8s Node VMs: Most cust

©2019 VMware, Inc.

looking to deploy K8sNo

today

Nested Network-Virtualiz

terminating the overlay tu

Node VM, we areextendi

Hypervisor vSwitch into t

using VLAN tagging. The

vSwitch (OVS) is ‘standal

gets programed by the N

Benefits:

• Enhanced security thro

isolation of the Node V

Control-Plane

• Less transport-nodes in

equates to higher scale

Container Interfaces

NSX-T Sub-VIF Interfaces

Kubelet: calls the NSX CNIplugin.

NSX CNI Plugin: is a simple python script th

translates between Kubelet and NSX Node

simple private protocol to talk to the Node

socket

NSX Node Agent: runs as a DeamonSet in

Mode and is responsible for:

1. The retrieval of IP/MAC/VLAN informatio

next 2 slides) through an NSX-RPC TCP c

Hypervisor

2. The creation and configuration of the upli

the Hypervisor vSwitch inOVS

3. The creation and configuration of downlin

connecting OVS to the Pods 'pause conta

NSX Kube Proxy: Is responsible for the crea

and Load Balancing rules in OVS to do the

East/West LB (will be covered later again).

responsible to create the needed flow/NAT

for the Node-Agent to Hyperbus communi

Node Agent &CNI Plugin

Node Agent & CNI Plugin

eth0

Minion Mgmt.

IP Stack

eth2

vla

n10

vla

n11

OVS

Pods

Hostnetwork-Mode

Node Agent & NSX Kube-Proxy

DeamonSet

NSX CNI Pluginkubelet

©2019 VMware, Inc.

©2019 VMware, Inc.

Pod attachment workflow

NC

M

Infra

K8s / OS

Adapter

CloudFoundr

y Adapter

Libnetwork

Adapter


More…

NSX

Manager

API

Client

NSX

Manager

K8s master

etcd

API-

Server

Schedule

r

NS: foo NS: bar

NSX/ K8s topology

1)2)

3)

4)

1. NCP creates a ‚watch‘ on any Pod events

2. A user creates a new K8s

3. The K8s API Server notifiethe change (addition) of P

4. NCP creates a logical port

a) Requests an IP from theSubnet

b) Request a MAC from the pool in NSX

c) Assigns a VLAN for theP d) Creates a logical port (Su

the Namespace LS and aMAC and VLAN to the lo

e) Adds all K8s Pod Labels logical port

Shared under NDA

K8s / NSX Workflows (1/2)Pod attachment workflow

©2019 VMware, Inc.

K8s / NSX Workflows (2/2)Pod attachment workflow Pod attachment workflow

Hypervisor

(ESXi &

KVM)

Node

VM

Vla

n4

09

4

Vla

n2

cif

kubelet

NSX

Manager

NSX

Controllers

NSX LCP

5)

6)

NSX

Hyperbus

Node Agent /

CNI Plugin

7)

8)

9)

5. NSX LCP will create the LPon

6. Hyperbus monitors LCP for neinterfaces and learns the CIF’s Id/IP/MAC/VLAN binding

7. Kubelet sees a new ‘PodSpec’Master and starts a new Pod. Ithe NSX cni plugin binary to d ‘network wiring’ of the Pod –proxy'd to the NSX Node Age

8. The NSX Node Agent gets theId/IP/MAC/VLAN binding dataHyperbus over the isolated anchannel (one-way connection e

9. The Node Agent creates theOright VLAN, and configures theinterface to connect to OVS witIP/MAC. After this, Kubelet isuthe Pod creation succeeds

Tenancy / Topology MappingPersistent IPs for K8s Namespaces

With NSX-T each Tenant (Kubernetes Namespace)either gets its own SNAT IP (NATMode), or is directly identifiable by itssource subnet (No NAT Mode)

Node VM

OpenvSwitch

10.12.5.5/2410.12.1.8/24

172.16.1.11/24

mgmt IP

vnic

PAS VMsT1router

VLAN Trunk

NSX-T Logical Switch

172.16.1.1/24

Namesp. T1router

10.12.1.1/24

Pods

Database

(VM based or Physical)

Physical DC Firewall

A new Son the TTenant f

Tenant: fTenant: bar

In NoA-TNMAoTdMeo,tdhee, ethxeteerxntaelrDnaCl FDiCreFwiarellwall aanndd tthhee DDBBccaann ddiissttiinngguuiisshh tteennaanntt ''ffoooo'' aanndd tteennaanntt ''bbaarr'' uussiinngg tthhee ssoouurrccee ISPNAT IP SthuabtnisetatllhoactaitseadlltoocatsepdetcoifaicsTpeencaifnict.Tenant.

©2019 VMware, Inc.

Firewall rules in exis

Firewalls to allow tr

workloads in K8s

The K8s user / Dev

applications that are

identifiable in the ph

With this feature as

Workloads (Pods) c

use a specific IP or

to source their traffi

Feature

Benefits

Infrastructure Team

Persistent SNAT IP per K8s ServiceSpecifying the source IP Kubernetes Workloads using the K8s service

Tier0 LR

Corporate network

DB

allow – from: 134.247.100.10 (App)

to: 134.247.200.9 (DB)

Tier1 LR

Kubernetes Namespace:

Foo

Web-Frontend

Pods

App Logic

Pods

K8s Svc for AppK8s Svc for Web

Namespace LS(s)

SNAT App Svc Pods to:

134.247.100.10For all other Pods

use namespace SNAT IP

©2019 VMware, Inc.

Kubernetes Metadata / NSX Logical Port Mapping

▶kubectl get pod nsx-demo-rc-c7x65 -o yaml

apiVersion: v1

kind: Pod

metadata:creationTimestamp: 2018-07-25T12:05:56Z

generateName: nsx-demo-rc-

labels:

app: nsx-demo

name: nsx-demo-rc-c7x65

namespace: nsx-ujo

Metadata within Kubernetes like Namespace, Pod names, Labels allge copied to the NSX Logical Port as Por

©2019 VMware, Inc.

NSX can be configured to collect ports and switches in dynamic security group

Tags (Kubernetes Metadata) and apply Firewall rules on them

Pre-Created Security Groups / Firewall rules (admin rule

Match on Port Tags

MaPoof t

GroFiresrc

©2019 VMware, Inc.

Support of Kubernetes Network Policy---

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: ps-ing-to-fe

spec:

podSelector:

matchLabels:

app: planespotter-frontend

policyTypes:

-Ingress

ingress:

- from:

- ipBlock:

cidr: 100.64.0.0/16

ports:

- protocol: TCP

port: 80

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: ps-fe-to-app

spec:

podSelector:

matchLabels:

app: planespotter-app

policyTypes:

- Ingress

ingress:

- from:

- podSelector:

matchLabels:


ports:

- protocol: TCP

port: 80©2019 VMware, Inc.

©2019 VMware, Inc.

Policy Support – Security per Category

CONFIDENTIA

Environ

Health-checks Admin Rules

Applica

Kubernetes Network Default rule:1.Allow Cluster2. Allow Namespace

3. None

Built-in Load BalancingBuilt-in support for Ingress (L7 HTTP/HTTPS) and Svc Type LB (L4 TCP/UDP) i

K8s integration. Most other K8s networking choice don't support Svc Type LB

you need an additional technology like NGINX from Ingress (L7).

NCM

Infra

K8s / OS

Adapter

CloudFoundry

Adapter

Libnetwork

Adapter


More…

NSX

Manager

API

Client

NSX

Manager

K8s master

etcd

API-

Server

Scheduler

Virtual Server10.114.209.209HTTP and/or

HTTPStraffic

Server Pool 1

Server Pool 2Rule 2/bar/

Rule 1/ f oo/

LB Service

NCM

Infra

K8s / OS

Adapter

CloudFoundry

Adapter

Libnetwork

Adapter

NSX Containe

More…

NSX

Manager

K8s master

etcd

API-

Server

Scheduler

Virtual Server10.114.209.212TCP and/or

UDP traffic

Server Pool

LB Service

©2019 VMware, Inc.

---

apiVersion: v1

kind: Service

metadata:

name: planespotter-frontend

labels:


spec:

loadBalancerIP: 78.11.24.19

type: LoadBalancer

ports:

# the port that this service should serve on

- port: 80

selector:


[root@master1 ~]# oc describe service/planespotter-frontend

Name:

Namespace:

Labels:

Annotations:

planespotter-frontend

default

app=planespotter-frontend

kubectl.kubernetes.io/ last-applied-

nnotations":{},"labels":{"app":"planespotter-configuration={"apiVersion":"v1","kind" :"Service" ,"metadata" :{"a

frontend"},"name":"planespotter-frontend","namespace":"d...

ncp/internal_ip_for_policy=100.64.64.1

Selector:

Type:

IP:

IP:

app=planespotter-frontend

LoadBalancer

172.30.253.255

78.11.24.19

LoadBalancer Ingress: 78.11.24.19, 100.64.64.1

Port:

TargetPort:

NodePort:

Endpoints:

Session Affinity:

<unset> 80/TCP

80/TCP

<unset> 32688/TCP

10.4.3.4:80,10.4.3.6:80

None

External Traffic Policy: Cluster

Events: <none>

master1 ]#

Persistent IP for Service of type LB

K8s / NSXWorkflows

©2019 VMware, Inc.

Installation ImprovementContainerize OVS and NCP Bootstrap initContainer

nsx-ncp-bootstrap-xkfgm 1/1 Running 0 131m 10.114.209.215 node2 <none> <none>

nsx-ncp-bootstrap-zrnq5 1/1 Running 0 131m 10.114.209.212 master2 <none> <none>

nsx-node-agent-4zfrj 3/3 Running 0 131m 10.114.209.212 master2 <none> <none>

nsx-node-agent-7gr6t 3/3 Running 0 131m 10.114.209.214 node1 <none> <none>

nsx-node-agent-g25v5 3/3 Running 0 131m 10.114.209.213 master3 <none> <none>

nsx-node-agent-n2z4p 3/3 Running 0 131m 10.114.209.211 master1 <none> <none>

nsx-node-agent-z5q87 3/3 Running 0 131m 10.114.209.215 node2 <none> <none>

root@master1:~#

root@master1:~# kubectl get pods -n nsx-system -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS

GATESnsx-ncp-bcf5c8778-q67wg 1/1 Running 0 120m 10.114.209.215 node2 <none> <none>

Bootst

initContainer installs:1. Installs/Upgrades t 2.Loads the ncp-app3. Installs/Upgrades/

nsx-ncp-bootstrap-4xq5f 1/1 Running 0 131m 10.114.209.214 node1 <none> <none> OVS kernel module ifnsx-ncp-bootstrap-grdqs nsx-ncp-bootstrap-pmhcx

1/1

1/1

Running Running

0

0

131m

131m

10.114.209.211

10.114.209.213

master1 master3

<none>

<none>

<none>

<none>

4. Stops OVS user sprunning on the host m

NSX node

Containers:1. nsx-node-age2. nsx-kube-pro

3. nsx-ovs

©2019 VMware, Inc.

Hipster Shop

©2019 VMware, Inc.

Cloud-Native Microservices Demo Application

https://github.com/GoogleCloudPlatform/microservices-demo

Why NSX for Kubernetes?

Muti-tenancy Load Balancing and

services

Secure Containers,

VMs and any other

endpoints with

overarching Firewall

Policies

Provide

troubl

tools to

containe

in the

©2019 VMware, Inc.

Join the NSX VMUG Community

vmug.com/nsx

Connect with yourPeers

communities.vmware.com

Embrace the NSXMindset

nsxmindset.com

Find NSX Resources

vmware.com/products/nsx

Read the Network Virtualization Blog

blogs.vmware.com/networkvirtualization

©2019 VMware, Inc.

Where to Get Started

Showcases, breakouts, quick talks & gro

discussions

Visit the VMwareBooth

Product overviews, use-case demos

Visit Technical Partner Booths

Integration demos – Infrastructure,secuoperations, visibility, and more

Meet the Experts

Join our experts in an intimate roundtab

Free Hands-on Labs

Test drive NSX with expert-led or self-paces hands-on

labs

labs.hol.vmware.com

VMware Education - Training and Certific

vmware.com/go/nsxtraining

Free NSX Training on Coursera

vmware.com/go/coursera

Engage and Learn Experience

Attend the Networking and Security Sess

Try Take

NSX-T Deep Dive: Kubernetes Networkingadmin@k8s -master:~$ kubectl run nginx foo --image=nginx n foo...

Documents

Transcript of NSX-T Deep Dive: Kubernetes Networkingadmin@k8s -master:~$ kubectl run nginx foo --image=nginx n foo...