Wolfram Richter | Red Hat

OpenShift Container Netzwerkaus Sicht der Workload

Why this session?

“OpenShift is great for web applications,

but we want to do X… will this work?“

X ∈ {“Analytics”, “Non-HTTP”, “High-

Performance Computing”, “Big Data”,

“Object storage”, “NAS”, “Replicated

Databases”, …}

Let’s take a look from a networking PoV!

Agenda

• What is OpenShift?

• How plain Docker Networking works

and what OpenShift does differently

• Container Networking across nodes

• Kubernetes Services

• Ingress: OpenShift Router

• Egress Pods and Network Policy

OpenShift

● Platform & Container as a Service

● Built for both traditional and cloud-native applications

● An integrated hybrid cloud application platform for application development and deployment

● Develop, build, and manage container based applications

● Easily turn source code into running applications with source-to-image capabilities

OpenShift High-Level Architecture

Container Networking

Problem Statements

• As an X,

I want my

containerized

applications

to be able to

connect to other

services, so that

they can perform

meaningful work.

• As an X,

I want my

containerized

applications

to be accessible

externally,

so that a

wide range of users

can use them.

RFC1918 IP

assigned by

docker daemon

Docker host

Docker bridge

Host NIC

Container

So we can use the

“ip” command

Container

Indicates which IF

the veth device is

connected to

Docker host

Endpoint of the

container’s veth

device

Source IP

appears to be

Node IP

Docker host

Inbound traffic is

forwarded to

container

Outbound traffic is

masqueraded

Container Networking

Problem Statement

• As an X, I want to use network attached

storage from within my container, so I

can provide stateful services (*).

• (*) and storage traffic shouldn’t share application network bandwith

Container Networking

Problem Statement

• As a X, I want name resolution to work

inside the container like they would on a

dedicated machine, so that I don’t have

to care about them.

/etc/hosts

/etc/hostname

/etc/resolv.conf

Container

Files in the

container fs are

overwritten

OpenShift Networking

Problem Statement

• As a X, I want container networking to

work seamlessly across multiple nodes,

so that I don’t have to worry where

which containers run (*).

• (*) while still maintaining compatibility with plain docker containers

OpenShift node

Host NICs

Docker bridge

docker bridge <-> ovs bridge

ovs bridge

ovs bridge <-> host NICs

OpenShift node

Look, there’s no

container

connected

OpenShift node

Container veth

endpoints on ovs

bridge

OpenShift node

Pod IP address

Container

Ping works from

container on same

host

Container

Ping fails from

container on

different host

Each node has

specific IP

range

OpenShift node

IP Range node 1

IP Range node 2

IP packets

destined for pod

on other node is

encapsulated via

VXLAN…

... and sent out

via the node‘s IP

stack

(MTU impact!)

OpenShift node

Port 1 is VXLAN

OpenShift node

Flow rules that

trigger VXLAN

encapsulation

Destination node

IP address

OpenShift node

Two pods in the

same namespace

on different

nodes…

OpenShift pod

…can

communicate with

each other

OpenShift Networking

Problem Statement

• As a X, I want to ensure that a rogue

pod cannot access pods in another

project, so that I have a base level of

security.

Project-specific

VXLAN ID

OpenShift pod

Pod in a different

namespace…

…cannot be

reached

OpenShift Networking

Problem Statement

• As a X, I want to be able to connect to

other containerized services using a

stable endpoint, so I don’t have to

reconfigure my application when other

containers come and go.

OpenShift node

Service IP

Address

OpenShift pod

Name resolution

via OpenShift

dnsmasq on node

Namespace in

search suffix list

OpenShift pod

Service name is

resolved into IP

adress

OpenShift pod

Communication

via service IP

Kubernetes Service Modes

• User-space mode– IPTables rules forward packages destined to the

service IP address to the kube-proxy

– Kube-proxy will in turn initiate connections to the actual destination IP and proxy between the two endpoints

– Key advantage: can detect non-responding pods and retry connection to other pods

• IPTables mode– kube-proxy continuously updates the node’s IPTables

rules

– forward packets directly to one of the target pod’s IP

– Key advantage: increased throughput

OpenShift Networking

Problem Statement

• As an X, I want my containerized

applications to be accessible externally,

so that a wide range of users can use

them (*).

• (*) without having to care on which node a container/pod runs

Ingress router

pod bound to

host port

Host Port = port

exposed on the

node

(containerized)

haproxy

OpenShift Routing

• Layer 7 Routing: HTTP(S), TLS-SNI

• To properly route other protocols,

deploy dedicated customized routers

• Alternatively instrument external load

balancers such as F5, etc.

OpenShift Networking

Problem Statement

• As an operator, I want to be able to fall

back to the known working version of a

service when deploying a new version

so I have a safety net (blue/green

deployments)

Router reconfiguration allows

blue/green deployments

oc patch route/api-gateway -p '{ "spec": {

"to": { "name": "api-gateway-green" }}}’

oc patch route/api-gateway -p '{ "spec": {

"to": { "name": "api-gateway-blue" }}}'

OpenShift Networking

Problem Statement

• As an operator, I want my containerized

applications to use specific source IP

addresses to access external services,

so I can restrict service access via

(external) firewall rules.

Egress source IP

Egress default

GWEgress target IP

(points to external

service “IPA”)

Points to egress-1

pod

OpenShift pod

Retrieving from

egress-1 service

works

External service

Egress

source IP

Egress

target IP

if2: node’s eth0

OpenShift egress pod

OpenShift Networking

Problem Statement

• As an operator, I want to control which

services my containerized applications

can access, so I can limit access via

internal means.

Egress Network Policy

{"kind": "EgressNetworkPolicy",

"apiVersion": "v1",

"metadata": { "name": "default” },

"spec": {

"egress": [{ "type": "Allow", "to": { "cidrSelector": "1.2.3.0/24” }

},{ "type": "Deny", "to": { "cidrSelector": "0.0.0.0/32” } }

]}

}

Summary

If the question is “OpenShift is great for

web applications, but we want to do X…

will this work?“, the answer is most likely

yes (from a networking point of view) (*).

(*) keep in mind that there is an MTU impact, multiple processing hops

which impact latency, etc…